OKLMdocumentation v1.0.3
On this page

Testing

How OKLM is tested: one binary per module, double-precision references, documented tolerances, and every run repeated under AddressSanitizer + UBSan and ThreadSanitizer.

Status: maintained.

Layout

  • tests/oklm_test.h: the harness. OKLM_CHECK, OKLM_CHECK_INT_EQ, OKLM_CHECK_OK, OKLM_CHECK_NEAR, OKLM_CHECK_ARRAY_NEAR, a deterministic splitmix64 generator, OKLM_TEST_BEGIN/OKLM_RUN/OKLM_TEST_END. No external framework.
  • tests/tolerances.h: every numerical tolerance, with the reason for its value.
  • tests/<module>/test_*.c: one binary per concern. make test runs them all; a binary exits with status 1 if any check failed. Their scratch directories are fixed (build/test_jobs/...) whatever the build, so OKLM_TEST_BEGIN takes a lock named after the source file (build/.test-locks/): the same test from the release, ASan and TSan builds run at once waits its turn instead of deleting the other's files (not on Windows).
  • tests/<module>/bench_*.c: benchmarks, run by make bench, never gating.
  • Snapshots: tests/harness/test_prompt.c compares the agent's system prompt with tests/fixtures/prompt/*.txt and prints a diff when they part; after a deliberate change, OKLM_UPDATE_SNAPSHOTS=1 and the test binary rewrite them, and the new snapshot is reviewed in the commit.

References

Each op is compared with a straightforward double-precision implementation written in the test from the definition (see adr/0015-test-references.md). Format code is checked against independent decoders:

  • f16/bf16: exhaustive over all 65 536 halves; float→half against a nearest-value search and against the F16C instructions on 16.7 million float patterns.
  • Legacy quants: hand-computed bytes for known inputs; element-wise decoding of the byte layout; round-trip error bounds per block.
  • K-quants: every dequantizer against a per-element decoder written from the bit layout, on random blocks (every bit pattern is a valid block).
  • Dot products and matmul: against the double-precision product of the dequantized operands as the kernel sees them.

Public models and reference tools

The canary set

A small set of published files stays on the disk and is what make test compares at every run: one file per kind of model, 10.2 GB in 66 files, named in tests/canary_models.txt. Every other family is verified when its code is touched, its files fetched, compared once and removed (below); its standing record is the figures of ROADMAP_DONE.md and of the page of its module.

Kind Files under models/ Size
Dense, the smollm2 tokenizer smollm2-135m/ Q8_0, Q4_K_M and F16; the checkpoint smollm2-135m-hf/ 145, 105, 271 and 269 MB
Dense, the Qwen tokenizer qwen3-0.6b/Qwen3-0.6B-Q8_0.gguf 639 MB
Dense, the Llama 3 tokenizer llama3.2-1b/Llama-3.2-1B-Instruct-Q4_K_M.gguf 808 MB
The default model qwen3-8b/Qwen3-8B-Q4_K_M.gguf (Qwen's own file, the catalogue's) 5.03 GB
Routed experts granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf 822 MB
Convolution and attention blocks lfm2-350m/LFM2-350M-Q4_K_M.gguf 229 MB
Vision smolvlm-256m/ Q8_0 and its F16 projector 175 and 190 MB
Recognition whisper-base/ (the checkpoint), samples/jfk.wav 294 MB
Speech outetts-0.2-500m/ Q8_0 and the WavTokenizer vocoder 537 and 130 MB
Embedding bge-small-en-v1.5/ (checkpoint, F32 and Q8_0 files) 133, 134 and 37 MB
Reranking ms-marco-MiniLM-L-6-v2/ (checkpoint and F16 file) 91 and 46 MB
Small files tiny-gpt2-sharded/, tokenizers/, templates/, bge-small-en/, t5-small/ 66 MB

SmolLM2-135M is there in three types because tests need each: Q8_0 is the model of most of them, Q4_K_M the second file of test_opencl_q8, F16 the reference of the checkpoint conversion and of check_quantize.sh.

Where a file is pinned and how it is fetched:

  • tests/public_models.txt (the corpus of the CI job) and tests/public_models_large.txt (larger files and the other families) hold the sha256, the destination and the URL of every file: they are the only place where a file is pinned. tests/canary_models.txt names destinations of those two lists.
  • sh tools/fetch_models.sh fetches the canary set and nothing else. A file already there with the right sha256 is kept; a canary name that no list pins is an error.
  • More is asked for by name: sh tools/fetch_models.sh --family gemma3-1b olmoe (the files under those folders, from either list), sh tools/fetch_models.sh tests/public_models.txt (a whole list), sh tools/fetch_models.sh --all (both lists whole, tens of GB), sh tools/fetch_models.sh --set tests/ci_models.txt (another set).
  • make test-public fetches the canary set, creates a Python venv in build/venv with numpy, gguf-py and safetensors (tools/setup_venv.sh), records the references below for the files that are there and runs the tests that compare them. It is the only make target that downloads a model.

What CI downloads: make test, make test-tsan, the DEBUG=1 run under AddressSanitizer, make check, make doc, make docs-lint and make fuzz-run fetch no model and no reference, none of them depends on a target that calls tools/fetch_models.sh, and the build-test, check and fuzz jobs run on the fixtures in git alone. The manual public-models job (started by hand, never by a push) runs make test-public PUBLIC_SET=tests/ci_models.txt: the CI set, a list of its own so that it cannot be taken for the canary set, 214 MB in 28 files (SmolLM2-135M Q8_0, 145 MB; the eight tokenizers, 61 MB; SmolLM2's, bge-small-en's and t5-small's tokenizers, the six templates, the sharded tiny GPT-2 and samples/jfk.wav, 8 MB), with Tiny Shakespeare (1 MB, tools/fetch_datasets.sh) and llama.cpp's sources at the pinned release for the references. It never takes the canary set nor a pinned list whole, and a file is added to tests/ci_models.txt only if it is small.

make test runs on what is under models/ and build/ref/: a test that wants a file outside the canary set, or a reference that was not recorded, prints skip: with the path it misses and compares nothing in its place. With the canary set alone these skip: every family of test_infer_public but the canary's and the five other SmolLM2 types; test_public_models's five other SmolLM2 files; test_mm_public but SmolVLM-256M (its tile against llama.cpp, its answer on the devices against the CPU); test_archs_public but LFM2-350M; test_gemma4_public, test_qwen35_public and test_bonsai_public whole; the Soprano and OuteTTS 0.3 parts of test_speech and test_soprano; test_train_gpt2 (llm.c's files, sh tools/fetch_llmc.sh); the grid IQ files of tools/gen_iq_models.sh; the templates of the files that are not there in test_jinja.

Recording the references

Everything under build/ref/ is recorded from the files under models/ by the tools of the next section, each of which leaves out a file that is not there (the shell tools name it). After the canary set is fetched, in this order (make test-public runs them all):

for q in Q8_0 Q4_K_M f16; do                      # gguf-py's dequantization, 0.5 GB a file
    build/venv/bin/python tools/dump_gguf_reference.py models/smollm2-135m/SmolLM2-135M-Instruct-$q.gguf \
        build/ref/smollm2-135m/SmolLM2-135M-Instruct-$q
done
sh tools/fetch_datasets.sh                        # data/tinyshakespeare.txt
build/venv/bin/python tools/gen_tok_fixture.py --expect build/ref/tok
build/venv/bin/python tools/bpe_train_ref.py data/tinyshakespeare.txt 4096 2 build/ref/tok/bpe_shakespeare_4096.json
sh tools/gen_llama_ref.sh                         # llama.cpp's tokens and logits, build/ref/llama/
sh tools/gen_infer_ref.sh                         # the float64 passes, build/ref/infer/
sh tools/gen_mm_ref.sh                            # llama.cpp's image embeddings, build/ref/mm/
build/venv/bin/python tools/gen_jinja_ref.py      # jinja2's renderings, build/ref/jinja/

To record one again, remove its files under build/ref/ first: gen_llama_ref.sh, gen_infer_ref.sh and the dump keep a reference that is already there (the others write theirs again at each run).

Verifying another family

When the code of a family outside the canary set changes, its published file is compared once and removed:

sh tools/fetch_models.sh --family gemma3-1b       # pinned in tests/public_models*.txt
OKLM_MODELS_DIR=models oklm models pull qwen3.5-9b   # or a catalogue entry that the lists do not hold
sh tools/gen_llama_ref.sh && sh tools/gen_infer_ref.sh   # and gen_mm_ref.sh for a vision or audio model
build/release/bin/tests/model/test_infer_public   # the test of the family; its lines are no longer "skip:"
rm -r models/gemma3-1b build/ref/llama/gemma3-1b-Q4_K_M.* build/ref/infer/gemma3-1b-Q4_K_M.json

The figures the test prints go into the family's entry of ROADMAP_DONE.md when they changed. One fetched file at a time: the disk holds the canary set and the file under verification.

Reference tools

The corpus includes a sharded safetensors repository (a tiny random GPT-2 in three shards, models/tiny-gpt2-sharded: read shard by shard and converted). tools/dump_gguf_reference.py dumps gguf-py's dequantization of every tensor of a SmolLM2 file into build/ref/, which tests/io/test_public_models.c compares with OKLM's.

Small fixtures are committed under tests/fixtures/, each generated once by a script that uses the reference implementation of its format: tools/gen_gguf_fixtures.py (gguf-py), tools/gen_safetensors_fixtures.py (safetensors), tools/gen_hf_fixtures.py (tiny Hugging Face checkpoints). tools/gen_tok_fixture.py (the 10 000-entry tokenizer corpus), tools/gen_tok_unit_fixtures.py (NFC cases and tiny tokenizers trained by Hugging Face, with their expected ids), tools/gen_unicode_tables.py (the Unicode tables compiled into the library). Reference ids for the downloaded tokenizers are written to build/ref/tok/ by tools/gen_tok_fixture.py --expect, and the trainer reference by tools/bpe_train_ref.py. tools/gen_model_fixtures.py writes three tiny random-weight models (llama, qwen3, gpt2) with gguf-py and their expected logits from tools/infer_ref.py, an independent numpy float64 forward pass (tests/model/test_model.c), and runs tools/gen_whisper_fixtures.py: two tiny Whisper checkpoints (80 mel bins in F32 holding half-precision values, as OpenAI's files do, 128 in F16 with one more language token) and what a numpy float64 Whisper decodes from a test signal: the text without timestamps, and with them the language found, the timed segments, a given language and the translation task (tests/mm/test_audio.c), and tools/gen_speech_fixtures.py: a tiny vocoder in the layout of WavTokenizer's GGUF file with the rows and samples a numpy float64 implementation computes from fixed codes, inverse transforms by numpy's irfft, and the prompts and normalizations of fixed texts (tests/mm/test_speech.c; the same script runs its network on a published vocoder file with --run, speech.md); tools/gen_infer_ref.sh records the same reference for the downloaded SmolLM2 files into build/ref/infer/ (tests/model/test_infer_public.c). tools/gen_llama_ref.sh builds llama.cpp from a pinned release into build/llama.cpp and records, with tools/llama_ref.c, its tokenization, greedy continuations and logits for every corpus model into build/ref/llama/ (same test), and for Qwen3.5-9B a continuation of 100 tokens and a prompt of some 300, which tests/model/test_qwen35_public.c compares with OKLM's (that prompt in batches of 64, two sequences in one pass, a sequence's recurrent state through a file, a copy and a cut; on the CPU, then on OpenCL and on Vulkan when the machine has them, where the device is also compared with the CPU step by step and run with a paged cache and with half of the blocks on the CPU; OKLM_Q35_BACKENDS=cpu,vulkan chooses the backends), and that continuation a second time with llama.cpp's keys and values kept as F32 (LLAMA_REF_KV=f32; a change of its last places only), which the test sets against the first: llama.cpp's distance from itself is the measure of OKLM's distance from it (hybrid-layers-design.md); the tiny-qwen35 fixture's reference is llama.cpp's too, recorded by tools/gen_model_fixtures.py. For Gemma 4 12B, whose whole Q4_K_M file moves by units of logit under one rounding, the script also writes two of its blocks alone in F16 (tools/gguf_cut_blocks.py) and records llama.cpp on them: there the two programs must agree closely (test_llama_cpp_gemma4). tests/model/test_gemma4_public.c runs the published Gemma 4 files that are under models/ (12B, E2B, E4B, 26B-A4B) on OpenCL and on Vulkan against those recordings and against a CPU context step by step, then with a paged cache, with half of the blocks on the CPU and, for the 26B-A4B, with its experts in host memory (OKLM_G4_BACKENDS=vulkan and OKLM_G4_FILES=e2b,12b choose). tests/model/test_archs_public.c does the same for the published files of the architectures with an attention output gate, of gpt-oss, of lfm2 and lfm2moe, of starcoder2, of mistral3 and of DeepSeek-Coder that are under models/ (Spark-X2.6 1.7B and 4B, Muse Glimmer 30B, gpt-oss-20b, LFM2-350M, Cyber-Prime 1.1 2.6B, Tini-Cybersec-8B-A1B, StarCoder2-3B, Devstral Small 2 24B, DeepSeek-Coder 1.3B and 6.7B; OKLM_PUB_BACKENDS=vulkan and OKLM_PUB_FILES=spark,muse,gpt-oss choose). For the Ternary Bonsai 2 files the script records the same prompts and 100 greedy tokens (tests/model/test_bonsai_public.c, which runs each file on the CPU, on OpenCL and on Vulkan, a device also against the CPU, with a paged cache and with half of its blocks on the CPU; OKLM_BONSAI_BACKENDS=vulkan and OKLM_BONSAI_FILES=4b choose): the Q2_0 files with the pinned build, the PQ2_0 and PTQ1_0 files, which upstream llama.cpp does not read, with PrismML's fork built beside it at a pinned commit (build/llama.cpp-prism, CPU only, only when one of those files is in models/); the hashes tests/core/test_quant_p128.c holds for these two types come from that fork's dequantizers and reference quantizers. Chat templates are checked against jinja2 in Hugging Face's environment (tools/jinja_ref.py): committed cases from tools/gen_jinja_fixtures.py (the GLM-4.5 and GLM-4.7 templates among them), published templates rendered into build/ref/jinja/ by tools/gen_jinja_ref.py (tests/model/test_jinja.c). Grammar token masks are checked against feeding every token's text to the grammar, over a byte-level vocabulary and through split UTF-8 characters, and constrained generation with SmolLM2 must produce JSON and tool calls that parse back, as must a GLM-4.5 call generated by a tiny random model (tests/model/test_grammar.c). tools/gen_case_fixtures.py records Python's str.lower / upper / title / capitalize of 1 500 random strings over every cased character, and Python's simple case folds (tests/fixtures/tok/case.json); tools/gen_fp8_fixture.py writes an FP8 checkpoint with block scales and its dequantized twin; tools/check_split.sh compares oklm split / oklm merge with llama.cpp's gguf-split, and tools/check_quantize.sh oklm quantize with llama-quantize (every file type, byte for byte). test_quant_k and test_quant check the K-quant, IQ4, FP4, ternary and low-bit quantizers, and the decoding of the latter, against hashes of ggml's output for fixed rows and blocks (tests/core/iq_blocks.h), and the weighted (importance matrix) quantizers likewise; check_quantize.sh also quantizes with bartowski's importance matrix (legacy format) and with one written by llama-imatrix (GGUF format). The grid IQ types are decoded against gguf-py's dequantization of fixed blocks (test_quant), and tools/gen_iq_models.sh makes SmolLM2 in every grid type with llama-quantize, whose greedy choices and logits test_infer_public compares with llama.cpp on both backends; tools/gen_iq_grids.py extracts ggml's codebook tables into src/core/iq_grids.inc. Their quantizers are OKLM's own: test_quant bounds their error on fixed rows, checks that importance weights lower the weighted error and that a power of two scales the result exactly; the perplexity comparison with llama-quantize is in convert.md. Training is checked against finite differences of the loss and the float64 references of the tiny GPT-2 and Llama fixtures (tests/train/test_train.c), and against llm.c (tests/train/test_train_gpt2.c, after sh tools/fetch_llmc.sh): its PyTorch reference state of GPT-2 124M (logits, loss and gradients of one batch, losses of ten steps) and the CPU run on Tiny Shakespeare its README publishes (training.md). Python is only used to produce or check data; no C code calls it.

Hidden states against llama.cpp

When the logits of a model differ from llama.cpp's by more than its neighbours', the two forward passes are compared step by step on the same tokens:

L=build/llama.cpp
cc -O2 -c tools/llama_dump.c -I$L/include -I$L/ggml/include -o build/llama_dump.o
c++ -o build/llama_dump build/llama_dump.o $L/build/src/libllama.a $L/build/ggml/src/libggml.a \
    $L/build/ggml/src/libggml-cpu.a $L/build/ggml/src/libggml-base.a -fopenmp -lpthread -lm
build/llama_dump model.gguf ref.bin "The capital of France is"
OKLM_DUMP=own.bin oklm run -m model.gguf -p "The capital of France is" -n 1 --backend cpu
python3 tools/dump_compare.py ref.bin own.bin            # every step, in the graph's order
python3 tools/dump_compare.py ref.bin own.bin --step l_out   # the stream after each block

tools/llama_dump.c (a test tool, linked against the pinned llama.cpp, never into OKLM) writes every tensor llama.cpp's graph names (attn_norm-3, Qcur-3, kqv_out-3, ffn_out-3, l_out-3...) through its evaluation callback; OKLM_DUMP makes the llama family's forward pass (src/model/arch_llama.c, src/model/dump.c) write its own under the same names; tools/dump_compare.py prints, for each step both have, the largest difference and the norm of the difference over the norm of the reference. A dump holds every step of every block for every token of the prompt: a few tokens are enough (a 12B model, 6 tokens: 150 MB). In a gemma4 block with experts the router's logits and probabilities, the joined gate | up product of each token's experts, their gated product, their outputs and their sum are steps too (a tensor llama.cpp keeps as a strided view, such as the gate half alone, is not in its dump).

A difference grows from block to block, so the first block that departs says little about the others. Two more variables separate them:

  • OKLM_DUMP_FEED=ref.bin: every block starts from llama.cpp's stream (l_out of the block before) instead of OKLM's own, so each block's figure is its own;
  • OKLM_DUMP_FEED_OPS=1 with it: inside a block too, the attention takes llama.cpp's rotated queries and keys and its values, the output projection its heads' outputs, the feed-forward its input and its gated product (the experts of a gemma4 block theirs). Each matrix product and the attention are then compared on equal operands, and the step that is computed differently is the one whose figure stays large.

OKLM_ATTN_F16=1 and OKLM_GELU_F16=1 make OKLM round where ggml does (the queries and the probabilities of attention over an F16 cache, the GELU table): a step whose figure falls to rounding noise with one of them differs by that rounding and by nothing else (inference.md has the 12B's figures). tools/ref_compare.py compares two records of tools/llama_ref.c made on the same tokens (ids:1,2,3|4,5 feeds a continuation): llama.cpp with itself, flash attention on (LLAMA_FLASH=1) and off, or its cache in single precision (LLAMA_KV_F32=1) and in its default F16, which gives the distance between two runs of the reference.

The terminal chat on a server

tests/harness/test_cli_chat_url.c types into oklm chat --url against a small OpenAI-style server of its own, which records each request and answers by the words the request holds (a reply cut, a refusal of an open last message, a prompt too long, reasoning alone, a summary); no model is loaded, so it runs everywhere make test does. tests/client/test_remote_models.c reads the lists of GET /v1/models the same way. tests/client/test_request_members.c runs the rule the chats and the agent's backend share for a request a server refuses by one of its members (src/client/request_members.c): the member named in the refusals of several servers, a name that is only part of a word, the members an answer cannot do without, and the request rebuilt without its first, a middle and its last member.

sh tools/check_chat_url.sh runs the chat on real servers, on loopback and on the CPU, with Qwen3-0.6B: llama.cpp's server (the pinned build of the test reference, LLAMA_SERVER=path when it is elsewhere; without it those checks are skipped with a line), the same with --api-key, its router over two models (when SmolLM2-135M is there), and oklm serve. PORT=N is the first of the four ports it listens on (18480). 33 checks, a little over a minute in all on the machine it was written on (the last two on the stand-in of the window's check, which lists three models: Tab after /model on a pseudo-terminal completes a name the server lists, and lists several): two turns, the reasoning shown and not kept, --resume, a cut reply gone on with as an open last message, /compact, the full-screen UI on a pseudo-terminal, the key missing, wrong and right, the list of two models (refused for input that is not typed, chosen in a picker on a terminal, /model N), and on oklm serve its context read and resized, a reply its context cut gone on with in a larger one, --resume and /compact.

OpenCL

tests/core/test_opencl.c checks every OpenCL kernel against the CPU (or a double-precision reference for matrix products) and the CPU fallback; tests/model/test_model.c and tests/model/test_infer_public.c repeat their end-to-end checks on the OpenCL backend. All three skip the OpenCL parts when no device is found (CI). The sanitizer targets leave OpenCL out; make asan-opencl covers it (see opencl.md).

Vulkan

tests/core/test_vulkan.c checks the Vulkan backend's core (copies, ordering, wide dispatches, the fallback, the fused residual tails) and its element-wise ops against the CPU, and tests/core/test_vulkan_<group>.c (norm, matmul, attention, gemm, q8, moe, ssm) one file per group of ops; all run on every Vulkan GPU present (tests/core/vk_test_util.h: llvmpipe left out unless OKLM_VK_DEVICE names it, OKLM_VK_DEVICES=none skips them) and skip without one (see vulkan.md).

Testing on another GPU

tools/gpu/report.sh is for someone with a GPU OKLM has not run on (NVIDIA first): from a fresh clone it checks the prerequisites, records the system, builds OKLM, runs the GPU tests on every device and writes one plain-text file, oklm-gpu-report-<host>-<date>.txt, to send back. It installs nothing, limits every step in time, goes on after a failure and exits 0 once the report is written.

What to install (the script names what is missing):

  • a C17 compiler (gcc or clang), GNU make, git, coreutils' timeout;
  • the GPU driver with its OpenCL and Vulkan parts (on NVIDIA, the proprietary driver brings both), the OpenCL ICD loader (ocl-icd-libopencl1, ocl-icd) and the Vulkan loader (libvulkan1, vulkan-icd-loader);
  • optional: clinfo and vulkaninfo (vulkan-tools) to list the devices, glslc (shaderc, the Vulkan SDK) for the Vulkan backend, which a build without it leaves out.

How to run it:

git clone <OKLM repository> oklm && cd oklm
git clone --depth 1 --branch next https://git.nilorea.net/gullradriel/nilorea-library external/nilorea-library
sh tools/gpu/report.sh --models --download   # about 2 GB of models, up to an hour

--clone-nilorea clones nilorea-library itself when none is found (it is looked for in --nilorea-dir, NILOREA_DIR, external/nilorea-library, ../nilorea-library, ../../nilorea-library).

Option Effect
--quick the device tests only: no make test, no speed table
--models the speed table: SmolLM2-135M Q8_0, Qwen2.5-0.5B Q4_K_M, Qwen3-0.6B Q8_0, Llama-3.2-1B Q4_K_M (OLMoE-1B-7B Q4_K_M when present), 128 greedy tokens after "The capital of France is" on the CPU and each device
--download with --models: fetch the missing ones with tools/fetch_models.sh (checksums from tests/public_models.txt)
--keep-going after a failed build, test the binaries already built
--models-dir DIR the models' directory (default models/)
--nilorea-dir DIR, --clone-nilorea where nilorea-library is, or clone it into external/
--out DIR, --jobs N where the report goes, make -j N

What the report holds, section by section: the prerequisites found and missing; the OS, kernel, CPU, memory, GPUs (lspci), drivers (nvidia-smi, clinfo, vulkaninfo --summary and the Vulkan extensions OKLM's fast paths use), compiler versions and the OKLM commit; the build's warnings and errors; the devices oklm devices finds, each selected by index (OKLM_CL_DEVICE, OKLM_VK_DEVICE); per device, test_opencl, test_opencl_q8, test_opencl_wmma, test_model and test_kv_file, and every test_vulkan* binary the build has (with test_model and test_kv_file again, OpenCL left out), each with its checks, failures, failing lines, exit code and time; without --quick, make test's result; with --models, generation and prompt t/s per backend and device and whether the greedy text matches the CPU's and the first device's (its first 240 bytes). Of the environment it writes only OKLM's device, backend, thread and kernel variables (build.md) that are set, and the home directory as ~.

What to send back: the report file. A test that fails on a new vendor is what the report is for; the run goes on to the next one.

Windows

make wine-test builds the Windows binaries (MinGW, statically linked) and runs every test under Wine, public models included when present. It found two portability bugs the Linux runs could not: the safetensors and GGUF overlap checks depended on the order the C library's qsort leaves equal keys in. It also showed that mingw-w64's roundf is wrong for the float just below a half, which the quantizers no longer call (test_quant, test_round_below_half).

sh tools/check_kept_servers.sh keeps a server of the test model across logins for real, with systemd's user manager (oklm serve --keep, a kill the manager answers by starting it again, --stop, systemctl --user start, --rekeep, --unkeep; a session with another XDG_RUNTIME_DIR reaching the manager by itself; 42 checks). Its unit carries the script's name and its pid and is removed when the script ends, however it ends; it is skipped where the user manager does not answer, where tests/harness/test_kept.c still runs the module against a script in place of systemctl.

sh tools/check_gui_kept.sh does the same from the server manager's window under Xvfb, with the tool's own presses (--press keep:NAME, start, stop:NAME, restart:NAME, ask-rekeep:NAME, rekeep:NAME, unkeep:NAME, profile:NAME, yes, no): each step is a window that does its presses, writes what it said and ends, and no place of the form is clicked, so a change of the layout does not break it. By default the service manager is a stand-in (tools/standin_systemctl.py through OKLM_SYSTEMCTL): its units are under build/check-gui-kept, its services are processes it starts and ends, and neither a unit of the user's nor the user manager is touched; 21 checks. REAL=1 sh tools/check_gui_kept.sh runs the same steps against systemd's user manager, with a unit named after the script and its pid among the user's, removed when the script ends: a run for a person to start, skipped where the user manager does not answer.

sh tools/check_gui_form.sh [tiny|canary|device] presses, under Xvfb and without a pointer, the buttons of the two tools' options that a screenshot does not: the chat window's Open in the server manager..., the server manager's Re-keep with form (against a stand-in for systemctl: no unit of the user's is touched) and its Start server with every tab of the Advanced window set, the server started being compared with the form and stopped (gui.md); make gui-test runs it on the tiny models. make gui-device-test runs its device mode, which is not part of gui-test (that one passes without a GPU): on each device backend that answers (oklm devices; BACKENDS names one), one server at a time started from the form with Start server, compared with its own account and stopped: --gpu-layers 20 of the 28 layers of Qwen3-0.6B (n_gpu_layers of GET /api/context, device_memory of /api/stats), --n-cpu-moe 8 --expert-gpu 0.25 on Granite 3.1 1B A400M (n_cpu_moe, expert_gpu, and the tier's slots in /api/stats), and --devices 2 where two devices of a backend answer (n_devices; Vulkan with OKLM_VK_DEVICES=all, a discrete and an integrated card, else two OpenCL devices), skipped with a line where there is one. With Qwen3-8B Q4_K_M under models/ it then checks the memory line against the server it describes, on each backend: at 40960 tokens the line says It fits with a smaller cache, the server started from the form tells the cache the line planned (cache_tokens, within 2 %), and the chat window's Apply restarts its model with that cache; at 131072 tokens, where the device holds the model with less than one context of cache, the line says the layers that go there and the cache on each side, and the server started from the form has those layers and that cache and answers; with every layer asked for there the line says It does not fit and Apply refuses; a Cache size of 4096 MiB is the server's cache to the token; a model loaded in place with 131072 tokens kept for it is served with them, fewer layers on the device, and one whose kept settings do not fit (every layer asked for too) is loaded with the window's own. It ends with what shares the device: a server of SmolVLM-256M with its projector and one of Qwen3-8B with Qwen3-0.6B as its draft model, each started from the form, whose cache is the one the line planned once it counted them, and, for the projector, whose device_memory tells the share the line counted; then the models beside the model where their backends put them (Qwen3-0.6B with bge-small on the device, whose device_memory.embed is the share the line counted; then with bge-small and MiniLM-L6 on the CPU, whose contexts as the line counted them are what the server tells in host_memory, to the byte); then a model loaded in place of another with nothing kept for it (Qwen3-0.6B after Qwen3-8B with 20 layers on the device: all its 28 layers there; Qwen3-8B at 32768 tokens on a device held to 8 GiB: loaded with the layers that leave the device a context, and answering); then a model larger than its device (Qwen3-8B on the device held to 4 GiB: the line says the layers that go there and the cache on each side, and the server started from the form has those layers and that cache, to the token); then what the line counts beside the model against the server's own figures, to the byte: SmolVLM-256M's projector on the CPU (host_memory.vision), OuteTTS 0.2 on the CPU loaded at the start (host_memory.speak and speak_call), bge-small on a second Vulkan device where one answers (side_devices, with OKLM_VK_DEVICES=all), and a chat window's line counting the embedding and the reranking models it finds by itself for a long document; and last one server seen by the three lines that speak of it: for a server started from the form at a context the form says fits (Qwen3-0.6B at 8192 tokens) or fits with a smaller cache (Qwen3-8B at 40960), the server manager's line for it (--press context:N) and the line of a chat window attached to it are not red at that context and are written for the cache the server tells (cache_tokens); at 131072 tokens, above that cache, the two tools' lines are red, since a server that runs keeps its layers, and the form is red with every layer asked for and says the layers it would place with them left to what fits; a server whose cache ends before its own context (--kv-mem 64 at 16384 tokens) is red in both tools at the context it runs with (tiny checks the same on the CPU, where the cache is in host memory). Without a device backend, or without the two models under models/, it says skipped and ends with status 0. sh tools/docs/gui_shots.sh takes the pictures of gui.md that the tools' unattended options reproduce.

On 2026-11-15, after make MINGW=1 BUILD=build/mingw tests apps gui (which no longer built: tests/harness/test_cli_chat.c included two headers of static functions it uses on POSIX only), the nine test_gui_* and test_mem_plan passed under Wine 11 as make wine-test runs them, test_gui_settings after its expectations of a quoted value took the system's quote (five checks expected '...' of a Windows command line), and test_server.exe passed with 2575 checks, its Vulkan and OpenCL tests included. The options' graphical flows run there without a pointer are in gui.md.

sh tools/check_gui_wine.sh runs the graphical tools of the Windows build (make MINGW=1 BUILD=build/mingw apps gui, after make windows-deps) under Wine and Xvfb, in a Wine prefix of its own under build/mingw/wine-gui-check, on the CPU with the tiny model of the server test, and checks what each flow leaves: files, processes, answers on a port, what a window wrote on its error stream. Every button is pressed by the tool's own test aid (--press, gui.md): no place of a window is clicked; the system's file dialog is answered from the keyboard, and a window that stays is closed with Alt+F4. The chat window (a message, a PDF attached, a recording transcribed, two files dropped on it, the two exports through the system's dialog), the chat window started with nothing named (a server chosen, a file loaded in the window, a load refused, the system's file dialog answered from the keyboard), the launcher from a staged folder named in three scripts (a tool that outlives it, the documentation's page handed to a stand-in for xdg-open), the model manager's list, the server manager (a window a step: a server started from the form and compared with it, restarted under another pid, its Dashboard and Chat, stopped; a server found by its port alone, Stop asking first, and stopped), the conversion (a GGUF file, a Hugging Face folder) and the inspector, the training window (a run started, left by its window, found by another, stopped by itself with a checkpoint; the Fine-tune, LoRA and RL tabs, Merge adapter, Export adapter..., Open a chat), the sessions browser (Attach on a listening agent, which oklm.exe agent --stop-listening then ends, Resume in a console, Export .md, Delete), an index built by the retrieval manager, searched, and a judged set scored by Evaluate, a server kept across logins by oklm.exe (the Run key written, listed, started as at login, changed, removed). Some checks need the models of MODELS=DIR (a store as oklm models pull fills it, models by default): the Chat data and Advanced tabs with smollm2-135m, a picture with smolvlm-256m, a sound with ultravox-1b, each skipped with a line when its model is not there. On 2026-11-16, with the canary set (no ultravox-1b: the sound skipped), 58 checks passed in 186 seconds; the group start, added after, alone (CHECKS=start): 13 checks in 34 seconds. The group saved, added on 2026-11-17 (CHECKS=saved, 5 checks): a server saved from the chat window's Connect window with its key, the settings file under the prefix's %APPDATA% holding it, the key in no output, and forgotten again. The window gives that file an access list of its owner alone (oklm_file_owner_only(), which tests/io/test_sys.c runs on every system) and reads one back: Wine takes the list, the calls succeed, and keeps none, what it reads back being made from the file's Unix mode (three entries, not protected), so the check takes either answer there and only Windows itself shows the list protected with the owner's entry alone. CHECKS="servers train" runs those groups only; MINGW_DEPS_DIR as for make; WINE_GUI_PREFIX=DIR names another Wine prefix than the check's own (never the user's ~/.wine). It is skipped with a message when Wine, Xvfb, xdotool, ImageMagick's import or the MinGW build is missing; the drop needs python3-xlib (tools/xdnd_drop.py is the source side of the X drop protocol, which Wine hands on as the system's) and is left out with a line without it. The screenshots are left in build/mingw/wine-gui-check/shots, and what each window wrote in build/mingw/wine-gui-check/logs. sh tools/check_gui_press.sh, part of make gui-test, drives the same presses of the launcher, the training window, the sessions browser and the retrieval manager on Linux (34 checks), and sh tools/check_gui_start.sh, part of it too, starts the chat window with nothing named (42 checks: no model loaded, a server, a saved server or an agent chosen, an installed model or a file loaded in the window once the memory line judged it, a load refused, gui.md), sh tools/check_gui_remote.sh links it to a server that is not OKLM's (115 checks on a stand-in and on oklm serve, 15 more on llama.cpp's server where it is built: what the window says of it, OKLM's own routes asked once, the context as the window's setting, a reply gone on with, a compaction by estimate, a server saved with its key in a file of mode 600 and chosen at a start with nothing named, gui.md), and sh tools/check_gui_advanced.sh opens every tab of the Advanced window of both tools in a window of 760 by 480 pixels, then sets the chat window's Other field and its MoE disk options (38 checks, gui.md). sh tools/check_gui_full.sh, part of it too, has replies cut and gone on with while they are read aloud and in a spoken conversation, and a conversation with pictures compacted, on the canary models that are there (19 checks, gui.md). These are Wine's runs, not Windows': the access lists of the registry's folder, a real console, a sound card, the system's dialogs and its certificate store are Wine's own there.

Paths in UTF-8 on Windows (build.md): tests/io/test_sys.c (test_utf8_paths, on every system) makes, writes, lists, maps, renames and removes files named with accents, Greek, Japanese and a character outside the basic plane, a name of 253 bytes, and reads a variable of the environment with such a value back; test_procreg.exe starts a program with the argument été Ελληνικά 日本語 and reads it back from its output.

A job's output above 2 GiB: tests/harness/test_jobs.c (test_big_output, on every system, the first of that file's tests of a started job that the Windows build runs) gives the log of a job that ended a hole of 3 GiB and 5 MiB of lines after it, then reads its size, a place above 32 bits, a reader's last lines and its last line; a sparse file, so 5 MiB are written.

The Windows build's float functions (expf, cosf, sinf, powf, logf) do not round like glibc's, so its CPU logits are not the Linux build's bits: on the prompt of 100 arbitrary tokens test_opencl_q8 gives SmolLM2-135M, the two CPU references are 0.50 (Q8_0) and 0.54 (Q4_K_M) apart, while the OpenCL device returns the same bits from both builds. That test's limit on the device against the CPU is therefore 0.75 on Linux (0.607 and 0.625 measured) and 1.1 on Windows (0.905 and 0.651).

Fuzzing

make fuzz                     # build fuzz/fuzz_{gguf,safetensors,regex,tokenizer,jinja,grammar,pdf,toolcalls,shsplit} (clang, libFuzzer, ASan, UBSan)
make fuzz-run FUZZ_TIME=60    # run each for 60 s; seeds in fuzz/corpus/, new inputs in build/fuzz/corpus/

Invariants checked

  • Quantized SIMD kernels are bit-identical to the scalar kernels.
  • Matmul results are bit-identical for 1, 3 and 7 threads.
  • Several application threads running ops concurrently get the same bits as one thread alone (test_threads, meant for ThreadSanitizer).
  • The passes of a device run in the order they asked for it, a yield hands the device to the passes that wait and takes it back with the same nesting, and a pass on several devices takes them in the order of their registration whatever the order they are named in (test_threads: test_pass_handoff, test_pass_devices, on devices that compute nothing).
  • A context spread over two devices, a context on one of them and an image encoder on the other, each in a thread of its own, give the floats each gives alone (test_multi_device: two OpenCL instances, two Vulkan instances, and two different Vulkan GPUs when the machine has them).

Sanitizers and static analysis

make asan-test    # ASan + UBSan (-fno-sanitize-recover=undefined)
make tsan-test    # TSan
make check        # cppcheck, gating
make CC=clang BUILD=build/clang test

A change is mergeable when all four are clean.

CI runs them in a Debian 13 container, whose tools are older than a recent machine's: cppcheck 2.17 reports what 2.22 does not, GCC 14 refuses what GCC 16 accepts. A clean run on the release machine does not say CI passes. Before a version tag the jobs of .gitea/workflows/ci.yml are run in a debian:trixie container on the tree to tag (the build and the suite with GCC and with clang, make format-check, make check, make doc, make docs-lint, make fuzz-run, packaging/ci-packages.sh), and the tag is set on a commit whose run on the server is green.

Logging during tests

Tests trigger errors on purpose (out-of-range ids, full arenas, bad shapes). The harness sets the log level to LOG_CRIT so those do not clutter the output; set OKLM_LOG_LEVEL=DEBUG to see everything.