Testing
How OKLM is tested: one binary per module, double-precision references, documented tolerances, and every run repeated under AddressSanitizer + UBSan and ThreadSanitizer.
Status: maintained.
Layout
tests/oklm_test.h: the harness.OKLM_CHECK,OKLM_CHECK_INT_EQ,OKLM_CHECK_OK,OKLM_CHECK_NEAR,OKLM_CHECK_ARRAY_NEAR, a deterministic splitmix64 generator,OKLM_TEST_BEGIN/OKLM_RUN/OKLM_TEST_END. No external framework.tests/tolerances.h: every numerical tolerance, with the reason for its value.tests/<module>/test_*.c: one binary per concern.make testruns them all; a binary exits with status 1 if any check failed. Their scratch directories are fixed (build/test_jobs/...) whatever the build, soOKLM_TEST_BEGINtakes a lock named after the source file (build/.test-locks/): the same test from the release, ASan and TSan builds run at once waits its turn instead of deleting the other's files (not on Windows).tests/<module>/bench_*.c: benchmarks, run bymake bench, never gating.- Snapshots:
tests/harness/test_prompt.ccompares the agent's system prompt withtests/fixtures/prompt/*.txtand prints a diff when they part; after a deliberate change,OKLM_UPDATE_SNAPSHOTS=1and the test binary rewrite them, and the new snapshot is reviewed in the commit.
References
Each op is compared with a straightforward double-precision implementation written in the test from the definition (see adr/0015-test-references.md). Format code is checked against independent decoders:
- f16/bf16: exhaustive over all 65 536 halves; float→half against a nearest-value search and against the F16C instructions on 16.7 million float patterns.
- Legacy quants: hand-computed bytes for known inputs; element-wise decoding of the byte layout; round-trip error bounds per block.
- K-quants: every dequantizer against a per-element decoder written from the bit layout, on random blocks (every bit pattern is a valid block).
- Dot products and matmul: against the double-precision product of the dequantized operands as the kernel sees them.
Public models and reference tools
The canary set
A small set of published files stays on the disk and is what
make test compares at every run: one file per kind of
model, 10.2 GB in 66 files, named in
tests/canary_models.txt. Every other family is verified
when its code is touched, its files fetched, compared once and removed
(below); its standing record is
the figures of ROADMAP_DONE.md and of
the page of its module.
| Kind | Files under models/ |
Size |
|---|---|---|
Dense, the smollm2 tokenizer |
smollm2-135m/ Q8_0, Q4_K_M and F16; the checkpoint
smollm2-135m-hf/ |
145, 105, 271 and 269 MB |
| Dense, the Qwen tokenizer | qwen3-0.6b/Qwen3-0.6B-Q8_0.gguf |
639 MB |
| Dense, the Llama 3 tokenizer | llama3.2-1b/Llama-3.2-1B-Instruct-Q4_K_M.gguf |
808 MB |
| The default model | qwen3-8b/Qwen3-8B-Q4_K_M.gguf (Qwen's own file, the
catalogue's) |
5.03 GB |
| Routed experts | granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf |
822 MB |
| Convolution and attention blocks | lfm2-350m/LFM2-350M-Q4_K_M.gguf |
229 MB |
| Vision | smolvlm-256m/ Q8_0 and its F16 projector |
175 and 190 MB |
| Recognition | whisper-base/ (the checkpoint),
samples/jfk.wav |
294 MB |
| Speech | outetts-0.2-500m/ Q8_0 and the WavTokenizer
vocoder |
537 and 130 MB |
| Embedding | bge-small-en-v1.5/ (checkpoint, F32 and Q8_0
files) |
133, 134 and 37 MB |
| Reranking | ms-marco-MiniLM-L-6-v2/ (checkpoint and F16 file) |
91 and 46 MB |
| Small files | tiny-gpt2-sharded/, tokenizers/,
templates/, bge-small-en/,
t5-small/ |
66 MB |
SmolLM2-135M is there in three types because tests need each: Q8_0 is
the model of most of them, Q4_K_M the second file of
test_opencl_q8, F16 the reference of the checkpoint
conversion and of check_quantize.sh.
Where a file is pinned and how it is fetched:
tests/public_models.txt(the corpus of the CI job) andtests/public_models_large.txt(larger files and the other families) hold the sha256, the destination and the URL of every file: they are the only place where a file is pinned.tests/canary_models.txtnames destinations of those two lists.sh tools/fetch_models.shfetches the canary set and nothing else. A file already there with the right sha256 is kept; a canary name that no list pins is an error.- More is asked for by name:
sh tools/fetch_models.sh --family gemma3-1b olmoe(the files under those folders, from either list),sh tools/fetch_models.sh tests/public_models.txt(a whole list),sh tools/fetch_models.sh --all(both lists whole, tens of GB),sh tools/fetch_models.sh --set tests/ci_models.txt(another set). make test-publicfetches the canary set, creates a Python venv inbuild/venvwith numpy, gguf-py and safetensors (tools/setup_venv.sh), records the references below for the files that are there and runs the tests that compare them. It is the only make target that downloads a model.
What CI downloads: make test,
make test-tsan, the DEBUG=1 run under
AddressSanitizer, make check, make doc,
make docs-lint and make fuzz-run fetch no
model and no reference, none of them depends on a target that calls
tools/fetch_models.sh, and the build-test,
check and fuzz jobs run on the fixtures in git
alone. The manual public-models job (started by hand, never
by a push) runs
make test-public PUBLIC_SET=tests/ci_models.txt: the CI
set, a list of its own so that it cannot be taken for the canary set,
214 MB in 28 files (SmolLM2-135M Q8_0, 145 MB; the eight tokenizers, 61
MB; SmolLM2's, bge-small-en's and t5-small's tokenizers, the six
templates, the sharded tiny GPT-2 and samples/jfk.wav, 8
MB), with Tiny Shakespeare (1 MB, tools/fetch_datasets.sh)
and llama.cpp's sources at the pinned release for the references. It
never takes the canary set nor a pinned list whole, and a file is added
to tests/ci_models.txt only if it is small.
make test runs on what is under models/ and
build/ref/: a test that wants a file outside the canary
set, or a reference that was not recorded, prints skip:
with the path it misses and compares nothing in its place. With the
canary set alone these skip: every family of
test_infer_public but the canary's and the five other
SmolLM2 types; test_public_models's five other SmolLM2
files; test_mm_public but SmolVLM-256M (its tile against
llama.cpp, its answer on the devices against the CPU);
test_archs_public but LFM2-350M;
test_gemma4_public, test_qwen35_public and
test_bonsai_public whole; the Soprano and OuteTTS 0.3 parts
of test_speech and test_soprano;
test_train_gpt2 (llm.c's files,
sh tools/fetch_llmc.sh); the grid IQ files of
tools/gen_iq_models.sh; the templates of the files that are
not there in test_jinja.
Recording the references
Everything under build/ref/ is recorded from the files
under models/ by the tools of the next section, each of
which leaves out a file that is not there (the shell tools name it).
After the canary set is fetched, in this order
(make test-public runs them all):
for q in Q8_0 Q4_K_M f16; do # gguf-py's dequantization, 0.5 GB a file
build/venv/bin/python tools/dump_gguf_reference.py models/smollm2-135m/SmolLM2-135M-Instruct-$q.gguf \
build/ref/smollm2-135m/SmolLM2-135M-Instruct-$q
done
sh tools/fetch_datasets.sh # data/tinyshakespeare.txt
build/venv/bin/python tools/gen_tok_fixture.py --expect build/ref/tok
build/venv/bin/python tools/bpe_train_ref.py data/tinyshakespeare.txt 4096 2 build/ref/tok/bpe_shakespeare_4096.json
sh tools/gen_llama_ref.sh # llama.cpp's tokens and logits, build/ref/llama/
sh tools/gen_infer_ref.sh # the float64 passes, build/ref/infer/
sh tools/gen_mm_ref.sh # llama.cpp's image embeddings, build/ref/mm/
build/venv/bin/python tools/gen_jinja_ref.py # jinja2's renderings, build/ref/jinja/To record one again, remove its files under build/ref/
first: gen_llama_ref.sh, gen_infer_ref.sh and
the dump keep a reference that is already there (the others write theirs
again at each run).
Verifying another family
When the code of a family outside the canary set changes, its published file is compared once and removed:
sh tools/fetch_models.sh --family gemma3-1b # pinned in tests/public_models*.txt
OKLM_MODELS_DIR=models oklm models pull qwen3.5-9b # or a catalogue entry that the lists do not hold
sh tools/gen_llama_ref.sh && sh tools/gen_infer_ref.sh # and gen_mm_ref.sh for a vision or audio model
build/release/bin/tests/model/test_infer_public # the test of the family; its lines are no longer "skip:"
rm -r models/gemma3-1b build/ref/llama/gemma3-1b-Q4_K_M.* build/ref/infer/gemma3-1b-Q4_K_M.jsonThe figures the test prints go into the family's entry of ROADMAP_DONE.md when they changed. One fetched file at a time: the disk holds the canary set and the file under verification.
Reference tools
The corpus includes a sharded safetensors repository (a tiny random
GPT-2 in three shards, models/tiny-gpt2-sharded: read shard
by shard and converted). tools/dump_gguf_reference.py dumps
gguf-py's dequantization of every tensor of a SmolLM2 file into
build/ref/, which
tests/io/test_public_models.c compares with OKLM's.
Small fixtures are committed under tests/fixtures/, each
generated once by a script that uses the reference implementation of its
format: tools/gen_gguf_fixtures.py (gguf-py),
tools/gen_safetensors_fixtures.py (safetensors),
tools/gen_hf_fixtures.py (tiny Hugging Face checkpoints).
tools/gen_tok_fixture.py (the 10 000-entry tokenizer
corpus), tools/gen_tok_unit_fixtures.py (NFC cases and tiny
tokenizers trained by Hugging Face, with their expected ids),
tools/gen_unicode_tables.py (the Unicode tables compiled
into the library). Reference ids for the downloaded tokenizers are
written to build/ref/tok/ by
tools/gen_tok_fixture.py --expect, and the trainer
reference by tools/bpe_train_ref.py.
tools/gen_model_fixtures.py writes three tiny random-weight
models (llama, qwen3, gpt2) with gguf-py and their expected logits from
tools/infer_ref.py, an independent numpy float64 forward
pass (tests/model/test_model.c), and runs
tools/gen_whisper_fixtures.py: two tiny Whisper checkpoints
(80 mel bins in F32 holding half-precision values, as OpenAI's files do,
128 in F16 with one more language token) and what a numpy float64
Whisper decodes from a test signal: the text without timestamps, and
with them the language found, the timed segments, a given language and
the translation task (tests/mm/test_audio.c), and
tools/gen_speech_fixtures.py: a tiny vocoder in the layout
of WavTokenizer's GGUF file with the rows and samples a numpy float64
implementation computes from fixed codes, inverse transforms by numpy's
irfft, and the prompts and normalizations of fixed texts
(tests/mm/test_speech.c; the same script runs its network
on a published vocoder file with --run, speech.md);
tools/gen_infer_ref.sh records the same reference for the
downloaded SmolLM2 files into build/ref/infer/
(tests/model/test_infer_public.c).
tools/gen_llama_ref.sh builds llama.cpp from a pinned
release into build/llama.cpp and records, with
tools/llama_ref.c, its tokenization, greedy continuations
and logits for every corpus model into build/ref/llama/
(same test), and for Qwen3.5-9B a continuation of 100 tokens and a
prompt of some 300, which tests/model/test_qwen35_public.c
compares with OKLM's (that prompt in batches of 64, two sequences in one
pass, a sequence's recurrent state through a file, a copy and a cut; on
the CPU, then on OpenCL and on Vulkan when the machine has them, where
the device is also compared with the CPU step by step and run with a
paged cache and with half of the blocks on the CPU;
OKLM_Q35_BACKENDS=cpu,vulkan chooses the backends), and
that continuation a second time with llama.cpp's keys and values kept as
F32 (LLAMA_REF_KV=f32; a change of its last places only),
which the test sets against the first: llama.cpp's distance from itself
is the measure of OKLM's distance from it (hybrid-layers-design.md);
the tiny-qwen35 fixture's reference is llama.cpp's too,
recorded by tools/gen_model_fixtures.py. For Gemma 4 12B,
whose whole Q4_K_M file moves by units of logit under one rounding, the
script also writes two of its blocks alone in F16
(tools/gguf_cut_blocks.py) and records llama.cpp on them:
there the two programs must agree closely
(test_llama_cpp_gemma4).
tests/model/test_gemma4_public.c runs the published Gemma 4
files that are under models/ (12B, E2B, E4B, 26B-A4B) on
OpenCL and on Vulkan against those recordings and against a CPU context
step by step, then with a paged cache, with half of the blocks on the
CPU and, for the 26B-A4B, with its experts in host memory
(OKLM_G4_BACKENDS=vulkan and
OKLM_G4_FILES=e2b,12b choose).
tests/model/test_archs_public.c does the same for the
published files of the architectures with an attention output gate, of
gpt-oss, of lfm2 and lfm2moe, of
starcoder2, of mistral3 and of DeepSeek-Coder
that are under models/ (Spark-X2.6 1.7B and 4B, Muse
Glimmer 30B, gpt-oss-20b, LFM2-350M, Cyber-Prime 1.1 2.6B,
Tini-Cybersec-8B-A1B, StarCoder2-3B, Devstral Small 2 24B,
DeepSeek-Coder 1.3B and 6.7B; OKLM_PUB_BACKENDS=vulkan and
OKLM_PUB_FILES=spark,muse,gpt-oss choose). For the Ternary
Bonsai 2 files the script records the same prompts and 100 greedy tokens
(tests/model/test_bonsai_public.c, which runs each file on
the CPU, on OpenCL and on Vulkan, a device also against the CPU, with a
paged cache and with half of its blocks on the CPU;
OKLM_BONSAI_BACKENDS=vulkan and
OKLM_BONSAI_FILES=4b choose): the Q2_0 files with the
pinned build, the PQ2_0 and PTQ1_0 files, which upstream llama.cpp does
not read, with PrismML's fork built beside it at a pinned commit
(build/llama.cpp-prism, CPU only, only when one of those
files is in models/); the hashes
tests/core/test_quant_p128.c holds for these two types come
from that fork's dequantizers and reference quantizers. Chat templates
are checked against jinja2 in Hugging Face's environment
(tools/jinja_ref.py): committed cases from
tools/gen_jinja_fixtures.py (the GLM-4.5 and GLM-4.7
templates among them), published templates rendered into
build/ref/jinja/ by tools/gen_jinja_ref.py
(tests/model/test_jinja.c). Grammar token masks are checked
against feeding every token's text to the grammar, over a byte-level
vocabulary and through split UTF-8 characters, and constrained
generation with SmolLM2 must produce JSON and tool calls that parse
back, as must a GLM-4.5 call generated by a tiny random model
(tests/model/test_grammar.c).
tools/gen_case_fixtures.py records Python's str.lower /
upper / title / capitalize of 1 500 random strings over every cased
character, and Python's simple case folds
(tests/fixtures/tok/case.json);
tools/gen_fp8_fixture.py writes an FP8 checkpoint with
block scales and its dequantized twin; tools/check_split.sh
compares oklm split / oklm merge with
llama.cpp's gguf-split, and
tools/check_quantize.sh oklm quantize with
llama-quantize (every file type, byte for byte).
test_quant_k and test_quant check the K-quant,
IQ4, FP4, ternary and low-bit quantizers, and the decoding of the
latter, against hashes of ggml's output for fixed rows and blocks
(tests/core/iq_blocks.h), and the weighted (importance
matrix) quantizers likewise; check_quantize.sh also
quantizes with bartowski's importance matrix (legacy format) and with
one written by llama-imatrix (GGUF format). The grid IQ types are
decoded against gguf-py's dequantization of fixed blocks
(test_quant), and tools/gen_iq_models.sh makes
SmolLM2 in every grid type with llama-quantize, whose greedy choices and
logits test_infer_public compares with llama.cpp on both
backends; tools/gen_iq_grids.py extracts ggml's codebook
tables into src/core/iq_grids.inc. Their quantizers are
OKLM's own: test_quant bounds their error on fixed rows,
checks that importance weights lower the weighted error and that a power
of two scales the result exactly; the perplexity comparison with
llama-quantize is in convert.md. Training is checked
against finite differences of the loss and the float64 references of the
tiny GPT-2 and Llama fixtures (tests/train/test_train.c),
and against llm.c (tests/train/test_train_gpt2.c, after
sh tools/fetch_llmc.sh): its PyTorch reference state of
GPT-2 124M (logits, loss and gradients of one batch, losses of ten
steps) and the CPU run on Tiny Shakespeare its README publishes (training.md). Python is only used
to produce or check data; no C code calls it.
Hidden states against llama.cpp
When the logits of a model differ from llama.cpp's by more than its neighbours', the two forward passes are compared step by step on the same tokens:
L=build/llama.cpp
cc -O2 -c tools/llama_dump.c -I$L/include -I$L/ggml/include -o build/llama_dump.o
c++ -o build/llama_dump build/llama_dump.o $L/build/src/libllama.a $L/build/ggml/src/libggml.a \
$L/build/ggml/src/libggml-cpu.a $L/build/ggml/src/libggml-base.a -fopenmp -lpthread -lm
build/llama_dump model.gguf ref.bin "The capital of France is"
OKLM_DUMP=own.bin oklm run -m model.gguf -p "The capital of France is" -n 1 --backend cpu
python3 tools/dump_compare.py ref.bin own.bin # every step, in the graph's order
python3 tools/dump_compare.py ref.bin own.bin --step l_out # the stream after each block
tools/llama_dump.c (a test tool, linked against the
pinned llama.cpp, never into OKLM) writes every tensor llama.cpp's graph
names (attn_norm-3, Qcur-3,
kqv_out-3, ffn_out-3, l_out-3...)
through its evaluation callback; OKLM_DUMP makes the llama
family's forward pass (src/model/arch_llama.c,
src/model/dump.c) write its own under the same names;
tools/dump_compare.py prints, for each step both have, the
largest difference and the norm of the difference over the norm of the
reference. A dump holds every step of every block for every token of the
prompt: a few tokens are enough (a 12B model, 6 tokens: 150 MB). In a
gemma4 block with experts the router's logits and
probabilities, the joined gate | up product of each token's experts,
their gated product, their outputs and their sum are steps too (a tensor
llama.cpp keeps as a strided view, such as the gate half alone, is not
in its dump).
A difference grows from block to block, so the first block that departs says little about the others. Two more variables separate them:
OKLM_DUMP_FEED=ref.bin: every block starts from llama.cpp's stream (l_outof the block before) instead of OKLM's own, so each block's figure is its own;OKLM_DUMP_FEED_OPS=1with it: inside a block too, the attention takes llama.cpp's rotated queries and keys and its values, the output projection its heads' outputs, the feed-forward its input and its gated product (the experts of agemma4block theirs). Each matrix product and the attention are then compared on equal operands, and the step that is computed differently is the one whose figure stays large.
OKLM_ATTN_F16=1 and OKLM_GELU_F16=1 make
OKLM round where ggml does (the queries and the probabilities of
attention over an F16 cache, the GELU table): a step whose figure falls
to rounding noise with one of them differs by that rounding and by
nothing else (inference.md has the
12B's figures). tools/ref_compare.py compares two records
of tools/llama_ref.c made on the same tokens
(ids:1,2,3|4,5 feeds a continuation): llama.cpp with
itself, flash attention on (LLAMA_FLASH=1) and off, or its
cache in single precision (LLAMA_KV_F32=1) and in its
default F16, which gives the distance between two runs of the
reference.
The terminal chat on a server
tests/harness/test_cli_chat_url.c types into
oklm chat --url against a small OpenAI-style server of its
own, which records each request and answers by the words the request
holds (a reply cut, a refusal of an open last message, a prompt too
long, reasoning alone, a summary); no model is loaded, so it runs
everywhere make test does.
tests/client/test_remote_models.c reads the lists of
GET /v1/models the same way.
tests/client/test_request_members.c runs the rule the chats
and the agent's backend share for a request a server refuses by one of
its members (src/client/request_members.c): the member
named in the refusals of several servers, a name that is only part of a
word, the members an answer cannot do without, and the request rebuilt
without its first, a middle and its last member.
sh tools/check_chat_url.sh runs the chat on real
servers, on loopback and on the CPU, with Qwen3-0.6B: llama.cpp's server
(the pinned build of the test reference, LLAMA_SERVER=path
when it is elsewhere; without it those checks are skipped with a line),
the same with --api-key, its router over two models (when
SmolLM2-135M is there), and oklm serve. PORT=N
is the first of the four ports it listens on (18480). 33 checks, a
little over a minute in all on the machine it was written on (the last
two on the stand-in of the window's check, which lists three models: Tab
after /model on a pseudo-terminal completes a name the
server lists, and lists several): two turns, the reasoning shown and not
kept, --resume, a cut reply gone on with as an open last
message, /compact, the full-screen UI on a pseudo-terminal,
the key missing, wrong and right, the list of two models (refused for
input that is not typed, chosen in a picker on a terminal,
/model N), and on oklm serve its context read
and resized, a reply its context cut gone on with in a larger one,
--resume and /compact.
OpenCL
tests/core/test_opencl.c checks every OpenCL kernel
against the CPU (or a double-precision reference for matrix products)
and the CPU fallback; tests/model/test_model.c and
tests/model/test_infer_public.c repeat their end-to-end
checks on the OpenCL backend. All three skip the OpenCL parts when no
device is found (CI). The sanitizer targets leave OpenCL out;
make asan-opencl covers it (see opencl.md).
Vulkan
tests/core/test_vulkan.c checks the Vulkan backend's
core (copies, ordering, wide dispatches, the fallback, the fused
residual tails) and its element-wise ops against the CPU, and
tests/core/test_vulkan_<group>.c (norm, matmul,
attention, gemm, q8, moe, ssm) one file per group of ops; all run on
every Vulkan GPU present (tests/core/vk_test_util.h:
llvmpipe left out unless OKLM_VK_DEVICE names it,
OKLM_VK_DEVICES=none skips them) and skip without one (see
vulkan.md).
Testing on another GPU
tools/gpu/report.sh is for someone with a GPU OKLM has
not run on (NVIDIA first): from a fresh clone it checks the
prerequisites, records the system, builds OKLM, runs the GPU tests on
every device and writes one plain-text file,
oklm-gpu-report-<host>-<date>.txt, to send
back. It installs nothing, limits every step in time, goes on after a
failure and exits 0 once the report is written.
What to install (the script names what is missing):
- a C17 compiler (gcc or clang), GNU make, git, coreutils'
timeout; - the GPU driver with its OpenCL and Vulkan parts (on NVIDIA, the
proprietary driver brings both), the OpenCL ICD loader
(
ocl-icd-libopencl1,ocl-icd) and the Vulkan loader (libvulkan1,vulkan-icd-loader); - optional:
clinfoandvulkaninfo(vulkan-tools) to list the devices,glslc(shaderc, the Vulkan SDK) for the Vulkan backend, which a build without it leaves out.
How to run it:
git clone <OKLM repository> oklm && cd oklm
git clone --depth 1 --branch next https://git.nilorea.net/gullradriel/nilorea-library external/nilorea-library
sh tools/gpu/report.sh --models --download # about 2 GB of models, up to an hour--clone-nilorea clones nilorea-library itself when none
is found (it is looked for in --nilorea-dir,
NILOREA_DIR, external/nilorea-library,
../nilorea-library,
../../nilorea-library).
| Option | Effect |
|---|---|
--quick |
the device tests only: no make test, no speed
table |
--models |
the speed table: SmolLM2-135M Q8_0, Qwen2.5-0.5B Q4_K_M, Qwen3-0.6B Q8_0, Llama-3.2-1B Q4_K_M (OLMoE-1B-7B Q4_K_M when present), 128 greedy tokens after "The capital of France is" on the CPU and each device |
--download |
with --models: fetch the missing ones with
tools/fetch_models.sh (checksums from
tests/public_models.txt) |
--keep-going |
after a failed build, test the binaries already built |
--models-dir DIR |
the models' directory (default models/) |
--nilorea-dir DIR, --clone-nilorea |
where nilorea-library is, or clone it into
external/ |
--out DIR, --jobs N |
where the report goes, make -j N |
What the report holds, section by section: the prerequisites found
and missing; the OS, kernel, CPU, memory, GPUs (lspci),
drivers (nvidia-smi, clinfo,
vulkaninfo --summary and the Vulkan extensions OKLM's fast
paths use), compiler versions and the OKLM commit; the build's warnings
and errors; the devices oklm devices finds, each selected
by index (OKLM_CL_DEVICE, OKLM_VK_DEVICE); per
device, test_opencl, test_opencl_q8,
test_opencl_wmma, test_model and
test_kv_file, and every test_vulkan* binary
the build has (with test_model and
test_kv_file again, OpenCL left out), each with its checks,
failures, failing lines, exit code and time; without
--quick, make test's result; with
--models, generation and prompt t/s per backend and device
and whether the greedy text matches the CPU's and the first device's
(its first 240 bytes). Of the environment it writes only OKLM's device,
backend, thread and kernel variables (build.md) that are set, and
the home directory as ~.
What to send back: the report file. A test that fails on a new vendor is what the report is for; the run goes on to the next one.
Windows
make wine-test builds the Windows binaries (MinGW,
statically linked) and runs every test under Wine, public models
included when present. It found two portability bugs the Linux runs
could not: the safetensors and GGUF overlap checks depended on the order
the C library's qsort leaves equal keys in. It also showed that
mingw-w64's roundf is wrong for the float just below a
half, which the quantizers no longer call (test_quant,
test_round_below_half).
sh tools/check_kept_servers.sh keeps a server of the
test model across logins for real, with systemd's user manager
(oklm serve --keep, a kill the manager answers by starting
it again, --stop, systemctl --user start,
--rekeep, --unkeep; a session with another
XDG_RUNTIME_DIR reaching the manager by itself; 42 checks).
Its unit carries the script's name and its pid and is removed when the
script ends, however it ends; it is skipped where the user manager does
not answer, where tests/harness/test_kept.c still runs the
module against a script in place of systemctl.
sh tools/check_gui_kept.sh does the same from the server
manager's window under Xvfb, with the tool's own presses
(--press keep:NAME, start,
stop:NAME, restart:NAME,
ask-rekeep:NAME, rekeep:NAME,
unkeep:NAME, profile:NAME, yes,
no): each step is a window that does its presses, writes
what it said and ends, and no place of the form is clicked, so a change
of the layout does not break it. By default the service manager is a
stand-in (tools/standin_systemctl.py through
OKLM_SYSTEMCTL): its units are under
build/check-gui-kept, its services are processes it starts
and ends, and neither a unit of the user's nor the user manager is
touched; 21 checks. REAL=1 sh tools/check_gui_kept.sh runs
the same steps against systemd's user manager, with a unit named after
the script and its pid among the user's, removed when the script ends: a
run for a person to start, skipped where the user manager does not
answer.
sh tools/check_gui_form.sh [tiny|canary|device] presses,
under Xvfb and without a pointer, the buttons of the two tools' options
that a screenshot does not: the chat window's Open in the server
manager..., the server manager's Re-keep with
form (against a stand-in for systemctl: no unit of
the user's is touched) and its Start server with every
tab of the Advanced window set, the server started being compared with
the form and stopped (gui.md);
make gui-test runs it on the tiny models.
make gui-device-test runs its device mode,
which is not part of gui-test (that one passes without a
GPU): on each device backend that answers (oklm devices;
BACKENDS names one), one server at a time started from the
form with Start server, compared with its own account
and stopped: --gpu-layers 20 of the 28 layers of Qwen3-0.6B
(n_gpu_layers of GET /api/context,
device_memory of /api/stats),
--n-cpu-moe 8 --expert-gpu 0.25 on Granite 3.1 1B A400M
(n_cpu_moe, expert_gpu, and the tier's slots
in /api/stats), and --devices 2 where two
devices of a backend answer (n_devices; Vulkan with
OKLM_VK_DEVICES=all, a discrete and an integrated card,
else two OpenCL devices), skipped with a line where there is one. With
Qwen3-8B Q4_K_M under models/ it then checks the memory
line against the server it describes, on each backend: at 40960 tokens
the line says It fits with a smaller cache, the server
started from the form tells the cache the line planned
(cache_tokens, within 2 %), and the chat window's
Apply restarts its model with that cache; at 131072
tokens, where the device holds the model with less than one context of
cache, the line says the layers that go there and the cache on each
side, and the server started from the form has those layers and that
cache and answers; with every layer asked for there the line says
It does not fit and Apply refuses; a
Cache size of 4096 MiB is the server's cache to the
token; a model loaded in place with 131072 tokens kept for it is served
with them, fewer layers on the device, and one whose kept settings do
not fit (every layer asked for too) is loaded with the window's own. It
ends with what shares the device: a server of SmolVLM-256M with its
projector and one of Qwen3-8B with Qwen3-0.6B as its draft model, each
started from the form, whose cache is the one the line planned once it
counted them, and, for the projector, whose device_memory
tells the share the line counted; then the models beside the model where
their backends put them (Qwen3-0.6B with bge-small on the device, whose
device_memory.embed is the share the line counted; then
with bge-small and MiniLM-L6 on the CPU, whose contexts as the line
counted them are what the server tells in host_memory, to
the byte); then a model loaded in place of another with nothing kept for
it (Qwen3-0.6B after Qwen3-8B with 20 layers on the device: all its 28
layers there; Qwen3-8B at 32768 tokens on a device held to 8 GiB: loaded
with the layers that leave the device a context, and answering); then a
model larger than its device (Qwen3-8B on the device held to 4 GiB: the
line says the layers that go there and the cache on each side, and the
server started from the form has those layers and that cache, to the
token); then what the line counts beside the model against the server's
own figures, to the byte: SmolVLM-256M's projector on the CPU
(host_memory.vision), OuteTTS 0.2 on the CPU loaded at the
start (host_memory.speak and speak_call),
bge-small on a second Vulkan device where one answers
(side_devices, with OKLM_VK_DEVICES=all), and
a chat window's line counting the embedding and the reranking models it
finds by itself for a long document; and last one server seen by the
three lines that speak of it: for a server started from the form at a
context the form says fits (Qwen3-0.6B at 8192 tokens) or fits with a
smaller cache (Qwen3-8B at 40960), the server manager's line for it
(--press context:N) and the line of a chat window attached
to it are not red at that context and are written for the cache the
server tells (cache_tokens); at 131072 tokens, above that
cache, the two tools' lines are red, since a server that runs keeps its
layers, and the form is red with every layer asked for and says the
layers it would place with them left to what fits; a server whose cache
ends before its own context (--kv-mem 64 at 16384 tokens)
is red in both tools at the context it runs with (tiny
checks the same on the CPU, where the cache is in host memory). Without
a device backend, or without the two models under models/,
it says skipped and ends with status 0.
sh tools/docs/gui_shots.sh takes the pictures of gui.md that the tools' unattended options
reproduce.
On 2026-11-15, after
make MINGW=1 BUILD=build/mingw tests apps gui (which no
longer built: tests/harness/test_cli_chat.c included two
headers of static functions it uses on POSIX only), the nine
test_gui_* and test_mem_plan passed under Wine
11 as make wine-test runs them,
test_gui_settings after its expectations of a quoted value
took the system's quote (five checks expected '...' of a
Windows command line), and test_server.exe passed with 2575
checks, its Vulkan and OpenCL tests included. The options' graphical
flows run there without a pointer are in gui.md.
sh tools/check_gui_wine.sh runs the graphical tools of
the Windows build (make MINGW=1 BUILD=build/mingw apps gui,
after make windows-deps) under Wine and Xvfb, in a Wine
prefix of its own under build/mingw/wine-gui-check, on the
CPU with the tiny model of the server test, and checks what each flow
leaves: files, processes, answers on a port, what a window wrote on its
error stream. Every button is pressed by the tool's own test aid
(--press, gui.md): no place of
a window is clicked; the system's file dialog is answered from the
keyboard, and a window that stays is closed with Alt+F4. The chat window
(a message, a PDF attached, a recording transcribed, two files dropped
on it, the two exports through the system's dialog), the chat window
started with nothing named (a server chosen, a file loaded in the
window, a load refused, the system's file dialog answered from the
keyboard), the launcher from a staged folder named in three scripts (a
tool that outlives it, the documentation's page handed to a stand-in for
xdg-open), the model manager's list, the server manager (a
window a step: a server started from the form and compared with it,
restarted under another pid, its Dashboard and
Chat, stopped; a server found by its port alone,
Stop asking first, and stopped), the conversion (a GGUF
file, a Hugging Face folder) and the inspector, the training window (a
run started, left by its window, found by another, stopped by itself
with a checkpoint; the Fine-tune, LoRA and RL tabs, Merge
adapter, Export adapter..., Open a
chat), the sessions browser (Attach on a
listening agent, which oklm.exe agent --stop-listening then
ends, Resume in a console, Export .md,
Delete), an index built by the retrieval manager,
searched, and a judged set scored by Evaluate, a server
kept across logins by oklm.exe (the Run key
written, listed, started as at login, changed, removed). Some checks
need the models of MODELS=DIR (a store as
oklm models pull fills it, models by default):
the Chat data and Advanced tabs with smollm2-135m, a
picture with smolvlm-256m, a sound with
ultravox-1b, each skipped with a line when its model is not
there. On 2026-11-16, with the canary set (no ultravox-1b:
the sound skipped), 58 checks passed in 186 seconds; the group
start, added after, alone (CHECKS=start): 13
checks in 34 seconds. The group saved, added on 2026-11-17
(CHECKS=saved, 5 checks): a server saved from the chat
window's Connect window with its key, the settings file under the
prefix's %APPDATA% holding it, the key in no output, and
forgotten again. The window gives that file an access list of its owner
alone (oklm_file_owner_only(), which
tests/io/test_sys.c runs on every system) and reads one
back: Wine takes the list, the calls succeed, and keeps none, what it
reads back being made from the file's Unix mode (three entries, not
protected), so the check takes either answer there and only Windows
itself shows the list protected with the owner's entry alone.
CHECKS="servers train" runs those groups only;
MINGW_DEPS_DIR as for make;
WINE_GUI_PREFIX=DIR names another Wine prefix than the
check's own (never the user's ~/.wine). It is skipped with
a message when Wine, Xvfb, xdotool, ImageMagick's import or
the MinGW build is missing; the drop needs python3-xlib
(tools/xdnd_drop.py is the source side of the X drop
protocol, which Wine hands on as the system's) and is left out with a
line without it. The screenshots are left in
build/mingw/wine-gui-check/shots, and what each window
wrote in build/mingw/wine-gui-check/logs.
sh tools/check_gui_press.sh, part of
make gui-test, drives the same presses of the launcher, the
training window, the sessions browser and the retrieval manager on Linux
(34 checks), and sh tools/check_gui_start.sh, part of it
too, starts the chat window with nothing named (42 checks: no model
loaded, a server, a saved server or an agent chosen, an installed model
or a file loaded in the window once the memory line judged it, a load
refused, gui.md),
sh tools/check_gui_remote.sh links it to a server that is
not OKLM's (115 checks on a stand-in and on oklm serve, 15
more on llama.cpp's server where it is built: what the window says of
it, OKLM's own routes asked once, the context as the window's setting, a
reply gone on with, a compaction by estimate, a server saved with its
key in a file of mode 600 and chosen at a start with nothing named, gui.md),
and sh tools/check_gui_advanced.sh opens every tab of the
Advanced window of both tools in a window of 760 by 480 pixels, then
sets the chat window's Other field and its MoE disk options (38 checks,
gui.md).
sh tools/check_gui_full.sh, part of it too, has replies cut
and gone on with while they are read aloud and in a spoken conversation,
and a conversation with pictures compacted, on the canary models that
are there (19 checks, gui.md).
These are Wine's runs, not Windows': the access lists of the registry's
folder, a real console, a sound card, the system's dialogs and its
certificate store are Wine's own there.
Paths in UTF-8 on Windows (build.md):
tests/io/test_sys.c (test_utf8_paths, on every
system) makes, writes, lists, maps, renames and removes files named with
accents, Greek, Japanese and a character outside the basic plane, a name
of 253 bytes, and reads a variable of the environment with such a value
back; test_procreg.exe starts a program with the argument
été Ελληνικά 日本語 and reads it back from its output.
A job's output above 2 GiB: tests/harness/test_jobs.c
(test_big_output, on every system, the first of that file's
tests of a started job that the Windows build runs) gives the log of a
job that ended a hole of 3 GiB and 5 MiB of lines after it, then reads
its size, a place above 32 bits, a reader's last lines and its last
line; a sparse file, so 5 MiB are written.
The Windows build's float functions (expf,
cosf, sinf, powf,
logf) do not round like glibc's, so its CPU logits are not
the Linux build's bits: on the prompt of 100 arbitrary tokens
test_opencl_q8 gives SmolLM2-135M, the two CPU references
are 0.50 (Q8_0) and 0.54 (Q4_K_M) apart, while the OpenCL device returns
the same bits from both builds. That test's limit on the device against
the CPU is therefore 0.75 on Linux (0.607 and 0.625 measured) and 1.1 on
Windows (0.905 and 0.651).
Fuzzing
make fuzz # build fuzz/fuzz_{gguf,safetensors,regex,tokenizer,jinja,grammar,pdf,toolcalls,shsplit} (clang, libFuzzer, ASan, UBSan)
make fuzz-run FUZZ_TIME=60 # run each for 60 s; seeds in fuzz/corpus/, new inputs in build/fuzz/corpus/Invariants checked
- Quantized SIMD kernels are bit-identical to the scalar kernels.
- Matmul results are bit-identical for 1, 3 and 7 threads.
- Several application threads running ops concurrently get the same
bits as one thread alone (
test_threads, meant for ThreadSanitizer). - The passes of a device run in the order they asked for it, a yield
hands the device to the passes that wait and takes it back with the same
nesting, and a pass on several devices takes them in the order of their
registration whatever the order they are named in
(
test_threads:test_pass_handoff,test_pass_devices, on devices that compute nothing). - A context spread over two devices, a context on one of them and an
image encoder on the other, each in a thread of its own, give the floats
each gives alone (
test_multi_device: two OpenCL instances, two Vulkan instances, and two different Vulkan GPUs when the machine has them).
Sanitizers and static analysis
make asan-test # ASan + UBSan (-fno-sanitize-recover=undefined)
make tsan-test # TSan
make check # cppcheck, gating
make CC=clang BUILD=build/clang testA change is mergeable when all four are clean.
CI runs them in a Debian 13 container, whose tools are older than a
recent machine's: cppcheck 2.17 reports what 2.22 does not, GCC 14
refuses what GCC 16 accepts. A clean run on the release machine does not
say CI passes. Before a version tag the jobs of
.gitea/workflows/ci.yml are run in a
debian:trixie container on the tree to tag (the build and
the suite with GCC and with clang, make format-check,
make check, make doc,
make docs-lint, make fuzz-run,
packaging/ci-packages.sh), and the tag is set on a commit
whose run on the server is green.
Logging during tests
Tests trigger errors on purpose (out-of-range ids, full arenas, bad
shapes). The harness sets the log level to LOG_CRIT so
those do not clutter the output; set OKLM_LOG_LEVEL=DEBUG
to see everything.