OKLMdocumentation v1.0.3
On this page

ADR-0015: Test references without PyTorch

Records how numerical tests obtain their expected values while PyTorch is not available on the development machine.

Status: accepted (2026-10-29).

Context

The intended reference for layer tests is golden tensors generated once with PyTorch in double precision (tools/gen_golden.py) and stored as small F32 GGUF files. PyTorch is not installed on the development machine, and the GGUF writer does not exist yet.

Decision

  • Every op test computes its expected values with a straightforward double-precision implementation written in the test from the mathematical definition (softmax, norms, RoPE including ggml's YaRN formulas, attention, matrix products), independent of the library's code structure.
  • Format tests use independent decoders written from the byte layout (per-element decoding of every block type), exhaustive checks where the domain is small (all 65 536 halves), and the hardware F16C instructions as a second oracle for half conversion.
  • Tolerances live in tests/tolerances.h with their rationale.
  • No PyTorch (decided 2026-10-30). Training adds two references that need no third-party framework: gradients checked by finite differences in the tests themselves, and a hand-written float64 backward pass in numpy like the forward one. End to end, llm.c (plain C) is built as a test tool, the way llama.cpp is, and its published reference state is downloaded.

Consequences

Tests run anywhere with no data files and no Python. Exact agreement with PyTorch's numerics is not a goal; agreement with llama.cpp (inference) and llm.c (training) is.

End-to-end references for inference

  • An independent forward pass in numpy float64 (tools/infer_ref.py), written from the model definitions and reading weights through gguf-py's dequantizers: committed tiny random models (tests/model/test_model.c) and the SmolLM2 files.
  • llama.cpp itself, as a test tool only: tools/gen_llama_ref.sh clones a pinned release into build/llama.cpp, builds it, links tools/llama_ref.c against it and records its tokenization, greedy continuations and logits for every model of the corpus (tests/model/test_infer_public.c). Nothing of llama.cpp is compiled into OKLM or needed to build or test it offline.
  • Python's jinja2 in Hugging Face's environment for chat templates (tools/jinja_ref.py, ADR-0006).