OKLMdocumentation v1.0.3
On this page

How it works

A guided tour of OKLM for someone who knows a terminal but is new to the inside of language models. For each topic: the idea, how OKLM does it, and commands to try. The details live in the reference pages linked from each section; this page explains why things are the way they are and gives the commands that show it.

Status: maintained. Every command below was run with the CPU backend on the small models of models/; the outputs are those runs, cut short.

oklm stands for build/release/bin/oklm (see build.md). oklm help lists every command and option; each command also prints its own usage when its arguments are wrong.

Contents

  1. Model files
  2. Quantization
  3. From a checkpoint to a quantized model, end to end
  4. Memory and the KV cache
  5. Backends and speed
  6. Generation
  7. Documents
  8. Embeddings
  9. The server
  10. The coding agent
  11. Training
  12. Images, sounds and speech
  13. Refusal analysis
  14. Developer tooling
  15. Files, caches and environment
  16. Limits worth knowing

Model files

A model is a list of named tensors (arrays of numbers: the weights) plus metadata: the architecture, its sizes, the tokenizer, the chat template. Two file families carry them.

  • Hugging Face checkpoints: a directory with config.json (the architecture and sizes), one or more .safetensors files (the tensors, with an index when sharded), tokenizer.json and tokenizer_config.json (the vocabulary and the chat template). This is how models are published for training frameworks, usually in BF16.
  • GGUF: one file (or a few shards) holding the tensors and every piece of metadata, tokenizer included. It is llama.cpp's format, and the one block-quantized weights (Q4_K, Q8_0, ...) are stored in.

OKLM reads both and writes GGUF. The byte layouts are in formats/gguf.md and formats/safetensors.md.

oklm convert turns a checkpoint into a GGUF file: it renames each tensor to the GGUF name, reverses the shape order, permutes the Q and K rows of Llama-style models into the layout GGUF expects, keeps norms in F32 and writes the hyperparameters and tokenizer as metadata. Target types are f32, f16 (the default), bf16, q8_0, q4_0, q4_1, q5_0, q5_1; the K-quants come from oklm quantize afterwards. The supported architectures and what each needs are in convert.md.

$ oklm convert models/smollm2-135m-hf --out smollm2-f16.gguf --type f16
convert: models/smollm2-135m-hf -> smollm2-f16.gguf (272 tensors, 0 fell back to F16)

Why run loads a checkpoint directly. Safetensors files are raw arrays at known offsets, so OKLM maps them and uses F32, F16 and BF16 matrices in place, with the hyperparameters from config.json and the tokenizer from tokenizer.json. FP8 matrices are widened to BF16 at load time. Nothing is written to disk. Converting still pays off for a single file, faster loading, and quantization (inference.md).

$ oklm run -m models/smollm2-135m-hf -p "The capital of France is" --temp 0 -n 12 --backend cpu
The capital of France is Paris. Paris is the largest city in France and the capital

Inspecting. oklm info model.gguf prints the metadata, the tensor count per type and a verdict (loadable, loadable with warnings, unsupported, with reasons); -t lists every tensor. --json prints the same as one JSON object for programs, with the model's layout: each block's tensors grouped by role (attention, feed-forward, experts, norms) with their types and bytes, the embeddings and the head (convert.md).

Shards. Large models are often published as several GGUF files. oklm split writes such shards and oklm merge joins them; the files are those of llama.cpp's gguf-split, byte for byte (formats/gguf.md), so either tool reads the other's. Every oklm command opens a split model from any of its shards.

$ oklm split SmolLM2-135M-Instruct-Q8_0.gguf part --split-max-size 64M
3 shard(s): part-00001-of-00003.gguf ...
$ oklm info part-00002-of-00003.gguf | grep -i 'split\|verdict'
  split.no                                     = 0
  split.tensors.count                          = 272
  split.count                                  = 3
verdict     : loadable
$ oklm merge part-00001-of-00003.gguf merged.gguf

--split-max-tensors N splits by tensor count instead of size.

Quantization

A weight stored in F32 takes 32 bits, in F16 or BF16 16. Generation reads every weight once per token, so a smaller file is both less memory and less to read: on a CPU, decoding speed follows memory bandwidth. Quantization stores weights in fewer bits at the cost of a small error.

Blocks. The ggml quants group weights into blocks that share a scale: Q8_0 stores 32 weights as 32 signed bytes plus one F16 scale (34 bytes, 8.5 bits per weight). The K-quants use super-blocks of 256 with sub-block scales, themselves quantized. oklm types lists every element type with its block size and bytes per block; bits per weight are bytes x 8 / block:

Type Block Bytes Bits per weight Kind
F32 1 4 32 float
F16, BF16 1 2 16 float (BF16 keeps F32's range, fewer digits)
Q8_0 32 34 8.5 legacy block quant
Q6_K 256 210 6.56 K-quant
Q5_K 256 176 5.5 K-quant
Q5_0 32 22 5.5 legacy
Q4_K 256 144 4.5 K-quant
Q4_0 32 18 4.5 legacy
IQ4_XS 256 136 4.25 non-linear 4-bit
MXFP4 32 17 4.25 4-bit float
Q3_K 256 110 3.44 K-quant
Q2_K 256 84 2.63 K-quant
IQ2_XXS 256 66 2.06 codebook (grid)
IQ1_S 256 50 1.56 codebook (grid)

The dequant and matmul columns of oklm types say whether OKLM can read a type and multiply with it directly (Q8_1 and Q8_K have no matrix product of their own). The layouts are in formats/quant.md.

File types are mixes. oklm quantize takes llama-quantize's file types (Q4_K_M, Q5_K_M, Q8_0, IQ4_XS, ...), not single element types. Q4_K_M keeps most matrices in Q4_K, but the output matrix in Q6_K, attention V and some FFN down matrices a step or two higher, and norms in F32. A matrix whose rows are not a whole number of blocks falls back to a type with smaller blocks: SmolLM2's rows are 576 wide, not a multiple of 256, so its "Q4_K_M" is mostly Q5_0:

$ oklm info smollm2-q4_k_m.gguf
...
tensors     : 272, 134.52 M parameters, 98.87 MiB
  F32      61
  Q5_0     166
  Q8_0     15
  Q4_K     16
  Q6_K     14

--pure puts every matrix in the base type instead. The output equals llama-quantize's byte for byte for every file type except the grid IQ types, whose blocks come from OKLM's own codebook search (convert.md).

The trade-off. Fewer bits mean a smaller file and faster decoding, and more error. The table of the next section measures it on one model: Q8_0 is within 0.1 % of F16's perplexity, Q4_K_M within 3 %, at 54 % and 39 % of the F16 size. Below 4 bits the error grows quickly, which is what the importance matrix is for.

Importance matrices. Not every weight matters equally. A weight multiplied by inputs that are large most of the time changes the output more than one whose inputs are near zero. oklm imatrix runs the model over a calibration text and records, for each matrix of each block, the sum of the squares of every input column (llama-imatrix's method and file format). oklm quantize --imatrix divides those sums by their counts and weights the quantizer's error by them, so the scales are chosen to keep the important columns exact at the expense of the others. The legacy quants, Q2_K to Q6_K, IQ4_NL, IQ4_XS and the grid IQ types use it. oklm quantize warns when a type meant for one (Q2_K_S, the IQ1 and IQ2 types, IQ3_XXS) gets none, where llama-quantize refuses the 1- and 2-bit ones; the 1-bit types are unusable without one.

Choosing the calibration text. The importance matrix records what inputs the model sees on that text. Calibrate on text like the real use (code for a coding model, the target language for a translator) so the columns that matter in use are the ones protected. Measure on a different text: perplexity on the calibration text itself rewards the quantizer for fitting that one text and overstates the quality.

From a checkpoint to a quantized model, end to end

SmolLM2-135M-Instruct from its Hugging Face checkpoint to a Q4_K_M GGUF, checked at each stage. The calibration text is the first 400 kB of Tiny Shakespeare, the evaluation text its last 100 kB, so the two do not overlap.

head -c 400000 data/tinyshakespeare.txt > calib.txt
tail -c 100000 data/tinyshakespeare.txt > eval.txt

1. Convert to F16. oklm plan reads GGUF headers only, so the checkpoint is converted first (it refuses a checkpoint directory with "convert a Hugging Face checkpoint first").

$ oklm convert models/smollm2-135m-hf --out smollm2-f16.gguf --type f16

2. Check the memory it needs.

$ oklm plan -m smollm2-f16.gguf -c 2048
model       : smollm2-f16.gguf (llama, 30 layers, 1 file)
context     : 2048 tokens x 1 sequence, batch 512, KV F16 / F16, backend cpu
...
host memory :
  weights resident              256.6 MiB
  KV cache                       45.0 MiB  (30 layers x 1.5 MiB)
  activations                   114.0 MiB
  scratch                         1.6 MiB
  total                         417.2 MiB
RAM         : 105.50 GiB available of 124.64 GiB, 3.78 GiB kept free
verdict     : fits resident

3. Baseline perplexity. Perplexity is the exponential of the mean negative log-probability the model gives each actual next token: how many choices it is, on average, unsure between. Lower is better. oklm ppl cuts the text into windows of -c tokens (512 by default) and scores the second half of each, so every scored token has at least half a window of context, as llama.cpp does.

$ oklm ppl -m smollm2-f16.gguf -f eval.txt -c 512 --backend cpu
...
perplexity: 23.4932 over 16065 tokens (512-token windows)

4. Importance matrix on the calibration text (100 chunks of 512 tokens; about two minutes on 8 threads here):

$ oklm imatrix -m smollm2-f16.gguf -f calib.txt -o smollm2.imatrix.gguf -c 512 --chunks 100
imatrix: 210 matrices over 100 chunks of 512 tokens -> smollm2.imatrix.gguf

Without --chunks every chunk of the text is used. --process-output also collects the output matrix.

5. Quantize.

$ oklm quantize --imatrix smollm2.imatrix.gguf smollm2-f16.gguf smollm2-q4_k_m.gguf Q4_K_M
quantize: 210 importance matrix entries from smollm2.imatrix.gguf
quantize: smollm2-f16.gguf -> smollm2-q4_k_m.gguf (Q4_K_M), 211 tensors converted, 180 fallbacks

The 180 fallbacks are the 576-wide rows above, each logged as a warning. oklm quantize --dry-run with the same arguments says the size the file will have without writing it (convert.md).

6. Perplexity again, with the same file and the same -c; anything else changes the number for reasons other than the quantization:

$ oklm ppl -m smollm2-q4_k_m.gguf -f eval.txt -c 512 --backend cpu
perplexity: 24.1685 over 16065 tokens (512-token windows)

7. A quick look at the output:

$ oklm run -m smollm2-q4_k_m.gguf -p "The capital of France is" --temp 0 -n 24 --backend cpu
The capital of France is Paris. Paris is a city that is known for its historical landmarks, art, fashion, and culture. It is the
prompt: 5 tokens in 0.012 s (404.5 t/s); generation: 24 tokens in 0.163 s (147.1 t/s)

The same steps with two other outputs, all on the same evaluation text:

File Size Perplexity Change from F16
F16 258.3 MiB 23.4932
Q8_0 (quantize ... Q8_0) 138.1 MiB 23.5109 +0.08 %
Q4_K_M with the importance matrix 100.6 MiB 24.1685 +2.9 %
Q4_K_M without it 100.6 MiB 24.5347 +4.4 %

Reading the change. The ratio of the two perplexities is what matters, not the absolute value, which depends on the text and the context size. A change within a few tenths of a percent is noise-level; a few percent is a real but often acceptable loss; when it grows larger, or the use is sensitive to small errors, try the next larger type (Q5_K_M, Q6_K, Q8_0) and measure again. Perplexity is an average over text: it is the quick check, and a test on the real task (the prompts the model will get) is the final one.

The same in a window. oklm-gui --tool convert does steps 1, 5, 3 and 6 from one window (gui.md): it names the source and says what it is, shows the size the result will have before anything is written, starts oklm convert, oklm quantize and oklm ppl in the background with the command lines on screen, follows their log and progress, and puts the two perplexities and sizes side by side.

Memory and the KV cache

Running a model takes four kinds of memory, which oklm plan computes from the GGUF headers alone, before anything is loaded, with the same functions the context allocates with:

  • Weights: the file's tensors. They are mapped, not copied; on the CPU some matrices are repacked into a faster layout, and plan counts the copy in place of the mapped bytes (a MoE model's routed experts, repacked when there is room, on top of them).
  • KV cache: see below. It grows with the context length.
  • Activations: the buffers of one batch (512 tokens by default) flowing through the layers.
  • Scratch: small temporary space.

It compares the total with the available RAM (less a margin of 3 %, at least 256 MiB) and, with --backend opencl, the device memory, and says "fits resident", "fits with N layers on the device" or "does not fit" (exit status 3). run, chat and serve make the same projection and warn, but do not refuse: mapped weights can still page in from disk, slowly. Details in memory-plan.md.

No model comes with OKLM. oklm models lists the ones it offers to download, by use (general chat, coding, images, sounds, security, retrieval), with their size and licence; oklm models pull qwen3-8b downloads one into ~/.local/share/oklm/models, resuming an interrupted download and keeping the file only when its SHA-256 is the one compiled into oklm; after that -m qwen3-8b works wherever a model file is expected. Details in models.md.

The KV cache. Attention lets each new token look at every earlier one. For that it needs, per layer, a key and a value vector for each earlier token. Recomputing them for every new token would redo the whole prompt each time; the KV cache keeps them, so a token costs one pass through the model. Its size is

layers x 2 (key and value) x KV heads x head size x bytes per element x tokens

For SmolLM2-135M in F16, 30 x 2 x 3 x 64 x 2 bytes = 22.5 KiB a token, so 2048 tokens take 45 MiB, as plan printed above. (Gemma 4's blocks are of two widths, and its E2B and E4B models have blocks that read another block's cache and keep none: the plan adds each layer's own and prints (N layers, X for the widest). A hybrid stack has blocks without attention, which keep no rows: 24 of Qwen3.5-9B's 32 blocks, 22 of Cyber-Prime's 30; the plan prints (8 of 32 layers x 32.0 MiB).) The context length multiplies it directly, so it dominates at long contexts:

$ oklm plan -m qwen2.5-0.5b-instruct-q4_k_m.gguf -c 32768
...
  weights resident              477.4 MiB  (387.9 MiB repacked for the CPU, in place of 373.4 MiB mapped)
  KV cache                      384.0 MiB  (24 layers x 16.0 MiB)
  activations                   337.3 MiB
  total                          1.17 GiB
$ oklm plan -m qwen2.5-0.5b-instruct-q4_k_m.gguf -c 32768 -ctk q8_0 -ctv q8_0
  KV cache                      204.0 MiB  (24 layers x 8.5 MiB)
  total                         1020.1 MiB

-ctk / -ctv (on run, chat, ppl, serve and plan) store the cache in f32, f16 (the default), bf16 or the block quants q8_0, q4_0, q4_1, q5_0, q5_1, iq4_nl. A quantized cache holds rotated heads (a Walsh-Hadamard transform, as llama.cpp does), which spreads outliers before rounding: Q8_0 stays within 0.2 % of F16's perplexity on Llama-3.2-1B (adr/0005-kv-cache-precision.md). --parallel N multiplies the cache by the number of sequences.

Listing what fits. oklm plan --list [DIR] projects every GGUF model under models/ (or DIR) at 8192 tokens (capped at each model's trained context) and --recommend names the largest that fits:

$ oklm plan --list models --recommend
      size       need  verdict                                  model
  63.0 MiB  374.7 MiB  fits resident on the CPU                 models/smollm2-135m-iq/SmolLM2-135M-Instruct-IQ1_S.gguf
...
 782.1 MiB   1.32 GiB  fits resident on the CPU                 models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf [MoE 8/32: resident 1.32 GiB, with the expert cache 656.4 MiB + 348.8 MiB]
...
recommended, resident: models/glm-4.7-flash/GLM-4.7-Flash-Q4_K_M.gguf (fits resident on the CPU)
recommended, with the expert cache: none (no MoE model here needs it to fit)

For a mixture-of-experts model the row also says what stays resident if its experts are read from disk instead (see MoE models below).

Saving the cache: chat --kv-save --resume. With --kv-save, oklm chat writes the conversation to .oklm/chat/last.json and the cache's rows to .oklm/chat/last.kv after each turn (--chat-file FILE names another place). --resume reloads both, so the first turn evaluates only the new message instead of the whole history:

$ printf 'My name is Ada.\n/exit\n' | oklm chat -m SmolLM2-135M-Instruct-Q8_0.gguf --backend cpu --temp 0 --kv-save --stats
> (prompt 35 tokens: 0 cached, 35 evaluated)
...
$ printf 'What is my name?\n/exit\n' | oklm chat -m SmolLM2-135M-Instruct-Q8_0.gguf --backend cpu --temp 0 --kv-save --resume --stats
(continued: 2 messages from .oklm/chat/last.json; 95 tokens of the cache loaded from .oklm/chat/last.kv)
> (prompt 111 tokens: 95 cached, 16 evaluated)
My name is Ada, ...

The KV file (OKLMKVC, version 1; version 2 when the conversation holds an image's or a sound's rows, which then also have their media hashes and positions in it: oklm describe --kv-save, and oklm chat --mmproj --kv-save once a picture or a sound was attached) starts with a header that identifies the model (a hash of its architecture, sizes and tensors), the adapter, the context and the per-layer cache types, then holds chunks: each one the token ids of a range of positions and every layer's key and value rows for them, closed by a trailer with a hash. A turn appends a chunk; a crash in the middle leaves the previous turn's state. A file made by another model, cache type or adapter is refused rather than misread. Here 95 tokens took 2.2 MB, the 22.5 KiB a token above (inference.md). The agent's --kv-save keeps the same kind of file beside each session. oklm serve --prefix-dir DIR does it for the server's prefix cache: a finished request's full blocks go to a file in DIR, and a restarted server loads the files before it takes requests, so a repeated long prompt is found cached (server.md).

Backends and speed

CPU and OpenCL. The CPU backend is always there; the OpenCL backend is compiled in but loads the system's OpenCL library at run time, so a machine without one still runs the same binary. --backend auto (the default of run, chat, ppl, serve and train; plan defaults to cpu) takes the first OpenCL GPU when there is one, else the CPU; --backend cpu gives the same numbers on every machine. --gpu-layers N puts the last N layers on the device and the rest on the CPU when the model does not fit. On an RDNA3 GPU the prompt products of the common quantized types (Q4_K, Q5_K, Q6_K, Q8_0, Q4_0, Q5_0) run on its 8-bit matrix units by default: the activations rounded to 8 bits exactly as the CPU rounds them, and the CPU's products bit for bit, so prompts follow the CPU's greedy choices (OKLM_CL_Q8=0: the float kernels); the Q4_K and Q5_K tiles, those of MoE prompts' indexed products included, have waves that load the next step of weights and activations while the others multiply. Decoding does the same for the larger K-quant products (Q4_K, Q5_K, Q6_K, rows of 2048 values and more: each work-group rounds its activation column in local memory, then 8-bit dot products; OKLM_CL_Q8_MV=0: the float row kernels); the other one-column products keep the float kernels, which measured faster. The prompt products of the other quantized types (the grid IQ types of a 2-bit file, Q3_K, MXFP4, the ternary types: no 8-bit kernel) run on the same device's float matrix units, their activations rounded to F16 as Vulkan rounds them (OKLM_CL_WMMA=0: the F32 kernels, three times slower on such a file). OKLM_CL_WMMA=1 runs every prompt product on the F16 matrix units instead, and the prompt attention of F16 caches too (not the CPU's bits, so it stays opt-in; decoding keeps the 8-bit products; OKLM_CL_WMMA_ATT=0 keeps the attention's half tiles). The backend also fuses a MoE layer's routing and Qwen3's query and key norms with their rotation into one launch each, and folds a decode step's RMS norm into the 8-bit product that reads it (the same bits), and latent attention's prompts (GLM-4.7-Flash, DeepSeek) have their own tiled kernel, as has the bidirectional attention of an encoder's rows (the vision encoders, Whisper's), which also takes a span of keys a row: the blocks of Qwen2.5-VL's encoder that attend within windows are one launch each, on OpenCL and on Vulkan. An image's rows that see each other (Gemma 3) take the prompt tiles with a last key, for heads up to 128. See opencl.md.

Vulkan. The Vulkan backend (--backend vulkan) is the main GPU backend on AMD, NVIDIA and Intel (ADR-0023): it loads the system's Vulkan library at run time like OpenCL, runs GLSL shaders compiled into the binary at build time, and reaches tensors through their buffer device addresses. Every op of the dense, MoE, hybrid and latent-attention models runs on the GPU (norms, attention, one-column, prompt (cooperative matrices where the device has them), 8-bit and MoE products); an op a shader does not take runs on the CPU through the fallback. A model runs on it as on OpenCL (--backend vulkan, --gpu-layers, its caches in device memory, weights and caches larger than one allocation split over several); a decode step is recorded once into command buffers and replayed on the next tokens, its per-token values read from device memory; the recording is cut into parts submitted as the host reaches them, so the device does not wait for the host's pass over the step. Its time on a discrete GPU is mostly the latency of its few hundred dependent dispatches, so the decode shaders are written for latency (no 64-bit division, few loads per invocation and all of them before the arithmetic, no long chain of arithmetic in one invocation: vulkan.md). As on OpenCL, a MoE layer's routing (softmax, expert choice, weights) and its gate and up products with their SwiGLU are held back op by op and run as one dispatch each, and so is the end of a residual branch (its last product or MoE sum, the scale of its output, the add into the stream), with the bits of the ops one by one. The server, layers spread over several Vulkan devices, weight streaming and the device expert tier (its uploads on a transfer queue) work on it as on OpenCL; auto picks it on every GPU it finds, OpenCL where Vulkan is not there (OKLM_BACKEND_ORDER overrides). Decode products of Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 and Q5_0 weights can run on 8-bit integer dot products with the activations rounded as the CPU rounds them, giving the CPU's products bit for bit; they are taken where they were measured not slower than the float products (OKLM_VK_Q8), the K-quants in one of two layouts by vendor: eight lanes a super-block with the rounding in the work-group where memory bounds the product (AMD), a rounding dispatch then a weight row per invocation where the shader's arithmetic does (Intel: Llama-3.2-1B decodes at 19.8 t/s on a UHD 750, 6.4 before, llama.cpp's Vulkan backend 15.6; OKLM_VK_Q8_ROWS). See vulkan.md.

$ oklm devices
backends:
  cpu      priority 0  (selected)
cpu:
  features  : sse3 avx avx2 fma f16c avx512f avx512bw avx512vl avx512vnni
  kernels   : avx512 (best available: avx512)
  threads   : 8
opencl: not available

features are the instruction sets the CPU has; kernels the SIMD level OKLM picked (scalar, AVX2 or AVX-512; OKLM_SIMD forces one). The SIMD kernels give the same bits as the scalar ones. A quantized product first rounds its input to bytes, once, on the calling thread; that rounding runs on vector instructions too (the scalar loop was 8 % of a SmolLM2-135M decoding step, all of it with the memory idle), and a token's rotation computes its cosines and sines once for all its heads. The scalar rounding is OKLM's own as well, not the C library's roundf: the Windows build's returns 1 for the float just below a half. The quantized matrices of the CPU layers also get a repacked copy, eight rows interleaved, which prompts read; decoding reads it too with one thread (two when the copy is barely larger), where waiting on one row's sum at a time would cost more than the memory, and at any thread count when the copy is no larger than the file (Q8_0, Q5_0); with more threads a larger copy loses, memory being the limit. Same bits either way. A MoE model's routed experts get the same copy when it fits in memory beside the mapped file (not with an expert cache), which makes its prompts about 1.45 times faster. On Linux the copies are cut one after the other from chunks aligned on 2 MiB that the kernel backs with huge pages (OKLM_CPU_PACK_HUGE=0: an allocation each): a decoding step reads all of them once, and on 4 KiB pages every page of a small matrix costs an address translation miss and ends the prefetcher's stream (SmolLM2-135M decoding 3 to 5 % faster on the chunks).

The worker team. Decoding a token runs hundreds of small parallel loops a few microseconds apart. Starting threads for each would cost more than the work, so the CPU backend keeps a team of threads alive (one per physical core by default, OKLM_THREADS to change it) that claim chunks of each loop from a shared counter. Each output element is computed by one thread in a fixed order, so the result does not depend on the thread count (adr/0016-cpu-worker-team.md). The products that read the same input (a layer's Q, K and V, its gate and up) run as one loop, which saves the team's start and end of the others. A decoding step's products are bound by memory, which half the cores already read at full speed with the packed copies' light kernels: more readers only contend, so those loops take at most that many threads (OKLM_MATVEC_THREADS), as do the file layout's under 4 MiB of weights, and the larger ones at most one per core (Qwen3-8B Q8_0 decoding from 8 threads 3.53 to 3.68 tokens/s). The workers a loop leaves out keep waiting, and those no loop of a step takes go to sleep, so extra threads cost a decoding step nearly nothing (SmolLM2-135M at four, eight or sixteen threads 250 tokens/s, where sixteen gave 220). With few keys, a decoding step's attention runs its tiles as they are, one loop, instead of splitting them into three.

oklm tune. The CPU kernels cut their work into tasks (rows of a matrix product per task, attention tile sizes). The best split depends on the machine and the model. oklm tune times a 512-token prompt (-n) with other splits for up to 60 s (--budget), keeps a candidate only if its logits equal the defaults' bit for bit and it is at least 3 % faster, and caches the result in ~/.cache/oklm/tuning/ (or $XDG_CACHE_HOME/oklm/tuning/, or --cache-dir), keyed by the CPU, the backend, the model's shapes and types, and the kernel sources. Contexts apply a cached result (entries older than 90 days are ignored); OKLM_TUNE=0 turns that off, --force tunes again, run --tune tunes first when nothing is cached.

$ oklm tune -m SmolLM2-135M-Instruct-Q8_0.gguf --backend cpu --budget 10 --cache-dir tunecache
...
setting       default  kept
matmul rows        16    32
attn rows          32    16
attn keys          64    64
prompt of 512 tokens: 227.4 ms with the defaults, 202.6 ms kept (11 candidates, 0 rejected by the logit check)

Results across machines are in tuning/06-results.md.

MoE models

A mixture-of-experts (MoE) layer replaces one large feed-forward block with many small ones, the experts, and a router that picks a few of them per token (8 of 32 for Granite-3.1-1B-A400M). The model is large on disk but each token reads only a fraction of it.

  • Grouped expert products. In a batch, different tokens pick different experts. OKLM's indexed matrix product groups the batch's (token, expert) pairs by expert, so each chosen expert's weights are read once for all the tokens that need it, on the CPU and on the GPU (where the grouping itself runs on the device) (inference.md).
  • The expert cache (--expert-cache GB|auto, CPU, on run, chat, ppl and serve): the routed experts stay in the file and are read into a per-layer cache of that size, least recently used first out; auto takes the largest cache that fits beside the rest. --expert-loaders N (4 by default) read missing experts on threads while the experts already in memory compute; --expert-predict last|router reads the next layer's likely experts ahead (its last routing, or its router applied to the current state); --expert-direct bypasses the page cache; --pin-gb GB pins the most used experts of the usage file (--expert-usage FILE, by default ~/.cache/oklm/usage/<model hash>.json, made on the first run and updated as the model runs); --expert-stats prints hit rates at exit; oklm expert-pack writes a copy of the experts with each one's three matrices together, which --expert-pack FILE reads in one read a miss instead of three, and --expert-mirror FILE sends every other read to a copy of that file on a second drive. The output is the same, bit for bit, as with every expert resident.
$ oklm run -m granite-3.1-1b-a400m-instruct-Q4_K_M.gguf -p "The capital of France is" -n 16 --temp 0 --backend cpu \
      --expert-cache 0.2 --expert-loaders 2 --expert-predict router --expert-stats
The capital of France is Paris.
...
expert cache: 24 layers, 216 slots, 0 pinned, 194.6 MiB; hits 2948, misses 656 (81.8 % hits); 2186.9 MiB read in 0.639 s (3424 MiB/s)
expert cache: waited 0.064 s for reads; read ahead 1747, used 1583 (90.6 %); predicted 3152, right 2882 (83.5 % of 3452 routed)
  • --n-cpu-moe N|all (device backend): the routed experts of the first N layers (or all) stay in host memory and run on the CPU, while the rest of those layers (attention, router, shared experts) goes to the device. Experts are most of the weights but few are read per token, so this keeps attention on the GPU for a model larger than the GPU (moe.md).
  • --expert-gpu GB|auto (device backend): a device tier holding the hottest experts of those CPU layers, chosen from the usage file's counts and refilled from the routing; in each step the device computes the experts it holds while the CPU computes the others, and a prompt batch copies the experts it lacks to the device instead (moe.md).

Generation

Tokens

A model reads and writes token ids, not characters. The tokenizer cuts text into pieces from a learned vocabulary (byte-level BPE or SentencePiece BPE); Ġ marks a leading space.

$ oklm tokenize --model models/smollm2-135m-hf --pieces "Hello world, quantization!"
 19556  'Hello'
   905  'Ġworld'
    28  ','
  3324  'Ġquant'
  1237  'ization'
    17  '!'
$ oklm tokenize --model models/smollm2-135m-hf --decode 19556,905
Hello world

--bos adds the start token, --no-special reads special-token text as plain text. oklm train-tokenizer learns a byte-level BPE vocabulary from text files (--pre picks the pre-tokenizer: gpt-2, llama-bpe, qwen2, smollm; --special adds special tokens, --chat-template embeds a template):

$ oklm train-tokenizer --vocab 1024 --out tok.gguf data/tinyshakespeare.txt
1024 tokens -> tok.gguf

See tokenizer.md.

Sampling

For each position the model outputs a score (logit) for every token of the vocabulary. The sampler turns those into one choice, in this order (src/model/sampling.c):

  1. Penalties over the last 64 accepted tokens: --repeat-penalty P divides positive logits by P and multiplies negative ones (1 = off); the library and the server also have frequency and presence penalties (0 = off).
  2. Temperature 0 is greedy: the highest score, lowest id on ties.
  3. Top-k keeps the k best tokens (--top-k, 0 = all).
  4. Top-p keeps the smallest set of best tokens whose probability sums to p (--top-p, 1 = off), after the temperature.
  5. Min-p drops tokens less than min-p times as likely as the best (--min-p, 0 = off).
  6. Draw one of the rest, by probability, from a xoshiro256** generator seeded with --seed, so a seed reproduces a run on every platform.
Option (run, chat, serve) Default
--temp T 0.8
--top-k K 40
--top-p P 0.95
--min-p P 0.05
--repeat-penalty P 1 (off)
--seed S 1234
-n N (tokens to generate) 128
-c N (context) the model's trained context, at most 4096

describe and agent take the same options. A server request may set temperature, top_p, top_k, min_p, seed, frequency_penalty, presence_penalty and repeat_penalty; the command-line values are the defaults of what it leaves out.

$ oklm run -m SmolLM2-135M-Instruct-Q8_0.gguf -p "List three fruits:" -n 30 --temp 0.7 --top-k 40 --top-p 0.9 \
      --min-p 0.05 --repeat-penalty 1.1 --seed 7 --backend cpu
List three fruits: apple, banana and grape.

oklm run completes the raw prompt; it does not apply a chat template. oklm chat does. With --url URL in place of -m, oklm chat is the same chat on the model of an OpenAI-compatible server (oklm serve, llama.cpp's server, another, at http[s]://host[:port] or under a base path the URL names, which then stands for /v1), which applies its own template: the conversation goes as chat messages in a streamed request, the model is chosen among those the server lists, and the context's use is an estimate (inference.md).

Chat templates

Chat models are trained on conversations written in a particular format, with markers for each role. The model file carries that format as a Jinja template (tokenizer.chat_template). OKLM has its own Jinja interpreter and renders the template as Hugging Face's apply_chat_template() does, with the variables messages, add_generation_prompt, bos_token, eos_token, and tools and documents set to none, plus the functions templates call (raise_exception, strftime_now, range, namespace, dict). oklm chat --template-vars JSON adds or replaces variables, such as Qwen3's enable_thinking; the server takes the same as chat_template_kwargs. A DeepSeek-Coder instruct file written before GGUF carried templates gets the published one, known by its BOS and EOS tokens. A file without a template is given one with --chat-template FILE|NAME (oklm chat, run, serve, agent, describe): a Jinja file, or the name of a built-in format (vicuna, chatml, llama2, llama3, mistral, mistral-v3, gemma, phi3, alpaca, deepseek-coder, foundation-sec), each a Jinja template compiled in; a catalogue model whose file has none names its format in the catalogue (LLaVA 1.5: vicuna) and needs no option. Without either, the command says once that the file carries no template and writes ChatML. A template given that way is rendered as it is: oklm run --rag writes its turn with it, nothing replaces it (not the template OKLM renders an LFM2 file's tools with), and the agent's small model gets it when its own file carries none. A template the interpreter rejects falls back to built-in ChatML, Llama 2, Llama 3, Mistral, Gemma or Phi-3 renderings (inference.md, adr/0006-chat-templates.md).

$ printf 'What is 2+2?\n/exit\n' | oklm chat -m Qwen3-0.6B-Q8_0.gguf --backend cpu --temp 0 --template-vars '{"enable_thinking": false}'
> 2 + 2 = 4.

Each chat turn renders the whole conversation again but evaluates only the tokens after the part already in the cache.

A conversation longer than the context does not stop the chat. Past a share of the context the older messages are summarized by the model into a note that takes their place (/compact), and a reply that the context or the token limit cut is gone on with: the conversation is rendered again with the cut reply as the open start of the model's turn, no end of turn after it, so the model writes on from the reply's last token (/continue). oklm chat, the chat window, the dashboard's chat and oklm agent do both by themselves, and either can be left to the person; a limit the person gave (-n, /max-tokens) still ends a reply for good. When the context is what cut the reply, each of the four first asks for a larger context where the memory plan (or the server's own, through the dry run of its resize) holds one, and compacts only when none is made. The agent asks for the larger context before a turn too, when the conversation leaves less than a turn's room, makes no summary that would be longer than what it replaces, and, when its own system prompt and tools leave no room in the context, says so with the figures instead of sending a request; oklm agent -p then ends with status 1, as for any message the context refused (agent-sessions.md). A model that writes its reasoning in a channel of its own has its turn opened again in the channel it was cut in, and afresh when it was cut inside the first header of its turn, whose markup is no text to go on from. A reply cut inside a tool call is gone on with under the calls' grammar, which first reads the call's text so far, so the call comes out whole and is never run as the half that was written, through the chat completions and through /v1/messages alike (inference.md).

Constrained output

A grammar restricts which tokens may come next: at each step the sampler checks its choice against the grammar and, if refused, masks every token the grammar does not allow and chooses again.

  • --grammar FILE.gbnf: a GBNF grammar, llama.cpp's syntax.
  • --json-schema FILE.json: a JSON schema turned into a grammar.
  • --json: any JSON value.
$ oklm run -m SmolLM2-135M-Instruct-Q8_0.gguf -p "A person as JSON:" --json-schema person.json -n 40 --temp 0 --backend cpu
A person as JSON:{"name": "John", "age": 30}

(person.json: an object with a required string name and integer age.) Tool calls use the same machinery: with chat --tools tools.json (OpenAI-style definitions), generation is free until the model writes the opening marker of a call, then constrained to a valid call of a declared tool. The wire format is the model's: hermes (<tool_call>), llama3, mistral ([TOOL_CALLS]), glm45 or glm4 (GLM-4-0414: the function's name opens the turn, its arguments follow as JSON, the result comes back as an observation turn), harmony (gpt-oss: a message addressed to=functions.f), lfm2 (Liquid's LFM2: a Python list of calls between <|tool_call_start|> and <|tool_call_end|>, its literals turned into JSON arguments) or gemma4 (Gemma 4: call:f{key:value} between <|tool_call> and <tool_call|>, strings between <|"|> marks, turned into JSON arguments), recognized from the template (for LFM2 from the vocabulary too: a file whose plain ChatML template says nothing of tools is then rendered, in oklm serve, oklm chat --tools and oklm agent, with a template that writes them as Liquid's does, unless --chat-template gave one) or set with --tool-format. A reply's reasoning is told apart from its answer by one splitter (src/harness/think.c): a <think> block, the analysis channel of a harmony reply, the message a Muse Glimmer reply addresses to itself ( to=self<|message|> ... <|eom|>), or Gemma 4's thinking channel (<|channel>thought ... <channel|>), which oklm chat, /v1/messages (a thinking block) and /v1/chat/completions (reasoning_content) show without the channels' markup; a prompt that ends on an opening tag makes the reply start as reasoning. --tool-choice required forces a call. Results go back with /tool RESULT. See inference.md and adr/0009-tool-calls-and-grammars.md.

Speculative decoding

Checking k proposed tokens costs the large model about one batched pass, not k passes. Speculative decoding guesses a few tokens cheaply and lets the model verify them together; every emitted token is still drawn from the model's own logits with the same random draw, so the output is the one plain generation gives with the same seed. Only the speed changes.

Option Source of the guesses
--draft MODEL (-md), --draft-n 5 a smaller model with the same vocabulary
--spec-ngram K prompt lookup: the K tokens that followed the last 1 to 3 tokens earlier in the text
--mtp, --mtp-model FILE the model's own multi-token prediction blocks (DeepSeek V3 files that have them)
--spec-grammar the text a grammar forces (a JSON key, a tool call's fixed text)
--spec-guard RATE not a source: no guesses while the expert cache's recent hit rate is below RATE
$ oklm run -m SmolLM2-135M-Instruct-Q8_0.gguf -p "Repeat: the cat sat on the mat, the cat sat on the mat, the cat" \
      --spec-ngram 4 -n 24 --temp 0 --backend cpu
...
draft: 20 proposed, 20 accepted (100.0 %)
$ oklm run -m SmolLM2-135M-Instruct-Q8_0.gguf --draft SmolLM2-135M-Instruct-Q4_K_M.gguf -p "The capital of France is" -n 16 --temp 0 --backend cpu
...
draft: 20 proposed, 16 accepted (80.0 %)

A draft pays off when it is much smaller than the model; the second example only shows the mechanism (inference.md).

Decisions by scoring

Some questions have a fixed set of answers ("positive or negative?"). Instead of letting the model write an answer and parsing it, OKLM scores each allowed answer as the continuation of the prompt: it sums the log-probabilities of the answer's tokens (teacher-forced: the tokens are given, not generated) and turns the sums into probabilities over the options with a softmax. Nothing is generated, the answer is always one of the options, and the probabilities show how sure the model is relative to the alternatives (they are not calibrated frequencies). The sum, not the mean, is used so a long option is not favoured for its length.

oklm chat's lines that start with / are commands, read from one table (include/oklm/client/chat_commands.h) that the chat window and the server's GET /api/commands share: /help lists them, an unknown one is answered with the nearest names and is not sent to the model, //text sends a text starting with / (inference.md). The commands change what the loop works with while the conversation stays: /context N makes another context and /model loads another model (chosen in a list of the catalogue by group, downloaded only once the chat asked), and since each turn renders the whole conversation again and evaluates what the cache does not hold, the next turn simply evaluates everything with the new context or model (inference.md).

In oklm chat, /decide A | B | ... asks about the conversation so far:

$ printf 'I love this phone.\n/decide positive | negative\n/exit\n' | oklm chat -m SmolLM2-135M-Instruct-Q8_0.gguf --backend cpu --temp 0
...
  positive                  64.86 %
  negative                  35.14 %
decision: positive (confidence 0.06)

Confidence is 1 minus the entropy of the distribution over that of a flat one: 0 when every option is as likely, 1 when one is certain. The server's /v1/decide (also /v1/systemone) does the same for several named questions about a state, with yes/no, choice and score questions, sharing the prompt's cache between options (server.md).

Looking inside a run

--observe FILE on oklm run and oklm chat writes, for the prompt's last token and each generated one, a JSON line of what the model did inside: the norm of what each block's attention and feed-forward part added to the residual stream, the experts each router picked with their weights and, with --observe-attn, the attention of that token over the positions before it, a row a head:

$ oklm run -m models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf --backend cpu --temp 0 -n 6 \
      -p "The capital of France is" --observe run.jsonl --observe-attn 5
$ head -c 150 run.jsonl
{"seq":0,"pos":4,"token":438,"piece":" is","serial":1,"n_layer":24,"n_observed":24,"embd":7.30355,"resid":[3.72...

The blocks a device runs are recorded too: the rows are copied aside on the device and read back once when the pass ends (what it costs: observe.md); off, it costs a pointer test per block. --observe-lens K adds what the output head would say after each block (the residual stream at that depth through the final norm and the head: for "The capital of France is", Llama 3.2 1B's stream reads " Paris" from block 11 of 16 on).

The norms say how large a write is, not what it does. --ablate runs the prompt again without a part (a block's attention write, its feed-forward write, one head: zeroed, or with --ablate-mean replaced by its mean over the prompt) and says how far the next token's distribution moved; --ablate-scan blocks does it for every block's two parts in turn:

$ oklm run -m models/llama3.2-1b/Llama-3.2-1B-Instruct-Q4_K_M.gguf -p "The capital of France is" --temp 0 --ablate-scan blocks
part          KL (nats)   p(plain)  likeliest after
0:attn          1.25425     0.9550  " Paris" p=0.9550
0:ffn           8.45301     0.0000  " out" p=0.0463
...

The stream is kept before the part and put back after it, through host memory, so every kind of block and every backend takes it (observe.md). A server started with --observe gives the same record for the token it decoded last (GET /api/activations), and the model inspector draws both (gui.md). The figures are exact, and say less than they seem to: a large write is not an important one, and attention is not an explanation (observe.md).

Documents

oklm ingest FILE... prints a document as the text a model reads, and oklm run --doc FILE puts that text before the prompt, inside <document> tags. The format is chosen by extension, then by content (src/mm/ingest.c); files up to 64 MiB:

Format How the text is extracted
Plain text, Markdown, CSV, TSV, code as it is, without a byte order mark or carriage returns
JSON parsed, then indented two spaces a level
HTML (.html, .htm, .xhtml, or a file starting with <html / <!doctype html) rendered: scripts and styles dropped, entities decoded, blocks on their own lines
DOCX (or any ZIP holding word/document.xml) the paragraphs; headings as # lines, list items as - , tables as Markdown rows
PDF (or a file starting with %PDF-) each page's text in order, spaces and line breaks placed from where the glyphs land; fonts decoded through their ToUnicode maps or encodings; documents encrypted with an empty user password are decrypted
$ oklm ingest r.html
Report

Sales rose & costs fell.
$ oklm ingest tests/fixtures/pdf/simple.pdf
Hello, World!
Kerning stays one gap
...
$ oklm run -m model.gguf --doc r.html -p "What happened to sales?"
Document r.html:
<document>
Report
...

Details in multimodal.md. The agent's Read tool shows Word and PDF files the same way.

Embeddings

oklm embed -m MODEL TEXT... (or --file F, one text a line) prints one vector a text: the first step of retrieval (rag.md). Three kinds of model run:

  • Text encoders, bert (BGE, MiniLM, e5; RoBERTa and XLM-R as llama.cpp writes them) and nomic-bert, from GGUF or a Hugging Face checkpoint (src/model/arch_bert.c, src/io/names_bert.c): the token, type and position embeddings summed and normalized, then post-norm blocks whose attention sees every token of the text in both directions (nomic-bert rotates queries and keys and uses SwiGLU). No KV cache: each forward pass holds whole texts, several at once.
  • Decoder embedding models (Qwen3-Embedding): the usual forward pass without its output head; the last token's final normalized state is the embedding.
  • The pooling (the first token for bert, the mean for nomic-bert, the last token for decoders, or what the file says) gives one vector a text, scaled to unit length.
$ oklm embed -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf "What is the boiling point of water?"
embed: 1 texts, 10 tokens in 0.005 s: 195.4 texts/s, 1954 tokens/s (cls pooling, 384 dimensions)
-0.01469593 -0.004681007 0.008391812 -0.03536071 0.05449276 ...

The library call is oklm_embed() (include/oklm/rag/embed.h), on an embedding context (oklm_context_params.embeddings); the server answers POST /v1/embeddings with --embed-model. The vectors match a numpy float64 encoder written from the model definitions and llama.cpp's llama-embedding (rag.md).

Retrieval

Retrieval answers from documents the model was not trained on (rag.md is the user guide and the measures). It runs in four stages, each a C interface a program can replace: index, search, judge, answer.

The index. oklm rag index PATH... --out INDEX reads each file as oklm ingest reads it and cuts it into chunks (src/rag/chunk.c, chunk_code.c: Markdown sections, paragraphs and sentences; C-like and Python functions; fixed windows; or semantic breaks). Each chunk is embedded through an encoder interface (include/oklm/rag/encoder.h: an embedding model through oklm_embed(), or a hashed bag of words that needs no model), and everything goes into one GGUF file (src/rag/index.c, ADR-0022): the chunks and their sources, the vectors as one [dim, n] tensor, the BM25 postings (src/rag/bm25.c) and the embedding model's identity, so a search with another model is refused. Run again, it hashes the files and embeds only what changed. Optionally a chat model extracts each chunk's entities and relations into the index (--graph), and a vision model captions the documents' figures, each caption a chunk (--caption-model, its question --caption-prompt). A program that drives the commands reads JSON instead of the text: the build's progress as lines (--progress-json: each file added, updated, unchanged or removed, the chunks embedded so far, the result), the index and whether each source file changed since (rag info --sources --json), and a search with its times and each hit's scores (rag search --json-full) (rag.md).

The search. oklm rag search maps the file. A dense search is one oklm_matmul() of the query against the vectors where they lie in the mapping and an oklm_topk(): exact, 0.04 ms over ten thousand chunks (384 dimensions, F16) and 27 ms over a million, where it reads the 768 MB of vectors at the memory's bandwidth, so no approximate index is needed. BM25 reads the postings of the query's terms; the default, hybrid, fuses the two lists by their scores, min-max normalized and weighted 0.5 / 0.5, or by reciprocal rank (--fusion rrf; src/rag/fuse.c). With a graph (--rag-graph on) the search also walks from the question's entities, and those of its three best hits, to the chunks that mention them or an entity related to them, and fuses that list too (include/oklm/rag/graph.h). A reranker can reorder the first candidates (--rerank, oklm rerank, /v1/rerank; src/rag/rerank.c): a cross-encoder (ms-marco-MiniLM-L-6-v2) is a bert encoder with a classification head that reads query and passage together as [CLS] query [SEP] passage [SEP], the passage's tokens adding the second row of the token type table (oklm_eval_batch_types()), the head (cls, a dense layer and tanh, then cls.output) turning the [CLS] row into one logit (oklm_model_classify()); without one, a chat model scores each passage by a yes / no decision through decide.c.

oklm rag eval measures the search on a BEIR-format set (src/rag/eval.c: the corpus indexed, with its graph when --graph --gen-model is given, every judged query searched, nDCG@10, MRR@10, recall@k and precision@k as BEIR scores them), and make rag-eval runs it on SciFact and on 120 questions over these docs (NFCorpus on request: 34.38 dense against the model card's 34.31, 36.67 hybrid; 112 questions over OKLM's own sources on request, where cutting code at functions scores no better than cutting at paragraphs and lines, 59.32 against 60.12 nDCG@10, and Qwen3-Embedding-0.6B is the best encoder, 63.63 dense). With bge-small-en-v1.5, SciFact's nDCG@10 is 71.27 dense (MTEB's published figure), 67.41 with BM25, 71.29 fused by reciprocal rank and 73.04 by weights (the default fusion for that); on the docs 60.48, 64.53, 69.78 and 70.61. The MiniLM reranker lowers SciFact's 71.64 to 68.95 over the hybrid candidates, as the BEIR paper's figure for that model predicts, so reranking is off by default; on the docs it puts the right section first more often (recall@1 30.2 to 37.4 %) without changing the answers beyond one question of 30. tools/rag_stats.py puts bootstrap intervals on these rows: only the larger differences hold on sets of 38 to 323 questions.

The judge. With --rag INDEX, run, chat, serve (a request's rag field) and the agent answer from an index (include/oklm/rag/rag.h). oklm_rag_run() chains the stages: the retriever (the index's search), an optional reranker, the evaluator, the rewriter, an optional web search. The evaluator is the generator itself: each passage becomes the state of a yes / no decision by scoring ("Does the passage above contain information that answers the question?", decide.h), one forward pass from the shared question; the probability of yes makes it relevant (0.5 or more), irrelevant (under 0.2) or ambiguous. When none is relevant the generator rewrites the question (a short greedy decode) and it is searched again; then, for the agent only, the web through its WebSearch tool; then the prompt asks for an answer saying that no reliable source was found. A passage the graph found is kept beside a relevant one, since the second fact of a chain does not answer the question alone. oklm_rag_run_turn() adds three optional stages, the generator again: a condenser rewrites a follow-up ("and how do I turn it off?") as a standalone query from the earlier turns, and the previous question is searched with it (on by default, --rag-followup); a router decides by one three-way decision whether to search not at all, once or several times (--rag-route, off); and for several, a decomposer writes the sub-questions, each retrieved and judged apart, their passages merged rank by rank.

The answer. The relevant and ambiguous passages go into one user message, numbered with their file, lines and heading, within a token budget, after instructions to cite them as [n] and to say when they do not answer. With --rag-critique on each cited statement is then asked against its sources and the answer against the question (critique.h, decisions by scoring); an unsupported statement gets the answer generated once more with the critique fed back, the better of the two kept and flagged when still unsupported. An answer already shown cannot be taken back, so a streamed one (--rag-critique flag, a server stream) is checked once complete and only flagged: the verdict follows it (a stream's last chunk, a line after a chat reply), and a server stream sends a second answer only when the client asks. The agent checks a final answer against the passages its message retrieved and, with on, sends the model back once with the critique as a reminder, through the loop's answer_check hook. The server runs the judge, the rewrites and the critique as scoring and generation requests of its scheduler and returns the run in a rag member. Every run is one record (query, rewrites, route, passages with scores and verdicts, fallback, critique) written as a JSON line to --rag-log, to the agent's debug log, and counted in /api/stats.

In the agent, Retrieve is a built-in tool over an index of the project it keeps itself (--rag-project, src/harness/rag_index.c): the files Glob would list that the include globs select (the documentation by default), stamped by name, size and modification time before each search and updated by content hash when the stamp changed, so the model's own edits are found by its next search. It reads only, so the permission modes run it as they run Read. With --rag-memory it also searches the project's earlier session transcripts and the memory files, and with --rag-inject always|route each user message gets its passages as a reminder through the agent's rag_context hook. On a model served elsewhere (--url) the judge, router and critic are requests to the server's /v1/decide, or on another server chat completions read by their log-probabilities or their text, and the rewrites chat completions at temperature 0 (src/rag/rag_http.c). The agent selects its tools by retrieval too (--tool-rag, The coding agent).

On 38 questions about OKLM's documentation, Qwen3-8B answers 16 % right without retrieval, 89 % with the top five passages and 92 % with corrective retrieval, which also declines all 8 questions the docs do not answer; self-critique raises the share of cited statements the same model finds supported from 87.5 % to 96.1 %, accuracy unchanged. On 30 questions that need two linked articles the graph brings both into the top five for 23 instead of 8; on 20 questions answered only by charts, SmolVLM2-2.2B's captions let the generator answer 18 instead of none (SmolVLM-256M's 9). On eight agent tasks that need the project's docs, the agent passes 2 without retrieval and 7 or 8 with it. Every figure, its set and its run are in rag.md.

The server

oklm serve puts a model behind an HTTP API on 127.0.0.1:7430 (--host, --port). Concurrent requests share one context: each step gives every generating request its next token in one batch, so serving many users costs much less than running them one after another. Full cache blocks are kept as a prefix cache, so requests that share a system prompt or a conversation reuse its computed rows (server.md). The GPU's products pick their kernel by a piece's rows, so a prompt found cached, whose last rows go as a small piece, can answer otherwise at temperature 0 than the same prompt cold; --deterministic puts the pieces on a fixed grid of positions and uses a cached prefix only up to its last line, so a request alone answers the same either way (server.md).

Endpoint API
POST /v1/chat/completions OpenAI chat (streaming, tools, response_format JSON and JSON schema)
POST /v1/completions OpenAI raw completion
GET /v1/models the model and each --lora-module adapter
POST /v1/messages, POST /v1/messages/count_tokens Anthropic Messages (streaming, tools, thinking with its budget_tokens cap)
POST /v1/decide, POST /v1/systemone decisions by scoring
GET /api/stats, /api/experts, /api/profile the dashboard's data
GET /api/activations with oklm serve --observe: what each block wrote, the experts picked and the attention asked for, for the token decoded last, or with serial=S for one of the last 512 records (observe.md)
GET /api/commands the commands a chat on this server takes (/context, /rag...), each with whether the server can do it
GET /api/model the model's layout: its blocks by role, where each layer is (CPU or device), the context's memory by part
GET /health {"status":"ok"}
GET / the dashboard
GET /activations with --observe: a page that draws what /api/activations answers (each block's writes, the lens, the attention of the blocks named, the experts), following the last token or stepping through the kept records

Embeddings and reranking are served when the server is given their models (--embed-model: POST /v1/embeddings; --rerank-model: POST /v1/rerank), as are transcriptions, speech and documents (server.md).

The servers that run. A server writes a file for the user alone at its start (pid, port, address, model, command line) and removes it when it stops. oklm serve --list reads them (--json for a program) and oklm serve --stop PORT|PID stops one; the graphical tools start their servers and jobs through the same registry (server.md). On Windows it is the same with that system's means: a command started without a console, an access list of one entry in place of the modes, and a stop asked through a named event the server waits on, since a process without a console cannot be sent Ctrl-C (run under Wine 11, not on Windows itself).

Servers kept across logins. oklm serve --keep NAME ... writes the same command as a unit of systemd's user manager (~/.config/systemd/user/oklm-serve-NAME.service, for the user alone), enables it and starts it: the server then starts at each login and again when it fails, its output appended to a log file the registry names, and it registers itself like any server, marked with its service's name. oklm serve --kept lists the kept ones with what the manager says of them, oklm serve --unkeep NAME removes one; the server manager does the same from its form (Keep across logins) and its list, where a kept server that is stopped or failed keeps its row (gui.md). The arguments are quoted by the rules of ExecStart= (% and $ doubled, quotes and backslashes escaped), a key given inline is refused, and where the user manager does not answer (a container, another init) the commands say so and do nothing. --rekeep NAME ... changes the command kept under a name, and so does the server manager's Re-keep with form for the kept server selected, with the form's settings. On Windows the same commands write a value of the user's Run key, which runs oklm serve --start-kept NAME at login: the server is started in the background through the registry, and is not started again when it fails; on macOS a launchd agent, written and never built (server.md, include/oklm/harness/kept.h).

Access. On a loopback address no key is needed unless --api-keys FILE is given. Any other address needs TLS (--tls-cert, --tls-key) and keys, or the server refuses to start (--insecure overrides it). The keys file holds one key per line, blank lines and # comments ignored, and must not be readable by group or others (chmod 600). Keys come as Authorization: Bearer KEY or x-api-key: KEY; a wrong or missing one gets 401.

A session against SmolLM2-135M on the CPU:

$ printf '# one key a line\nsecret-key-1\n' > keys.txt; chmod 600 keys.txt
$ oklm serve -m SmolLM2-135M-Instruct-Q8_0.gguf --backend cpu --port 18431 -c 2048 --api-keys keys.txt
serving SmolLM2-135M-Instruct-Q8_0 on http://127.0.0.1:18431/ (512 requests at once, 2048 tokens each, API keys required)

OpenAI style:

$ curl -s http://127.0.0.1:18431/v1/chat/completions \
    -H 'Authorization: Bearer secret-key-1' -H 'Content-Type: application/json' \
    -d '{"messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":24,"temperature":0}'
{"id":"chatcmpl-1","object":"chat.completion","created":1794094858,"model":"SmolLM2-135M-Instruct-Q8_0",
 "choices":[{"index":0,"message":{"role":"assistant","content":"The capital of France is Paris."},"finish_reason":"stop"}],
 "usage":{"prompt_tokens":37,"completion_tokens":7,"total_tokens":44,"prompt_tokens_details":{"cached_tokens":0}}}

With "stream":true the answer comes as server-sent events, one chat.completion.chunk per piece, then data: [DONE].

Anthropic style:

$ curl -s http://127.0.0.1:18431/v1/messages \
    -H 'x-api-key: secret-key-1' -H 'anthropic-version: 2023-06-01' -H 'Content-Type: application/json' \
    -d '{"model":"any","max_tokens":24,"system":"Answer in one sentence.",
         "messages":[{"role":"user","content":"What is the capital of France?"}]}'
{"id":"msg_0000000000000003","type":"message","role":"assistant","model":"SmolLM2-135M-Instruct-Q8_0",
 "content":[{"type":"text","text":"The capital of France is Paris."}],"stop_reason":"end_turn","stop_sequence":null,
 "usage":{"input_tokens":26,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":7}}

A decision:

$ curl -s http://127.0.0.1:18431/v1/decide -H 'Authorization: Bearer secret-key-1' -H 'Content-Type: application/json' \
    -d '{"state":"The order arrived two weeks late and the box was crushed.",
         "questions":{"complaint":{"type":"noul","instructions":"Is the customer complaining?"},
                      "topic":{"type":"choice","criteria":{"delivery":"shipping and delays","billing":null}}}}'
{"id":"decide-4","object":"systemone.result","model":"SmolLM2-135M-Instruct-Q8_0","answers":{
 "complaint":{"type":"noul","label":"yes","probabilities":{"yes":0.669319449,"no":0.330680551},"confidence":0.0843798223,"p_yes":0.669319449},
 "topic":{"type":"choice","label":"delivery","probabilities":{"delivery":0.949540009,"billing":0.0504599911},"confidence":0.711652238}},
 "usage":{"prompt_tokens":215,"cached_tokens":80,"scored_tokens":5}}

Ctrl-C (or SIGTERM) stops the server.

Capacity. --parallel N is how many requests generate at once. With the default paged cache a waiting sequence costs only its block table, so by default it is the batch's 512 rows (or the cache's blocks, the fewer) and memory is the real bound; --kv-mem MB sizes the shared cache, --queue N (256) how many requests may wait, -c (4096) each request's context.

Adapters. --lora FILE merges one LoRA adapter into the model for every request. --lora-module NAME=FILE (repeatable) serves an adapter beside the model instead: a request whose model is NAME runs with it, others without, in the same batches, and the base weights stay as they are (a quantized model stays quantized):

$ oklm serve -m SmolLM2-135M-Instruct-f16.gguf --backend cpu --port 18432 --lora-module shakespeare=lora/a-000003.gguf
$ curl -s http://127.0.0.1:18432/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"model":"shakespeare","messages":[{"role":"user","content":"Hello"}],"max_tokens":12,"temperature":0}'
{"id":"chatcmpl-1","object":"chat.completion",...,"model":"shakespeare",...}

The dashboard (http://127.0.0.1:7430/) shows the figures of /api/stats (requests, tokens, cache use, prefix hits), a chat that takes the chat's commands (/help lists them) and files dropped, pasted or chosen (images and sounds for a server with a projector, documents read by the server, one too long for a message kept there for retrieval when the server has an embedding model, a recording transcribed into the message on a server started with --transcribe-model) and the microphone (a Record button and /listen, on localhost or over HTTPS: what is said is written in the message box) and, on a server started with --speak-model, a voice (a Speak button on each reply, /speak, a switch that reads every reply: the page asks /v1/audio/speech for samples and plays them with WebAudio while they are made; a reply read by itself is spoken while it is written, each sentence asked for once no later text can change it) and, with both, a spoken conversation (the Talk switch, /talk: the microphone stays open with the browser's echo cancellation, a detector on its level ends a turn at a pause, the turn is transcribed and sent, the reply spoken, and a voice at the microphone stops the voice and the reply and starts the next turn), for a MoE model an expert map (one row per layer, one cell per expert, lit by how often the router picks it), and a profile of the last requests (queue, prompt, decode times and the distribution of step times) (server.md). The chat window does the same with the system's microphone (apps/gui/chat_listen.h, gui.md): Record or /listen keeps the fragments of Allegro's audio recorder in memory, at 16 kHz mono or resampled to it, with a level meter; while it grows, the recording so far is transcribed again every two seconds (partial=true, which a busy recognizer refuses at once) and the words two of those readings agree on are shown as said, in the window's status line as under the dashboard's meter: Whisper reads a whole sound, so the words while speaking are that, not a streaming decoder; at its end the recording goes to POST /v1/audio/transcriptions as a WAV file (the window's own server is started with the smallest installed Whisper model, a server at an address has to say it transcribes), or, attached to an agent, through the library in the window itself, and the text lands in the message box. It reads replies aloud too (apps/gui/chat_speak.h, gui.md): Speak beside a reply, /speak or the Read aloud box take what the dashboard reads of a reply (the same rule, ported), cut it at sentence ends into the pieces a server takes, and ask POST /v1/audio/speech for raw samples, read while they are made (oklm_client_post_data()), from the window's own server when it was started with --speak-model or from a server at an address that has a voice; elsewhere the window loads the installed speech model itself at the first reading. The samples go to a queue that a thread feeds to a stream of Allegro's audio, or, for the tests, to a WAV file. With Read aloud on a reply is said while it is written: each time the window draws, what was written since is read (speak_stream), and the sentences that nothing written later can change (no mark of Markdown left open before them, the same whatever closing mark follows) join the queue of the thread making the sound, the first piece in two passes so that its sound comes early. A voice is the model's built-in one or a speaker file chosen in the window, which loads its synthesizer again, or starts its own server again, with it. And a spoken conversation (gui.md, apps/gui/chat_talk.h): Talk leaves the microphone open; a turn starts when its level held above a threshold for a quarter of a second and ends at a pause (listen_turn); its words are transcribed and sent without a press, the reply is said while it is written, and the window listens again once the voice ended. The same rule hears somebody speak over the voice: the voice and the reply stop, and what is being said is the next turn. oklm-gui holds the native graphical tools, one program with --tool NAME choosing among them (gui.md): launcher, a home window with a button for each tool and one for the documentation that came with the program, models, the model manager (the catalogue's models downloaded, listed and deleted, and a chat, a server or the inspector opened on one), servers, a manager that starts oklm serve in the background from a form (with what the context takes of the machine's memory), lists every server of the machine with its load and stops, restarts, follows and resizes them (gui.md). The form and the chat window share the controls of oklm serve's options: one table (apps/gui/serve_opts.h) says for each option that has a control what its value is, its range and when it applies, and the command line, a kept profile, a command line read back and the memory plan's parameters are made from it; an offload row (the layers on the device, the cache quantized, the experts on the CPU, the cache's size) feeds the memory line, which is the plan of a server: its cache is sized by the rule oklm serve applies itself (oklm_kv_cache_rule()), so that the line says It fits, It fits with a smaller cache or It does not fit as the server would turn out (a model larger than its device, or one the device holds with less than one context of cache beside it, has the cache of the layers that go there, and the line says how many, the cache on each side and, for the second, that it is slower than whole on the device; a model beside the model on a device of its own is counted and judged on that device); one Advanced window holds the rest, a tab a group, the last one a text field for the options without a control: the server manager adds them to the command line, the chat window gives its own server those it can take (gui.md). A chat window started with nothing named loads no model: it lists the servers, the agents, the installed models and a row for a GGUF file, and a model chosen there is judged by the memory line before it is loaded in the window (gui.md). In the chat window the same controls are the model settings: the sampling goes with each request, and a change of where the window's own model computes restarts it after asking, the conversation kept, while a server at an address tells its own settings and its sampling defaults in GET /api/context (gui.md). A server at an address that is not OKLM's (llama.cpp's, any OpenAI-compatible one) is found out once, when the window is linked to it, by asking each of OKLM's own routes: the window then asks it for models and replies only, counts on a context of its own setting, compacts by estimate, and says so where the figures would be; such a server is kept with its key, its model's name and that context in the window's settings file, which only its owner reads (gui.md). The memory line of a model the window already serves counts as free what its server holds on its device and in the host's memory (device_memory.held and host_memory.held of GET /api/stats), since a restart frees it first; the line counts what the server loads beside the model (a draft model, the projector's encoder, the sound models, the embedding and the reranking models) where the server places it, on the device or on the CPU with what it works with there, sized from the files by the functions the server uses, and in the chat window a speech model that is only installed, which the window opens at the first reading (gui.md); a model loaded in place of another runs with the window's backend, context, cache types and sampling, not with the layers, the cache's size or the draft model chosen for the model before, which are automatic again; and what a model of the window's own runs with is kept for it, in the window's settings file, for the next window on that model. convert, which makes a GGUF file from a checkpoint or requantizes one and measures what was lost, train, which builds an oklm train, rl, refusals or abliterate command from a form, starts it in the background and draws its loss from the --progress-json file as it is written, rag, the retrieval manager, inspect, the model inspector, and chat, the default. The inspector draws a model as a map, a bar a block sized by its bytes and split by role, from the JSON the layout already gives: a file's tensor table and the memory planner (no weight loaded), with the backend, the context and the layers on the device chosen in the window, or a running server's /api/model with its figures read every second; it also runs a prompt through a file's model on the CPU (as it is, or through the chat template), or follows a server started with --observe and steps back through the records it keeps, and draws what each block wrote, the attention of the block selected, the experts picked, the tokens the output head would give after each block, and how far the output moves when a block's write or a head is taken out (gui.md). The retrieval manager works on one index: it starts oklm rag index as a background process and follows the JSON lines of its progress, shows each source of the index with the state of its file (the stale ones marked, and an update that embeds only those), searches it in each mode with or without the reranker, shows the table of oklm rag eval, and starts a chat window whose answers come from the index (oklm-gui -m MODEL --rag INDEX). The chat window is on the same API: given a model it serves it itself, and started alone it loads nothing and asks what to talk to, a server already running on the machine or an installed model, since a second copy of a model a server holds would take its memory twice. From its side panel it downloads, uses and deletes the models of the catalogue, changes the context's size while a conversation is open, and connects to something else without closing: a server found on this machine or given by its address, an agent started with oklm agent --listen, or an agent session of the folder, read only. Files go with a message (dropped on the window, chosen with Attach, or --attach): an image, a sound or a video is sent in the request as the server takes them (a video whole, as a video_url part the server samples; the window decodes it once for its first frame and its length), to a model whose projector the window's server opened (the catalogue's, or --mmproj), and refused with the reason before sending when the link has none; a PDF, DOCX, HTML, JSON or text file is read by the window with the library's reader and its text sent under its name; one longer than the context leaves room for is indexed by the window, with an installed embedding model and the retrieval module, in a file of the user's cache named by the text's hash and the embedding model, so that the same document attached to another window is not embedded again; each message is sent the passages found in it for that message, put in order by an installed reranker when there is one, or passages spread over the document when the message is about the whole, which the model is told (its beginning only, when no embedding model is there and the user chose so); an attached agent tells its context and is sent passages too (gui.md). A message that starts with / is a command: the window gives the commands' parser the place it is in (its own server, a server at an address, an agent) and what that link can do, runs what the parser finds right (the system prompt, the model, the context, the sampling, saving and exporting, retrying and rewinding), and writes the answer as a note in the conversation that no request carries; a list above the message completes the commands as they are typed (gui.md). sessions is the browser of a project folder's agent sessions: listed with their date, title, turns, model and size, searched by what their messages hold, read with the tools' calls folded, written as Markdown or JSON, resumed in a terminal, attached to when an agent runs them, or deleted (gui.md).

The context is the number of tokens a model works on at once. Each model has the one it was trained with; -c chooses another at start, and a running server takes POST /api/context to change it without loading the model again: the cache is made again for the new size, so the next message of a conversation is read whole once. "dry_run": true asks what a size would take in memory first (models.md, server.md). The model itself is changed the same way, POST /api/model with a name: the server opens the new file first, waits for the requests under way, frees the old model and its cache, and makes the cache again for the new one with the settings it was started with; a model it cannot serve leaves the one before serving (server.md). The dashboard's /model NAME and the chat window's Use this model do that, and since a chat sends its whole conversation with each message, the conversation goes on with the new model. The new cache is filled whole when it is made, a second for 6 GiB and a minute for tens of them, so the chat window asks for a resize on a thread of its own and says when it ended (gui.md).

The coding agent

oklm agent reads, searches and edits the files of a project and runs commands there, through a model that writes tool calls. See user-guide.md for daily use and harness.md for the mechanics.

Local or remote model. With -m MODEL the model runs in the same process. With --url URL the agent talks to any OpenAI-compatible server (oklm serve or another) through streamed chat completions; --remote-model NAME names the model there and the key comes from --api-key, else OKLM_API_KEY, else OPENAI_API_KEY. With neither, the agent uses the baseUrl setting (a server) or the model setting (oklm agent config set model PATH), else looks for Qwen3-8B-Q4_K_M.gguf in $OKLM_MODELS, the user's models directory (~/.local/share/oklm/models), ./models and the source tree's models/. The in-process context defaults to 32768 tokens within the model's trained context.

Permission rules. Every tool call goes through rules before it runs. Rules are Tool or Tool(pattern) in allow, ask and deny lists, in settings files read from the highest priority down: managed (/etc/oklm/managed-settings.json), the command line (--allow, --ask, --deny), .oklm/settings.local.json (personal, where "always" answers go), .oklm/settings.json (shared with the project), then the user's ~/.config/oklm/settings.json. Deny beats ask beats allow, whatever the file. A Bash line is split into its commands, and each must match an allow rule; make:* matches make followed by anything. Path rules resolve .. and symbolic links first, so Read(./secrets/**) also catches secrets/../secrets/key.pem. Calls no rule matches are decided by the mode (default: reads inside the project run, the rest asks; acceptEdits: edits inside run too; plan: reads only, until the user accepts a plan). A list of dangerous commands (rm -rf /, git push --force, sudo, ...) always asks even when allowed (agent-permissions.md).

oklm permissions shows what the rules decide for a call, with exit status 0 (allow), 3 (ask) or 4 (deny):

$ cat proj/.oklm/settings.json
{"permissions":{"allow":["Bash(make:*)","Edit(./src/**)"],"ask":["Bash(git push:*)"],"deny":["Read(./secrets/**)","Bash(rm -rf:*)"]}}
$ oklm permissions --project proj Bash "make test && rm -rf build"
deny: Bash(rm -rf:*) (project settings), a command matches a deny rule
$ oklm permissions --project proj Read "secrets/../secrets/key.pem"
deny: Read(./secrets/**) (project settings), matches a deny rule
$ oklm permissions --project proj Bash "git push origin main"
ask: Bash(git push:*) (project settings), a command matches an ask rule
$ oklm permissions --project proj Bash "ls"
ask: default mode, not covered by a rule
$ oklm permissions --project proj --mode acceptEdits Edit "src/a.c"
allow: Edit(./src/**) (project settings), matches an allow rule

The audit log. Each decision is appended to .oklm/audit/<session>.jsonl, one JSON object a line, written before the tool runs: time, session, tool, args, decision, by (rule or mode), source, rule, reason, answer (the user's answer to an ask). oklm audit lists them, filtered by --tool, --decision and --session, from the project's logs or the files given. After a scripted session (--mock replays a fixture of tool calls instead of a model) in that project:

$ oklm audit
2026-11-07T23:44:50Z  deny  rule Bash       {"command":"rm -rf build"}                                   Bash(rm -rf:*)
2026-11-07T23:44:50Z  allow rule Bash       {"command":"make -v"}                                        Bash(make:*)
$ oklm audit --decision deny
2026-11-07T23:44:50Z  deny  rule Bash       {"command":"rm -rf build"}                                   Bash(rm -rf:*)

Sessions. Each run is a session, its conversation appended to .oklm/sessions/<id>.jsonl as it goes. --resume resumes the latest, --resume ID a given one (/resume lists them), --fork-session copies it into a new one. oklm agent sessions lists them without a model, sessions export ID writes one as Markdown or JSON and sessions rm ID removes one. With --kv-save and an in-process model, the model's cache is kept beside the transcript (<id>.kv, the KV file above), so a resumed session's first turn evaluates only what is new instead of the whole conversation: a warm resume (agent-sessions.md).

With many tools (MCP servers above all), --tool-rag K selects them by retrieval: the agent keeps the descriptions of its tools, skills, plugin commands and sub-agent types in a retrieval index (src/harness/tool_index.c over the RAG index of Retrieval, cached in ~/.local/state/oklm/rag/ and re-embedded only where a description changed), grouped Type, Service, Tool. The prompt then counts each MCP server's tools instead of naming them, and before each message the K entries that fit it best (hybrid: bge-small-en-v1.5 on the CPU and BM25, or BM25 alone without the model; toolSearch.mode dense or bm25 ranks by one list) are looked up; the deferred tools among them come with the message, as definitions in a reminder for the in-process model or in the tools array for a server. ToolSearch's word queries use the same index, and every decision is a line of the debug log (rag.md).

The agent has more: background jobs, sub-agents, skills, hooks, MCP servers, plugins, a sandbox for commands. Sub-agents report to the agent that started them (no channel between them): a commit or push asked while background ones still run waits for their reports, so it is decided with them read (agent-tools.md); each report ends with the files the sub-agent wrote, from its edits; lines a sub-agent wrote are kept from the others running beside it; and eval task 26 checks two run at once for undoing each other's work or a report claiming what was not done. oklm agent config, skills, mcp, plugin, agents, attach, send and stop manage them from the shell (extending.md, processes.md, commands.md).

Training

oklm train trains on the CPU, OpenCL or ROCm (--backend auto|cpu|opencl|rocm); cpu gives the same bits on every machine. Parameters, gradients and the AdamW optimizer's two moments are held in F32 (four floats per parameter), and oklm train prints what it will use before it starts. Checkpoints are <prefix>-<step>.gguf models (loadable by run) with a .train sidecar holding the optimizer state and the run's options, for --resume. The directories of --out are created when missing (before any work, so a path that cannot be made fails at once). See training.md.

train, rl, abliterate and refusals take --progress-json FILE (- for standard output, the report then on standard error): the run as one JSON object a line, flushed as written — a start line (the model, the data, the options in effect, the device, the steps planned), a step line an update (loss, validation loss when measured, learning rate, tokens a second, an estimate of the time left), checkpoint and sample lines, the layers and prompts of the two refusal commands, and always a last end line (ok or the error, the final figures, the files written). The fields of each line are in training.md.

From scratch. With --vocab N a tokenizer is first trained on the text; then a new model of the given shape (--layers 4 --embd 128 --heads 4 --ctx 256 by default) learns to predict the next token.

$ oklm train -f data/tinyshakespeare.txt --out nano/nano --vocab 512 --layers 2 --embd 64 --heads 2 --ctx 64 \
      --batch 4 --steps 20 --eval-every 10 --backend cpu
tokenizer: 512 tokens trained on data/tinyshakespeare.txt
model: gpt2, 136960 parameters, 2 layers, 64 channels, 2 heads (2 key / value), MLP 256, context 64
data: 575809 tokens, 518229 for training, 57580 for validation; batches of 4 x 64, 1 per update; 20 steps; on cpu
...
final val loss 6.1651
saved nano/nano-000020.gguf

Twenty steps learn nothing useful; longer runs and their losses are in training.md. The architecture changes how a block is built:

GPT-2 (default) Llama (--arch llama)
Positions learned position embeddings added to the tokens rotary embeddings (RoPE) applied to queries and keys, --rope-theta
Norm LayerNorm RMSNorm
MLP 4 x width, GELU gated (gate, up, down), about 8/3 x width (--ff)
Attention heads as many key / value heads as query heads fewer key / value heads allowed (--kv-heads, grouped-query attention)

--arch qwen2 adds biases to Q, K and V and qwen3 norms on each head's Q and K, both on Llama blocks.

Full fine-tuning (-m MODEL, a GGUF file or a Hugging Face directory) updates every weight of an existing model, starting from its tokenizer and shape. It needs the most memory: for SmolLM2-135M, gradients 513 MiB and optimizer 1026 MiB beside the 513 MiB of weights.

LoRA. A low-rank adapter leaves the model frozen and learns a small correction to some of its matrices: a weight W (outputs x inputs) becomes

W' = W + (alpha / R) x B x A        A: R x inputs, B: outputs x R

R, the rank, is much smaller than the matrix's sides, so B x A has the matrix's shape but only R x (inputs + outputs) free numbers. A starts random and B at zero, so a fresh adapter changes nothing. Only A and B get gradients and optimizer state:

$ oklm train -m SmolLM2-135M-Instruct-f16.gguf -f data/tinyshakespeare.txt --out lora/a --lora 8 --lora-targets all \
      --batch 2 --seq 64 --steps 3 --eval-every 3 --lr 1e-3 --backend cpu
LoRA: rank 8, alpha 8, targets all; 2442240 trainable parameters, the model frozen
...
memory: gradients 9.3 MiB, optimizer 18.6 MiB (F32), activations kept for the backward pass 132.2 MiB (F32); checkpoints F32
saved lora/a-000003.gguf

2.4 M trainable parameters instead of 135 M; gradients and optimizer 28 MiB instead of 1.5 GiB; the adapter file 9.8 MB against 540 MB for a full F32 checkpoint (its sidecar 19.6 MB against 1.08 GB). The adapter is in llama.cpp's LoRA format.

  • Targets: --lora-targets lists the matrices: any of q, k, v, o, gate, up, down (for GPT-2 qkv, o, up, down), or all; the default is the attention's (q,k,v,o).
  • Rank: a larger rank can express a larger change and costs proportionally more parameters, memory and file size. The adapter's scale is alpha / R; --lora-alpha defaults to the rank (scale 1). Start small (8 to 16) and raise it only when the training and validation losses stop improving while a larger change is needed; compare on the validation loss the run prints.
  • Use at runtime (run, chat, ppl, serve --lora FILE [--lora-scale S]): the adapter is merged into a temporary copy of the model as it loads; serve --lora-module serves several beside one base. Merge (oklm lora-merge MODEL ADAPTER OUT [--scale S]) writes W + S x (alpha / R) x B x A into a new model file; F32, F16 and BF16 weights keep their type, quantized ones become F16 (quantize the merged file again if needed).
$ oklm run -m SmolLM2-135M-Instruct-f16.gguf --lora lora/a-000003.gguf -p "ROMEO:" -n 12 --temp 0 --backend cpu
lora: SmolLM2-135M-Instruct-f16.gguf + lora/a-000003.gguf (scale 1) -> /tmp/oklm-lora-2741031-0.gguf, 210 weights merged
...
$ oklm lora-merge SmolLM2-135M-Instruct-f16.gguf lora/a-000003.gguf merged.gguf
  • Limits: a low-rank update changes how the model uses what it already knows (tone, format, following a kind of instruction, using a tool) well; it has few parameters to store new facts in, so teaching facts this way is unreliable. Give facts in the prompt (documents) instead.

Saving memory.

  • --recompute keeps only each block's input and recomputes its activations in the backward pass: much less activation memory (longer sequences, larger batches) for about one more forward pass, the same gradients.
  • --quantized-base (with --lora) keeps the frozen model's matrices in the file's types (Q8_0, Q4_K, ...) and unpacks each just before its product: SmolLM2-135M Q8_0 takes 147 MiB instead of 522 MiB of weights, with the same results.
  • --act-type bf16 stores the activations kept for the backward pass in BF16, --opt-type bf16 the AdamW moments (with stochastic rounding) and --grad-type bf16 the gradients (each block's computed in an F32 window, then added in with stochastic rounding), on the CPU or the GPU, every computation still in F32; --save-type bf16 writes the checkpoints in BF16 (training.md).
  • --matmul-precision bf16|f16 (--backend rocm) runs the large products on the GPU's matrix units with 16-bit inputs; F16 needs --loss-scale dynamic, which computes the gradients of a scaled loss (so small ones do not vanish in F16) and divides them back in the update, skipping an update that overflowed (training.md). The weights' 16-bit copy is kept on the GPU between updates (2 bytes a parameter, when it fits): each weight is rounded once after the optimizer changes it, not at each of its products, with the same results.

Chat data. --chat-data conv.jsonl (lines of {"text": ...}, conversations rendered through the model's template, as oklm agent --save-training FILE writes them) trains on the assistant turns only.

Reinforcement learning. oklm rl fine-tunes from rewards instead of example answers: for each task it samples a group of completions (--group 8, --max-new 32 tokens at --temp 1), scores each 1 or 0, and pushes up the completions that scored above their group's mean (group-relative advantages, GRPO). --updates N reuses each batch for several clipped updates and --kl W (with --ref MODEL, default the starting model) keeps the model near the reference.

tasks.jsonl holds one {"prompt": ..., "answer": ...} per line. The prompt is tokenized as written, special tokens included, so it may carry the chat template's markers. A completion earns 1 when the answer is a number and the completion's last number equals it ("13 + 6 = 19." gives 19; 7 and 7.0 agree), or, for any other answer, when the completion starts with it (leading spaces dropped, not followed by a letter or digit). --val-frac (0.1) of the tasks are held out and their greedy accuracy printed every --eval-every steps.

$ cat tasks.jsonl
{"prompt": "2+3=", "answer": 5}
{"prompt": "4+4=", "answer": 8}
...
{"prompt": "Capital of France:", "answer": "Paris"}
$ oklm rl -m SmolLM2-135M-Instruct-f16.gguf -f tasks.jsonl --out rl/rl --steps 2 --tasks 2 --group 4 --max-new 8 \
      --eval-every 1 --val-frac 0.2 --backend cpu
model: llama, 134515008 parameters; 4 tasks for training, 1 held out; 2 tasks a step, groups of 4, 8 new tokens at temperature 1; on cpu
step 1/2: reward 0.000, solved 0.000, length 7.5, 0 tasks learned from, grad norm 0, 5.2 s
step 2/2: reward 0.125, solved 0.125, length 8.0, 1 tasks learned from, grad norm 7.77, 6.2 s
...
saved rl/rl-000001.gguf

A group whose completions all score the same teaches nothing (the first step above made no update, so the checkpoint counts one). Rewards only reinforce what the model already does sometimes (training.md).

Images, sounds and speech

oklm describe: a language model reads text embeddings; an image or a sound has to become vectors of the same kind. The projector file (--mmproj, llama.cpp's mmproj GGUF) holds an encoder and a projector that do this: for SmolVLM, a SigLIP vision encoder whose patch embeddings are pixel-shuffled and projected (idefics3); for Gemma 3, the same encoder at 896 pixels, its patches averaged in groups of 16 and projected to 256 embeddings, which the model reads attending to one another in both directions; for LLaVA 1.5, a CLIP encoder whose 576 patch rows go through two linear layers; for LLaVA 1.6, the same encoder on an overview and on two to four tiles of the image, the tiles' rows put back side by side as one grid, the part that covers the padding around the image cut away and each row of the grid ended by an embedding the model learned for that; for Qwen2-VL and Qwen2.5-VL, an encoder that reads the image at its own size (one embedding per square of 28 pixels, up to 4 096), whose rows the language model places by their row and column in the image: its rotary positions have three parts (time, height, width), and the text after an image continues from the largest one used; for Ultravox, a Whisper audio encoder and a projector. Their output rows take the place of token embeddings in the prompt, where the marker <__media__> is (or before the prompt: the question right after the picture, as the templates of SmolVLM, Gemma 3, Qwen2-VL and Qwen2.5-VL write a picture and a text, or after a new line for LLaVA, whose own conversations have one; a sound for Ultravox after the question and a space, the order its own pipeline writes and the one it answers about the sound in). A chat request to oklm serve --mmproj holding a picture and a question is written the same way, by the same function: the model reads the same tokens. A video in a request (video_url, base64) is decoded, sampled and laid out by the code oklm describe uses, the frames it may decode bounded. The server keeps the embeddings of recent media by a hash of their pixels or samples, and remembers which bytes decoded to which of them: a chat window sends a conversation's pictures and videos again with every message, and the same bytes are then neither decoded nor encoded (server.md). oklm chat --mmproj does the same for one conversation: /attach FILE gives a picture, a sound or a video to the next message, each turn's prompt is rows (tokens, and each item's embeddings named by its content's hash) compared with the rows the context holds, and an item is encoded once and its embeddings kept, so a question after a picture costs its own tokens (SmolVLM-256M on the CPU: 14 rows evaluated of 375, the 320 of the picture among the 361 cached; inference.md). The projector file decides whether images or sounds are expected. --tiles SIDE cuts large images into tiles plus an overview; videos give frames (--fps 1, --frames 8) or, for an audio model, their sound: Ogg Theora natively, other containers through ffmpeg only with --ffmpeg or OKLM_FFMPEG=1. Qwen2-VL and Qwen2.5-VL read the frames as their own code reads a video: two by two through the two kernels of the patch embedding, one grid of embeddings a pair, and in the language model a time position that advances from a pair to the next (by one for Qwen2-VL, by the seconds a pair covers times two for Qwen2.5-VL); the other models get each frame as a still image with its time written before it. With the language model on a device (--backend), the vision encoder runs there too, from the image's patches to the projector's rows (a prompt with a picture: 8.6 s to 0.35 s for Qwen2-VL 2B on Vulkan), and the audio encoder, from its first convolution to its projector (a prompt with 11 seconds of speech: 5.3 s to 0.26 s for Ultravox on Vulkan); the image is cut into patches and the sound's spectrogram computed on the CPU.

$ oklm describe -m SmolVLM-256M-Instruct-Q8_0.gguf --mmproj mmproj-SmolVLM-256M-Instruct-f16.gguf \
      --frames 2 -n 30 -p "What is shown?" tests/fixtures/mm/testsrc.ogv
[tests/fixtures/mm/testsrc.ogv: 2 frames, 0.0 s of sound]
 A colorful image consisting of multiple vertical lines with varying shades of color.[640 media embeddings, prompt 713 tokens in 11.65 s; 14 tokens in 0.15 s]

oklm transcribe takes a Hugging Face Whisper directory: config.json (a Whisper configuration; the mel bin count is read from it), a single model.safetensors and tokenizer.json. It decodes greedily, a window of 30 seconds at a time, with Whisper's timestamp tokens: the tokens between two timestamps are a segment with its times (--timestamps, --format srt|vtt|json), and the next window starts where the last closed segment ended, so the cuts fall in the pauses and not every 30 seconds. The language is the most probable language token after the start token, found from the first window (--language CODE gives it instead), and --translate asks for English with the other task token (multimodal.md). The layer counts and the special tokens' ids are read from the files too, and the weights may be F32 or F16, so the large-v3 checkpoints load like the small ones. Checked with Whisper base, small, large-v3-turbo and large-v3 against transformers on the same files. --backend opencl|vulkan runs the encoder (its two convolutions and its blocks) and the decoder on a device (a 30-second window of large-v3: 11.4 s on the CPU, 0.71 s on Vulkan; multimodal.md); oklm speak --backend does the same for the language model of a speech model and for its vocoder's network, whose convolutions are products over views of the rows each position of a kernel reads and whose group normalizations are a layer normalization between two copies through a view that lays each group out as a row (speech.md). A server takes the same devices for its image encoder (with its model), its recognizer (--transcribe-backend) and its synthesizer (--speak-backend): each computes in a thread of its own, one pass on the device at a time, the passes in the order they asked for the device. A decode step is a pass of a few milliseconds and an encoder's is tenths of a second, so an encoder lets the passes that wait in between two of its blocks, where its stream is whole in its own buffer: it first waits for the blocks it queued, then hands the device over. It does so on a device another thread uses: one whose pass waits, or that said it keeps taking passes there, as the scheduler does for as long as it has requests to decode, however slow a step. A model whose layers are spread over two devices holds both for its forward pass, always taken in the same order, so two such models cannot each hold the device the other waits for, and lets the waiting passes in where its stream goes from one device to the other. The server's cache, which takes what the device has free, is sized after what the encoders and the sound models will take there has been counted, each as it allocates, with what the backend's own buffers are left. A request that is decoding waits for one block of an image's or a window's encoder (0.013 to 0.08 s on Vulkan where it waited 0.20 to 0.70 s; server.md).

$ oklm transcribe -m models/whisper-base models/samples/jfk.wav
[11.0 s of audio in 0.92 s]
And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.

A server does the same for its clients: oklm serve --transcribe-model whisper-base answers POST /v1/audio/transcriptions and POST /v1/audio/translations (OpenAI's routes: a multipart file, or the sound as the body; verbose_json holds the language and the timed segments), and every server reads a document sent to POST /api/ingest into text, as oklm ingest does (server.md, src/server/transcribe.c, src/server/upload.c). A document too long for its client is cut, unless the server holds an embedding model (--embed-model, or the one of its --rag index) and the client names a conversation: the text is then cut into passages, each embedded and kept in memory under that name, and POST /api/ingest/search gives the passages nearest a question (or, for a question about the whole document such as "summarize this", passages at equal steps over it: oklm_ingest_whole_question(), which the chat window uses the same way), which the dashboard's chat puts before each message (src/server/docs.c: bounded in conversations, documents and bytes, dropped after an hour without a question or when the client says so; server.md). The server checks the sound models' files when it starts and loads their weights at the first request of each route (--preload: at start; /api/stats says loaded).

The other way too: oklm serve --speak-model outetts-0.2-500m answers POST /v1/audio/speech (OpenAI's route) with the synthesis described next. Each request's synthesis runs in a thread of its own, one at a time, and hands the sound over pass by pass; the HTTP thread sends each pass as a chunk (pcm, a WAV file whose header leaves its lengths open, or the FLAC frames the pass completes) or the whole file at the end, and stops the synthesis when its client is gone. A request names its voice among the speaker files the server offers (GET /v1/audio/voices) and its speed (server.md, src/server/speak.c).

oklm speak goes the other way, with two models (speech.md): OuteTTS 0.2 or 0.3, a qwen2 language model that continues a prompt of words and audio codes (a voice's, then the text's words) with the codes of the text, 75 a second, and WavTokenizer's decoder, a convolutional network with one attention block that turns the codes into a spectrum, which an inverse short-time Fourier transform makes a waveform at 24 kHz. A text is spoken in passes, a sentence each (a long one in several), each handed over when its last word is written, with a pause after a sentence's end. The voice's codes come after the text in the prompt, so a language model's cache cannot keep them from one text to the next: what is saved is prompts, the passes that hold together being written under one, the vocoder reading each pass with a second of the codes before it; a first pass cut short gives an earlier first sound for no prompt more. A speed is a change made on the samples that keeps the pitch (overlap-add of frames placed where the waveforms agree), and a FLAC file is written by OKLM's own encoder (fixed predictors and Rice codes). On the CPU, or with the language model and the vocoder's network on a device (--backend: a sentence of 7.0 s in 1.48 s on Vulkan, the sound within one step of the CPU vocoder's 16-bit samples); on the CPU the vocoder's output is within 131 dB of a double-precision implementation's, and the prompt's tokens are llama.cpp's.

Soprano (speech.md, the catalogue's soprano-1.1-80m) is the model that speaks faster than it is heard on a CPU. It has no voice in its prompt and no audio codes: the text is cleaned as the model's own package cleans it (numbers, dates, amounts and abbreviations in words, src/mm/soprano_text.c), each sentence is a prompt of three tokens more than its text, an 80 M-parameter qwen3 model writes tokens until [STOP], and what goes to the decoder is the final hidden state that predicted each token (oklm_context_params.hidden), 16 a second. The decoder is the ConvNeXt half of the vocoder above without its attention: the states interpolated to four frames each, eight blocks, a spectrum, and the inverse transform at 32 kHz. Since nothing in it reaches further than four states, the first sound of a sentence is handed over while the rest is written, and the pieces are the samples of one call. The tokens written are llama.cpp's, the decoder's output within 107 dB of a double-precision implementation's.

$ oklm speak -m outetts-0.2-500m -o cat.wav "The catalogue name finds both files."
[2.6 s of speech after 6.0 s]
cat.wav: 2.60 s of speech in 6.00 s (0.43 s of sound per second)

Audio formats: PCM WAV (8, 16, 24, 32-bit integer and 32-bit float) always; FLAC, Ogg Vorbis, Opus and MP3 when OKLM is built with Allegro 5. Images: PNM always; PNG, JPEG, BMP, TGA, PCX with Allegro 5. Sounds are mixed to mono and resampled to 16 kHz. See multimodal.md.

Refusal analysis

Chat models are tuned to decline some requests. Research on these models (Arditi et al., 2024) found that this behaviour is largely carried by one direction in the model's internal state.

This weakens the model's safety behaviour. oklm abliterate exists for research and interpretability: measuring where refusal lives in a model, and preparing a base whose behaviour is then retrained deliberately. Do not deploy an abliterated model in place of the original.

  • oklm refusals answers each prompt of a file greedily (-n 48 tokens) through the chat template and counts the answers that read as refusals (the usual markers). With --ref MODEL --harmless FILE it also measures the mean KL divergence of the first token's distribution from a reference model on prompts the model should answer: how much a change moved the model's ordinary behaviour.
  • oklm abliterate runs two prompt sets through the model, one it declines (--harmful) and one it answers (--harmless), and reads the residual stream (the running state between blocks) at each prompt's last token, entering every layer. Per layer, the difference of the two sets' mean states, normalized, is the candidate refusal direction; the layer that separates the two sets best within a band of the depth (0.3 to 0.7, --band) is chosen, or --layer N. With --out, every matrix that writes into the residual stream (each block's attention output and FFN down projection, and the token embeddings when the head is not tied to them) is made orthogonal to that direction, W' = W - s r (r^T W), so the model can no longer write along it. --scale s (1) weakens the removal below 1; --save-dirs / --load-dirs keep the directions to write other layers or scales without recomputing.
$ oklm abliterate -m SmolLM2-135M-Instruct-f16.gguf --harmful h32.txt --harmless l32.txt --out abl.gguf
abliterate: 32 + 32 prompts read in 2 s
abliterate: layer 11's direction removed (scale 1), abl.gguf written in 0 s
layer  separation  gap
...
   11       4.368  0.093  <- chosen
...
$ oklm refusals -m abl.gguf --ref SmolLM2-135M-Instruct-f16.gguf --harmless answered8.txt --backend cpu
first-token KL divergence from the reference: 0.1127 (mean of 8 answered prompts)

(The prompt files are 32 lines of models/abliterate/'s training sets, fetched by tools/fetch_abliterate_data.sh.) SmolLM2-135M refused none of the 8 held-out harmful prompts tried here even before; the measured effect on a model that does refuse (Qwen3-8B: 97.5 % to 7.5 %) and its cost are in training.md.

Developer tooling

Command What it does When
make library, apps, tests always
make test every test binary; skips what needs files not downloaded every change
make asan-test the tests under AddressSanitizer and UBSan (GPU drivers left out by the build itself) before merging; memory errors and undefined behaviour
make tsan-test the tests under ThreadSanitizer (GPU drivers left out) before merging; data races
make check cppcheck over src, apps, tests, failing on findings before merging
make CC=clang BUILD=build/clang test the tests built with clang before merging
make fuzz, make fuzz-run FUZZ_TIME=60 libFuzzer targets (GGUF, safetensors, regex, tokenizer, Jinja, grammar, PDF, tool calls, shell splitting), each run FUZZ_TIME seconds after touching a parser
make test-public downloads a pinned corpus of public models, builds a Python venv, and checks OKLM's files, dequantization, tokenization, templates and outputs against gguf-py, Hugging Face, jinja2 and llama.cpp's files after touching formats, kernels or the forward pass
make bench matrix product micro-benchmark kernel work
sh tools/bench_vs_llama.sh prompt and decode t/s against llama.cpp's llama-bench on the same files, the CPU at 1 to 16 threads and the GPU (benchmarks.md) speed work
sh tools/gpu/report.sh on a GPU OKLM was never tested on: prerequisites, system and drivers, build, the OpenCL and Vulkan device tests on each device, a speed table with --models; one report file (testing.md) a new GPU or driver
make doc Doxygen HTML in build/doc/html, warnings in build/doc/doxygen_warnings.log after changing headers
make docs-html README.md and docs/ converted by pandoc to an HTML set in build/doc/docs-html (open index.html): page list, table of contents, offline search; fails on a dead link or a missing anchor to read or hand over the docs outside a Markdown viewer
make deb, make appimage, make installer-windows, make docker-build the packages in dist/: a .deb, an AppImage, a Windows installer, and the first two built in a Debian container (build.md) a release, or a machine without the build tools
make docs-lint trailing spaces, tabs, a Status: line on every page, relative links and their anchors, the README's quick start, every environment variable documented; and, read from the sources without running anything, every option of the usage texts, every route of the server and of a listening agent, every slash command and every oklm-gui tool documented, the synopses of the command reference (cli.md) those of the usage text after changing docs, an option, a route or a command
make eval the agent's evaluation tasks with EVAL_MODEL agent changes
make rag-eval retrieval on SciFact and the docs set, dense / sparse / hybrid with bge-small-en-v1.5: the published retrieval table (rag.md) embedding, chunking, index or search changes
make tool-eval tool selection by retrieval: accuracy and prompt tokens over tests/rag/tool_queries.jsonl tool index or ToolSearch changes
make gui-test, make wine-test, make asan-opencl the GUI under Xvfb, the Windows build under Wine, the OpenCL tests under ASan the matching area
sh tools/check_gui_wine.sh the Windows build's graphical tools under Wine and Xvfb, the flows of each, driven by the tools' own presses, and a server kept across logins (58 checks with the canary models; testing.md) a tool's Windows paths, the registry, the listener

A change is mergeable when the tests, both sanitizers, cppcheck and the clang build are clean (testing.md, build.md).

Files, caches and environment

What OKLM writes and where:

Path What
~/.cache/oklm/tuning/ ($XDG_CACHE_HOME) oklm tune results; safe to delete
~/.cache/oklm/usage/<model hash>.json MoE routing counts of the expert cache and the device tier
.oklm/chat/last.json, last.kv oklm chat --resume / --kv-save
.oklm/sessions/<id>.jsonl, <id>.kv agent sessions; removed after cleanupPeriodDays (30)
.oklm/audit/<session>.jsonl the agent's permission decisions
.oklm/jobs/ the agent's background jobs
.oklm/settings.json, .oklm/settings.local.json, ~/.config/oklm/settings.json, /etc/oklm/managed-settings.json agent settings and permission rules
~/.local/state/oklm/logs/ the agent's debug log (--debug)
~/.local/state/oklm/rag/tools-<hash>.gguf the agent's tool index (--tool-rag), one a project; safe to delete
~/.local/state/oklm/rag/project-<hash>.gguf, memory-<hash>.gguf the agent's index of the project's documentation (--rag-project) and of its earlier sessions (--rag-memory), one a project; rebuilt when deleted
~/.local/share/oklm/models/ one place the agent looks for its default model

The environment variables met most often (every one, with its default and where it is read, is in build.md):

Variable Effect
OKLM_THREADS CPU threads (default: physical cores)
OKLM_BACKEND auto, cpu, opencl or vulkan when --backend is not given
OKLM_BACKEND_ORDER the backends auto tries first, in order (vulkan,opencl,cpu)
OKLM_VK_DEVICES which Vulkan device(s) the vulkan backend uses (by default every discrete GPU, integrated ones only without a discrete one); all: every GPU; none: no Vulkan device
OKLM_CL_DEVICES which OpenCL device(s) to use (by default every discrete GPU, integrated ones only without a discrete one); none: the CPU alone
OKLM_GPU_MEM_MB the device memory to assume when fitting layers
OKLM_STATE_KEEP states a context of a model with recurrent blocks keeps for cut sequences (default 8; 0: none)
OKLM_GPU_MAX_ALLOC_MB the largest device allocation to assume (weights and caches split into allocations of that size)
OKLM_LOG_LEVEL, OKLM_LOG_FILE log verbosity and destination
OKLM_MODELS directories where the agent looks for models
OKLM_CL_WMMA 1, f16 or bf16: every OpenCL prompt product on the RDNA3 float matrix units; unset: those of the quantized types without an 8-bit kernel; 0: none
OKLM_CL_Q8, OKLM_CL_Q8_MV 0: quantized prompt products, or the K-quants' one-column decode products, on the float kernels instead of the 8-bit ones (default on)
OKLM_CL_FUSE, OKLM_CL_MVG, OKLM_CL_RES, OKLM_CL_ROPE_KV, OKLM_CL_ROUTE_SIGMOID, OKLM_CL_ATT_MLA 0: no fused OpenCL launches (routing, query / key norms with rotation, a decode norm folded into its product; OKLM_CL_MVG=0 only a MoE layer's gate, up and SwiGLU, OKLM_CL_RES=0 only the end of a residual branch with the norm after it, OKLM_CL_ROPE_KV=0 only the rotation with the cache writes, OKLM_CL_ROUTE_SIGMOID=0 only a sigmoid router's routing), or latent attention's prompts on the per-token kernel (default on)
OKLM_CPU_PACK_EXPERTS 0 / 1: never / always keep repacked copies of a MoE model's routed experts (default: when they fit)
OKLM_GELU_F16 1: the tanh GELU rounded as ggml's half-precision table rounds it (CPU), for comparisons with llama.cpp on the Gemma family (default off: single precision)
OKLM_ATTN_F16 1: attention over an F16 cache with the queries and the probabilities rounded to half precision, as ggml's products round them (CPU), for comparisons with llama.cpp (default off)
OKLM_Q8_ROUND_EVEN 1: Q8_0 and Q8_1 activations rounded as ggml's x86 code rounds an exact half (to the even step), for comparisons with llama.cpp (default off: away from zero, as ggml's portable code)
OKLM_DUMP, OKLM_DUMP_FEED, OKLM_DUMP_FEED_OPS the steps of a forward pass written to a file, and another program's taken in their place, for a comparison step by step with llama.cpp (testing.md)
OKLM_MATVEC_THREADS threads at most of a one-row (decode) product; fewer readers get more from the memory
OKLM_CPU_PACK_HUGE 0: the repacked copies as one allocation each instead of chunks on huge pages (Linux, default on)

Architectures. oklm info gives the verdict for a file. Inference covers llama, mistral, mistral3, qwen2, qwen3, gemma, gemma2, gemma3, gemma4, phi2, phi3, granite, granitemoe, qwen2moe, qwen3moe, deepseek, deepseek2 (V2 / V3), glm4, glm4moe, olmoe, gpt2, starcoder2, lfm2 and lfm2moe (Liquid's LFM2: short convolution blocks between attention blocks, whose per-sequence state, the last two gated inputs of each block, is rebuilt from the last tokens when a sequence is truncated, copied or read back), gpt-oss (attention sinks, experts with biases; on the CPU, OpenCL and Vulkan, src/model/arch_gptoss.c), spark2_5 and muse-glimmer (the attention output times a sigmoid gate computed from the block's input, a sliding window on three blocks out of four, the rotation set per kind of block; on the CPU, OpenCL and Vulkan, src/model/arch_gate.c), gemma4 (Gemma 4 12B and 26B-A4B: window blocks and blocks without window with heads of two sizes, each layer's cache in its block's shape, values taken from the key projection where a block has no attn_v, a scale on each block's output, routed experts beside the dense feed-forward; in the E2B and E4B an embedding per layer added at each block's end and last blocks that keep no cache and attend over an earlier block's; on the CPU and on both GPU backends, a block and the cache it reads in one memory, src/model/arch_gemma4.c), mimo2 (MiMo-V2: values with narrower heads than the queries and keys, kept in cells of the keys' size with zeros after them so the attention op and the cache stay as they are; the window and the count of key heads set block by block; on the CPU, checked on a tiny model only, src/model/arch_mimo2.c), and the hybrid qwen3next stack (checked on a synthetic model only), whose Gated DeltaNet blocks carry a fixed-size state per sequence instead of a KV cache: a short conv with its ring and the gated delta recurrence, two ops that run on the CPU or the GPU like the others, the states of GPU layers kept in GPU memory; training covers GPT-2, Llama, Qwen2 and Qwen3 blocks and that hybrid stack, its Gated DeltaNet blocks on the GPU as well. qwen35 (Qwen3.5-9B) is the published form of such a stack, with a decay per token and gated queries; it runs on the CPU, on OpenCL and on Vulkan (the recurrence's kernel takes both forms) and is checked against llama.cpp on the published file on each, as is qwen35moe (Qwen3.6-35B-A3B), the same stack with routed experts and a shared one in place of the dense feed-forward (hybrid-layers-design.md). On a few steps of a sentence it repeats, the 9B file's logits move by 10 and more when only the last places of a run change, in llama.cpp against itself as in OKLM against it: the model's own sensitivity, measured block by block in hybrid-layers-design.md. A sequence's recurrent state only moves forward, so the context keeps a few states (eight tokens before the end of each prompt, every 32 tokens of a growing sequence, before each draft) and a journal of what the recurrences read at the last positions: cutting a hybrid sequence back (a rejected draft, a turn asked again, a request sharing a system prompt) starts from the nearest kept state and costs the tokens after it, not the prefix (hybrid-layers-design.md); with the blocks on a device the states are kept in host memory as well and the journal is not written, a cut being a forward pass from the kept state. The Ternary Bonsai 2 files are that layout with ternary weights stored rotated: their prism.hadamard.* metadata lists the matrices that were multiplied by signs and a block Walsh-Hadamard matrix before being rounded, and each product with one of them reads its input through the same signs and transform (src/model/fold.c; on OpenCL and Vulkan one device op, oklm_hadamard_fold(), with the signs copied to the device); two of their three tensor types, PQ2_0 and PTQ1_0, are a fork's and are read like ggml's, on the CPU and in the devices' products (hybrid-layers-design.md). Tokenizers, templates and tensor types per family are in compatibility.md.

Limits worth knowing

  • oklm plan reads GGUF files only; convert a checkpoint first. A checkpoint's headers give every shape, but loading it builds tensors the file does not hold as such (per-expert matrices stacked, FP8 matrices widened, GPT-2's transposed), which reads their data; the plan would need a sizing path of its own for those, so a directory is refused instead.
  • A Hugging Face checkpoint loads directly only with F32, F16, BF16 or FP8 matrices; other stored types go through oklm convert.
  • oklm run completes the raw prompt; use chat (or the server) for a chat model's template. With --doc on a chat model, a raw prompt may get an immediate end of text.
  • The expert cache works on the CPU; --n-cpu-moe and --expert-gpu need a device backend.
  • The server serves one model (and its adapters), no embeddings, and n must be 1. The Windows build has no TLS.
  • oklm transcribe reads a single model.safetensors, not sharded Whisper checkpoints.
  • Against llama.cpp on the same machine the CPU is ahead on prompts and level or ahead on decoding from four threads; on the GPU some rows stay behind (benchmarks.md).