OKLMdocumentation v1.0.3
On this page

Inference

How OKLM runs a decoder-only transformer from a GGUF file: loading and binding the weights, the context and its KV cache, the Llama-family and GPT-2 forward passes, sampling, chat templates, the oklm run, oklm chat and oklm ppl commands, and how the results are verified. Mixture-of-experts models (the expert products, the expert cache, --n-cpu-moe, the device expert tier) are in moe.md; the memory projection of oklm plan in memory-plan.md.

Status: maintained.

API

#include "oklm/model/model.h"
#include "oklm/model/sampling.h"

oklm_model* m;
oklm_context* c;
oklm_context_params cp = oklm_context_default_params();   /* n_ctx 0: training context, max 4096 */
oklm_model_load("SmolLM2-135M-Instruct-Q8_0.gguf", &m);   /* maps the file, binds every tensor */
oklm_context_new(m, &cp, &c);                             /* KV cache + buffers */
oklm_eval(c, prompt_ids, n, 0);                           /* appends n tokens to the cache */
const float* logits = oklm_get_logits(c, -1);             /* n_vocab values of the last position */
  • oklm_model_load maps the GGUF file (weights are used in place, never copied), reads the hyperparameters, builds the tokenizer when the file has tokenizer.ggml.* keys, and binds every tensor by name with a shape check. Norm weights and biases must be F32; matrices may be any type that has a dot product (F32, F16, BF16, the legacy quants, the K-quants).
  • A context owns the KV cache (kv_type F16 by default, or F32), the activation buffers for n_batch positions and a scratch arena. Several contexts can share one model. The cache has rows for the blocks that attend: a block without attention (the linear blocks of qwen35 and qwen35moe, the convolution blocks of lfm2 and lfm2moe) reserves none, on the CPU or on a device (hybrid-layers-design.md).
  • oklm_eval appends tokens at position oklm_n_past(), splitting them into batches of n_batch. With all_logits (only for n <= n_batch) every position's logits stay readable with oklm_get_logits(c, i); otherwise only the last one is computed. Tokens beyond n_ctx give OKLM_ERR_SHAPE, out-of-range ids OKLM_ERR_INVALID.
  • oklm_kv_truncate(c, n) forgets everything after the first n positions, which is how oklm chat re-uses the common prefix of two turns and how a caller rewinds.
  • Several sequences: with cp.n_seq = S the cache holds n_ctx cells for each of S independent sequences. oklm_eval_batch(c, items, k) runs the new tokens of up to S sequences (oklm_seq_tokens: sequence, tokens, count, all logits or the last one; n_batch tokens in all) in one forward pass: the matrix products read each weight once for every token, attention keeps each sequence to its own cells. oklm_get_logits_item(c, item, i) reads the logits; oklm_seq_n_past, oklm_seq_truncate and oklm_seq_copy (a shared prompt evaluated once, then copied) manage the sequences; oklm_eval, oklm_n_past and oklm_kv_truncate act on sequence 0. The logits equal those of one context per sequence (bit for bit on the CPU and in the tests on OpenCL). Decode of Llama-3.2-1B Q4_K_M on 8 CPU threads: 33 tokens/s for one sequence, 200 for 8 together, 290 to 340 for 16 to 32 (tests/model/bench_batch.c).
  • A sequence's cache saved to a file and loaded back: oklm_kv_save() and oklm_kv_load(), in KV files below.

KV files

oklm_kv_save(c, seq, path, tokens, n, flags, &info) writes the cached state of a sequence's first n positions; oklm_kv_load(c, seq, path, tokens, n, &loaded) empties a sequence and gives it the rows of the longest prefix the file and tokens share (everything with tokens NULL). The caller then evaluates the tokens after loaded, as after a truncation. The context does not keep token ids, so the caller passes them (oklm_kv_file_tokens() reads them back from a file). The next token's logits after a load are those of the context that saved, bit for bit (tests/model/test_kv_file.c: one sequence, the second of two, paged to contiguous and back, F16 and F32, MLA, a hybrid stack).

A sequence that holds media rows (the embeddings of an image, a sound or a video) is saved and loaded with oklm_kv_save_media() and oklm_kv_load_media(), which take, beside the tokens, the rows' media hashes as oklm_eval_batch() takes them (oklm_seq_tokens.media: 0 for a token, else the item's content hash, the token then being the row's index within its item); oklm_kv_file_rows() reads both back. The file keeps the hashes and, per row, the positions the context gave it: for a model with three-part rotary positions (Qwen2-VL, Qwen2.5-VL: multimodal.md) an image's rows share a time position and the token after an image does not take the position of its row index, so the loading context is given what the saving one held, the time position of each row and the position a token after it takes. A load may stop inside an image (the prefix wanted ends there): the image is continued with oklm_eval_embd_grid() from its next row, as after a cut. A paged cache that loads the file records each row by its token or by its hash, so the full blocks are found again by a request with the same media. oklm_kv_save() on such a sequence keeps the positions too; it has no hashes to give, and its caller passes -1 as the token of a row given as an embedding, which a paged cache never takes for a cached prefix. Checked on the tiny qwen2vl model (test_kv_file: saved after an image, after a video, after an image of one column, inside them and in the text, loaded contiguous and paged; test_model's test_mrope: 100 saves over its 14 cases, the logits of the context that never left memory bit for bit on the CPU, and the reference's), on models whose image rows take one position each (plain embedding rows on the tiny llama model, rows that see each other on the tiny gemma3 one), and once on the published Qwen2-VL 2B (multimodal.md). A file whose hashes check and whose rows say positions no sequence has (a time position below zero, a position after a row that is not past the row's own, or, for a model whose rows take one position each, anything but the row's index and the next) is refused before a row is read: test_kv_file writes such files, the chunk's hash made that of the changed payload, and a changed file no rule forbids, which loads.

Format (little-endian, versions 1 and 2, src/model/kv_file.c):

part content
header OKLMKVC\0, version, header size; the model's identity (oklm_model_id(): a hash of the architecture, the sizes, every tensor's type and shape and the first bytes of its data; a tensor added to the layers since is counted only in the models that have it, so the identities written before it stay valid); the sequence's adapter; n_ctx, block size, rotation groups of a quantized cache, MLA, the layer range, the recurrent state's size; per layer its key and value types and row bytes; a hash of all that
chunk head (magic, first position p0, count n, payload bytes); payload: the n token ids; in version 2 the n media hashes (64 bits each), the n time positions and the n positions a token after each row takes (32 bits each); then per layer its n key rows and its n value rows (MLA: the compressed latent and rope rows only; a hybrid stack's linear blocks have none), then the recurrent state of each linear block; trailer (magic, positions held after it, the payload's hash, the head's offset)

Versions. Version 1 keeps a token id a row: every row is a token at the position of its index. Version 2 adds 16 bytes a row (the media hash and the two positions) and says so in the header (the word that was padding is 1). A sequence is written as version 2 only when it has what version 1 cannot say, a row with a media hash or a position that is not its index; a sequence of tokens alone, of a qwen2vl model too, is written as version 1, byte for byte what the builds before version 2 wrote, so they read it. Both versions are read; a version 1 file's rows load as tokens. A build from before version 2 refuses a version 2 file as not being a KV file (OKLM_ERR_FORMAT: it would otherwise read the rows at the wrong positions); this build refuses a version after 2 with OKLM_ERR_UNSUPPORTED and a log line naming the version it found and those it reads. The positions are under the chunk's hash like the rest of the payload: a cut or changed row section is a chunk that does not check. A file whose hashes check but whose positions no sequence can have (a time position below zero or not below the position after the row, positions going back, or, for a model with one-part positions, a row away from its index) is OKLM_ERR_FORMAT. An append to a version 1 file by a sequence that now holds an image writes the file whole as version 2.

oklm serve --prefix-dir DIR uses these files for the server's prefix cache: one file per distinct head, reloaded at start (server.md), heads holding images included. oklm describe --kv-save FILE and --kv-load FILE keep a conversation about an image in one (multimodal.md).

A chunk replaces what the file held from its p0 on. The rows are the cache's own bytes, so a file loads into a context whose cache has the same types per layer, contiguous or paged (a paged cache takes blocks for them and records their tokens: their full blocks become cached prefixes). Loading refuses another model, another cache type or row size, other rotation groups, another adapter and, for a model with LongRoPE factors, another context length (OKLM_ERR_SHAPE); a file without an intact chunk is OKLM_ERR_FORMAT. On any error the sequence is left empty. A hybrid stack's recurrent state is that of the file's end: it loads whole or not at all. A device cache is read back through the host (float types; a quantized device cache is OKLM_ERR_UNSUPPORTED).

Crash safety. Written whole, the file goes to path.tmp, is synced and renamed over path (oklm_file_replace(), MoveFileEx on Windows) and its directory synced: a crash leaves the previous file. With OKLM_KV_SAVE_APPEND, a compatible file is extended instead: one chunk with the rows from the first position where the file's tokens and the new ones differ (the new turn's tokens; after a divergence, from where it diverged), then synced. A reader walks the chunks while each trailer checks and its payload hashes right, so a crash in the middle of an append leaves the state of the previous save; the next append cuts the torn bytes and goes on from there. Saved again unchanged, nothing is written. Compaction: when the rows the file holds but no longer uses (a diverged history, superseded recurrent states) would outweigh those it uses, or the conversation diverged at its start, the file is written whole instead. An append costs the new rows; a load reads the file once to check it and once for the rows.

The size is the cache's: per token, every layer's key and value rows (23 KB for SmolLM2-135M in F16, about 128 KB for an 8B model with 8 KV heads). A block without attention has no rows in the file, as it has none in the context: its entry of the header's table is zero. That was so before a context stopped reserving rows for those blocks, so the files written then load unchanged (checked on LFM2-350M and Qwen3.5-9B: a conversation saved by the earlier build, continued by the later one, gives the same answer and the same file as the earlier build continuing it).

Backends

A context runs on the CPU (the default of oklm_context_default_params()), on OpenCL (cp.backend = "opencl", or "auto") or on Vulkan (cp.backend = "vulkan", see vulkan.md); the oklm commands use auto and accept --backend, --gpu-stream B (see opencl.md) and --gpu-layers N (the last N layers and the head on the device, the rest on the CPU; by default as many as fit in the device memory). On OpenCL the weights are copied to the device once per model, the KV cache and buffers live there, and quantized matrix products use float activations, so their logits are closer to the float64 reference than the CPU's. See opencl.md. Weights and caches larger than the device's largest allocation (4 GiB on RADV) are split over several allocations (OKLM_GPU_MAX_ALLOC_MB sets a smaller limit).

Hugging Face checkpoints

oklm_model_load() also takes a Hugging Face checkpoint: a directory with config.json and safetensors weights (one file or sharded with an index), or the path of its model.safetensors, index or config.json. Nothing is converted on disk:

  • the hyperparameters come from config.json, the tokenizer and chat template from tokenizer.json and tokenizer_config.json when present;
  • F32, F16 and BF16 matrices are used in place from the mapping (the BF16 dot product has an AVX2 kernel); FP8 matrices (float8_e4m3fn / e5m2, with <name>_scale_inv block scales as in DeepSeek-V3 and transformers' "fp8", or a per-tensor / per-row <name>_scale) are converted to BF16 with their scales at load time; other stored types must be converted with oklm convert first;
  • Q and K rows keep Hugging Face's half-split rotary layout and are rotated in that layout (NeoX) instead of being permuted as GGUF Llama weights are;
  • copies are made only where the forward pass needs another layout: norm and bias vectors in F32, GPT-2's Conv1D matrices transposed, the Llama 3 rope factors computed from rope_scaling.

oklm run -m models/smollm2-135m-hf ... therefore runs the BF16 checkpoint as published. The tiny llama, qwen2 and gpt2 checkpoints of tests/fixtures/hf give the logits of their own GGUF conversion (within 1e-6 for llama, where only the rotation order differs; bit-identical for the others), and the SmolLM2 checkpoint gives llama.cpp's greedy continuations on every prompt, like the published F16 GGUF.

Llama family

llama, mistral, mistral3, qwen2 and qwen3 share one forward pass (src/model/arch_llama.c). For each new position and each layer:

  1. RMSNorm of the residual, times attn_norm;
  2. Q, K, V projections (plus attn_*.bias for Qwen2);
  3. Qwen3 only: RMSNorm of each head of Q and K (attn_q_norm, attn_k_norm);
  4. rotary positions on the first rope.dimension_count values of each head: adjacent pairs for GGUF llama/mistral (their Q/K rows were permuted at conversion), first and second halves (NeoX) for Qwen; frequency scaling: linear (rope.scaling.type linear, or no type and a factor other than 1 under rope.scaling.factor or, in files of 2023, rope.scale_linear: llama.cpp's default), YaRN, or the Llama 3 per-pair factors (rope_freqs.weight); mistral3 then multiplies the rotated queries by its attention temperature, 1 + attention.temperature_scale x ln(floor(position / original context) + 1), which is 1 below the YaRN original context;
  5. K and V written to the cache at their positions;
  6. attention of every query head over all cached positions up to its own, grouped-query (each KV head serves n_head / n_head_kv query heads), softmax in F32 (denominator in double); on the CPU (AVX2), queries are computed in tiles of up to 32 rows (tokens x heads of a group) against blocks of 64 keys converted to float once (block-quant caches dequantized, the query rounded through the activation type of their dot product), each row's sums in a fixed order whatever the tiling or thread count; with fewer tiles than threads (decode, small batches) the same arithmetic is split by key blocks, rows and value columns, so a batch gives exactly the logits of its tokens one by one;
  7. output projection, added to the residual;
  8. RMSNorm times ffn_norm, SwiGLU (down(silu(gate(x)) * up(x))), added to the residual.

Then a final RMSNorm (output_norm) and the output head (output.weight, or the token embedding when the file ties them).

GPT-2

gpt2 (src/model/arch_gpt2.c): token plus learned position embeddings, pre-LayerNorm blocks with biases everywhere, one fused QKV matrix whose three thirds are used as views, multi-head attention, a GELU (tanh form) MLP, a final LayerNorm and the tied head. The context is capped at the training context because positions are learned.

phi2 and starcoder2 run the same function with rotary positions and no position table. Phi-2's block is parallel (the MLP reads the attention's normalized input). StarCoder2 has separate query, key and value projections with their biases, keys and values of attention.head_count_kv heads shared by the query heads (grouped-query attention, as in the llama family) and the NeoX rotation of whole heads at rope.freq_base; the rest is GPT-2's block.

LFM2

lfm2 and lfm2moe (Liquid's LFM2 and LFM2.5, src/model/arch_lfm2.c for what the llama family's forward pass lacks) mix two kinds of blocks. The file says which is which: attention.head_count_kv is an array with one entry per block, 0 for a short convolution block (the block then has the three shortconv.* tensors) and the key / value head count for an attention block. LFM2-350M has 16 blocks, 10 of them convolutions (0 0 8 0 0 8 0 0 8 0 8 0 8 0 8 0); the 2.6B has 30, 20 of them convolutions. OKLM checks the array against the tensors and wants one head count for all the attention blocks.

Both kinds start from the same norm, RMSNorm times attn_norm (the "operator norm" of the model's own code), with attention.layer_norm_rms_epsilon, and end with the same feed-forward.

  • Attention block: the llama family's, without biases: attn_q, attn_k, attn_v, grouped keys and values (32 query heads over 8 in the 2.6B), an RMSNorm of each head of Q and K (attn_q_norm, attn_k_norm, one weight vector of key_length shared by the heads) before the rotation, the NeoX rotation of whole heads at rope.freq_base (1e6 for the 350M, 1e7 for the 2.6B, 5e6 for the 8B MoE) with no scaling, scores times 1 / sqrt(head size), attn_output.
  • Convolution block: shortconv.in_proj [n_embd, 3 n_embd] gives three vectors of n_embd, in this order: a gate b, a scale c and a value x. The product u = b * x goes through a causal depthwise convolution over the tokens: y(t) = sum over j < L of conv[j] * u(t - (L - 1) + j) per channel, with L = shortconv.l_cache taps (3 in every published file), no bias and no activation; the taps before a sequence's first token read zeros (the padding is on the left only: the block never sees a later token). shortconv.conv is stored [L, n_embd], each channel's taps together; OKLM keeps an F32 copy [n_embd, L] for its convolution op (oklm_ssm_conv(), the one of the Gated DeltaNet blocks). The block's output is shortconv.out_proj (c * y), added to the residual.
  • Feed-forward after either: RMSNorm times ffn_norm, then down(silu(gate(x)) * up(x)) without biases; in lfm2moe, after leading_dense_block_count dense blocks (2), routed experts instead (sigmoid router, a selection bias, renormalized weights: DeepSeek V3's form without groups or shared experts).

The final norm is RMSNorm times the tensor the files name token_embd_norm (llama.cpp's "fix for wrong tensor name": it is the output norm), and the head is tied to the token embeddings when the file has no output.weight. The encoder variants (attention.causal false, decision.block_count) are refused.

The convolution state. A convolution block writes no key or value row. What it needs from the past is the last L - 1 columns of u of its sequence (two vectors of n_embd per block and sequence: 164 kB a sequence for the 2.6B), kept where the Gated DeltaNet blocks keep their state (oklm_context::lin_st, in device memory for a device layer). A batch reads them as the taps before its first token and leaves the last L - 1 columns of [state, batch] behind, so prompts in batches and tokens one by one give the same values, bit for bit on the CPU.

Unlike a recurrent state, this one is a function of a bounded past: the states at position p depend on the tokens of the (L - 1) x (convolution blocks) positions before p (40 for the 2.6B, 20 for the 350M) and on the key / value rows of the positions before them. So when a sequence does not continue where its states were left, OKLM rebuilds them (model_shortconv_sync()): it runs those tokens again from zero states, with the cache frozen: the attention blocks read the cached rows up to each position and write nothing (model_replay()), so a block whose input is not right yet (the first L - 1 positions of the replay for the first convolution block, L - 1 more for each one above) cannot spoil the rows the positions after it read. The context keeps each sequence's token ids for that. This is what each cache operation does with the state:

Operation Convolution states
a batch that continues the sequence carried on
oklm_seq_truncate() / oklm_kv_truncate() to position 0 zeroed by the next batch
a truncation elsewhere (a prefix reused, a draft rejected) rebuilt by the next batch: (L - 1) x blocks tokens run again
oklm_seq_copy() the destination's are rebuilt by its next batch
the paged cache (blocks shared between sequences, oklm_seq_attach_prefix(), a block copied on write) the rows are shared, the states are per sequence and rebuilt after an attach
oklm_seq_swap_in() rebuilt
oklm_kv_save() / oklm_kv_load() (the prefix cache on disk, a context resized through a file) the file stores them as it stores a recurrent state; the reader rebuilds them from the file's tokens, so a prefix of the file can be used too
rows given as embeddings (oklm_eval_embd()) carried on; they cannot be rebuilt, so a truncation into or just after them makes the next batch fail (OKLM_ERR_UNSUPPORTED) rather than run on wrong states

The cost of a rebuild is that of a prompt of (L - 1) x blocks tokens. llama.cpp keeps the state in its recurrent memory and, for a rollback of a few tokens, snapshots of it (n_rs_seq); a longer rollback evaluates the prompt again.

The Gated DeltaNet stacks (qwen35, qwen3next) are the other kind of hybrid: their state is recurrent and does not go back. Their contexts keep states instead (8 by default, shared by the sequences and found by a hash of the tokens before them) and journal what each recurrence read at a sequence's last 96 positions: a cut sequence starts again from the nearest kept state, through the journal alone when it covers the gap (a rejected draft, a cut of a few tokens), else through a forward pass over the tokens between, which reads the attention rows the cut left (hybrid-layers-design.md). With the linear blocks on a device the states are kept the same way (read back to host memory) and no journal is written: a cut is that forward pass, from the nearest kept state (on a device). The paged cache's shared prefixes work the same way; the swap stays off. An LFM2 context needs none of it: it keeps both and pays the short replay.

The cache. The convolution blocks keep no key nor value row: a context reserves rows for the attention blocks only (6 of LFM2-350M's 16 blocks, 8 of the 2.6B's 30), on the CPU, in the paged cache and on a device. Cyber-Prime 1.1 2.6B at 32768 tokens takes 512 MiB of cache where it took 1.88 GiB (the figures).

Chat form. ChatML turns (<|im_start|>role, <|im_end|>) after <|startoftext|>; the pre-tokenizer lfm2 is llama 3's (the tokenizer adds the BOS). LFM2-350M's template is one line of ChatML and knows no tools. The 2.6B tunes carry Liquid's full template: the tools as a JSON list in the system turn (List of tools: [...]), a call written by the model as a Python call list between two special tokens, <|tool_call_start|>[get_weather(city='Paris')]<|tool_call_end|>, results as tool turns, and the reply opened on <think> (the reasoning of earlier turns is dropped unless preserve_thinking is set). OKLM renders it (the generation tag, macros, raise_exception); the reasoning up to </think> goes to the reasoning field as for every template that opens the reply on <think>. The Python call list is not parsed into tool calls yet.

On OpenCL and Vulkan. A context runs on a device, every block or the last ones: the convolution blocks through the device ops of the linear blocks, the experts of lfm2moe through the llama family's. LFM2-350M, Cyber-Prime 1.1 2.6B and Tini-Cybersec-8B-A1B run there against llama.cpp and the CPU, with a paged cache, half of the blocks on the CPU and Tini's experts in host memory (the figures).

gpt-oss

gpt-oss (gpt-oss-20b and 120b; llama.cpp's graph calls it "openai-moe"; src/model/arch_gptoss.c for what the llama family's forward pass lacks). Every block is attention, then routed experts; there is no dense feed-forward. Audited block by block against llama.cpp:

  1. RMSNorm times attn_norm (attention.layer_norm_rms_epsilon).
  2. attn_q, attn_k, attn_v with biases. The attention is wider than the residual stream: 64 query heads of attention.key_length 64 (4096 values for a width of 2880) over 8 key / value heads.
  3. The NeoX rotation of whole heads (rope.freq_base 150 000) with YaRN (rope.scaling.factor 32 over original_context_length 4096, boundaries 32 / 1, the magnitude 1 + 0.1 ln 32 on queries and keys, no log multiplier), the same for every block.
  4. Attention with scores times 1 / sqrt(64). Even blocks see the last attention.sliding_window (128) positions, odd blocks every position (llama.cpp's pattern of 2 for this name; a file may give attention.sliding_window_pattern). The key / value cache keeps every position for every block: the window is applied when reading.
  5. Attention sinks: attn_sinks holds one score per query head. It joins the softmax of every query of that head as one more term, as it is (not scaled by 1 / sqrt(64)), inside or outside the window, and brings no value: the weights are exp(s_j) / (sum of exp(s) + exp(sink)), so a head can attend to nothing. The CPU attention op takes the sinks (oklm_attention_params::sinks) on its three paths (prompt tiles, the split decode path, the scalar pairs); the OpenCL and Vulkan attention kernels take them from a device tensor (below).
  6. attn_output with its bias, added to the residual.
  7. RMSNorm times the tensor stored as post_attention_norm (the norm before the experts, under the name llama.cpp's converter gave it).
  8. The router: ffn_gate_inp times the normalized input plus ffn_gate_inp.bias. The expert_used_count (4) best of the expert_count (32) scores choose the experts, and their weights are the softmax of those 4 scores (not of all 32, and no renormalization or scale after).
  9. Each chosen expert: gate and up products plus the expert's rows of ffn_gate_exps.bias and ffn_up_exps.bias, the activation min(g, 7) x sigmoid(1.702 x min(g, 7)) x (clamp(u, -7, 7) + 1) (the sigmoid form of GELU on the gate, the up value plus one, both clamped), the down product plus ffn_down_exps.bias; the four results summed with their weights, in rank order, and added to the residual.

Then RMSNorm times output_norm and an untied head (output.weight).

Against OKLM's other mixtures of experts: qwen3moe, granitemoe and olmoe take the softmax of all the scores and pick the best probabilities (the first two then renormalize the chosen ones, which gives the same weights as gpt-oss's softmax of the chosen scores, the third does not); deepseek2 and glm4moe use a sigmoid and a selection bias. None has a router bias, expert biases or this activation, so gpt-oss has its own function (model_gptoss_ffn()): the products through the same ops (oklm_matmul_id() reading the mapped experts), the biases and the activation as loops over host memory for a CPU layer, as ops for a device layer (below).

In the published Q4_K_M file the experts are MXFP4 (GGUF type 39): 32 values per block of 17 bytes, a shared power-of-two scale (one E8M0 byte: 2^(e - 127), stored halved because the value table is doubled) and 16 bytes of 4-bit codes, the low nibble of byte j for value j and the high one for value j + 16, each code one of {0, 0.5, 1, 1.5, 2, 3, 4, 6} with a sign bit (E2M1). OKLM has the type with its dot product against Q8_0 activations (core.md, blocks and dequantized values identical to ggml's); the rest of the file is Q5_0, Q8_0 and Q4_K.

Chat form (harmony). The template (17 kB: macros that write the tools as a TypeScript namespace, strftime_now, raise_exception) renders a system turn (identity, knowledge cutoff, the date, Reasoning: medium, the valid channels), an optional developer turn (the caller's system prompt as instructions, the tools), then <|start|>user<|message|>...<|end|> turns and <|start|>assistant. The model answers in channels, each a message of its own: <|channel|>analysis<|message|> reasoning <|end|>, then <|start|>assistant<|channel|>final<|message|> the answer <|return|>; a tool call is <|channel|>commentary to=functions.NAME <|constrain|>json<|message|>{...}<|call|>, its result a <|start|>functions.NAME to=assistant<|channel|>commentary<|message|> turn.

OKLM ends a reply at the first control token it samples. For a vocabulary that has all the harmony tokens, the ones inside a reply (<|start|>, <|end|>, <|message|>, <|channel|>, <|constrain|>) are therefore read as added tokens that are not special (harmony_tokens() in src/tok/tok_gguf.c): they are sampled like text and written as their own text, and the reply ends at <|return|> or <|call|>, which stay control tokens. The text of a reply is then the channels' markup as the model wrote it, and two readers take it apart:

  • the reasoning splitter (src/harness/think.c, the one of <think>) takes the analysis channel's message as reasoning and drops the final channel's header, so /v1/messages answers with a thinking block and a text block as for the other thinking models, and /v1/chat/completions with reasoning_content beside content (server.md);
  • the tool-call parser (src/model/tool_calls.c, format harmony, recognized from the template) takes a message addressed to=functions.NAME, the address before or after the channel, and its JSON arguments, and removes that message from the text. With tools, the call grammar starts at to=functions.: a tool's name, the end of the header, the arguments its parameters describe.

oklm chat shows the reply split the same way: in the full-screen UI (--tui) the analysis is folded or dimmed as other models' reasoning is; on standard output it is dimmed on a terminal and written between <think> and </think> when the output is a file or a pipe, then the answer, without the channels' markup. The conversation keeps the answer alone (the template writes the final channel's header again, and the format drops the analysis of earlier turns). /v1/completions and oklm run, which complete a prompt, give the text as the model wrote it: <|channel|>analysis<|message|>...<|end|><|start|>assistant<|channel|>final<|message|>....

On OpenCL and Vulkan. A context runs on a device, every block or the last ones (--gpu-layers), since 2026-11-14; a device layer reads nothing back. What the devices lacked, and how each part is done:

  • The sinks are an F32 tensor of one value per query head, copied to the device with its layer and given to the attention op, which reads them where it runs. On OpenCL the partial results (maximum, sum, weighted values) of an op with sinks always go through attention_merge, which adds each head's sink to its sum as one more term: the per-token kernel splits a query's keys in two ranges at least, the tiled prompt kernels write partials even for one range. On Vulkan attn_split and attn_tile add the sink where a row's output is normalized (one range, or where the ranges are combined). Both take the maximum over the scores and the sink, so a head whose sink dominates gives an output near zero, not a division by a small sum. Heads wider than 256 on Vulkan (the hybrid group's shader) have no sinks: there the op runs on the CPU, the sinks read back with its other operands (no model has both).
  • The router: the scores plus their bias (a broadcast add), the expert choice and the chosen scores (oklm_moe_select(), oklm_moe_weights() without normalization), then a softmax over the 4 values of each token: existing ops, no kernel of its own.
  • The experts' biases: each (token, rank) pair's row is gathered from the layer's table [width, 32] by the chosen expert (oklm_get_rows() on the pairs' indices) and added to the product's rows: two launches per bias instead of an indexed add, three biases a block. The gathered rows use the buffer of the shared expert's output, free in this architecture and as long as the model's width, so a model whose experts are wider inside than the model runs on the CPU (a notice; both published sizes have 2880 for both).
  • The activation is a new element-wise op, oklm_sigglu() (OKLM_OP_SIGGLU: g x sigmoid(1.702 g) x (clamp(u, -c, c) + 1) with g = min(gate, c)), on the CPU, in OpenCL's ew kernel and in Vulkan's ew shader. The CPU layers keep their own loop (the same expression), so the CPU's results are unchanged.
  • The products are the MXFP4 kernels both backends had.

The routed experts are computed where their layer runs: a context asked with n_cpu_moe, an expert cache or a device expert tier is created without them, with a notice (the first batch failed before). A file larger than the device therefore has the layer split alone.

test_gptoss_shapes (tests/core/test_opencl.c, tests/core/test_vulkan_attention.c; RX 7900 XT and Intel UHD 750) holds the operands to the CPU kernels at gpt-oss-20b's counts: attention with sinks of 64 query heads of 64 on 8 key / value heads, decode and prompt shapes, a window of 128 and none, a first token (one key and the sink), a Q8_0 cache, heads of 128, 256 and 512, on Vulkan a paged cache; the gated unit with values on both sides of its clamp; the gathered biases (the sums bit for bit). tests/core/test_ops.c holds the CPU's gated unit to a float64 expression. The fixture (test_gptoss_device) runs on OpenCL and on each Vulkan device with every block there and with the last two, F32 and F16 caches, batches, several sequences, a paged cache, and on OpenCL with the CPU layers streamed for prompts: 36 / 36 choices, within 2.9e-5 of the float64 reference (0.039 with an F16 cache); batches against single tokens bit for bit on OpenCL, within 1.6e-5 on Vulkan (two attention kernels, see the test). The fixture's experts are 16 values wide inside, which the indexed-product kernels do not take (a multiple of 32): its down product runs on the CPU for these operands on both backends, and Vulkan does not replay its decode steps; the published file's products are all on the device.

gpt-oss-20b Q4_K_M, every block on the RX 7900 XT (tests/model/test_archs_public.c: the 160 steps of the corpus on llama.cpp's tokens, in a device context and in a CPU one):

backend choices equal to llama.cpp's mean / max logit difference from llama.cpp choices equal to the CPU's mean / max from the CPU
CPU 153 / 160 0.081 / 3.96
OpenCL 158 / 160 0.081 / 4.05 155 / 160 0.067 / 2.88
Vulkan 157 / 160 0.081 / 4.05 156 / 160 0.067 / 2.69

OpenCL's row is with the experts' prompt products on the float matrix units, the default since 2026-11-14 (opencl.md); on the F32 tiles it was 0.082 / 4.03 and 0.066 / 2.50, the same choices.

Two sequences decoded together give the plain device context's choices on their first 8 steps with a paged cache (16 of 16 on both backends) and with the first 12 of the 24 blocks on the CPU (16 and 15 of 16). Vulkan replays the decode steps (172 of 179 in the run).

Tokens a second (bench_llama: a prompt of 512 tokens in one batch, 128 tokens one at a time, the best of three runs; one run on the CPU):

CPU, prompt / decode OpenCL Vulkan
gpt-oss-20b Q4_K_M (whole on the 20 GB device) 7.08 / 5.62 3323 to 3346 / 129.9 to 130.6 3656 / 163.6

Vulkan's prompt rate was 844 until the grouped expert products of a prompt went to the prompt products' tiles, and its rates 2149 / 96.3 until the shaders stopped reading MXFP4's values from a constant array (vulkan.md); OpenCL's was 1681 until the experts' prompt products went to the float matrix units (opencl.md), and its decode rate 127 to 128 until the end of a residual branch read its elements in batches (opencl.md: what keeps it below Vulkan's is 25 launches a block). The device rows are of 2026-11-14.

The models without sinks kept their rates (Llama-3.2-1B Q4_K_M, before and after: OpenCL 15 062 / 409.5 and 14 700 to 14 910 / 399 to 409 over three runs, Vulkan 16 247 / 511.2 and 16 210 to 16 520 / 507 to 516; Qwen3-8B on Vulkan 2670 / 111.9 and 2688 / 112.1).

Not done on a device: Vulkan's rates against OpenCL's (half the prompt rate, three quarters of the decode rate: its routing and its gated unit do not fit the fused MoE dispatches, 6 more launches a block for the biases; not looked into), a fused indexed add, a published file on the Intel UHD 750, the 120b (larger than the device: the layer split alone).

gpt-oss: perplexity between the backends

A perplexity of gpt-oss-20b over a thousand tokens is known to about 2 % either way, on one backend as between two (oklm ppl -c 512, the first 14 000 bytes of harness.md, 1275 tokens scored, 2026-11-14):

perplexity
llama.cpp (CPU, llama-perplexity -c 512 -b 512) 62.6978
CPU 64.3276
CPU, F32 cache 64.1297
OpenCL 61.6072
OpenCL, float kernels (OKLM_CL_Q8=0 OKLM_CL_WMMA=0) 63.0249
Vulkan 63.2014
Vulkan, F32 tiles (OKLM_VK_Q8=0 OKLM_VK_COOPMAT=0) 62.9504

The router keeps 4 experts of 32 on scores whose fourth and fifth are within 0.01 for 6 to 14 % of the tokens of a block, and the backends round a product's activations differently (the CPU to Q8_K for a K-quant, as llama.cpp and OpenCL's 8-bit kernels; the matrix units to F16): block 0's scores are 0.0004 to 0.0033 RMS apart between two backends, block 23's 0.07 to 0.10, and the experts chosen differ in 9 % of the (token, block) pairs, 8 % between the CPU's two cache types. A token's log-probability then differs by 0.08 to 0.11 on average (0.03 to 0.04 with the CPU's choices forced on a device), and over some windows the differences keep one sign. With both devices on float kernels the choices differ in 0.54 % of the pairs and the perplexities by 0.12 %: the kernels agree, the rounding is what parts the rows. The window of 128 is not involved (the differences do not step there, and the attention op handed to the CPU's kernel leaves them). Tini-Cybersec-8B-A1B has the same scatter per token and no drift (50.39 on the CPU, 50.24 on Vulkan, 50.12 on OpenCL). The measurements are in ROADMAP_DONE.md.

Spark-X2.6 and Muse Glimmer

spark2_5 (Spark-X2.6 1.7B and 4B) and muse-glimmer (Muse Glimmer 30B) are the llama block with a sliding window on three blocks out of four and a gate on the attention output: before the output projection, the attention's result is multiplied by the sigmoid of a projection of the block's normalized input (attn_gate). The forward pass is the llama family's, driven by the hyper-parameters; src/model/arch_gate.c binds the gate and applies it. Audited against llama.cpp's src/models/spark2-5.cpp and muse-glimmer.cpp.

spark2_5, block by block:

  1. RMSNorm times attn_norm (epsilon 1e-6).
  2. One fused attn_qkv, rows Q | K | V: 16 query heads of attention.key_length 256 over 4 key / value heads in the 4B (4096 query values for a width of 2560; 8 heads over 2 in the 1.7B). No biases, no Q/K norms.
  3. The NeoX rotation, with a count of dimensions per kind of block: a window block rotates rope.dimension_count_swa (256, the whole head) with the base rope.freq_base_swa (10 000); a block without window rotates the first rope.dimension_count (64: 32 pairs, value i with value i + 32) with the base rope.freq_base (5 000 000) and leaves the other 192 values of each head as they are. The blocks that see the whole context thus turn slowly, over a trained context of 1 048 576 positions.
  4. Attention with scores times 1 / sqrt(256). The blocks named by attention.sliding_window_pattern, one boolean per block (three window blocks, then a full one, repeated), see the last attention.sliding_window (512) positions. OKLM reads the list as a period (4) and refuses a list that is not one pattern repeated.
  5. The gate: attn_gate [n_embd, n_head] times the normalized input of step 1, a sigmoid, and each head's 256 output values times its gate.
  6. attn_output, added to the residual.
  7. RMSNorm times ffn_norm, then ffn_down(gelu(ffn_gate x) * ffn_up x) (the tanh form of GELU, as Gemma), added to the residual.

Then RMSNorm times output_norm and the head (output.weight; a file without one ties it to the embeddings). In the published Q4_K_M files the token embeddings and the gates are BF16 (the "one BF16 tensor a block" of the tensor list is the gate, a 2560 x 16 matrix kept at full precision), the head and attn_output Q8_0, attn_qkv and ffn_down Q6_K, ffn_gate and ffn_up Q4_K.

muse-glimmer differs from it as follows (its language side; the vision projector of the published files is another matter):

  • the token embeddings go through an RMS norm without weight before the first block (no scale by sqrt(n_embd));
  • separate attn_q, attn_k, attn_v (32 query heads of 128 over 2 key / value heads), then per-head RMS norms of queries and keys (attn_q_norm, attn_k_norm; the converter folds the model's query scale into the first);
  • only the window blocks rotate queries and keys (adjacent pairs, one base); a block without window has no position signal at all;
  • attention.sliding_window_pattern is a period (4; llama.cpp's default for this name when the key is absent);
  • the gate has one value per attention output value (attn_gate [n_embd, n_head x head_dim]);
  • norms after the attention's projection (post_attention_norm) and after the feed-forward (post_ffw_norm) as Gemma 2 and 3, with their own epsilon, 1e-8;
  • the feed-forward is SwiGLU;
  • the logits are multiplied by logit_scale (0.196 in the 30B), then soft-capped by final_logit_softcapping (20).

Muse Glimmer's replies. Its chat template (macros, strftime_now, raise_exception) writes a system turn when the conversation has none (identity, knowledge cutoff, the date, Reasoning strength: high, the valid recipients), then <|start|>user<|message|>...<|eot|> turns and <|start|>assistant. The model answers in messages that each name a recipient: to=self<|message|> its reasoning <|eom|>, then <|start|>assistant to=user<|message|> the answer <|eot|>; a tool call is a message to the tool's name. <|eom|> closes a message another follows, <|eot|> the turn. The published file types the four tokens as control tokens, and OKLM ends a reply at the first control token it samples: the first run of oklm chat on the file stopped after to=self, at <|message|>. For a vocabulary that has the four, the three written inside a reply (<|start|>, <|message|>, <|eom|>) are therefore read as added tokens that are not special (recipient_tokens() in src/tok/tok_gguf.c, as harmony's are): they are sampled like text and written as their own text, and the reply ends at <|eot|>, where llama.cpp ends it too. The reasoning splitter (src/harness/think.c) takes the message to self as reasoning and drops the header of the message to user, so oklm chat shows a thought and an answer as for the other thinking models, the conversation keeps the answer alone (the template writes its header again), and the server answers with reasoning_content or a thinking block. A message to another recipient stays in the text as the model wrote it: the template's tool calls (<atem:function_calls> blocks) have no parser and no grammar here yet.

On OpenCL and Vulkan. A context of either runs on a device, every block or the last ones (--gpu-layers), since 2026-11-14. No kernel was missing: the pass is the llama family's sequence of ops, and the device kernels take what these models add from their operands.

  • The gate: attn_gate is copied to the device with its layer's other matrices; the product, the sigmoid and the multiplication run there (for one gate per head, a multiplication broadcast over the head's 256 values).
  • The rotation: its count of dimensions and its base are parameters of each rotation op, so a block without window turns 64 of 256 values with one base and a window block the whole head with another; the values past the count are copied as they are. Spark's Q and K heads are rotated as one tensor (the rows Q | K of the fused projection).
  • The window is a parameter of each attention op (as Gemma's); heads of 256 on 4 key / value heads and of 128 on 2 are shapes both backends' attention kernels take, for prompts and for single tokens.
  • Muse Glimmer's norms without weight, its per-head norms of queries and keys, its norms with their own epsilon, and the scaled then soft-capped logits are the existing norm and element-wise ops.

test_attn_gate_shapes (tests/core/test_opencl.c, tests/core/test_vulkan_attention.c) holds these operand shapes to the CPU kernels at the published counts, on the RX 7900 XT and the Intel UHD 750. The fixtures (test_attn_gate_models) run on OpenCL and on each Vulkan device with every block there and with the last two (a block without window and a window block), F32 and F16 caches, batches, several sequences, a paged cache, and on OpenCL with the CPU layers streamed for prompts: 36 / 36 choices, within 1.6e-5 (tiny-spark) and 7.9e-6 (tiny-muse-glimmer) of the float64 reference with every block on the device (0.011 and 0.006 with an F16 cache); batches against single tokens and several sequences against one bit for bit on OpenCL, within 9.7e-6 on Vulkan (its prompts and single tokens are two attention kernels).

The published files, every block on the RX 7900 XT (tests/model/test_archs_public.c: the 160 steps of the corpus on llama.cpp's tokens, in a device context and in a CPU one):

file backend choices equal to llama.cpp's mean / max logit difference from llama.cpp choices equal to the CPU's mean / max from the CPU
Spark-X2.6-1.7B Q4_K_M OpenCL 158 / 160 (the CPU 159) 0.164 / 7.67 159 / 160 0.155 / 7.35
Vulkan 160 / 160 0.143 / 8.37 159 / 160 0.155 / 7.83
Spark-X2.6-4B Q4_K_M OpenCL 158 / 160 (the CPU 158) 0.172 / 2.89 158 / 160 0.216 / 15.2
Vulkan 158 / 160 0.174 / 3.19 158 / 160 0.221 / 15.0
Muse Glimmer 30B UD-IQ2_XXS OpenCL 156 / 160 (the CPU 157) 0.171 / 3.66 157 / 160 0.163 / 4.36
Vulkan 157 / 160 0.171 / 3.63 156 / 160 0.163 / 4.36

The devices are as far from llama.cpp as the CPU is (its means: 0.156, 0.268 and 0.194). The largest differences are single logits far below a step's best one (the 1.7B at step 109: -21.1 on OpenCL where llama.cpp has -28.7 under a best logit of -0.9). The 4B's sensitive step, where the CPU is 16.2 from llama.cpp, is the step where the devices are 15 from the CPU: they stay within 3.2 of llama.cpp there. Neither side's kernels are off at that step (step 15 of the code prompt, whose continuation is one token repeated): in the CPU's run the feed-forward parts of blocks 20 to 24 write 1.9 to 12 times what they write on a device (block 23: 73.0 against 6.0), the stream grows to a norm of 149 against 38, and block 34 takes it back (a write of 107 against 18), from a stream that enters block 19 within 0.8 % of the device's; the same excursion, smaller, at steps 12 to 14. A CPU context with an F32 cache does not make it (its stream at step 15 has the device's norms within 1 % in blocks 16 to 29), and with one rounding changed the CPU is as far from llama.cpp at another step: the mean difference of steps 14 and 15 is 1.43 and 7.12 as it is, 5.93 and 0.45 with OKLM_ATTN_F16=1, 5.63 and 4.90 with OKLM_Q8_ROUND_EVEN=1. It is a state of the model on that repeated token which a last place tips into, whoever computes (the CPU against llama.cpp: below). Two sequences decoded together give the plain device context's choices on their first 8 steps with a paged cache (16 of 16 for the three files on both backends) and with the first half of the blocks on the CPU (16, 15 and 15 of 16; 14 of 28, 18 of 36 and 26 of 52 layers on the device), the misses near ties.

Tokens a second (bench_llama: a prompt of 512 tokens in one batch, 128 tokens one at a time, the best of three runs; one run for the 30B on the CPU):

file CPU, prompt / decode OpenCL Vulkan
Spark-X2.6-1.7B Q4_K_M 482.4 / 22.6 8030 / 215.7 8179 / 285.3
Spark-X2.6-4B Q4_K_M 166.4 / 9.80 3469 / 117.9 3614 / 144.9
Muse Glimmer 30B UD-IQ2_XXS (whole on the 20 GB device) 20.8 / 2.41 875 / 29.2 831 / 36.7

Muse Glimmer's row is of 2026-11-14, after OpenCL's prompt products of its 2- and 3-bit matrices went to the float matrix units (253.3 before: opencl.md); its OpenCL decode rate is its products' (26 of a step's 31 ms of kernel time) and 902 launches a step (opencl.md).

Not done on a device: a published file on the Intel UHD 750 (the op tests and the fixtures ran there), a file larger than the device, a comparison past the window of 512 positions.

Verified: the tiny models tiny-spark and tiny-muse-glimmer (five blocks, a window of 4 positions, 8 and 16 rotated dimensions; see tools/gen_model_fixtures.py) against the float64 reference (tools/infer_ref.py), 36 / 36 greedy choices, logits within 2.6e-5 and 4.4e-6; batches and several sequences give the single tokens' logits bit for bit. llama.cpp run on the same two fixtures (written without a vocabulary, build/llama_ref) gives the reference's 36 / 36 tokens, logits within 0.021 and 0.006 (its F16 key / value cache). The published files are in the table of Against llama.cpp.

Muse Glimmer 30B, the published file (unsloth/Muse-Glimmer-30B-GGUF, Muse-Glimmer-30B-UD-IQ2_XXS.gguf, 10.7 GB, the smallest: 52 blocks at width 6656, 731 tensors, IQ2_XXS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, Q5_K and Q6_K among them; SHA-256 3e9bd0e6...0875ad9), run on the CPU on 2026-11-13 and removed. The forward pass written from llama.cpp's graph and checked on the tiny model needed no change: against llama.cpp at the pinned commit, 5 / 5 prompts tokenized alike (the llama4 pre-tokenizer, a vocabulary of 202 048), 157 / 160 greedy choices on its tokens, logits within 5.51, three of the five continuations identical and the others parting where llama.cpp's two logits are at most 0.10 apart; over 100 greedy tokens, 100 / 100 choices after a plain prompt (logits within 6.88) and 96 / 100 after a chat turn, the text parting at step 14 where llama.cpp's two logits are 0.10 apart (a file at about two bits a weight: close choices are common). oklm plan gives 16.75 GiB at 4096 tokens (the 2- and 3-bit matrices repacked for the CPU); 0.5 tokens a second on 6 threads of a machine under a load of 14 to 17, no figure for a quiet one. What the run showed was in the chat, not in the layers: the reply stopped after two tokens (see Muse Glimmer's replies above), and with that fixed oklm chat thinks, then answers, its first 69 tokens llama.cpp's greedy ones for the same rendered prompt; a second turn is rendered from the answer alone and answered. The file was downloaded again on 2026-11-14, run on OpenCL and Vulkan (the tables above) and removed. Not done: the perception encoder (mmproj), the template's tool calls, a larger file of the repository (Q4_K_XL is 15.9 GB).

MiMo-V2

mimo2 (Xiaomi's MiMo-V2 models; the published MiMo-V2.6-Flash files were looked at) is the llama block with values narrower than the keys, a sliding window and a count of key heads set block by block, and routed experts after a first dense block. The forward pass is the llama family's, driven by the hyper-parameters; src/model/arch_mimo2.c binds the tensors and holds the two steps of its own. Audited against llama.cpp's src/models/mimo2.cpp and against the metadata and tensor table of the published MiMo-V2.6-Flash-RL file.

Block by block:

xn = rms_norm(x) * attn_norm
q | k | v = attn_qkv xn        n_head heads of dk, the block's key heads of dk, as many of dv
q, k = rope(q), rope(k)        NeoX pairs on rope.dimension_count values of each head;
                               a window block with rope.freq_base_swa, the others with rope.freq_base
a = attention(q, K, V)         scores / sqrt(dk); a window block sees the last
                               attention.sliding_window positions and has sinks (attn_sinks)
x += attn_output a * attention.value_scale        a: n_head heads of dv
xn = rms_norm(x) * ffn_norm
x += ffn_down (silu(ffn_gate xn) * ffn_up xn)     a block without router (the first)
x += sum of w_e * down_e (silu(gate_e xn) * up_e xn)   a block with ffn_gate_inp
  • dk is attention.key_length and dv attention.value_length (192 and 128 in MiMo-V2.6-Flash). The attention op and the cache take one head size, so a block's values are written into heads of dk whose last dk - dv places are zero: the sums over those places are zero and the first dv places of each output head are the attention over values of dv, exactly; those are what attn_output reads. The cache keeps the padded values (192 places a head where 128 would do).
  • attention.sliding_window_pattern and attention.head_count_kv are lists, one entry a block. The published files open with a block without window, then repeat five window blocks and one without, and give 4 key heads to a block without window and 8 to a window block. They are read as leading blocks without window (oklm_hparams::n_swa_lead_full), a period, and one count of key heads per kind of block; a file laid out otherwise is refused. Each block's cache rows have its own count of heads, as gemma4's.
  • The router is as llama.cpp's graph calls it for this architecture, whatever expert_gating_func and expert_weights_norm say: sigmoid probabilities, the experts chosen on the probabilities plus exp_probs_b, their weights the probabilities divided by their sum. The count of leading dense blocks is not in the file: it is counted from the tensors; a dense block after a routed one is refused.

Not read: separate attn_q, attn_k and attn_v (llama.cpp takes them too; the published files have attn_qkv), the multi-token prediction blocks (three, in files of their own: mtp-*.gguf, refused with a message), the vision and audio projector. The model's tool calls, <tool_call><function=NAME><parameter=KEY>VALUE</parameter></function></tool_call>, are not one of the formats Tool calls parses.

On a device: none yet. The two steps of its own move activations in host memory, so a context asked on OpenCL or Vulkan says so and runs on the CPU.

Verified: the tiny model tiny-mimo2 (seven blocks, heads of 16 and values of 8, a window of 4 positions on five blocks, 1 and 2 key heads, sinks, a dense block then 8 experts; tools/gen_model_fixtures.py) against the float64 reference (tools/infer_ref.py): 36 / 36 greedy choices, logits within 1.4e-5; within 0.042 with an F16 cache; batches and several sequences on the contiguous and the paged cache give the single tokens' logits bit for bit; a sequence saved to a KV file and loaded gives the same logits. llama.cpp run on the same model (build/llama_ref) gives the reference's 36 / 36 tokens, logits within 0.062: its key / value cache is F16 for this architecture whatever is asked, and the same pass with queries, keys, values and attention weights rounded to F16 is within 1.2e-3 of it. The published MiMo-V2.6-Flash-RL-Q2_K set (126.4 GB in two parts, the smallest of that repository; ggml-org/MiMo-V2.6-Flash-RL-GGUF) was not run: the disk does not take it. Its metadata and its table of 472 tensors, read from the first megabytes of both parts, load and bind (48 blocks of 4096, 64 heads, 4 and 8 key heads, a window of 128, 256 experts of which 8 a token, one dense block), oklm plan gives 117.7 GiB of weights of which 111.6 GiB are experts (8.85 GiB resident with the experts streamed from disk), and its chat template renders as jinja2 renders it on 12 conversations.

Gemma 4

gemma4 (Gemma 4 12B, 26B-A4B, 31B, and the E2B / E4B models) is Gemma 3's block with two kinds of block that do not have the same heads, values that are normalized, a scale on each block's output, in the 26B-A4B routed experts beside the dense feed-forward, and in the E2B and E4B an embedding per layer and blocks that read earlier blocks' keys and values. Audited from the headers of unsloth/gemma-4-12b-it-GGUF, gemma-4-26B-A4B-it-GGUF and gemma-4-E2B-it-GGUF (their first 40 MB by an HTTP range, the files extended sparse to their size; the E2B and E4B files whole since) and from llama.cpp's src/models/gemma4.cpp at the pinned commit. Licence: Apache-2.0 (general.license of the three files and the repositories' cards).

12B 26B-A4B E2B
blocks, width 48, 3840 30, 2816 35, 1536
query heads 16 16 8
window blocks: head size, key / value heads 256, 8 256, 8 256, 1
other blocks: head size, key / value heads 512, 1 512, 2 512, 1
window, pattern 1024, five then one 1024, five then one 512, four then one
feed-forward 15360 2112, and 128 experts of 704, 8 a token 6144 then 12288 (a list)
attn_v in the blocks without window no no yes
embedding per layer, blocks sharing keys and values no no 256 a layer, the last 20
trained context 262 144 262 144 131 072

What the three share, block by block:

  1. RMSNorm times attn_norm (epsilon 1e-6).
  2. Queries, keys and values from attn_q, attn_k, attn_v. The blocks are named one by one by attention.sliding_window_pattern (a list of booleans). A window block has heads of attention.key_length_swa values (256); a block without window has heads of attention.key_length values (512) and its own count of key / value heads (attention.head_count_kv is a list). The count of query heads is the same, so attn_q and attn_output are twice as wide in a block without window (8192 against 4096 in the 12B). In the 12B and the 26B-A4B a block without window has no attn_v: its values are its key projection, taken before the key norm.
  3. Queries and keys through a per-head RMSNorm with a weight (attn_q_norm, attn_k_norm, of the block's head size); the values through a per-head RMSNorm without weight.
  4. Rotation in NeoX pairs: a window block rotates rope.dimension_count_swa (256) dimensions with the base rope.freq_base_swa (10 000); a block without window rotates rope.dimension_count (512) with the base rope.freq_base (1 000 000), each pair's angle divided by rope_freqs.weight (256 values, one tensor for the model).
  5. Attention with no scaling of the scores (Gemma 3 divides by the square root of the head size) and no soft-cap; a window block sees the last attention.sliding_window positions.
  6. attn_output, RMSNorm times post_attention_norm, added to the stream.
  7. RMSNorm times ffn_norm, the GELU-gated feed-forward, RMSNorm times post_ffw_norm, added to the stream.
  8. The stream multiplied by the block's layer_output_scale (one value).

Before the first block the embeddings are multiplied by the square root of the width; after the last, RMSNorm times output_norm, the head tied to token_embd, and the logits soft-capped by final_logit_softcapping (30).

The experts of the 26B-A4B. Every block has a router. The dense feed-forward stays and the experts run beside it, each side with its own norms:

  • the dense side is step 7's feed-forward, its output through post_ffw_norm_1;
  • the router does not read the normalized input: it reads the stream through an RMSNorm without weight, times 1 / sqrt(width), times a learned scale per dimension (ffn_gate_inp.scale), then ffn_gate_inp; softmax over the 128 experts, the 8 best, their probabilities divided by their sum;
  • the experts read the stream through pre_ffw_norm_2; each is a GELU-gated feed-forward of 704 whose gate and up rows are stored in one tensor, ffn_gate_up_exps [2816, 1408, 128] (gate rows first), and whose output is multiplied by a scale of its own (ffn_down_exps.scale, 128 values); the weighted sum goes through post_ffw_norm_2;
  • the two sides are added, then post_ffw_norm, then the stream.

Of OKLM's expert forms this is the softmax router with renormalized weights of Qwen3-MoE, with three things of its own: the dense feed-forward as an unconditional second branch with separate norms (not a shared expert added to the experts' sum), the router's own input, and the joined gate | up tensor with a scale per expert. There is no selection bias, no group, no shared-expert gate.

The E2B and E4B models add three things the others do not have (the E4B: 42 blocks of width 2560, 8 query heads over 2 key / value heads, a window of 512 on five blocks out of six, a feed-forward of 10 240, the same 256 values a layer):

  • an embedding per layer. per_layer_token_embd holds, for each token, 256 values for every block (262 144 rows of 35 x 256 values in the E2B, 2.3 of the file's 4.6 billion parameters: they are looked up, never multiplied). Before the first block, each token's row is multiplied by sqrt(256) and added to a projection of its scaled token embedding (per_layer_model_proj, times 1 / sqrt(width), each block's 256 values through an RMSNorm times per_layer_proj_norm), the sum times 1 / sqrt(2). Each block then ends, after its feed-forward and before its scale, with x += rms_norm(proj (gelu(inp_gate x) * e)) * post_norm, e being its 256 values;
  • blocks that compute no keys and values (attention.shared_kv_layers: the last 20 of the E2B's 35, the last 18 of the E4B's 42). They project their queries only and attend over an earlier block's cache: as llama.cpp, a window block over that of the block two before the first of them, a block without window over that of the block just before it (the last block of each kind that has a cache). The files still carry attn_k, attn_v and attn_k_norm for them, which nothing reads;
  • in the E2B a feed-forward width that is a list (feed_forward_length): 6144 in the first 15 blocks, 12 288 in the 20 that share. A block without window has its attn_v in these files.

Images. With an image, llama.cpp lets the rows of the window blocks see each other in both directions and keeps the blocks without window causal. The projectors (mmproj, 175 MB for the 12B) are not read.

The chat template (18 922 characters in the 12B and the 26B-A4B files, Unsloth's edition of Google's):

  • turns are <|turn>ROLE\n ... <turn|>\n, the assistant's role being model; the prompt opens with the BOS token;
  • a system turn is written when there is a system message, tools, or thinking: <|think|> on its first line turns the thinking on (enable_thinking, off by default), then the system text, then each tool as <|tool>declaration:NAME{...}<tool|>;
  • the model's thinking is a channel: <|channel>thought\n ... <channel|> before the answer. With thinking off the generation prompt ends with an empty one (<|turn>model\n<|channel>thought\n<channel|>), so the model answers directly; past turns' thinking is dropped unless preserve_thinking keeps it with tool calls;
  • a tool call is <|tool_call>call:NAME{key:value,...}<tool_call|> (the arguments in the model's own notation, strings between <|"|> marks), its result <|tool_response>response:NAME{...}<tool_response|> in the same model turn; the template reads the OpenAI messages (an assistant turn's tool_calls, then tool turns) and writes them so. The E2B and E4B files carry a later edition (18 808 characters): after a tool's result it writes no generation prompt, and with thinking on it opens the channel itself (<|channel>thought\n), so the reply starts inside its reasoning; an assistant turn's reasoning_content is written back as a channel before that turn's tool calls;
  • images, audio and video are placeholders (<|image|>, <|audio|>, <|video|>).

The tokenizer is the GGUF model gemma4 (BPE over raw UTF-8 with 514 906 merges, 262 144 tokens, add_bos_token true, EOS 106), which OKLM already reads; the 12B lists two tokens to suppress (tokenizer.ggml.suppress_tokens, which llama.cpp keeps out of the sampling: OKLM reads the list and its samplers never choose those tokens, Sampling; the E2B and E4B files carry none). The tokens a reply is made of (<|channel>, <channel|>, <|tool_call>, <tool_call|>, <|"|>) are user-defined tokens in these files: they are sampled and written as their own text, which the reasoning splitter and the call parser read. Two tokens that end a reply are not marked so: <|tool_response> (a user-defined token: the model writes it after its calls, where the template then puts the results) and <eos> (a plain token: the E2B ends a call with it after its thinking; the file's end-of-sequence id is <turn|>). Left as they are, a reply ran on into a result the model made up, or into <eos> until the token limit. Both are read as control tokens (src/tok/tok_gguf.c), where generation ends, as llama.cpp does.

In OKLM these forms run on the CPU and on the OpenCL and Vulkan backends (below, "On OpenCL and Vulkan"). The forward pass is the llama family's (src/model/arch_llama.c), driven by the hyper-parameters src/io/hf_config.c reads (head_dim_swa, n_head_kv_swa, n_rot_swa, value_norm, rope_freqs_full, moe_branch, and for the E2B and E4B n_embd_per_layer, n_kv_shared, n_ff_first; the list of window blocks must be one pattern repeated, as it is in every file looked at, and the list of feed-forward widths one width then a wider one). src/model/arch_gemma4.c binds the tensors and holds what that pass calls for a block of this architecture: the work views with the block's own heads, the projections with the values taken from the keys where there is no attn_v, the block scale, and the feed-forward of a block with experts. Each layer's key / value cache has its block's shape (2048 values a token in a window block of the 12B, 512 in a block without window), so the contiguous and the paged cache, the KV files and the quantized cache types take them as they are. The joined gate | up tensor is bound as the gate stack with twice the rows: one indexed product gives both halves of each (token, expert) pair, in the buffer of the gathered inputs, which holds them as long as an expert is at most half as wide as the stream (704 against 2816; a file that is not so is refused at load). An expert's output scale is applied to its router weight. The expert cache and the device expert tier do not compute this layout (the experts' own norm, the joined product, the scales): a context asked with either is created without it, with a notice, and the experts are computed from the mapped file or on their layer's device.

For the E2B and E4B, the embeddings per layer of a batch are computed once before the first block (model_gemma4_ple_inputs(): one lookup in the quantized table, one product with per_layer_model_proj) and kept beside the context's buffers for the blocks' ends (model_gemma4_ple()). A block that shares has no cache of its own: its entry in the context's list of caches is the earlier block's (model_kv_source()), it writes nothing there, and the attention reads the rows that block wrote in the same pass. So the E2B keeps 15 caches for 35 blocks and the E4B 24 for 42, on the contiguous and on the paged cache, with any cache type. What walks every layer's cache (KV files, a sequence copied or swapped out) goes over those rows once per block that reads them: the result is right, a KV file of these models is larger than it needs to be. oklm plan and the context's own report count each layer's cache at its block's width and none for a block that shares: for the E2B at 8192 tokens (F16) 144 MiB of cache, where every layer at the widest block's width gave 560 MiB, and 3.78 GiB resident in all (the batch's embeddings per layer, 36.7 MiB, are in its scratch line).

On OpenCL and Vulkan a context of these models runs on the device, every block or the last ones. The pass is the same sequence of ops; what a device needed is in the placement and in src/model/arch_gemma4.c, no kernel or shader was added:

  • the two head sizes: each block's work views and its cache have the block's own heads, and the rotation and attention kernels take the head size from their operands. OpenCL's attention takes heads of 256 and of 512 alike. Vulkan's attention group takes heads up to 256 (the window blocks); the heads of 512 go on to the shaders written for latent attention (mla_attn, vulkan.md), which read F32 and F16 caches: with another cache type (BF16, a quantized one) the blocks without window run their attention on the CPU for these operands (the CPU's result, checked on a Q8_0 cache in test_vulkan_attention; no model was run that way);
  • the projections: a device's joined Q, K, V matrix is not used for this architecture (the rows of the three differ from block to block and a block may have no attn_v): two or three products, the values copied from the keys where there is no attn_v, then normalized. The joined gate and up matrix is used, at the block's own width;
  • the block scale is one value: it stays in host memory and is read as a number;
  • the experts (26B-A4B) run on their layer's device through the indexed products: the joined product, two strided copies, the GELU, the down product; each chosen expert's scale is looked up on the device (a gather from the layer's 128 values) and multiplied into its weight. With n_cpu_moe the routing and the rows are read back, the experts and their sum computed on the CPU and the sum written to the device. Neither the expert cache nor the device expert tier computes this layout;
  • the embeddings per layer (E2B, E4B): the projection and its norm are placed with the head, the table is looked up in host memory as the token embeddings are, and the batch's embeddings live in a second set of two device buffers (17.5 MiB each for the E2B at a batch of 512); when the first blocks run on the CPU they are computed there and copied to the device with the stream;
  • the shared caches (E2B, E4B): a block and the cache it reads are in one memory. The device holds a cache for the blocks that keep one only (15 of the E2B's 35), and the split never falls between a block and its cache: a count of device layers that would is raised to the blocks that share plus the two they read (--gpu-layers 17 gives 22 for the E2B), and when that many do not fit the model stays on the CPU. Such a model is placed on one device (not spread over several GPUs) and its CPU layers are not streamed (stream_batch is not used).

Measured on the RX 7900 XT, every block on the device, an F16 cache, the backends' default products (tests/model/test_gemma4_public.c: the five prompts, 32 steps each on llama.cpp's tokens, in a device context and in a CPU one; the CPU's own line is the comparison above):

file backend choices equal to llama.cpp's logit difference, largest / mean choices equal to the CPU's largest / mean
12B Q4_K_M CPU 148 / 160 12.5 / 0.581
OpenCL 148 / 160 10.3 / 0.550 149 / 160 13.1 / 0.579
Vulkan 152 / 160 8.55 / 0.582 151 / 160 12.6 / 0.622
E2B Q4_K_M CPU 160 / 160 8.08 / 0.380
OpenCL 158 / 160 6.52 / 0.486 158 / 160 5.83 / 0.446
Vulkan 157 / 160 7.46 / 0.447 157 / 160 7.32 / 0.444
E4B Q4_K_M CPU 156 / 160 3.51 / 0.267
OpenCL 158 / 160 3.96 / 0.267 156 / 160 3.63 / 0.261
Vulkan 157 / 160 3.60 / 0.242 159 / 160 3.75 / 0.243
26B-A4B UD-Q4_K_M CPU 152 / 160 12.1 / 0.501
OpenCL 155 / 160 22.8 / 0.680 153 / 160 22.5 / 0.639
Vulkan 152 / 160 20.7 / 0.631 154 / 160 20.4 / 0.616

A device is as far from the CPU as the CPU is from llama.cpp, as on the other files of this kind. The 26B-A4B's largest differences are one step of the 160 (step 28 of "The three primary colors are"): there the whole vector of logits moves on a device, by 9.5 on average on OpenCL and 7.1 on Vulkan against the CPU, the best token the same on every side with a lower logit (11.2 and 14.7 against the CPU's 19.8 and llama.cpp's 19.1, the next token at 7 to 9), and the two devices 2.5 apart on average; against llama.cpp OpenCL's float products (OKLM_CL_Q8=0) give 157 / 160 choices and 14.2 at most, and with the experts computed on the CPU (n_cpu_moe) that step moves less (6.4 on average). Between the CPU and itself with one rounding changed (OKLM_GELU_F16=1, OKLM_Q8_ROUND_EVEN=1, OKLM_ATTN_F16=1) no step of that prompt moves by more than 3.1 on average. The softmax of that step is 0.028 (OpenCL) and 0.007 (Vulkan) of total variation from llama.cpp's.

That step is a choice of the model's routing, not of a device kernel. Traced with the records of an observed run (observe.md: each block's writes and its experts, a CPU context and a device one on the same tokens):

  • Where the two part. Up to block 3 the CPU and either device agree (the stream after block 3 within 3.5e-3 in norm, the same eight experts in blocks 0 to 3). In block 4 seven experts are the same and the eighth is not: the CPU takes expert 95, OpenCL and Vulkan take 107. The CPU's router gives the two logits of 3.370 and 3.353 (probabilities 0.02648 and 0.02605 of 128 experts): 0.017 apart, when one rounding switch of the CPU alone moves expert 107's logit there by up to 0.06 (3.354 with OKLM_GELU_F16=1, 3.292 with OKLM_Q8_ROUND_EVEN=1, 3.300 with OKLM_ATTN_F16=1; none of the three happens to put it above the eighth).
  • What follows. Block 4's write differs by 3.6 % in norm, and from there 26 of the 30 blocks choose another set of experts (the stream's norm up to 29 % apart, at block 24). On the 31 other steps of the prompt 2 to 16 blocks choose another set, the same routing noise, and the logits move by 0.24 to 2.4 on average. The 9.5 of step 28 is one offset of 9.5 common to every logit; 2.2 is left without it, which is why the softmax hardly moves.
  • The CPU does the same when it takes that expert. A CPU context whose router is made to exchange experts 95 and 107 at that one place (block 4 of that one token; a change made for the measure, not kept) is 9.6 on average from the plain CPU context at step 28 (22.2 at most, the best token unchanged at 10.2) and 0.66 on average from OpenCL (5.4 at most, an ordinary step), 2.6 from Vulkan.
  • The CPU against itself. The prompt evaluated a token at a time gives the bits of one batch. With an F32 cache step 28 is the step of the prompt that moves most from the F16 cache's (1.58 on average, the median of the steps 0.47), block 4 keeping expert 95.

So the CPU and llama.cpp, which round alike, fall on one side of a near tie of block 4's router and the two devices on the other, and this token's logits depend on that one expert by a common offset. A device that chose expert 95 there would stay within the CPU's bound. What the test bounds on a device is therefore 30 for the largest difference and 1.0 for the mean.

Other layouts of a device context give the plain device context's choices on the first 8 steps of two sequences decoded together, but where two tokens are all but tied: a paged cache (64 blocks) 16, 16, 15 and 14 to 15 of 16 for the four files; the first half of the blocks on the CPU 15 of 16 (12B, 24 of 48 layers on the device), 16 (E2B, 22 of 35), 15 (E4B, 21 of 42) and 14 to 15 (26B-A4B, 15 of 30); the 26B-A4B's experts in host memory 14 and 15.

Tokens a second (bench_llama: a prompt of 512 tokens in one batch, 128 tokens one at a time, the best of two runs):

file CPU, prompt / decode OpenCL Vulkan
12B Q4_K_M 64.8 / 3.90 1464 / 53.8 1616 / 68.5
E2B Q4_K_M 285.7 / 17.7 5029 / 120.8 6042 / 141.2
E4B Q4_K_M 154.1 / 9.07 3110 / 80.4 3598 / 115.0
26B-A4B UD-Q4_K_M (whole on the 20 GB device) 120.8 / 7.94 2220 / 71.8 2716 / 106.4

The 26B-A4B's Vulkan prompt rate was 1302 until the grouped expert products of a prompt went to the prompt products' tiles (vulkan.md); its row is that session's (OpenCL 2224 / 72.2 beside it).

The fixtures run on OpenCL and on each Vulkan device with every block there and with one layer asked (five for tiny-gemma4e), on the contiguous and the paged cache, the expert fixtures also with their experts in host memory: within 1.7e-5 of the float64 reference token by token (0.011 with an F16 cache), batches and several sequences bit for bit on OpenCL and within 8.9e-5 on Vulkan, whose prompts and single tokens go through two attention kernels (test_gemma4_models). The operand shapes of a block are held to the CPU kernels in test_gemma4_heads and test_gemma4_experts (tests/core/test_opencl.c, test_vulkan_attention.c, test_vulkan_moe.c).

Verified: the tiny models tiny-gemma4, tiny-gemma4moe and tiny-gemma4moe-split (six blocks, a window of 4 positions on two blocks out of three, heads of 8 over 2 key / value heads and of 16 over 1, one block without attn_v, 8 experts of 8 with 2 a token, joined or apart; tools/gen_model_fixtures.py) against the float64 reference (tools/infer_ref.py): 36 / 36 greedy choices each, logits within 8.0e-6 (0.011 with an F16 cache); batches and several sequences, on the contiguous and on the paged cache, give the single tokens' logits bit for bit (test_gemma4_models in tests/model/test_model.c). llama.cpp run on the same three fixtures (written without a vocabulary, build/llama_ref) gives the reference's 36 / 36 tokens, logits within 0.012 (dense) and 0.020 (experts), its key / value cache being F16.

The E2B and E4B's form has its own tiny model, tiny-gemma4e: the same six blocks with an embedding per layer of 4 values, the last three blocks reading the keys and values of the second (window) and third (no window) blocks, a feed-forward twice as wide in them, and the first of the three still carrying key and value tensors that nothing reads. Against the float64 reference: 36 / 36 greedy choices, logits within 7.3e-6 (0.0068 with an F16 cache), batches and several sequences bit for bit on the contiguous and the paged cache (the same test). llama.cpp on the same weights (in F16, without the logit cap, tools/gguf_cut_blocks.py) gives the reference's 36 / 36 tokens, logits within 0.011.

Gemma 4 E2B and E4B on the published files (unsloth/gemma-4-E2B-it-GGUF, gemma-4-E2B-it-Q4_K_M.gguf, 3.1 GB; unsloth/gemma-4-E4B-it-GGUF, gemma-4-E4B-it-Q4_K_M.gguf, 5.0 GB; CPU; test_llama_cpp_gemma4), on the five prompts of the other models, the chat turn with its control tokens among them:

tokenization greedy choices on llama.cpp's tokens largest logit difference continuations identical tokens before parting llama.cpp's margin where they part
E2B 5 / 5 160 / 160 8.08 5 / 5 160 / 160 -
E2B, OKLM_GELU_F16=1 5 / 5 158 / 160 6.38 3 / 5 100 / 160 1.0
E4B 5 / 5 156 / 160 3.51 1 / 5 79 / 160 0.341

The logit differences are of the 12B's kind (over the 262 144 logits of a step, in a model that turns the rounding of one activation function into units of logit: the second line is the same model with ggml's half-precision GELU table, which changes more than the two programs differ). The E2B's chat template renders as jinja2 renders it on the twelve conversations of tools/gen_jinja_ref.py (tools among them). oklm chat answers on both (the E4B: "The capital of Australia is Canberra."); the E4B's file was removed after its run (disk), the E2B's is the one the tests read.

Gemma 4 12B on the published file (unsloth/gemma-4-12b-it-GGUF, gemma-4-12b-it-Q4_K_M.gguf, 7.1 GB, CPU): tokenization identical to llama.cpp's on five prompts (the chat turn with its control tokens among them); on llama.cpp's tokens 148 of 160 greedy choices equal and logits within 12.5; one of the five 32-token continuations identical, 66 of 160 tokens before the others part, the widest gap in llama.cpp's own logits where they part being 1.64. Those figures are wider than the other families' and were looked into rather than accepted:

  • the model turns a rounding into units of logit. ggml computes the tanh GELU through a table of 65 536 half-precision values; OKLM computes it in single precision. OKLM_GELU_F16=1 makes OKLM round the same way (input and result to F16, src/core/cpu/ops_cpu.c): OKLM's own logits then move by up to 2.5 at a step of these prompts, and the figures against llama.cpp become 149 of 160, 15.4 and 2.09. So one relative error of 1e-3 in one activation function moves this model as much as the two programs differ, and the comparison of the whole Q4_K_M file cannot be tighter than that (in the K-quant products both programs round the activations to 8 bits, where a last-bit difference becomes a step);

  • so the blocks were compared without that amplification: tools/gguf_cut_blocks.py writes a model of two of the 12B's blocks (4 and 5: a window block, then a block without window), their matrices dequantized to F16, the first 16 384 tokens, and a logit cap that does nothing. On it (logits of 250 to 650) OKLM and llama.cpp agree within 0.25, 96 of 96 choices equal, the three continuations identical (gemma4-12b-cut in test_llama_cpp_gemma4); blocks 0 and 5: within 0.42, 96 of 96. With OKLM_GELU_F16=1 the best logits are the same to 0.02;

  • then the whole stack was compared block by block and step by step with llama.cpp's hidden states on the same tokens (testing.md: tools/llama_dump.c, OKLM_DUMP, tools/dump_compare.py), on "The capital of France is" (6 tokens) and on 72 tokens of text. The figures are the norm of the difference over the norm of llama.cpp's tensor.

    Each program on its own stream: the embeddings are equal, the first block's normalized input, queries, keys and values agree within 2.7e-7, and the first step that departs is the first block's attention (the heads' outputs, kqv_out-0: 1.5e-4), then its GELU (ffn_geglu-0). The stream is 3.8e-3 apart after block 0, 1.5e-2 after block 2, 5.6e-2 after block 6 and 0.12 after the last, the logits up to 6.4 apart.

    Each block of OKLM on llama.cpp's stream (OKLM_DUMP_FEED): no block stands out. A block adds 7e-4 to 1.8e-2 (median 4.0e-3), the blocks without window (5, 11, ... 47) as the window blocks.

    Each step on llama.cpp's own operands (OKLM_DUMP_FEED_OPS=1), over the 48 blocks:

    step smallest median largest
    queries, keys (projection, norm, rotation), normalized values 1.7e-7 3.3e-7 2.5e-6
    attention (heads' outputs) 1.3e-4 2.5e-4 3.7e-4
    the same with OKLM_ATTN_F16=1 2.0e-8 2.8e-8 3.9e-5
    output projection and its norm 9.2e-8 5.5e-7 5.9e-6
    feed-forward input, gate and up products 1.1e-7 2.8e-7 2.1e-6
    GELU-gated product 2.0e-5 1.8e-4 4.6e-4
    the same with OKLM_GELU_F16=1 4.9e-7 2.0e-6 2.3e-5
    down product, its norm, the block's output 8.4e-8 6.4e-7 2.8e-6

    and the output norm and the head within 8.8e-7 and 4.4e-7. On the 72 tokens the same: every step within 8.5e-5 with the two variables (rotation at 72 positions included), attention 7.6e-6 at the median. So two steps a block are computed differently, and they are llama.cpp's roundings, not a different formula:

    • attention over the F16 cache. llama.cpp (its graph of separate products, which the reference runs use) multiplies the F16 keys by the queries, and the F16 values by the probabilities, with ggml's matrix product, which converts its other operand to the type of the first: each query is rounded to half precision before the scores, each probability after the softmax (one below 3e-8 becomes 0, one below 6e-5 keeps 5.96e-8 of absolute precision). OKLM keeps the queries and the probabilities in single precision and reads only the cache in half precision. OKLM_ATTN_F16=1 rounds as ggml does (the plain kernel, several times slower): the step then agrees within 2.8e-8 at the median, the largest (3.9e-5) being a probability on the edge of two half-precision values;
    • the GELU table, as above.

    Everything else agrees to rounding noise on equal operands, and that noise does not stay small on each program's own stream: with both variables set, block 0 ends 3.5e-7 apart (3.8e-3 without) and the stream is again 1.6e-3 apart after block 1, 8.6e-3 after block 2 and 0.12 after the last. A K-quant product rounds its operand to 8 bits a block of 256 values and attention has no score scale, so a difference in the last place of a single-precision value moves a rounding, and each block multiplies what it receives by 5 to 20 (block 1: the heads' outputs 3e-5 apart, the output projection of them 4.1e-4, the gate and up products 1e-3). The variables are there to show what differs, not to make the logits equal: with both, the figures against llama.cpp are 151 of 160 choices, logits within 9.2 and a gap of 0.82 on the 12B, and on the E2B 155 of 160, 8.23 and 2.49, from 160 of 160 without them.

    Each program against itself gives the floor (tools/ref_compare.py), on the reference's 160 steps:

    the 12B's file, on the same tokens choices largest logit difference widest gap where they differ
    OKLM against llama.cpp (the reference: separate products, F16 cache) 148 / 160 12.5 2.2
    llama.cpp with flash attention against the reference 153 / 160 11.7 1.76
    llama.cpp with an F32 cache against the reference 150 / 160 12.3 1.76
    llama.cpp, the same run twice 160 / 160 0 -
    OKLM's scalar kernels against its AVX kernels 157 / 160 14.3 0.87

    Flash attention is llama.cpp's other attention kernel, the one its programs take when the choice is left to them; the F32 cache (LLAMA_KV_F32=1 in tools/llama_ref.c) takes the half-precision rounding out of its attention. The last line has no llama.cpp in it: OKLM's scalar and AVX kernels compute the same formulas, and of the first block only the sums of attention come in another order (the heads' outputs 5.5e-7 apart, every projection bit for bit equal); after block 1 the two streams are 1.3e-3 apart, after block 3 1.6e-2, and the logits of that step 4.4 apart. A difference in the last place of one sum is enough for logits 14 apart somewhere in 160 steps; llama.cpp's own roundings, hundreds of times larger, move a few more choices. The distance between OKLM and llama.cpp on this file is the distance between two runs of either program: rounding, with no step computed by another formula. Nothing was changed in OKLM's forward pass, and the test keeps the family's bounds.

oklm chat renders the file's template and answers: thinking off by default (the generation prompt closes an empty thinking channel), the answer ended by <turn|>; with --template-vars '{"enable_thinking": true}' the reply opens with <|channel>thought, reasons, closes with <channel|> and answers (17 x 23: three ways to 391, then the answer). (That markup was shown as the model wrote it at the time; it is told apart now, below.) oklm info gives the verdict and oklm plan the memory: 6.62 GiB of weights, 10.96 GiB resident with 8192 tokens of context (the cache was counted, and its arena sized, at the window blocks' width for every layer then; each layer is counted at its own width since, which by the 12B's shapes is 0.37 GiB less at that context). About 2 tokens/s on this machine's CPU while other work ran; 2.8 to 3.2 tokens/s (a prompt at 21 to 23) at six and eight threads on a calmer run.

Gemma 4 26B-A4B on the published file (unsloth/gemma-4-26B-A4B-it-GGUF, gemma-4-26B-A4B-it-UD-Q4_K_M.gguf, 16.9 GB, CPU; SHA-256 f2c28b3d...f63f293f). The file's matrices are Q8_0 (embeddings, projections, the dense feed-forward), its experts Q4_K (ffn_gate_up_exps) and Q5_1 (ffn_down_exps, rows of 704 values). Its window blocks have an attn_v; its blocks without window take their values from the key projection, as the 12B's. It runs as the tiny models did, with no change to the forward pass:

tokenization greedy choices on llama.cpp's tokens largest logit difference continuations identical tokens before parting llama.cpp's widest margin where the choices differ
26B-A4B 5 / 5 152 / 160 12.1 0 / 5 72 / 160 5.2
26B-A4B, OKLM_ATTN_F16=1 OKLM_GELU_F16=1 OKLM_Q8_ROUND_EVEN=1 5 / 5 153 / 160 13.0 1 / 5 58 / 160 2.49
llama.cpp with flash attention, on the reference's tokens 155 / 160 10.1 2.77
llama.cpp with an F32 cache, on the reference's tokens 150 / 160 15.0 3.56

Its hidden states were compared with llama.cpp's as the 12B's were (testing.md), on 6 tokens and on the 29 tokens that end where the two programs differ most. On llama.cpp's operands, over the 30 blocks: the router's logits and probabilities within 4.2e-6 (1.7e-7 at the median), the joined gate | up product of the eight experts within 3.4e-7 (the same experts in the same order for every token), the experts' weighted sum after its norm within 4.1e-7, the dense side, the projections and the head as on the 12B. The steps that differ are the 12B's two, attention (2.7e-4 at the median; 3.1e-8 with OKLM_ATTN_F16=1) and the GELU of both feed-forwards (1.8e-4; 1.2e-6 with OKLM_GELU_F16=1), and one more that Q8_0 matrices bring: the first block's queries are 4.6e-4 from llama.cpp's on the same input (its keys 3.5e-4, its values 8.4e-4), where every other block's are 2.3e-7. A Q8_0 product quantizes its input to 8 bits, and where a value falls exactly on a half OKLM rounds it away from zero (ggml's portable code) and llama.cpp on x86 to the even step (Numerics, below); the input of the first block, made of Q8_0 embeddings, has such values (35 of 16 896 on six tokens), the others do not. OKLM_Q8_ROUND_EVEN=1 rounds as x86 does (7.6e-5 left, a value one place from a half in one of the two programs).

The widest gap is a choice both programs are sure of. After

def fibonacci(n):
    if n <= 0:
        return 0
    elif n == 1:

llama.cpp, with either attention kernel, continues with twelve spaces (logit 17.9 against 12.7 for eight) and OKLM with eight (21.8 against 9.2), the indentation the code calls for. With OKLM_GELU_F16=1 alone OKLM's choice does not move (22.3 against 6.7), with OKLM_ATTN_F16=1 alone the margin narrows (17.6 against 12.0), and with both OKLM writes llama.cpp's twelve spaces: on this step the half-precision roundings decide the answer, and the two programs differ by them and by nothing else. The test's bound on that margin is 6.5 for this file (20 and 0.9 for the two others, as the family).

oklm plan counts 15.77 GiB of weights and 26.90 GiB resident at 8192 tokens of context (the routed experts repacked for the CPU: 8.67 GiB on top of their mapping); oklm chat answers "The capital of Australia is Canberra." and, with enable_thinking, reasons in its channel then answers "17 times 23 is 391.". 7.1 to 7.2 tokens/s at six and eight threads on this machine's CPU while other work ran (a prompt at 26 to 27). In the catalogue as gemma4-26b-a4b, marked run; the file was removed after its run (disk).

Tool calls and the thinking channel (the gemma4 tool-call format, Grammars). A call <|tool_call>call:get_weather{city:<|"|>Paris<|"|>,days:3}<tool_call|> comes back as get_weather with {"city":"Paris","days":3}; under the grammar a call names an offered tool and gives its keys sorted by name, as the template writes an earlier call's, the required ones always. The thinking channel is reasoning for the one splitter of src/harness/think.c: oklm chat shows it dimmed (between <think> and </think> when the output is no terminal) and the agent as a REASONING event; /v1/chat/completions gives it as reasoning_content and the rest as content, /v1/messages as a thinking block. A prompt that ends on the channel's opening tag (after a tool's result with thinking on) makes the reply start as reasoning (oklm_think_opened()), and the turn is kept with that tag so the template tells the reasoning from the answer when it renders the turn again. The chat template is handed an assistant turn's leading reasoning (the channel, or a <think> block) as reasoning_content when it reads that member and writes no </think> itself, as this one does.

Run on the published E2B file through oklm serve (CPU, temperature 0, two tools offered, "What is the weather in Paris, in celsius?"): tool_calls get_weather {"city":"Paris","unit":"celsius"} with finish_reason tool_calls (172 prompt tokens, 21 generated), whole and streamed; the result sent back as a tool message gives "The weather in Paris is 21 degrees Celsius and it is sunny." (219 prompt tokens, 192 of them cached); tool_choice required on "Hello!" gives a call (get_time {"zone":"UTC"}); a request that needs no tool gets a plain answer. With chat_template_kwargs {"enable_thinking": true}: "What is 17 times 23?" gives 542 tokens split into a reasoning_content and the content "17 times 23 is 391.", the same streamed; with the tools, 877 characters of reasoning then the same call, and after the result 681 characters of reasoning (the channel the prompt opened) then "The weather in Paris is 21 degrees Celsius and sunny.", whole and streamed. oklm chat --tools does the same three turns (/tool RESULT), the reasoning apart each time.

The same on the published 12B file (oklm serve, CPU, temperature 0, the same two tools; its template is the earlier edition of the same format): get_weather {"city":"Paris","unit":"celsius"} with finish_reason tool_calls (137 prompt tokens, 21 generated), whole and streamed; the result sent back gives "The current weather in Paris is 21°C and sunny." (187 prompt tokens, 128 of them cached); tool_choice required on "Hello!" gives get_time {"zone":"UTC"}; a question that needs no tool gets "The capital of Australia is Canberra.". With enable_thinking: "What is 17 times 23?" gives 278 tokens, 579 characters of reasoning_content and the content "17 times 23 is 391.", the same streamed; with the tools, 417 characters of reasoning then the same call, and after the result 146 characters of reasoning then the same sentence.

The 12B's suppressed tokens (tokenizer.ggml.suppress_tokens: 258883 <audio|> and 258882 <image|>, the marks that close a recording and an image) on the published file: after <|turn>user and "Describe this picture: <|image>" the best logit is <image|> (24.8; the next, <|image>, at 18.6) and the sampler takes the next (test_gemma4_suppress in tests/model/test_infer_public.c, skipped without the file).

Not done: the 26B-A4B's tool calls through oklm serve (its template is the 12B's), its expert cache and its smaller quantizations; for the E2B and E4B, the rows of a shared cache written once to a KV file, and their audio and image projectors; a comparison past the window of 1024 positions on the published file (the tiny models cross theirs); the devices; images (mmproj); a cap on the thinking channel's length (the reasoning budget closes <think> blocks only); the 31B; the HF conversion; the trainer.

Numerics

Products with quantized matrices quantize the activation row first, as llama.cpp does: Q8_0 for Q4_0/Q5_0/Q8_0 weights, Q8_1 for Q4_1/Q5_1, Q8_K for K-quants; then the dot product is integer arithmetic times the scales. F16/BF16 matrices are multiplied in F32. Every kernel is deterministic and independent of the thread count, and batched evaluation gives bit-identical logits to token-by-token evaluation (checked by tests/model/test_model.c).

The Q8_0 and Q8_1 rows are quantized as ggml's portable code writes them (the scale d = max / 127 in half precision, each value times 1 / d rounded to the nearest step, a value exactly halfway going away from zero), on every machine and at every SIMD level. ggml's x86 code is not that code: it multiplies by 127 / max and rounds a half to the even step, and its ARM code rounds to even with 1 / d. The results differ where a value falls exactly on a half, which ordinary activations do not and the first block's input does when the embeddings are Q8_0 themselves: on Gemma 4 26B-A4B (Q8_0 embeddings and projections) 35 of the 16 896 values of a 6-token prompt, and the first block's queries are 4.6e-4 from llama.cpp's on x86 for it. OKLM_Q8_ROUND_EVEN=1 rounds as the x86 code does, for comparisons. The default was left as it is on 2026-11-13: following the x86 code would move the bits of every model with Q8_0 or Q8_1 activation products, the pinned references and the devices' 8-bit kernels, for a difference seen on one block of such files; to be looked at again if a file shows more than that. The Q8_K rows agree with llama.cpp's to the last place (2e-7 on the queries of Gemma 4 12B).

Sampling

oklm_sampler_sample (include/oklm/model/sampling.h) applies, in order:

  1. repetition (CTRL style: positive logits divided, negative multiplied), frequency and presence penalties over the last penalty_last_n accepted tokens, each distinct token once;
  2. temperature 0: the arg-max (greedy, lowest id on ties);
  3. top-k (bounded heap, then sorted), top-p over the temperature-scaled distribution, min-p relative to the best token;
  4. a draw from what remains with xoshiro256** seeded by splitmix64, so a seed reproduces a run on every platform.

Defaults: temperature 0.8, top-k 40, top-p 0.95, min-p 0.05, no penalty, seed 1234.

A model's file may list tokens that are never to be sampled (tokenizer.ggml.suppress_tokens, an array of ids: Gemma 4 12B has one). The tokenizer keeps the list (oklm_tokenizer_suppress(), ids outside the vocabulary dropped, written back by oklm_tokenizer_to_gguf()) and a sampler takes it with oklm_sampler_set_tokenizer(), as oklm run, chat, serve and the agent do: those tokens then count as minus infinity before step 1, whatever the options and the grammar, as llama.cpp's logit bias does. A sampler with such a list no longer reads the best logit where the logits are (oklm_sampler_greedy() is 0) and asks for that many more of the best ones (oklm_sampler_top_needed()).

Grammars

include/oklm/model/grammar.h constrains generation with a GBNF grammar, the syntax of llama.cpp's --grammar files (ADR-0009):

root   ::= "- " item ("\n- " item){2}   # three dashed lines
item   ::= [a-z]+ (" " [a-z]+)?

Rules are name ::= alternatives; items are "literals", classes [a-z_] / [^"\\], ., rule names and ( groups ), each optionally followed by *, +, ?, {m}, {m,} or {m,n}. Characters are code points; a token that ends inside a UTF-8 character is allowed when some completion of it matches. Control and end-of-generation tokens are allowed only where the grammar can end.

oklm_grammar* g;
oklm_grammar_new(gbnf_text, "root", tok, &g, err, sizeof(err));
oklm_sampler_set_grammar(sampler, g);          /* the sampler only returns allowed tokens */
int32_t id = oklm_sampler_sample(sampler, logits);
oklm_grammar_accept(g, id);                    /* the caller advances the grammar */

The sampler checks the token it chose first and masks the vocabulary only when the grammar refuses it. oklm_json_schema_to_grammar() turns a JSON schema into a grammar (types, required and optional properties, arrays with item counts, enum, const, anyOf / oneOf, local $ref, string lengths, integer and number ranges (exact decimal bounds), date / time / uuid formats, regular expression patterns); oklm_grammar_json() is a grammar of any JSON value.

Speculative decoding

include/oklm/model/speculative.h: a draft model sharing the target's vocabulary proposes up to n tokens greedily; the target evaluates them, after the previous round's last token, in one batch, and its sampler draws position by position while it agrees with the proposals. Each emitted token is drawn from the target's logits after the same prefix with one random draw, so the output is the one plain generation gives with the same seed (greedy and sampled, grammars included); the draft changes only the speed. Caches are cut back to the accepted tokens with oklm_kv_truncate.

oklm_speculative* sp;
oklm_speculative_new(target_ctx, draft_ctx, 5, &sp);    /* both evaluated the prompt */
oklm_speculative_step(sp, sampler, grammar_or_NULL, out, &n_out);

The gain depends on the size ratio and the backend: Qwen2.5-1.5B with the 0.5B as draft, 200 greedy tokens, decodes 26.0 → 31.9 t/s on the CPU with 3 proposals (89 % kept) and does not gain on the OpenCL device, where the target is already fast compared with three draft steps.

Multi-token prediction draft

DeepSeek V3 files may carry multi-token prediction blocks (llama.cpp's NextN layers, <arch>.nextn_predict_layers) after the main stack; they count in block_count but are not part of the main stack, which runs without them. oklm_model_n_mtp() tells whether a file has their weights (files converted with --no-mtp, and DeepSeek V3 conversions older than llama.cpp's V3.2 support, have none).

The blocks can serve as the draft. A context created with oklm_context_params.mtp holds only their caches (one per block). With a device backend it runs where the model's head is: the device copy of the weights carries the blocks with the head (create the main context first; otherwise the head and blocks are placed alone and the main stack can no longer go to a device), its token table staying on the host; on the CPU otherwise. The blocks may also come from a file of their own, llama.cpp's --mtp export (the header of the full model, the blocks, the embeddings, final norm and head, none of the main stack): oklm_model_mtp_only() tells such a file, which only a multi-token prediction context can use, as the draft of the full model (oklm run --mtp-model FILE). oklm_eval_mtp() pairs the token at position p with the main stack's normalized hidden state of the token at p - 1, which oklm_get_hidden() returns for a context created with oklm_context_params.hidden. The block normalizes the embedding and the hidden state, projects them together (nextn.eh_proj), runs one MLA + MoE block, and applies its own final norm (nextn.shared_head_norm) and the output head. Its logits predict the token after the one it was given.

cp.hidden = 1;  oklm_context_new(m, &cp, &target);   /* the main stack */
cp.hidden = 0;  cp.mtp = 1;  oklm_context_new(m, &cp, &mtp);
oklm_eval(target, prompt, n_prompt, 0);               /* one call */
oklm_speculative_new_mtp(target, mtp, prompt, n_prompt, 3, &sp);
oklm_speculative_step(sp, sampler, grammar_or_NULL, out, &n_out);

In a round, the block's first proposal uses the target's hidden state of the last verified token. Further proposals chain on the block's own normalized output, as llama.cpp does. With several blocks (chained heads, oklm_set_mtp_head() choosing which one oklm_eval_mtp() runs), proposal s comes from block min(s, n - 1): a block taking over first re-evaluates the round's tokens so far into its own cells, each paired with the output of the evaluation that proposed it. After verification, the cells of every block are recomputed from the target's real hidden states of the tokens kept (the first block past its first cell, which already paired the pending token with the real hidden state; in a round without a pending token, the first one, several blocks hand the first proposal straight to the second block, so the first block's first cell is recomputed too; test_mtp_files checks that round on the chained fixture: every block's logits for one more token, from the draft the round left, are those of a draft that evaluated the kept tokens in one call). oklm_speculative_last_proposals() returns a round's proposals. The output is that of plain generation, as with a draft model. oklm run --mtp uses it. The tiny V3-like fixture checks the block's logits against the float64 reference (within 5e-4), and checks that generation, greedy and sampled, is unchanged with the target and the block on the CPU and on OpenCL; its --mtp export gives the block's logits bit for bit and the same proposals; with two chained blocks generation is unchanged, and a second block copying the first proposes, round by round, what the first proposes alone. No real V3 file fits this machine, so the speed gain has not been measured.

Prompt lookup, grammar-forced drafts and the guard

oklm run -m model.gguf -p TEXT [--spec-ngram K] [--spec-grammar] [--spec-guard RATE] ...
oklm serve ... [--spec-ngram K] [--spec-guard RATE]

Three more sources and one brake for the same draft-and-verify rounds; the verification is unchanged, so the output stays the one plain generation gives with the same seed (greedy and sampled).

  • Prompt lookup without a draft model (--spec-ngram K, oklm_speculative_new_lookup()): the last 3, 2 or 1 tokens found earlier in the prompt and the output propose the K tokens that followed them there, as the server's --spec-ngram does for greedy requests.
  • Grammar-forced drafts (--spec-grammar, oklm_speculative_set_grammar_drafts(), or oklm_speculative_new_grammar() when the grammar is the only source): when every way forward in the grammar allows exactly one character (and it cannot end there), that text is forced: a JSON schema's keys after their opening quote, the rest of an enum value or true / false once its first letter is chosen, a tool call's fixed text, closing brackets (oklm_grammar_forced_text()). Its tokens (oklm_grammar_forced_tokens(): the tokenizer's own tokens of the text, then the longest token at each point) are the round's proposals instead of the draft's, without sampling; a draft model or block then only evaluates the pending token and catches up with the accepted ones after the round. The model may write the text with other tokens (the text's last character often merges with the next free one, " into ":), which the verification turns down like any proposal. The grammars of --json-schema allow one optional space between tokens, so their forced runs stop at every space; a tool call's template text is forced whole.
  • The guard (--spec-guard RATE, oklm_speculative_set_guard(), the server's oklm_server_params::spec_guard): with an expert cache, a round makes no proposal (forced ones aside) while the cache's recent hit rate (oklm_expert_cache_recent_hit_rate(): the experts batches needed that were in memory over about the last 8 forward passes) is below RATE. Verifying k proposals runs k + 1 tokens through every MoE layer: their union of experts is read, and the rejected proposals' experts were read for nothing. The guard decides round by round, so it resumes once the rate climbs back. The server counts the steps it rested (spec_rested in /api/stats).

Grammar-forced drafts, Qwen2.5-0.5B-Instruct Q4_K_M on the CPU (4 threads, best of 3, the machine shared), a chat-formatted prompt; the output identical with and without, greedy and sampled (seed 11):

constrained by sampling plain forced drafts forced / kept
a tool call's grammar (hermes <tool_call> text, 4 arguments), 48 tokens greedy 41.2 t/s 141.5 t/s (3.4x) 42 / 39
same temperature 0.7 46.4 t/s 140.6 t/s (3.0x) 42 / 39
a JSON schema (5 keys, an enum, an array of objects), 160 tokens greedy 54.1 t/s 46.2 t/s 47 / 20
same, 83 tokens temperature 0.7 38.0 t/s 42.1 t/s 37 / 18

A tool call's fixed text is forced whole and nearly every forced token is kept. Under a JSON schema the forced runs are short (a key's letters after its quote, between optional spaces) and the small model writes the key's closing quote merged with what follows, so fewer than half are kept and the gain stays within the noise.

The guard, Granite-3.1-1B-A400M Q4_K_M with direct reads and 2 loaders, a copying prompt (117 tokens) then 96 greedy tokens, prompt lookup with 4 proposals (26 % kept by this model), best of 3:

cache plain prompt lookup lookup, guard 0.8 lookup, guard 0.95
10 % (cold: 10 % hits) 3.9 t/s, 15.3 GiB read 3.5 t/s, 18.3 GiB read 4.1 t/s (96 of 96 rounds rested) 4.3 t/s (96 of 96 rested)
100 % (warm: 96 % hits) 69.3 t/s 61.6 t/s 72.0 t/s (10 of 64 rested) 72.1 t/s (20 of 69 rested)

With a cold cache the verification batches read 20 % more and decoding slows by 10 %; the guard rests every round and gives the plain speed back. With a warm cache the guard rests only the first rounds, while the cache fills.

Tool calls

include/oklm/model/tool_calls.h reads the calls a model writes in its own syntax: Hermes <tool_call>{...}</tool_call> (Qwen 2.5 / 3, Hermes, SmolLM3), Llama 3 {"name", "parameters"} (after an optional <|python_tag|>), Mistral [TOOL_CALLS][...] and [TOOL_CALLS]f[ARGS]{...}, GLM-4.5 / 4.6 / 4.7 (glm45) <tool_call>f then <arg_key>k</arg_key><arg_value>v</arg_value> pairs and </tool_call> (a newline between the parts for GLM-4.5 / 4.6, none for GLM-4.7; a value is JSON when it parses as a number, a boolean, null, an object or an array, else a raw string; given the tools offered, oklm_tool_calls_parse_tools() keeps a string property's value a string whatever it reads as, <arg_value>42</arg_value> giving "42", as oklm chat, the agent and oklm serve do), and GLM-4-0414 (glm4): a turn that is a call starts with the function's name, then a newline and its arguments as a JSON object (<|assistant|>f\n{...}), a turn of text with the newline; the call ends the turn on <|observation|>, and the first line is a call only when it is the name of a tool offered; given the tools, a turn of text may hold calls too, a line naming a tool and its JSON object below it (in a code fence or not, the fence going with the call). The format is recognized from the chat template (<arg_key> beside <tool_call> for GLM-4.5; <|observation|> and an assistant turn's metadata for GLM-4-0414; <|channel|> and an address to= for gpt-oss's harmony, whose call is a message <|channel|>commentary to=functions.f <|constrain|>json<|message|>{...}; <|tool_call_start|> for Liquid's LFM2, lfm2; <|tool_call>call: for Gemma 4, gemma4) or set with --tool-format.

Gemma 4 writes each call as call: and the function's name, then its arguments between braces, between two tokens: <|tool_call>call:get_weather{city:<|"|>Paris<|"|>,days:3}<tool_call|>. The keys are bare words; a string stands between two <|"|> marks and is taken as it is (it has no escapes: quotes, braces, backslashes and newlines are its text); numbers, true, false, null, lists and nested dicts are written as JSON writes them. JSON quotes and a bare word are read as strings too, for a model that leaves the notation. The reply ends on <|tool_response>, after which the template writes the result in the same model turn; a result the model writes itself is dropped. A reply that starts with call: and the name of an offered tool is read without its opening token.

LFM2 and the models tuned from it (LFM2-350M, Cyber-Prime) write their calls as a Python list between two special tokens, <|tool_call_start|>[get_weather(city="Paris", days=3), now()]<|tool_call_end|>. The arguments' Python literals come back as JSON: a string with either quote and its escapes (\n, \', \xHH, \uHHHH...), a number (.5, 1., 1_000 and 0x1F too), True / False / None, a list or a tuple as an array, a dict as an object; an argument written without its keyword takes the name of the tool's property at that position, and a reply that is nothing but such a list of calls to offered tools is read as calls even without the two tokens. <|tool_call_start|> is a control token that would end the reply: with tools it opens the call instead, <|tool_call_end|> ends the turn. Some LFM2 GGUF files carry the plain ChatML template, which says nothing of tools: the format is then recognized from the vocabulary's two tokens, and oklm serve, oklm chat --tools and oklm agent render the conversation with a template of their own that writes what Liquid's does (the tools in the system turn between <|tool_list_start|> and <|tool_list_end|>, a result in a tool turn between <|tool_response_start|> and <|tool_response_end|>, an earlier call as the list; a template given with --chat-template is rendered instead, as it is). Calls come back as name + compact JSON arguments, and oklm_tool_calls_to_json() writes them as the OpenAI style tool_calls array chat templates expect in the assistant message (oklm_chat_msg.tool_calls); tool results are tool messages with tool_call_id and name. GLM-4-0414's template knows neither: oklm_chat_template_render() hands it the conversation in its own form, the assistant's text then one turn per call (the function's name in the turn's metadata, the arguments as JSON text in its content) and each result as an observation turn.

oklm_tool_call_grammar() builds the grammar of a call of the declared tools and returns the marker that opens a call (<tool_call>, [TOOL_CALLS], <|tool_call_start|>, <|tool_call>); generation is free until the model writes the marker, then constrained until the call is closed. For gemma4 the grammar is call:name{key:value,...}<tool_call|>: a tool's keys sorted by name (the order the template writes an earlier call in), the required ones always, the others or not; a string property a text between the marks that holds anything but the mark, an integer, number or boolean property that literal, an enum one of its values, an array with typed items a list of them. For lfm2 the grammar is the list: each call a tool's name and key=value arguments (the required keys in their order, then optional ones), a string property a quoted string, an integer, number or boolean property that Python literal, an enum one of its values. For glm45 the grammar takes the tools' names, each tool's property names as keys (the required ones first, in the schema's order), a string property's value as raw text up to </arg_value> and another's as the JSON its schema describes. glm4 has no marker, as Llama 3 when it writes no <|python_tag|>: its grammar covers the whole answer (a tool's name, a newline, the JSON its parameters describe), used from the start with --tool-choice required; otherwise the answer runs free and is parsed at its end (the server holds back the text of such an answer until then).

Chat templates

Models ship a Jinja template (tokenizer.chat_template). OKLM renders it with its own Jinja interpreter (include/oklm/model/jinja.h, ADR-0006) exactly as Hugging Face's apply_chat_template() does: the variables are messages, add_generation_prompt, bos_token, eos_token, and tools / documents set to none. When the tokenizer adds BOS itself, a BOS text at the start of the rendering is removed (as llama.cpp does), so it is not doubled. A template that raises an error (for example roles that must alternate) stops the turn with its message.

oklm_chat_template* t;
oklm_chat_template_new(oklm_tokenizer_chat_template(tok), "<s>", "</s>", &t);
oklm_chat_template_render(t, msgs, n, 1 /* generation prompt */, oklm_tokenizer_add_bos(tok),
                          "{\"enable_thinking\": false}" /* extra variables, or NULL */, &text, err, sizeof(err));

Extra variables are a JSON object whose members are added to (or replace) the defaults: enable_thinking for Qwen3, tools for templates with tool calls, date_string for Llama 3... oklm chat --template-vars JSON passes them.

Models without a template, or with one the interpreter rejects, use built-in renderings of the common formats recognized from the template's markers: ChatML (<|im_start|>), Llama 2 ([INST] <<SYS>>), Llama 3 (<|start_header_id|>), Mistral ([INST]), Gemma (<start_of_turn>) and Phi-3 (<|user|> ... <|end|>); ChatML when nothing matches.

A file without a template

GGUF files of 2023, and some later conversions, carry no tokenizer.chat_template, and nothing else in them names the format: LLaVA 1.5's file calls itself "LLaMA v2" and was trained on Vicuna's turns. Written as ChatML, the conversation reaches such a model between markers it was never trained on. --chat-template FILE|NAME, in oklm chat, oklm run (where it renders a chat turn: --rag), oklm serve, oklm agent and oklm describe, puts a template in place of the file's:

oklm chat -m model.gguf --chat-template vicuna         # a built-in format
oklm serve -m model.gguf --chat-template turns.jinja   # a Jinja file

A name of the list below is that format (write ./NAME for a file of that name); anything else is a Jinja file, which has to compile. A name that is neither ends the command with the list. The option also replaces a template the file does carry, when it is a wrong one.

The template asked for is the one rendered, wherever the command renders a chat turn:

  • oklm run renders one with --rag (and --rag-critique): the question and its sources go through the option's template. Without --rag the prompt is the text given, and the option changes nothing.
  • An LFM2 file whose template says nothing of tools is otherwise rendered, with tools, by a template of OKLM's own that writes them (tool calls). Not when a template was asked for, or named by the catalogue: that one is rendered as it is, with the tools if it writes them and without them if it does not (oklm_tokenizer_chat_template_given(), oklm_tool_format_model_template(), in oklm chat --tools, oklm serve and oklm agent alike).
  • The agent's small model (smallModel, which answers the asides) is another file than the one the option was written for: it keeps the template its file carries. When its file carries none, it gets the catalogue's, else the option's. When it is the main model's file, it gets what that one got.
Name The turns Defined by
vicuna the system text and an empty line; USER: ... and ASSISTANT: ... </s> lines; ASSISTANT: to answer llama.cpp's vicuna (Vicuna 1.1; LLaVA 1.5)
chatml <|im_start|>role, the text, <|im_end|> oklm_chat_apply()
llama2 [INST] <<SYS>> system <</SYS>> user [/INST] answer </s><s> oklm_chat_apply()
llama3 <|start_header_id|>role<|end_header_id|>, an empty line, the text, <|eot_id|> oklm_chat_apply()
mistral [INST] user [/INST] answer </s><s>, the system text in front of the first user turn oklm_chat_apply()
mistral-v3 [INST] user [/INST] answer </s>, the system text in front of the last user turn the template of Codestral 22B and Mistral 7B v0.3
gemma <start_of_turn>user or model, the text, <end_of_turn>, the system text in the first user turn oklm_chat_apply()
phi3 <|role|>, the text, <|end|> oklm_chat_apply()
alpaca a preamble (or the system text), ### Instruction: and ### Response: headings, an empty line between turns Stanford Alpaca's prompt without an input, repeated turn after turn
deepseek-coder its system text, ### Instruction: and ### Response:, an answer closed by <|EOT|> DeepSeek-Coder instruct's tokenizer_config.json
foundation-sec <|system|>, <|user|> and <|assistant|> lines, the text on the next, an answer closed by the end of the sequence Foundation-Sec-1.1-8B-Instruct's chat_template.jinja

One table serves every command (oklm_chat_builtin(), oklm_chat_builtin_names()): each entry is the Jinja template that writes the format, so everything that reads the model's template (the rendering, the tool-call format found from it, the server, the agent) sees the one chosen. oklm_chat_template_source() reads the argument, oklm_model_set_chat_template() puts it in the model, and oklm_models_chat_template() is what the commands and oklm_server_new() call. tests/model/test_model.c renders a conversation of two turns with a system prompt in each format and compares it with the definition, and the six formats of oklm_chat_apply() with what that function writes.

Two cases need no option. A DeepSeek-Coder instruct file is known by its BOS and EOS tokens (tokenizer.md). A catalogue model whose file carries no template names its format in the catalogue (models.md): oklm chat -m llava-1.5-7b chats in Vicuna's turns, and so does the file given by its path in the model's own folder (.../llava-1.5-7b/ggml-model-q4_k.gguf), which is how the chat window's own server gets it.

When the file has no template, none is given and the catalogue names none, the command says so once on stderr, with the option and the names, and goes on in ChatML:

$ oklm chat -m plain.gguf
oklm: this model's file carries no chat template: the turns are written as ChatML, which it may not have been trained on.
      --chat-template FILE|NAME gives it one (a Jinja file, or one of vicuna, chatml, llama2, llama3, mistral, mistral-v3, gemma, phi3, alpaca, deepseek-coder, foundation-sec)

Checked on LLaVA 1.5 7B (Q4_K, the catalogue's file) on the CPU, without its projector and without an option:

$ printf 'What is the capital of France?\n/exit\n' | oklm chat -m llava-1.5-7b --backend cpu --temp 0 -n 64 --stats
chat template: jinja. Commands: /system TEXT, /reset, /decide A | B | ..., /tui, /exit (/help lists them all)

> (prompt 16 tokens: 0 cached, 16 evaluated)
 The capital of France is Paris.

The prompt is Vicuna's turn (USER: and the question on a line, then ASSISTANT:), 16 tokens with the BOS, where ChatML's markers make it 35 (--chat-template chatml); the answer ends on </s>, well short of the limit of 64 tokens. oklm describe -m llava-1.5-7b --mmproj llava-1.5-7b on a screenshot answers as before through the same template (576 image rows in a prompt of 600 tokens, "A screenshot of a web page.").

Commands

oklm run  -m model.gguf -p "The capital of France is" [-n 128] [--temp 0] [--top-k 40] [--top-p 0.95] [--min-p 0.05] [--repeat-penalty 1.1] [--seed 1234] [-c CTX]
          [--grammar list.gbnf | --json-schema person.json | --json] [--draft small.gguf [--draft-n 5]]
oklm chat -m model.gguf [--system "Be brief."] [--chat-template FILE|NAME] [--template-vars '{"enable_thinking": false}'] [--temp 0.7] [-c CTX]
          [--tools tools.json [--tool-choice auto|required] [--tool-format hermes|llama3|mistral|glm45|glm4|harmony|lfm2|gemma4]]
          [--resume] [--kv-save] [--chat-file FILE] [--stats] [--tui]
          [--mmproj FILE|NAME|none [--tiles SIDE] [--fps 1] [--frames 8] [--ffmpeg]]
          [--auto-compact on|off] [--auto-continue on|off] [--continue-max 8]
oklm ppl  -m model.gguf -f data/tinyshakespeare.txt [-c CTX] [--progress-json FILE|-]

run, chat and ppl take -ctk TYPE / -ctv TYPE (--cache-type-k, --cache-type-v): the KV cache types, F16 by default; f32, bf16 and the block quants q8_0, q4_0, q4_1, q5_0, q5_1, iq4_nl as llama.cpp. A quantized cache holds Walsh-Hadamard rotated heads, as llama.cpp's (adr/0005-kv-cache-precision.md): on Llama-3.2-1B, Q8_0 keeps the perplexity within 0.2 % of F16 and Q4_0 within 5 %, with half and a quarter of the memory.

  • run prints the prompt and the continuation as it is generated, then the prompt and generation speeds on stderr. Generation stops after -n tokens, at the end-of-sequence token or at any control token. With a grammar (a GBNF file, a JSON schema, or --json for any JSON value) the output follows it and generation stops once it is complete and the model ends.
  • chat is a REPL. Each turn re-renders the whole conversation with the model's chat template, evaluates only the tokens after the part already in the cache, and streams the answer. /system TEXT sets the system prompt, /reset starts over, /exit (or end of input) quits; /help lists the commands (The chat's commands), which also change the model (/model, a selection), the context's size, the sampling and what the model answers from (In the terminal chat). With --tools (a JSON array of OpenAI style tool definitions, passed to the template as tools), calls are constrained by the tools' schemas once the model opens one (from the start with --tool-choice required), printed as a [tool calls] JSON array, and answered with /tool RESULT, which adds a tool message and lets the model continue. /decide label1 | label2 | ... asks the model to choose among the labels for the conversation so far, by scoring rather than generating (the server's /v1/systemone rule, server.md): the conversation is the state, each label is scored as the answer, and each label's probability is printed, then the most probable and the confidence (0 flat, 1 certain). The cache then holds the decision's prompt; the next turn evaluates again only what differs. With --resume, --kv-save or --chat-file, the conversation is written after each turn to .oklm/chat/last.json (in the current directory; --chat-file FILE another; without them nothing is written), and --resume goes on with it (its system prompt too, unless --system is given). --kv-save (off by default) also keeps the cache's rows in a KV file (above) beside it (last.kv), appended after each turn (the prompt's and the reply's tokens); --resume loads it when it is there and fits the model, so the first turn evaluates only the new message: on SmolLM2-135M, a third turn of 127 tokens evaluated 15, the 112 others coming from the file, with the answer of a cold continue (tools/check_chat_continue.sh). --stats prints each turn's prompt tokens, those found cached and those evaluated, what was written to the KV file and, for a model with recurrent blocks whose sequence was cut (/retry, /rewind, a turn whose prompt parts from the cached one), what bringing their state back cost: the tokens that went through the journal and through a forward pass (hybrid-layers-design.md). The reply's tokens are kept with the prompt's, so the next turn also finds the reply cached when the template renders it as generated. With a projector (--mmproj FILE, or the name of an installed catalogue model that carries one; a catalogue model given to -m gets its own without the option, and --mmproj none loads none) /attach also takes a picture, a sound or a video (A picture, a sound or a video). A turn's prompt is then rows: tokens, and for each picture or sound its embeddings, each row known by a hash of its item's content and its index within it. They are compared with the rows the context holds as token ids are, so a turn after a picture evaluates the new message only; --stats then tells the prompt's rows, how many are media, how many were cached and evaluated, how many media items the encoder ran on, and the time the prompt and the evaluation took. --tiles, --fps, --frames and --ffmpeg are oklm describe's (multimodal.md). --tui (or /tui typed in the chat, which also goes back to line mode) runs the chat in the agent's full-screen UI (agent-ui.md) on a terminal: the transcript, where each reply's Markdown is rendered as it streams and a thinking model's reasoning is folded into a [thought, N words] line (Ctrl-O shows it as it is written, and folds it again); the input, with the history (Up, Down, Ctrl-R) and multi-line messages (Alt-Enter, or a backslash before Enter); a status bar with the model, the context's use and the last reply's speed in tokens per second. Esc or Ctrl-C stops a reply (kept as written); keys typed meanwhile wait for the next prompt. The chat's commands work there as in line mode; Ctrl-D on an empty line quits. Without a terminal --tui says so and stays in line mode. tools/check_chat_tui.sh drives it on a pseudo-terminal with SmolLM2-135M on the CPU.
  • ppl computes the perplexity of a text: windows of -c tokens starting with BOS when the tokenizer adds one (not GLM's, whose BOS [gMASK] only the chat template writes), only the second half of each window scored, so every scored token has at least half a window of context (llama.cpp's method). A text shorter than one window scores nothing: the command says so and fails, instead of printing a perplexity of 1 over 0 tokens. --progress-json FILE (- for standard output, the figure's line then going to standard error) writes the run as JSON lines: start with the number of windows, a window line after each with the perplexity so far, end with the figure (training.md).

run, chat and serve check the memory a model would take before its context is made (memory-plan.md); a MoE model's options (--expert-cache, --n-cpu-moe, --expert-gpu) are in moe.md, which also has what each costs and gains on this machine (measured).

The chat's commands

A line that starts with / in a chat is a command, read by one module (include/oklm/client/chat_commands.h) for the terminal chat, the chat window and the server's dashboard, so that a command has the same name, arguments and help everywhere. The rules are the agent's (agent-ui.md): a command only at the line's start, names without case; //text (or a blank before the slash) sends a text that starts with /; the models' own switches /think and /no_think, and a first word that holds another slash (/usr/bin/cc), are text too. Arguments are words; quotes ("...", '...') hold a word with blanks and a backslash keeps the next character; a text (/system, /decide, /tool) is the rest of the line as it is, and so is a file's name or an address when it is not within quotes.

A line that is not a command that runs is never sent to the model, and the chat says why: an unknown name, with the names close to it (unknown command /modle: did you mean /model?); wrong arguments, with the usage; a command of other chats (/connect in a terminal); a command not written in this chat yet (not available here yet); a command that misses something where the chat is (a server whose context is fixed, no retrieval index, a model that cannot be changed from a remote page).

/help lists the commands of the chat it is typed in, /help NAME tells one with its other names and the chats it belongs to. In oklm chat, at the full-screen prompt and in line mode on a terminal, Tab completes the command's name: whole when one command starts with what was typed, up to what they share when several do, which are then listed. After a name, Tab completes the argument the same way: on or off, markdown or json, for /model the catalogue's names a chat can serve and the .gguf files (on a server, oklm chat --url, the names the server lists, asked for at the first Tab), for /attach, /save and /rag the files of the directory typed so far, and force after /context N. Line mode on a terminal reads key by key with the line editor of the full-screen prompt: the arrows move in the line and through the lines sent before, Ctrl-C clears the line, and what Tab lists is written under the line, which is drawn again after it. Input that is not a terminal (a pipe, a file) is read a whole line at a time and completes nothing. In the chat window a list above the message shows the matching commands with their help while one is typed, then the values of its argument, and a command's answer is a note in the conversation that the model is never sent (gui.md).

The dashboard's chat lists the matching commands while a / word is typed (arrows, Tab or Enter, Escape) and shows a command's answer as a note in the conversation; what each command does in a page is in server.md.

The table below is written by the module (oklm_chat_commands_markdown()) and compared with it by tests/client/test_chat_commands.c: yes where a command runs today, not yet where it belongs and is not written yet, - where it does not belong. The window attached to an agent passes the lines it does not take itself to the agent, which has its own commands.

Command What it does Terminal Window, own server Window, remote server Dashboard Window on an agent
/help [NAME], /? List the commands, or tell one in full yes yes yes yes yes
/system [TEXT|-] Show the system prompt, or set it and start a new conversation (- for none) yes yes yes yes -
/reset, /clear, /new Start a new conversation yes yes yes yes -
/decide A | B | ... Have the model choose one of the options, with its confidence; the conversation is kept as it is yes yes yes yes -
/tool RESULT Answer the model's tool call with its result yes - - - -
/model [NAME] Show the model, choose another among those listed, or switch to the one named yes yes yes yes -
/models List the models by group, with their sizes and whether they are installed yes yes yes yes -
/context [TOKENS [force]], /ctx, /context-size Show the context's size and its use, or change the size (the conversation is kept) yes yes yes yes -
/attach FILE Attach an image, a sound or a document to the next message yes yes yes yes yes
/detach [N] Remove what is attached, or the Nth file only yes yes yes yes yes
/rag INDEX|off Answer from a retrieval index's passages, or stop doing so yes yes yes yes -
/save FILE Write the conversation to a file, to be continued later yes yes yes yes yes
/export [markdown|json] Write the conversation as Markdown or JSON yes yes yes yes yes
/copy [N] Copy the Nth latest reply to the clipboard yes yes yes yes yes
/retry, /regenerate Drop the last reply and have it written again yes yes yes yes -
/rewind [N], /undo Go back to before the last message, or before the last N yes yes yes yes -
/continue Go on with the last reply where it was cut (by the context or by the token limit) yes yes yes yes -
/compact [FOCUS] Summarize the older messages into a note to free the context; the recent ones are kept as they are yes yes yes yes -
/autocompact [on|off] Show or set whether the conversation is compacted by itself when the context is nearly full yes yes yes yes -
/autocontinue [on|off|N] Show or set whether a cut reply is continued by itself, or how many times in a row at most yes yes yes yes -
/stop Stop the reply being written - yes yes yes yes
/stats, /usage The conversation's tokens, the context's use and the speed of the last reply yes yes yes yes -
/temp [T], /temperature Show or set the sampling temperature (0: the likeliest token) yes yes yes yes -
/top-k [K] Show or set top-k: sample among the K likeliest tokens (0: all) yes yes yes yes -
/top-p [P] Show or set top-p: sample among the likeliest tokens holding the probability P yes yes yes yes -
/min-p [P] Show or set min-p: leave out the tokens below P times the likeliest one's probability yes yes yes yes -
/seed [N] Show or set the sampling seed yes yes yes yes -
/max-tokens [N] Show or set the longest reply, in tokens yes yes yes yes -
/thinking [on|off] Have the model reason before it answers, or not yes yes yes yes -
/connect URL Chat with the server at an address in place of the current one - yes yes - -
/video [fps F] [frames N] | default Show or set how the videos attached next are sampled: frames a second and frames kept, at most - yes yes - -
/speak [on|off|stop|pause|resume|TEXT] Read the last reply aloud, every reply as it ends (on, off) or a text; stop ends what is being said, pause holds it and resume goes on - yes yes yes yes
/pause Pause the voice that reads aloud, or have it go on from where it was - yes yes yes yes
/listen [stop|send] Record the microphone and write what was said in the message - yes yes yes yes
/talk [on|off] Hold a spoken conversation: listen until a pause, send what was said, speak the reply, listen again - yes yes yes not yet
/tui Enter or leave the full-screen interface yes - - - -
/exit, /quit Leave yes - - - -

A server tells its clients the same table with GET /api/commands (server.md), each command with whether a chat on that server can run it.

In the terminal chat

oklm chat runs every command of its column, in line mode and in the full-screen UI (with --url, on a server's model: On a server says what each does there). A command that takes a value shows the current one without it. What each does there:

  • /system shows the system prompt, /system TEXT sets it and starts a new conversation, /system - removes it.
  • /model and /models: Choosing a model.
  • /context shows the context's size, its use and the range the model takes; /context N makes a context of N tokens (256 to 1048576) for the same model, the conversation kept and evaluated again on the next turn. A size the conversation does not fit in is refused. A size the memory plan says does not fit (memory-plan.md) is warned of in red and not made unless force follows (/context 65536 force); one beyond the context the model was trained with is made with a warning in red. The context in use is freed before the new one is made, so that a device holds one at a time; when the new one cannot be made the old size comes back.
  • /attach FILE reads a document as --doc does (PDF, DOCX, HTML, JSON, text: rag.md) and keeps it for the next message, which holds it before the text typed (Document NAME: and its text between <document> tags), so that /save, /export and a later model see it; its size in tokens is told, with a warning when it is more than the context has left. Several files can be attached; /detach removes them all, /detach N the Nth (the documents are numbered first, then the pictures, sounds and videos, as /stats lists them). With a projector the file may be a picture, a sound or a video (below); without one such a file is refused, with how the chat takes it (--mmproj FILE, or a catalogue model that carries a projector).
  • /rag INDEX opens a retrieval index as --rag does at start, with the other --rag-* options the chat was given, in place of the one in use (which stays when the new one cannot be opened); /rag off closes it.
  • /save FILE writes the conversation as --chat-file keeps it (oklm chat --resume --chat-file FILE goes on with it); /export writes it as Markdown and /export json as the body of a chat request ({"model", "messages"}), to conversation-DATE-TIME.md or .json in the current directory.
  • /copy puts the last reply, without its reasoning, in the clipboard (/copy 2 the one before): wl-copy, xclip, xsel, pbcopy or clip when one is there, else the terminal's own clipboard (OSC 52).
  • /retry drops the last reply and has it written again (at temperature 0 it comes out the same, which the chat says); /rewind goes back to before the last message typed, /rewind N before the last N.
  • /stats prints the model, the conversation's messages and the context's use, the last turn (its prompt's tokens, those evaluated, the reply's tokens and speed), the sampling, what is attached and the retrieval index.
  • /temp, /top-k, /top-p, /min-p, /seed and /max-tokens show or set what the options of the same names set at start, from the next reply on.
  • /thinking on|off sets the template's enable_thinking variable (Qwen3, GLM); the chat says so when the model's template has no such switch.
  • /compact [FOCUS] summarizes the older messages into a note, dwelling on the focus when one is given; /continue goes on with a reply that was cut, or answers a message the full context left unanswered; /autocompact on|off and /autocontinue on|off|N say whether the chat does either by itself (A full context). /stats tells where compaction starts, how a cut reply is continued and how many messages compactions replaced.

tests/harness/test_cli_chat.c types them into a chat on SmolLM2-135M (419 checks with the media's below and a full context's) and tools/check_chat_tui.sh drives the full-screen UI on a pseudo-terminal.

A picture, a sound or a video

oklm chat -m MODEL --mmproj FILE (since 2026-11-15; /attach took documents only) loads the model's projector as oklm describe and oklm serve do: a vision encoder, or an audio one. -m smolvlm-256m, or any installed catalogue model that carries a projector, loads it without the option. /attach FILE then reads a picture, a sound or a video (by the file's extension, as oklm describe tells them) and keeps it for the next message; a tool's result (/tool) takes none, and what is attached waits for the message after it. A video is its sound for an audio encoder, one item for Qwen2-VL and Qwen2.5-VL, and for another vision encoder its frames as pictures, each announced by its time in the message.

$ oklm chat -m smolvlm-256m --backend cpu --temp 0 --stats
(projector mmproj-SmolVLM-256M-Instruct-f16.gguf: with the next message, /attach FILE takes a picture, or a video as its frames)
> /attach front-page.ppm
(attached front-page.ppm: a picture of 640 x 488; it goes with the next message, /detach removes it)
> What is in this picture?
(prompt 348 rows, 320 of them media: 0 cached, 348 evaluated; 1 of 1 media items encoded, the prompt made in 4.32 s)
(348 rows evaluated in 0.22 s)
 A newspaper with an old picture of an astronaut on the front.
> What is the headline?
(prompt 375 rows, 320 of them media: 361 cached, 14 evaluated; 0 of 1 media items encoded, the prompt made in 0.05 s)
(14 rows evaluated in 0.02 s)
 Men walk on moon.

The conversation keeps a message's text and, beside it, the files that went with it. Each turn the messages are written as the projector's model writes a turn with media (oklm_mm_question(), oklm_mm_format_chat(): the functions of oklm describe and of the server's chat routes) and made rows by oklm_mm_build(), the function the server builds a request's rows with. The first time, an item's encoder runs; its embeddings are then kept by the hash of its content, the item is named by that hash and its pixels or samples are freed. A later turn's prompt takes the embeddings kept, so the encoder does not run again, and its rows are those the context already holds, so they are not evaluated again either. The same picture attached twice is encoded once.

What the other commands do with media in the conversation:

Command With a picture, a sound or a video in the conversation
/retry the reply is written again; the prompt's rows are in the context but for the last
/rewind [N] the messages dropped take their media and embeddings with them; the rows before them stay cached
/reset, /system TEXT the conversation's media go; what is attached to the next message stays
/detach [N] removes what is attached to the next message, documents and media alike
/context N the new context evaluates the whole conversation again from the embeddings kept: no encoder runs
/decide A | B scores the labels on the conversation's text: its pictures and sounds are not in the decision's prompt (said); the next turn evaluates the rows again, without encoding
/save, --chat-file, --resume the file names each message's media files ("media": ["front-page.ppm"]); --resume reads them again and encodes them once; with --kv-save their rows come back from the KV file (version 2) and are not evaluated again. A file that is gone, or a chat without the projector, starts a new conversation and says so
/export Markdown lists each message's files (Attached: NAME); the JSON holds the text
/model refused while the chat holds media: their rows are this model's projector's. Without media the model changes, and the projector with it: the catalogue's for the new model, or none
--rag, --rag-critique the question is searched as text; its sources go in the prompt beside the message's media, and the critique's retry is rendered the same way
/tool RESULT a tool message holds no media

A conversation that no longer fits the context says so with its size in rows (/rewind, /context N, /reset); a file the encoder cannot read, or a template with no place for a picture, drops the message with what was attached to it and the chat goes on. A projector whose embeddings are not of the model's width is refused at start (it is another model's projector). The context is not sized for the pictures as oklm describe's is: /attach tells the rows a picture will take when the encoder says so before it runs (Qwen2-VL, Qwen2.5-VL) and warns when they are more than the context has left.

Measured on the CPU (2026-11-15, --temp 0, the machine shared with other work): SmolVLM-256M (Q8_0), the front page of test_mm_public's answer case at 640 x 488: the turn with the picture 348 rows, 320 of them the picture's, encoded in 4.3 s and evaluated in 0.22 s; the next question 375 rows, 361 cached, 14 evaluated in 0.02 s, nothing encoded; /retry 1 row; /rewind and another question 11 rows. Ultravox v0.5 on Llama 3.2 1B (Q8_0), jfk.wav (11 s): 228 rows, 188 the sound's, encoded in 8.5 s; the next question 294 rows, 276 cached, 18 evaluated in 0.07 s; /retry 1 row. Qwen2-VL 2B (Q4_K_M), the same front page: 418 rows, 391 the picture's (9.4 s to encode, 1.7 s to evaluate); the next question 15 rows of 473; a second picture two turns later 292 rows of 773, one item of two encoded; --resume with --kv-save in a new process: 813 rows from the file (24.1 MB), 19 evaluated of 832, the two pictures encoded again and not evaluated. The turn with the second picture gives the same text when the whole conversation is evaluated from nothing (/context 8192 before it) as when 481 of its rows come from the cache. Gemma 3 4B (Q4_K_M), whose picture is 256 rows that see each other: 272 rows for the picture's turn, the headline read; the next question 22 rows of 312; /retry 1 row.

tests/harness/test_cli_chat.c (test_media) runs this on a tiny model and tiny encoders made by the test, a vision one and an audio one: the rows --stats tells for a picture's turn, the next one, /retry, /rewind, the same picture attached again (not encoded), /decide, /context N, /reset; the conversation's file and the Markdown export; --resume with and without the projector; a sound; a retrieval turn with its critique; a tool's result; a conversation longer than the context; a projector of another width; the catalogue's projector and /model away from it. tools/check_chat_tui.sh attaches a picture in the full-screen UI on SmolVLM-256M.

A full context

oklm chat, the chat window (gui.md) and oklm agent (agent-sessions.md) do two things by themselves when a conversation reaches the end of the context. Each of the two can be left to the person, in each program.

Compaction. Past a share of the context the older messages are summarized by the model into a note, and the recent ones stay as they are. The code is the agent's, in one place for the three programs (src/client/compaction.c):

  • When. The agent's threshold: a share of the context less the reply's reserve (the reply's token limit, a quarter of the context at most). The share follows the model's size: 0.60 under 2 billion parameters, 0.75 under 8 billion, 0.85 above or when the size is not known (a server at an address). With a context of 512 tokens, replies of 1024 and a small model: 230 tokens.
  • What stays as it is. The last exchange (the person's latest message and what follows it), always; the exchanges before it, the latest first, while together they take a quarter of the context at most; a message that carries a picture, a sound, a video or an attached document, with its reply; the system prompt.
  • The note. The summary of the other messages. The model reads it after the system prompt, in the same system message, after the sentence This session continues an earlier conversation that grew too long for the context; it was summarized:. A later compaction summarizes the earlier note with the messages that go.
  • The summary. Asked after the conversation when the context has room for the request and the answer (the cache is reused), else from a transcript of the messages that go, read in parts the context holds, each part's summary carried into the next. An eighth of the context at most (48 to 1024 tokens), with reasoning off.
  • Nothing to gain. When the messages that would go do not hold an eighth of the context more than the note may take, nothing is compacted: the note would free nothing worth a summary (oklm_compact_gains()). The agent holds its own compactions to the same rule (it did not: with a system prompt and tools that nearly filled a server's context, each compaction made the prompt longer, until the server refused it), and says so when its own part leaves no room for a conversation (agent-sessions.md). oklm chat and POST /api/compact were run on the same case, a system prompt of 535 tokens in a context of 512: nothing is compacted ("compacted":0), and the chat says The context is full (535 of 512 tokens): the message was not answered.
  • What was replaced is kept. /export writes the replaced messages before the note, under Compacted, and the conversation's file keeps them ("compacted") beside the note ("summary").

Continuing. A reply that the context or the token limit cut is gone on with: the conversation is given to the model again with the cut reply as the open start of its turn, with no end of turn after it, so the model writes on from the reply's last token. The rule is one function for the three programs (src/client/reply_continue.c):

  • Room first. When the context is what cut the reply, room is made before it goes on: a larger context where one fits, else a compaction, which then keeps the last exchange alone. A note says which. When neither makes room (the reply alone fills the context) the reply stays cut, and a note says that a larger context holds more.
  • The token limit. The limit a program starts with is the size of one piece: 512 tokens in oklm chat, 1024 in the window, 8192 in the agent. A reply goes on past it, to its own end. A limit the person gave (-n N, /max-tokens N, the window's Reply length moved, the agent's --max-tokens N) ends the reply for good, with no note: it is what was asked for. /continue still goes on with it, one piece.
  • A bound. A reply is continued 8 times in a row at most (--continue-max N, /autocontinue N). A reply that repeats itself (its last 160 bytes found twice more in the 4000 bytes before them, or a piece that added nothing) is not continued. Either way a note says so, and that /continue goes on.
  • Reasoning. The open turn is the reply as the model wrote it, its <think> block included: cut inside its reasoning, the model goes on reasoning and then answers. oklm chat and the in-process agent write the reply's own tokens after the prompt, so the rows are those the cache holds. Over a server the text is sent back (server.md); for a model that writes its reasoning in a channel (harmony, Gemma 4, Muse Glimmer) the server writes the turn again in the model's channels and leaves it open in the one the reply was cut in. In oklm chat such a reply cut inside its reasoning keeps its block open on the screen, and the next piece goes on in it: Gemma-4-E2B-it Q4_K_M on the CPU, thinking on, the question of the server's measure in a context of 128 tokens, is cut inside its reasoning (the context made 256), again inside its answer (made 512), and what is printed is byte for byte what a context of 4096 prints, 228 tokens. A reply of such a model cut inside the first header of its turn (<|channel|>, <|channel>, to=self: its first tokens) has written markup and nothing else: nothing of it is shown, kept or sent back as text, and the turn is opened afresh, the template writing the header, in the server, oklm chat and the agent alike (server.md).
  • Tool calls. A reply cut inside a tool call is gone on with like any other, and the call is not parsed, shown as a call nor run while it is half written. The calls' grammar reads the call's text so far (oklm_tool_call_open()) and the model goes on under it, so what comes out is a whole call, parsed with the whole reply. oklm chat and the in-process agent do it on the reply's own text; an agent on a server sends the call's text back as tool_call_open (server.md). It needs a format whose calls open with a marker (not Llama 3's bare JSON nor GLM-4's name line) and the calls' grammar (not oklm agent --no-tool-grammar). oklm chat --tools FILE --tool-choice required on SmolLM2-135M with replies of 8 tokens (-n 2): the call is cut inside, shown as text, and after the /continues that end it the one [tool calls] line is the line of the reply written in one go (test_call_cut).

The dashboard's chat does the same in a page: /compact, /continue, /autocompact, /autocontinue, a Continue on a cut reply, both by themselves unless set otherwise (kept in the browser), with the server's requests: the open last message, and POST /api/compact for the plan and the summary (server.md). Room for a reply the context cut is made as the chat window makes it: a larger context first, asked of the server's resize after its dry run, where the server is not --fixed-context; else a compaction (server.md).

One reply on the screen. The pieces of a continued reply are one reply wherever it is shown or kept. oklm chat in line mode writes them one after the other on standard output, the note about the room made between them on standard error. In the full-screen UI (--tui, /tui) they are one block: a piece that ends cut leaves the block open (the Markdown renderer and the reasoning's fold as they were), the next piece is written after its last word, and what was said meanwhile ((the context was full: it is now 256 tokens, and the reply goes on; ...), a compaction's two lines) is shown after the block, not inside it. A piece asked for later with /continue follows its own line of the transcript, as reasoning when the reply was cut inside its reasoning. /copy, /retry, /rewind, /export and the conversation's file take the conversation's message, which is one reply in both modes: tools/check_chat_tui.sh runs the same question in a context of 128 tokens in line mode and in the full-screen UI, where the words on each side of the cut follow one another, /copy takes the same 678 bytes, and the conversation's file and /export's are the same files.

oklm chat The chat window oklm agent
At start --auto-compact on|off, --auto-continue on|off, --continue-max N, over the settings for the run the same options the same options, over the settings for the run
Commands /compact [FOCUS], /continue, /autocompact [on|off], /autocontinue [on|off|N] the same, a Continue on a cut reply, two boxes in the model settings /compact [focus], /continue, /autocompact [on|off|auto|TOKENS], /autocontinue [on|off|N]
Kept autoCompact, autoContinue, continueMax in the user's settings, the agent's in the window's settings file autoCompact, autoContinue, continueMax in the user's settings
Room for a cut reply a context twice as large, up to the one the model was trained with, when the memory plan holds it; else a compaction the next context the Context slider offers, when the line under it has nothing to warn of; else a compaction in process as oklm chat; on a server that allows resizing, the same size asked of it after its dry run; else a compaction (agent-sessions.md)
Room before a turn the same larger context when the conversation leaves less than a turn's room (512 tokens), else a compaction when it frees room; when the system prompt and the tools alone leave no room, the message is not sent and the agent says so (agent-sessions.md)

One setting for the two terminal programs. oklm chat reads autoCompact, autoContinue and continueMax where oklm agent keeps them (agent-settings.md: the user's ~/.config/oklm/settings.json, $XDG_CONFIG_HOME/oklm/settings.json when that is set, with the settings of the directory it runs in over it, as the agent reads them). /autocompact on|off and /autocontinue on|off|N in either program write the user's file and say so, (saved in PATH, for oklm chat and oklm agent), so the other program starts with what was chosen. The start options are over the file for the run and write nothing. The chat window keeps its own three in its settings file (gui.md), which the terminal programs do not read.

Manual. With --auto-compact off a full context says so and names the ways out, and nothing is summarized until /compact:

(The context is full (271 of 256 tokens): the message was not answered. /compact summarizes the older messages, a larger context (/context N) holds more; /continue then answers the message.)

With --auto-continue off a cut reply says (The reply was cut by the context: /continue goes on with it.), or the same of the reply's token limit, and /continue goes on with it. /continue also answers a message the full context left unanswered, once there is room.

Measured on the CPU (tests/harness/test_cli_chat.c, test_full_context, SmolLM2-135M Q8_0, temperature 0):

  • "Write three sentences about the sea." in a context of 128 tokens: the reply is cut at the context's end, the context is made 256 tokens, and the reply ends at 138 tokens, the same text as in a context of 2048.
  • With --auto-continue off the same reply stops at 128 tokens of context with its note; /continue makes the context 256 and ends it at the same 138 tokens.
  • With -n 4 (16 tokens a reply) the reply stops at 16 tokens without a note; /continue adds 16, one token of the prompt evaluated, the rest in the cache.
  • "Write a long story about a lighthouse keeper and the sea." with --continue-max 1: continued once, then (The reply was continued 1 time in a row and is still not ended: ...). Asked in a context of 256 tokens without the bound, the context is made 512, and there the reply is not continued: it repeats itself, which the note says.
  • Five short questions in a context of 256 tokens with replies of 48: with compaction manual the fifth does not fit (271 tokens), /compact takes the conversation from 271 tokens to 131 (6 of 9 messages become the note) and /continue answers the message; automatic, the conversation is compacted twice on the way (163 tokens to 42, 169 to 40) and every message is answered.

A long answer continued across a compaction, on Qwen3-0.6B Q8_0 on the CPU (oklm serve -c 2048 --fixed-context, the chat window linked to it, /thinking off, temperature 0): after three questions about tides (444, 403 and 369 tokens of replies) an essay of about 1200 words on the history of lighthouses was asked for. Its prompt took 1339 tokens and the reply was cut by the context after 709. The six older messages became a note of 288 bytes, and the reply went on from a prompt of 832 tokens for 521 more, to its own end: 1230 tokens. Read as exported, it is one essay: one title, its sections in order from the ancient world to the future and a conclusion, no heading twice and no restart where it was cut. What it says is a 0.6B model's (it repeats its phrases, and its history is thin), before the cut as after it.

Choosing a model

/model without a name is a selection among the catalogue's models (models.md), listed under the names of their groups as the model manager lists them: each with its size, whether it is the model in use, installed or to download (and whether its licence asks for agreement), and its description. A model of two groups is in both. The models a chat cannot serve (images, sounds, retrieval) are listed dimmed, with what they are for in brackets, and cannot be chosen. In the full-screen UI the list is a panel: the arrows move the selection (over the groups' names), typing searches it, Enter chooses, Esc keeps the model in use.

+- Choose a model (arrows and Enter; type to search; Esc keeps the current one) ---------------+
| General chat                                                                                 |
| > smollm2-135m                  138 MiB  current                the smallest chat model: a f |
|   qwen3-0.6b                    610 MiB  installed              a small chat model with a th |
|   llama3.2-1b                   770 MiB  to download (licence)  Meta's 1B chat model         |
|   gemma3-1b                     769 MiB  to download (licence)  Google's 1B chat model (text |
|   phi3.5-mini                   2.2 GiB  to download            Microsoft's 3.8B chat model  |
|   ...                                                                                        |
| Coding                                                                                       |
|   qwen3-8b                      4.7 GiB  to download            the default general model fo |
|   ...                                                                                        |
| Images                                                                                       |
|   smolvlm-256m                  348 MiB  to download            [images: oklm describe] desc |
|   ...                                                                                        |
| Sounds                                                                                       |
|   whisper-base                  281 MiB  to download            [sounds: oklm transcribe, ok |
+----------------------------------------------------------------------------------------------+

In line mode the same list is printed with a number before each model a chat can serve, and the chat asks for a number or a name (Enter keeps the model in use). /models prints the list and asks nothing. /model NAME takes a catalogue name, a number of the list or a model file.

A model that is not installed is downloaded only once the selection asked: the question names its size, its licence and the address of the licence's text, and where the files go. For a model whose licence asks for agreement the answer is the agreement (yes typed in full in line mode), recorded as oklm models pull records it. In the full-screen UI the question is a panel whose first entry is "No":

+- llama3.2-1b is not installed: download 770 MiB? --------------------------------------------+
| > No: keep the current model                                                                 |
|    nothing is downloaded                                                                     |
|   Yes, I agree to its licence: download 770 MiB                                              |
|    licence llama3.2: https://www.llama.com/llama3_2/license/                                 |
+----------------------------------------------------------------------------------------------+

The download shows its progress; Ctrl-C (or Esc in the full-screen UI) stops it, what arrived is kept and the same choice goes on with it.

The model chosen is loaded in place of the one in use and the conversation is kept: the chat tells its messages and its tokens for the new model, which reads it again on the next turn. The new model's tokenizer, template and tool-call grammar are made, and the conversation is counted with them, before anything of the old model goes; the old context is then freed before the new one is made, and made again if the new one fails. Nothing is dropped from a conversation: when it is longer than the new model's context would be, the model in use stays and the chat says how to make room (/rewind, /reset, /context N). A file without a tokenizer or with a template that cannot render the conversation is refused the same way. With --kv-save, the cache's file is removed (it held the old model's rows). The agent's /model is the same selection (agent-ui.md).

On a server

oklm chat --url URL (since 2026-11-17) is the same chat, in line mode and in the full-screen UI, with the model of an OpenAI-compatible server in place of one in this process: an oklm serve, llama.cpp's server, or another. Each turn is a streamed POST /v1/chat/completions through the client oklm agent --url uses (the agent's HTTP backend, without tools; agent-loop.md): transient failures are sent again as the agent sends them. The key is --api-key KEY, else OKLM_API_KEY, else OPENAI_API_KEY. -m and --url exclude each other. With neither, the baseUrl of the settings is the server, and their model the name sent, as for the agent.

$ oklm chat --url http://127.0.0.1:8080
server: http://127.0.0.1:8080, model models/qwen3-0.6b/Qwen3-0.6B-Q8_0.gguf, a context of 32768 tokens (the default for a server that does not tell its own). Commands: /system TEXT, /reset, /model, /tui, /exit (/help lists them all)

The model. --remote-model NAME names it. Without it the chat asks the server (GET /v1/models, oklm_client_remote_models() in src/client/remote_models.c, which reads OKLM's answer, llama.cpp's and an OpenAI-style one): a server that lists one model gets that one; one that lists several shows them in a list to choose from on a terminal; one that lists none is sent default. When the input is not typed (a pipe, a script) and several are listed, none is taken: the chat ends with status 2, the list and the option's name, because the first of a list is the server's order and not a choice. /model lists the server's models with the one in use marked and, on a terminal, chooses among them; /model NAME or /model N asks for one from the next message, listed or not, the conversation kept; /models lists them.

The window. An oklm serve tells its context (GET /api/context) and the chat counts with it; /context N asks the server for another size (POST /api/context, force as in process). Another server tells none: the window is -c N when given, else the contextWindow setting, else 32768 as for the agent, and /context N sets the size this chat counts with. The conversation's size against the window is an estimate: the server's count of the last turn (its usage) and four bytes a token for what was added since, four bytes a token for everything when the server tells no count. /stats, /context and the status bar say "about".

A full context (as in process). Automatic compaction starts where that estimate passes the agent's threshold, with the share of a model whose size is unknown (0.85); the summary is asked of the server's model, without reasoning (enable_thinking false), and the figures the chat tells are estimates. A prompt the server refuses for its length (oklm serve's 400 with its counts, another server's 400 naming the context) is compacted once and sent again when the chat compacts by itself, and said with the ways out otherwise. A reply the token limit cut is gone on with as in process: the reply so far goes back as the last message, left open ("prefix": true, which oklm serve reads; llama.cpp's server goes on with a last message of the model by itself). A server that refuses an open last message (400 or 422) is told so once, and from then on a cut reply goes back closed, followed by a request to go on with it exactly where it stopped; the conversation keeps one reply either way. A reply that ends for its length with fewer tokens than its limit was cut by the server's context: an oklm serve that allows it is asked for a larger one, by the rule the agent uses; otherwise the conversation is compacted, or the chat says there is no room. A reply cut before its answer began (inside its reasoning, which a server sends apart) leaves nothing to go on with: the chat says so, the message stays, and /continue asks again.

Reasoning. Reasoning deltas (reasoning_content) are shown as a model in process writes its reasoning, within <think> tags: printed as they are in line mode, folded in the full-screen UI (Ctrl-O). The conversation and its file keep the answer alone.

What is sent. The system prompt and the messages; max_tokens (the reply's limit, -n and /max-tokens); temperature always, as the agent sends it; top_p, top_k, min_p and seed once the person gave them (--top-p..., /top-p...), because a server that does not know one of them may refuse the request, and what is not sent is the server's default; --template-vars and /thinking on|off as chat_template_kwargs, which a server whose template has no such switch ignores. --repeat-penalty is not sent. A server that refuses one of these members by name (a 400 or a 422: Unrecognized request argument supplied: top_k) is asked again without it at once, the chat says so once, (the server does not take top_k: --top-k has no effect on it, and is no longer sent), and no later request carries it; the same holds for stream_options (the tokens are then estimated) and chat_template_kwargs (agent-loop.md). A refusal that names no member the chat sent stays an error, shown as the server worded it.

What each command and option does on a server:

Command or option On a server
/help, /exit, /tui, /system, /reset, --system as in process
/save, /export, /copy, /retry, /rewind, --chat-file, --resume as in process: the conversation and its file are the chat's own, in the same format
/compact, /continue, /autocompact, /autocontinue, --auto-compact, --auto-continue, --continue-max as in process, by estimate (above)
/temp, /top-k, /top-p, /min-p, /seed, /max-tokens, /thinking, --template-vars, -n sent in each request (above)
/model, /models, --remote-model the server's models
/context, -c the server's context, or the size the chat counts with
/stats, --stats the server, its model, the estimated use, the last turn by the server's counts (the prompt, of it the tokens in the server's cache, the reply), the sampling sent
/attach FILE (a document), /detach as in process, its size estimated
/attach FILE (a picture, a sound, a video), --mmproj needs the projector in this process: said in a line (--mmproj: refused at start)
/rag, --rag and its options need the model in this process, which searches and judges: said in a line (--rag: refused at start)
/decide needs the model in this process, whose logits score the options: said in a line
/tool, --tools need the model in this process, which writes a call under a grammar: said in a line (--tools: refused at start; oklm agent --url gives a server's model tools)
--kv-save, --chat-template, --grammar, --json-schema, --json, --lora, --draft, --mtp, --observe need the model in this process: refused at start, status 2
--backend, --gpu-layers and the other options of where a model computes nothing to place: without effect

What a failure says: a key the server refuses, or wants and was not given, with its status and its reason and the two ways to give one (status 1: at start when the server keeps its list behind the key, at the first message otherwise); a server that cannot be reached; a model the server does not have, with /model to list them; any other error status with the server's own reason. After a failed turn the message stays in the conversation and /continue asks again.

tests/harness/test_cli_chat_url.c runs all of this against a server of its own that records each request (127 checks, no model loaded); tests/client/test_remote_models.c reads the lists (55 checks) and tests/client/test_request_members.c the member a refusal names and the request without it (34 checks); tools/check_chat_url.sh runs it on llama.cpp's server and on oklm serve with Qwen3-0.6B on the CPU (testing.md).

Verification

Test Reference Result
tests/model/test_model.c (committed fixtures) numpy float64 forward pass (tools/infer_ref.py) on three tiny random models: llama (GQA, Llama 3 rope factors), qwen3 (biases, Q/K norms, tied F16 embeddings), gpt2 max logit difference 8e-7 / 1.4e-5 / 3.9e-5 with an F32 KV cache, every greedy choice identical; F16 KV cache within 0.04
same token-by-token evaluation batched, multi-batch and truncate-and-replay logits bit-identical
tests/model/test_infer_public.c (make test-public) tools/infer_ref.py on the published SmolLM2-135M-Instruct GGUFs, 5 prompts x 32 greedy steps, replaying the reference's tokens F16: max logit difference 2.3e-5, 160/160 greedy choices equal; Q8_0: 1.69, 158/160; Q4_K_M: 2.11, 156/160
same llama.cpp v0.8.0 (tools/gen_llama_ref.sh) on the 8 corpus files and the SmolLM2 checkpoint loaded directly, same 5 prompts x 32 steps see the table below
tests/model/test_model.c the GGUF conversion of each tiny Hugging Face checkpoint logits within 1e-6 (llama) or identical (qwen2, gpt2)

The reference is written from the model definitions, not from OKLM's code, and reads the weights through gguf-py's own dequantizers, in float64 with float activations.

For quantized files OKLM (like llama.cpp) rounds each activation row to Q8 before the product, and the reference does not. How far apart two equally valid evaluations of a quantized model are was measured with tools/infer_ref.py --act-q8, which rounds the activations the way ggml does: against the plain reference, on the steps where both follow the same tokens, its largest logit difference is 1.31 (Q8_0) and 1.78 (Q4_K_M) with 97 % of greedy choices equal. That is the size of OKLM's own difference, and it cannot be made smaller by emulating the rounding: across 30 layers, float32 / float64 differences flip a few roundings, and each flip changes the next layers' inputs. The test bounds are therefore 2.5 (Q8_0) and 3.0 (Q4_K_M) with 95 % agreement; the exact behaviour of each quantized kernel is checked separately, bit for bit, in tests/core/.

Perplexity of the first 100 kB of Tiny Shakespeare (oklm ppl -c 512, 15 300 scored tokens): F16 30.94, Q8_0 31.08, Q4_K_M 31.77 (the windows start with their own first token: SmolLM2 has a BOS id but does not add it, as llama.cpp does); llama-perplexity 30.94, 31.08, 31.79.

Against llama.cpp

llama.cpp is built from a pinned release by tools/gen_llama_ref.sh and used through tools/llama_ref.c, which records its tokenization of each prompt, its greedy continuation (32 tokens) and its logits at every step. The test checks three things: the prompt tokenizes to the same ids; on llama.cpp's tokens, OKLM's logits agree; and OKLM's own free-running greedy continuation is compared with llama.cpp's. Where the two continuations part, everything before is identical, so llama.cpp's logits for that step are known: the last column is the margin by which llama.cpp itself preferred its token over OKLM's.

file prompt ids choices equal on llama.cpp's tokens max logit difference identical continuations tokens before parting llama.cpp margin where they part
SmolLM2 F16 5/5 160/160 0.047 5/5 160/160 -
SmolLM2 BF16 checkpoint, loaded directly 5/5 160/160 0.039 5/5 160/160 -
SmolLM2 Q8_0 5/5 156/160 1.71 2/5 129/160 0.15
SmolLM2 Q4_K_M 5/5 154/160 2.10 1/5 93/160 0.11
Qwen2.5-0.5B Q4_K_M 5/5 160/160 1.72 5/5 160/160 -
Qwen3-0.6B Q8_0 5/5 158/160 0.82 3/5 104/160 0.08
Llama-3.2-1B Q4_K_M 5/5 157/160 1.06 2/5 143/160 0.09
TinyLlama-1.1B Q4_K_M 4/5 156/160 0.80 2/5 110/160 0.06
GPT-2 Q4_K_M 5/5 152/160 139 1/5 62/160 0.26
Qwen3-8B Q8_0 (made by OKLM) 5/5 159/160 1.07 4/5 145/160 0.13
Qwen3-8B Q4_K_M (made by OKLM, importance matrix) 5/5 155/160 2.21 2/5 104/160 0.43
OLMoE-1B-7B-0125-Instruct Q4_K_M (64 experts, 8 per token) 5/5 157/160 2.04 3/5 117/160 0.12
DeepSeek-Coder-1.3B-Instruct Q4_K_M (llama.rope.scale_linear) 5/5 159/160 7.26 4/5 143/160 0.16
DeepSeek-Coder-6.7B-Instruct Q4_K_M (the same keys) 5/5 159/160 2.81 4/5 157/160 0.08
Qwen3.5-9B Q4_K_M (qwen35, Gated DeltaNet blocks) 5/5 159/160 13.8 4/5 154/160 0.44
Qwen3.8-27B UD-Q4_K_M (qwen35, 64 blocks) 5/5 160/160 3.22 5/5 160/160 -
Cyber-Ornith 1.5 9B Q4_K_M (qwen35, a tune of Qwen3.5-9B) 5/5 158/160 3.58 3/5 131/160 0.25
MiMo-V2.6-Distill-Qwen-9B Q4_K_M (qwen35, a tune of Qwen3.5-9B) 5/5 158/160 2.85 3/5 120/160 0.08
BugTraceAI-CORE-Ultra 27B Q4_K_S (qwen35, a tune on Qwen3.8-27B's layout) 5/5 157/160 3.59 3/5 115/160 0.08
Uraion Cyber V1.0 Q4_K_M (qwen35moe, a tune of Qwen3.6-35B-A3B) 5/5 159/160 4.40 4/5 158/160 0.38
Imperum-CybersecurityLLM v1.0 Q4_K_M (qwen35moe, the same base) 5/5 154/160 2.59 2/5 97/160 0.26
CyberStrike-OffSec-35B Q4_K_M (qwen35moe, the same base, five shards) 5/5 154/160 2.30 1/5 94/160 0.14
Spark-X2.6-1.7B Q4_K_M (spark2_5) 5/5 159/160 4.53 4/5 129/160 0.03
Spark-X2.6-4B Q4_K_M (spark2_5) 5/5 158/160 16.2 3/5 122/160 0.20
Muse Glimmer 30B UD-IQ2_XXS (muse-glimmer) 5/5 157/160 5.51 3/5 114/160 0.10
Gemma-4-12B Q4_K_M (gemma4; see Gemma 4 for why it is wide) 5/5 148/160 12.5 1/5 66/160 1.64
Gemma-4-12B, blocks 4 and 5 alone in F16 (tools/gguf_cut_blocks.py; prompts given as ids) - 96/96 0.25 3/3 96/96 -
Gemma-4-E2B Q4_K_M (gemma4 with an embedding per layer and blocks sharing keys and values) 5/5 160/160 8.08 5/5 160/160 -
Gemma-4-E4B Q4_K_M (the same form) 5/5 156/160 3.51 1/5 79/160 0.34

Reading it:

  • With F16 weights the continuations are identical. The 0.047 of logit difference is llama.cpp rounding activations to F16 for F16 products; OKLM keeps them in F32 and matches the float64 reference to 2.3e-5.
  • With quantized weights both engines round activations to 8 bits, then sum in different orders, which moves logits by 1 to 2. Continuations stay identical until two candidates are nearly tied: wherever they part, llama.cpp's own margin was at most 0.15 (0.26 for GPT-2), an order of magnitude below the noise between the engines. OKLM never disagrees with llama.cpp on a clear choice. The test bounds that margin at 0.5.
  • Qwen3.5-9B's 13.8 is on single logits far below the best one, at four steps of the prompt it answers by repeating one sentence, where the next token is all but certain; every other step is within 0.4 to 0.7 and the mean difference over all logits is 0.116. Those steps are that sensitive to rounding in OKLM alone: with an F32 instead of an F16 cache its own logits move by 0.121 on average and by up to 10 at the same steps (tests/model/test_qwen35_public.c has the other checks of that file). And in llama.cpp alone: with F32 keys and values, or its prompt given a token a pass, it moves its own logits by 0.19 to 0.23 on average over 100 steps and by 13.7 to 14.9 at those steps, and the hidden states of two of its runs part in the same blocks as OKLM's and its own (hybrid-layers-design.md).
  • Spark-X2.6-4B's 16.2 is one step of the code prompt (5.5 and 4.8 on the steps beside it) and its 9.4 one step of the first prompt; every other step is under 2.6. The same sensitivity: with an F32 cache OKLM gives 2.76 and 0.69 at those two steps, 3.88 at most over the 160, and 159 choices. On the 1.7B's unquantized file (BF16 weights, 3.4 GB) the two engines agree on 160 of 160 choices with logits within 0.35, llama.cpp rounding the activations of a BF16 product to BF16 (8 bits of mantissa against F16's 11, hence more than SmolLM2's 0.047). The five prompts stay inside the window of 512 positions; past it (the licence's first 3600 bytes, 769 tokens, then 16 tokens; a copy of tools/llama_ref.c with a context of 2048, not kept) the 1.7B gives 14 of 16 choices and logits within 1.01, the 4B 16 of 16 and 1.42.
  • GPT-2 has activation outliers of very large magnitude: 8-bit rounding of activations moves single logits by up to ~140 in both engines (the float reference with ggml's rounding, tools/infer_ref.py --act-q8, gives the same greedy tokens as OKLM and llama.cpp; without the rounding it does not).
  • Qwen3-8B, the default model, is made from the official BF16 weights by tools/make_qwen3_8b.sh (OKLM's conversion, importance matrix and quantization) and its rows are skipped when the files are missing. On OpenCL: Q8_0 159/160 (1.12), Q4_K_M 158/160 (2.68). Its perplexity on the last 30 KB of Tiny Shakespeare (512-token windows, 4080 tokens): Q8_0 6.997 in llama.cpp, 6.972 on OKLM's CPU, 6.984 on OpenCL; Q4_K_M 7.830, 7.820 and 7.697. The OpenCL figure is the exact one for those weights: the same Q4_K weights unpacked to an F32 file give 7.697 on the CPU, where no activation is rounded. Both CPU engines round activations to 8 bits for K-quant products, which costs this model 1.6 % (Qwen3-4B 0.4 %).
  • OLMoE-1B-7B (allenai's GGUF, tests/public_models_large.txt): the three choices that differ on llama.cpp's tokens and the two places where the free-running continuations part are near ties (llama.cpp's own margin at most 0.12). Perplexity on the last 30 KB of Tiny Shakespeare (512-token windows, CPU): 2.2015 in llama.cpp, 2.1963 in OKLM (the file has a BOS id but does not add it, so neither engine puts BOS at the window starts). On OpenCL (float kernels, as test_infer_public runs them): 157/160 choices equal on llama.cpp's tokens, max difference 1.84, 3/5 continuations identical, llama.cpp's margin where they part 0.04. With the default 8-bit matrix units, against OKLM's CPU on the CPU's greedy tokens: all layers on the device 155/160 (max difference 2.08), 8 of 16 layers 157/160, --n-cpu-moe 8 158/160, --n-cpu-moe all 155/160 and the same with a 50 % expert cache, --n-cpu-moe all --expert-gpu 1.8 154/160; every choice that differs is a near tie on the CPU (its margin at most 0.26). Perplexity on the same text: 2.1959 (all layers), 2.1997 (--gpu-layers 8), 2.1969 (--n-cpu-moe 8), 2.1958 (--n-cpu-moe all, and with --expert-cache 1.8), 2.2015 (--expert-gpu 1.8), against 2.1963 on the CPU.
  • The TinyLlama file stores merges with all vocabulary scores 0: llama.cpp ignores the merges and merges by score, leftmost first on ties; OKLM uses the merges, which gives Hugging Face's tokenization of the model ("def fibonacci(n):" is ▁fib on acci in both OKLM and Hugging Face, ▁fib ona cc i in llama.cpp).
  • Without tokenizer.ggml.add_bos_token, OKLM adds BOS by the same rule as llama.cpp: for SentencePiece and for the Llama 3 family of pre-tokenizers (llama-bpe...).

Published files on a device

tests/model/test_archs_public.c runs the published files of the architectures whose device paths needed no kernel of their own on OpenCL and on Vulkan, every block on the RX 7900 XT: the 160 steps of the corpus on llama.cpp's tokens in a device context and in a CPU one (2026-11-14; the CPU's choices equal to llama.cpp's in brackets).

file backend choices equal to llama.cpp's mean / max logit difference from llama.cpp choices equal to the CPU's mean / max from the CPU
LFM2-350M Q4_K_M (lfm2) OpenCL 158 / 160 (157) 0.137 / 2.13 157 / 160 0.134 / 1.67
Vulkan 158 / 160 0.127 / 1.82 157 / 160 0.124 / 1.46
Cyber-Prime 1.1 2.6B Q4_K_M (lfm2) OpenCL 152 / 160 (154) 0.228 / 4.44 150 / 160 0.214 / 4.00
Vulkan 151 / 160 0.225 / 7.36 149 / 160 0.216 / 7.52
Tini-Cybersec-8B-A1B Q4_K_M (lfm2moe) OpenCL 155 / 160 (156) 0.315 / 7.30 157 / 160 0.295 / 8.70
Vulkan 155 / 160 0.327 / 5.65 159 / 160 0.314 / 5.42
StarCoder2-3B Q4_K_M (starcoder2) OpenCL 159 / 160 (159) 0.193 / 1.72 158 / 160 0.187 / 1.63
Vulkan 157 / 160 0.187 / 1.60 156 / 160 0.189 / 1.70
Devstral Small 2 24B Q4_K_M (mistral3) OpenCL 159 / 160 (158) 0.095 / 4.01 157 / 160 0.093 / 2.46
Vulkan 158 / 160 0.090 / 4.57 158 / 160 0.085 / 2.89
DeepSeek-Coder-1.3B-Instruct Q4_K_M (llama) OpenCL 159 / 160 (159) 0.581 / 6.13 158 / 160 0.626 / 7.42
Vulkan 157 / 160 0.534 / 6.04 156 / 160 0.601 / 7.33
DeepSeek-Coder-6.7B-Instruct Q4_K_M (llama) OpenCL 157 / 160 (159) 0.136 / 3.73 156 / 160 0.129 / 1.55
Vulkan 159 / 160 0.087 / 1.28 158 / 160 0.087 / 1.91

The devices are as far from llama.cpp as the CPU is (its means on the seven files: 0.128, 0.190, 0.237, 0.186, 0.073, 0.557 and 0.089), and the choices that differ are the near ties the CPU's own differ on; the largest differences are single logits far below a step's best one (Tini on OpenCL at step 25: 36.6 where llama.cpp has 29.3 under a best logit of 47.4). Cyber-Prime is the widest: 149 to 152 of 160 on a device where the CPU has 154. Its records block by block (an observed CPU context beside a device one, the five prompts) show no block that parts: the stream's norms differ by 1.5 % at the median of the 160 steps, growing along the blocks. The choices that differ from the CPU's or from llama.cpp's (12 steps on Vulkan, 11 on OpenCL) are ties: the best logit is 0.001 to 0.78 above the second on every side at each of them, where the logits of those steps differ by 0.07 to 1.07 on average. Vulkan's 7.5 is step 16 of the chat prompt, after llama.cpp's token 124895 seven times in a row, where the three sides give three tokens with best logits of 0.39 to 0.55: the difference grows from 1 % at block 2 to 29 % at block 23 without a step between two blocks (OpenCL is 1.04 at most there). Two sequences decoded together give the plain device context's choices on their first 8 steps with a paged cache (15 or 16 of 16 for every file on both backends), with the first half of the blocks on the CPU (14 of 16 for Cyber-Prime, 15 or 16 for the others) and, for Tini, with its routed experts in host memory (15 and 16 of 16).

Prompt and decode, tokens a second (bench_llama, 512 and 128 tokens, the best of two; Devstral on the CPU 128 and 32 tokens, one run):

file CPU OpenCL Vulkan
LFM2-350M 2480 / 123.0 29134 / 631.2 38828 / 1007.9
Cyber-Prime 1.1 2.6B 310.3 / 16.9 6626 / 222.0 8156 / 271.3
Tini-Cybersec-8B-A1B 451.2 / 25.6 7265 / 250.9 7720 / 341.1
StarCoder2-3B 202.2 / 14.3 4838 / 139.4 4952 / 203.0
Devstral Small 2 24B (14.3 GB, whole on the 20 GB device) 29.2 / 2.08 1017 / 38.8 1026 / 44.2
DeepSeek-Coder-1.3B 538.5 / 33.2 10845 / 341.1 11479 / 446.0
DeepSeek-Coder-6.7B 109.1 / 7.10 3083 / 111.2 2961 / 135.3

Tini's CPU rates were measured while a file was being downloaded. Vulkan's prompt rate on Tini was 2523, a third of OpenCL's, until the grouped expert products of a prompt went to the prompt products' tiles (vulkan.md); its row is that session's (OpenCL 6895 to 6958 / 254.8 beside it).

Limits

  • Architectures (per family details and what was checked in compatibility.md): llama, mistral, mistral3 (YaRN with the log multiplier, an attention temperature by position), qwen2, qwen3, gemma, gemma2, gemma3, phi3, phi2, granite, granitemoe (embedding, attention, residual and logit scales), qwen2moe, qwen3moe (MoE: softmax router, top-k experts, optional renormalization and sigmoid-gated shared expert), deepseek (dense leading layers, scaled routing, shared experts), deepseek2 (DeepSeek V2 / V3 and GLM-4.7-Flash: multi-head latent attention in llama.cpp's absorbed form, YaRN, sigmoid routing with a selection bias, expert groups), glm4 (GLM-4-0414: norms after attention and the FFN too), glm4moe (GLM-4.5 to 4.7: DeepSeek V3's MoE with QKV biases and partial rotation), olmoe (Q/K norms over the whole projections, 64 experts, weights not renormalized), gpt2, starcoder2 (GPT-2's block with separate projections, grouped keys and values, rotary positions), lfm2 and lfm2moe (short convolution blocks with a state per sequence between attention blocks; sigmoid routing with a selection bias for the MoE), gpt-oss (attention sinks, a window on every second block, experts with biases and a clamped activation), spark2_5 and muse-glimmer (a gate on the attention output, a window on three blocks out of four, a count of rotated dimensions per kind of block or no rotation outside the window), mimo2 (values narrower than the keys, the window and the key heads block by block; on the CPU, checked on a tiny model only). Mixtral loads as llama. MoE layers run on either backend, routing included (moe.md).
  • Against llama.cpp on the same machine, the CPU's prompts are ahead (1.02 to 1.94 times its rate) and its decoding level or ahead from four threads (1.00 to 1.09); the GPU rows still behind are in benchmarks.md.
  • Sequences of a context have fixed, equal shares of the cache (no cells shared between sequences: a common prefix is copied).
  • KV cache types are fixed for the life of a context (ADR-0005).