Inference
How OKLM runs a decoder-only transformer from a GGUF file: loading
and binding the weights, the context and its KV cache, the Llama-family
and GPT-2 forward passes, sampling, chat templates, the
oklm run, oklm chat and oklm ppl
commands, and how the results are verified. Mixture-of-experts models
(the expert products, the expert cache, --n-cpu-moe, the
device expert tier) are in moe.md; the memory
projection of oklm plan in memory-plan.md.
Status: maintained.
API
#include "oklm/model/model.h"
#include "oklm/model/sampling.h"
oklm_model* m;
oklm_context* c;
oklm_context_params cp = oklm_context_default_params(); /* n_ctx 0: training context, max 4096 */
oklm_model_load("SmolLM2-135M-Instruct-Q8_0.gguf", &m); /* maps the file, binds every tensor */
oklm_context_new(m, &cp, &c); /* KV cache + buffers */
oklm_eval(c, prompt_ids, n, 0); /* appends n tokens to the cache */
const float* logits = oklm_get_logits(c, -1); /* n_vocab values of the last position */oklm_model_loadmaps the GGUF file (weights are used in place, never copied), reads the hyperparameters, builds the tokenizer when the file hastokenizer.ggml.*keys, and binds every tensor by name with a shape check. Norm weights and biases must be F32; matrices may be any type that has a dot product (F32, F16, BF16, the legacy quants, the K-quants).- A context owns the KV cache (
kv_typeF16 by default, or F32), the activation buffers forn_batchpositions and a scratch arena. Several contexts can share one model. The cache has rows for the blocks that attend: a block without attention (the linear blocks ofqwen35andqwen35moe, the convolution blocks oflfm2andlfm2moe) reserves none, on the CPU or on a device (hybrid-layers-design.md). oklm_evalappends tokens at positionoklm_n_past(), splitting them into batches ofn_batch. Withall_logits(only forn <= n_batch) every position's logits stay readable withoklm_get_logits(c, i); otherwise only the last one is computed. Tokens beyondn_ctxgiveOKLM_ERR_SHAPE, out-of-range idsOKLM_ERR_INVALID.oklm_kv_truncate(c, n)forgets everything after the firstnpositions, which is howoklm chatre-uses the common prefix of two turns and how a caller rewinds.- Several sequences: with
cp.n_seq = Sthe cache holdsn_ctxcells for each ofSindependent sequences.oklm_eval_batch(c, items, k)runs the new tokens of up toSsequences (oklm_seq_tokens: sequence, tokens, count, all logits or the last one;n_batchtokens in all) in one forward pass: the matrix products read each weight once for every token, attention keeps each sequence to its own cells.oklm_get_logits_item(c, item, i)reads the logits;oklm_seq_n_past,oklm_seq_truncateandoklm_seq_copy(a shared prompt evaluated once, then copied) manage the sequences;oklm_eval,oklm_n_pastandoklm_kv_truncateact on sequence 0. The logits equal those of one context per sequence (bit for bit on the CPU and in the tests on OpenCL). Decode of Llama-3.2-1B Q4_K_M on 8 CPU threads: 33 tokens/s for one sequence, 200 for 8 together, 290 to 340 for 16 to 32 (tests/model/bench_batch.c). - A sequence's cache saved to a file and loaded back:
oklm_kv_save()andoklm_kv_load(), in KV files below.
KV files
oklm_kv_save(c, seq, path, tokens, n, flags, &info)
writes the cached state of a sequence's first n positions;
oklm_kv_load(c, seq, path, tokens, n, &loaded) empties
a sequence and gives it the rows of the longest prefix the file and
tokens share (everything with tokens NULL).
The caller then evaluates the tokens after loaded, as after
a truncation. The context does not keep token ids, so the caller passes
them (oklm_kv_file_tokens() reads them back from a file).
The next token's logits after a load are those of the context that
saved, bit for bit (tests/model/test_kv_file.c: one
sequence, the second of two, paged to contiguous and back, F16 and F32,
MLA, a hybrid stack).
A sequence that holds media rows (the embeddings of an image, a sound
or a video) is saved and loaded with oklm_kv_save_media()
and oklm_kv_load_media(), which take, beside the tokens,
the rows' media hashes as oklm_eval_batch() takes them
(oklm_seq_tokens.media: 0 for a token, else the item's
content hash, the token then being the row's index within its item);
oklm_kv_file_rows() reads both back. The file keeps the
hashes and, per row, the positions the context gave it: for a model with
three-part rotary positions (Qwen2-VL, Qwen2.5-VL: multimodal.md) an
image's rows share a time position and the token after an image does not
take the position of its row index, so the loading context is given what
the saving one held, the time position of each row and the position a
token after it takes. A load may stop inside an image (the prefix wanted
ends there): the image is continued with
oklm_eval_embd_grid() from its next row, as after a cut. A
paged cache that loads the file records each row by its token or by its
hash, so the full blocks are found again by a request with the same
media. oklm_kv_save() on such a sequence keeps the
positions too; it has no hashes to give, and its caller passes -1 as the
token of a row given as an embedding, which a paged cache never takes
for a cached prefix. Checked on the tiny qwen2vl model
(test_kv_file: saved after an image, after a video, after
an image of one column, inside them and in the text, loaded contiguous
and paged; test_model's test_mrope: 100 saves
over its 14 cases, the logits of the context that never left memory bit
for bit on the CPU, and the reference's), on models whose image rows
take one position each (plain embedding rows on the tiny llama model,
rows that see each other on the tiny gemma3 one), and once on the
published Qwen2-VL 2B (multimodal.md).
A file whose hashes check and whose rows say positions no sequence has
(a time position below zero, a position after a row that is not past the
row's own, or, for a model whose rows take one position each, anything
but the row's index and the next) is refused before a row is read:
test_kv_file writes such files, the chunk's hash made that
of the changed payload, and a changed file no rule forbids, which
loads.
Format (little-endian, versions 1 and 2,
src/model/kv_file.c):
| part | content |
|---|---|
| header | OKLMKVC\0, version, header size; the model's identity
(oklm_model_id(): a hash of the architecture, the sizes,
every tensor's type and shape and the first bytes of its data; a tensor
added to the layers since is counted only in the models that have it, so
the identities written before it stay valid); the sequence's adapter;
n_ctx, block size, rotation groups of a quantized cache,
MLA, the layer range, the recurrent state's size; per layer its key and
value types and row bytes; a hash of all that |
| chunk | head (magic, first position p0, count n,
payload bytes); payload: the n token ids; in version 2 the
n media hashes (64 bits each), the n time
positions and the n positions a token after each row takes
(32 bits each); then per layer its n key rows and its
n value rows (MLA: the compressed latent and rope rows
only; a hybrid stack's linear blocks have none), then the recurrent
state of each linear block; trailer (magic, positions held after it, the
payload's hash, the head's offset) |
Versions. Version 1 keeps a token id a row: every
row is a token at the position of its index. Version 2 adds 16 bytes a
row (the media hash and the two positions) and says so in the header
(the word that was padding is 1). A sequence is written as version 2
only when it has what version 1 cannot say, a row with a media hash or a
position that is not its index; a sequence of tokens alone, of a
qwen2vl model too, is written as version 1, byte for byte
what the builds before version 2 wrote, so they read it. Both versions
are read; a version 1 file's rows load as tokens. A build from before
version 2 refuses a version 2 file as not being a KV file
(OKLM_ERR_FORMAT: it would otherwise read the rows at the
wrong positions); this build refuses a version after 2 with
OKLM_ERR_UNSUPPORTED and a log line naming the version it
found and those it reads. The positions are under the chunk's hash like
the rest of the payload: a cut or changed row section is a chunk that
does not check. A file whose hashes check but whose positions no
sequence can have (a time position below zero or not below the position
after the row, positions going back, or, for a model with one-part
positions, a row away from its index) is OKLM_ERR_FORMAT.
An append to a version 1 file by a sequence that now holds an image
writes the file whole as version 2.
oklm serve --prefix-dir DIR uses these files for the
server's prefix cache: one file per distinct head, reloaded at start (server.md), heads holding
images included. oklm describe --kv-save FILE and
--kv-load FILE keep a conversation about an image in one
(multimodal.md).
A chunk replaces what the file held from its p0 on. The
rows are the cache's own bytes, so a file loads into a context whose
cache has the same types per layer, contiguous or paged (a paged cache
takes blocks for them and records their tokens: their full blocks become
cached prefixes). Loading refuses another model, another cache type or
row size, other rotation groups, another adapter and, for a model with
LongRoPE factors, another context length (OKLM_ERR_SHAPE);
a file without an intact chunk is OKLM_ERR_FORMAT. On any
error the sequence is left empty. A hybrid stack's recurrent state is
that of the file's end: it loads whole or not at all. A device cache is
read back through the host (float types; a quantized device cache is
OKLM_ERR_UNSUPPORTED).
Crash safety. Written whole, the file goes to
path.tmp, is synced and renamed over path
(oklm_file_replace(), MoveFileEx on Windows)
and its directory synced: a crash leaves the previous file. With
OKLM_KV_SAVE_APPEND, a compatible file is extended instead:
one chunk with the rows from the first position where the file's tokens
and the new ones differ (the new turn's tokens; after a divergence, from
where it diverged), then synced. A reader walks the chunks while each
trailer checks and its payload hashes right, so a crash in the middle of
an append leaves the state of the previous save; the next append cuts
the torn bytes and goes on from there. Saved again unchanged, nothing is
written. Compaction: when the rows the file holds but
no longer uses (a diverged history, superseded recurrent states) would
outweigh those it uses, or the conversation diverged at its start, the
file is written whole instead. An append costs the new rows; a load
reads the file once to check it and once for the rows.
The size is the cache's: per token, every layer's key and value rows (23 KB for SmolLM2-135M in F16, about 128 KB for an 8B model with 8 KV heads). A block without attention has no rows in the file, as it has none in the context: its entry of the header's table is zero. That was so before a context stopped reserving rows for those blocks, so the files written then load unchanged (checked on LFM2-350M and Qwen3.5-9B: a conversation saved by the earlier build, continued by the later one, gives the same answer and the same file as the earlier build continuing it).
Backends
A context runs on the CPU (the default of
oklm_context_default_params()), on OpenCL
(cp.backend = "opencl", or "auto") or on
Vulkan (cp.backend = "vulkan", see vulkan.md); the oklm commands use
auto and accept --backend,
--gpu-stream B (see opencl.md)
and --gpu-layers N (the last N layers and the head on the
device, the rest on the CPU; by default as many as fit in the device
memory). On OpenCL the weights are copied to the device once per model,
the KV cache and buffers live there, and quantized matrix products use
float activations, so their logits are closer to the float64 reference
than the CPU's. See opencl.md. Weights and
caches larger than the device's largest allocation (4 GiB on RADV) are
split over several allocations (OKLM_GPU_MAX_ALLOC_MB sets
a smaller limit).
Hugging Face checkpoints
oklm_model_load() also takes a Hugging Face checkpoint:
a directory with config.json and safetensors weights (one
file or sharded with an index), or the path of its
model.safetensors, index or config.json.
Nothing is converted on disk:
- the hyperparameters come from
config.json, the tokenizer and chat template fromtokenizer.jsonandtokenizer_config.jsonwhen present; - F32, F16 and BF16 matrices are used in place from the mapping (the
BF16 dot product has an AVX2 kernel); FP8 matrices (float8_e4m3fn /
e5m2, with
<name>_scale_invblock scales as in DeepSeek-V3 and transformers' "fp8", or a per-tensor / per-row<name>_scale) are converted to BF16 with their scales at load time; other stored types must be converted withoklm convertfirst; - Q and K rows keep Hugging Face's half-split rotary layout and are rotated in that layout (NeoX) instead of being permuted as GGUF Llama weights are;
- copies are made only where the forward pass needs another layout:
norm and bias vectors in F32, GPT-2's Conv1D matrices transposed, the
Llama 3 rope factors computed from
rope_scaling.
oklm run -m models/smollm2-135m-hf ... therefore runs
the BF16 checkpoint as published. The tiny llama, qwen2 and gpt2
checkpoints of tests/fixtures/hf give the logits of their
own GGUF conversion (within 1e-6 for llama, where only the rotation
order differs; bit-identical for the others), and the SmolLM2 checkpoint
gives llama.cpp's greedy continuations on every prompt, like the
published F16 GGUF.
Llama family
llama, mistral, mistral3,
qwen2 and qwen3 share one forward pass
(src/model/arch_llama.c). For each new position and each
layer:
- RMSNorm of the residual, times
attn_norm; - Q, K, V projections (plus
attn_*.biasfor Qwen2); - Qwen3 only: RMSNorm of each head of Q and K
(
attn_q_norm,attn_k_norm); - rotary positions on the first
rope.dimension_countvalues of each head: adjacent pairs for GGUFllama/mistral(their Q/K rows were permuted at conversion), first and second halves (NeoX) for Qwen; frequency scaling: linear (rope.scaling.type linear, or no type and a factor other than 1 underrope.scaling.factoror, in files of 2023,rope.scale_linear: llama.cpp's default), YaRN, or the Llama 3 per-pair factors (rope_freqs.weight);mistral3then multiplies the rotated queries by its attention temperature, 1 +attention.temperature_scalex ln(floor(position / original context) + 1), which is 1 below the YaRN original context; - K and V written to the cache at their positions;
- attention of every query head over all cached positions up to its
own, grouped-query (each KV head serves
n_head / n_head_kvquery heads), softmax in F32 (denominator in double); on the CPU (AVX2), queries are computed in tiles of up to 32 rows (tokens x heads of a group) against blocks of 64 keys converted to float once (block-quant caches dequantized, the query rounded through the activation type of their dot product), each row's sums in a fixed order whatever the tiling or thread count; with fewer tiles than threads (decode, small batches) the same arithmetic is split by key blocks, rows and value columns, so a batch gives exactly the logits of its tokens one by one; - output projection, added to the residual;
- RMSNorm times
ffn_norm, SwiGLU (down(silu(gate(x)) * up(x))), added to the residual.
Then a final RMSNorm (output_norm) and the output head
(output.weight, or the token embedding when the file ties
them).
GPT-2
gpt2 (src/model/arch_gpt2.c): token plus
learned position embeddings, pre-LayerNorm blocks with biases
everywhere, one fused QKV matrix whose three thirds are used as views,
multi-head attention, a GELU (tanh form) MLP, a final LayerNorm and the
tied head. The context is capped at the training context because
positions are learned.
phi2 and starcoder2 run the same function
with rotary positions and no position table. Phi-2's block is parallel
(the MLP reads the attention's normalized input). StarCoder2 has
separate query, key and value projections with their biases, keys and
values of attention.head_count_kv heads shared by the query
heads (grouped-query attention, as in the llama family) and the NeoX
rotation of whole heads at rope.freq_base; the rest is
GPT-2's block.
LFM2
lfm2 and lfm2moe (Liquid's LFM2 and LFM2.5,
src/model/arch_lfm2.c for what the llama family's forward
pass lacks) mix two kinds of blocks. The file says which is which:
attention.head_count_kv is an array with one entry per
block, 0 for a short convolution block (the block then
has the three shortconv.* tensors) and the key / value head
count for an attention block. LFM2-350M has 16 blocks,
10 of them convolutions (0 0 8 0 0 8 0 0 8 0 8 0 8 0 8 0);
the 2.6B has 30, 20 of them convolutions. OKLM checks the array against
the tensors and wants one head count for all the attention blocks.
Both kinds start from the same norm, RMSNorm times
attn_norm (the "operator norm" of the model's own code),
with attention.layer_norm_rms_epsilon, and end with the
same feed-forward.
- Attention block: the llama family's, without
biases:
attn_q,attn_k,attn_v, grouped keys and values (32 query heads over 8 in the 2.6B), an RMSNorm of each head of Q and K (attn_q_norm,attn_k_norm, one weight vector ofkey_lengthshared by the heads) before the rotation, the NeoX rotation of whole heads atrope.freq_base(1e6 for the 350M, 1e7 for the 2.6B, 5e6 for the 8B MoE) with no scaling, scores times 1 / sqrt(head size),attn_output. - Convolution block:
shortconv.in_proj[n_embd, 3 n_embd] gives three vectors of n_embd, in this order: a gateb, a scalecand a valuex. The productu = b * xgoes through a causal depthwise convolution over the tokens:y(t) = sum over j < L of conv[j] * u(t - (L - 1) + j)per channel, withL = shortconv.l_cachetaps (3 in every published file), no bias and no activation; the taps before a sequence's first token read zeros (the padding is on the left only: the block never sees a later token).shortconv.convis stored [L, n_embd], each channel's taps together; OKLM keeps an F32 copy [n_embd, L] for its convolution op (oklm_ssm_conv(), the one of the Gated DeltaNet blocks). The block's output isshortconv.out_proj (c * y), added to the residual. - Feed-forward after either: RMSNorm times
ffn_norm, thendown(silu(gate(x)) * up(x))without biases; inlfm2moe, afterleading_dense_block_countdense blocks (2), routed experts instead (sigmoid router, a selection bias, renormalized weights: DeepSeek V3's form without groups or shared experts).
The final norm is RMSNorm times the tensor the files name
token_embd_norm (llama.cpp's "fix for wrong tensor name":
it is the output norm), and the head is tied to the token embeddings
when the file has no output.weight. The encoder variants
(attention.causal false, decision.block_count)
are refused.
The convolution state. A convolution block writes no
key or value row. What it needs from the past is the last
L - 1 columns of u of its sequence (two
vectors of n_embd per block and sequence: 164 kB a sequence for the
2.6B), kept where the Gated DeltaNet blocks keep their state
(oklm_context::lin_st, in device memory for a device
layer). A batch reads them as the taps before its first token and leaves
the last L - 1 columns of [state, batch] behind, so prompts
in batches and tokens one by one give the same values, bit for bit on
the CPU.
Unlike a recurrent state, this one is a function of a bounded past:
the states at position p depend on the tokens of the
(L - 1) x (convolution blocks) positions before p (40 for
the 2.6B, 20 for the 350M) and on the key / value rows of the positions
before them. So when a sequence does not continue where its states were
left, OKLM rebuilds them (model_shortconv_sync()): it runs
those tokens again from zero states, with the cache frozen: the
attention blocks read the cached rows up to each position and write
nothing (model_replay()), so a block whose input is not
right yet (the first L - 1 positions of the replay for the
first convolution block, L - 1 more for each one above)
cannot spoil the rows the positions after it read. The context keeps
each sequence's token ids for that. This is what each cache operation
does with the state:
| Operation | Convolution states |
|---|---|
| a batch that continues the sequence | carried on |
oklm_seq_truncate() / oklm_kv_truncate()
to position 0 |
zeroed by the next batch |
| a truncation elsewhere (a prefix reused, a draft rejected) | rebuilt by the next batch: (L - 1) x blocks tokens run
again |
oklm_seq_copy() |
the destination's are rebuilt by its next batch |
the paged cache (blocks shared between sequences,
oklm_seq_attach_prefix(), a block copied on write) |
the rows are shared, the states are per sequence and rebuilt after an attach |
oklm_seq_swap_in() |
rebuilt |
oklm_kv_save() / oklm_kv_load() (the
prefix cache on disk, a context resized through a file) |
the file stores them as it stores a recurrent state; the reader rebuilds them from the file's tokens, so a prefix of the file can be used too |
rows given as embeddings (oklm_eval_embd()) |
carried on; they cannot be rebuilt, so a truncation into or just
after them makes the next batch fail (OKLM_ERR_UNSUPPORTED)
rather than run on wrong states |
The cost of a rebuild is that of a prompt of
(L - 1) x blocks tokens. llama.cpp keeps the state in its
recurrent memory and, for a rollback of a few tokens, snapshots of it
(n_rs_seq); a longer rollback evaluates the prompt
again.
The Gated DeltaNet stacks (qwen35,
qwen3next) are the other kind of hybrid: their state is
recurrent and does not go back. Their contexts keep states instead (8 by
default, shared by the sequences and found by a hash of the tokens
before them) and journal what each recurrence read at a sequence's last
96 positions: a cut sequence starts again from the nearest kept state,
through the journal alone when it covers the gap (a rejected draft, a
cut of a few tokens), else through a forward pass over the tokens
between, which reads the attention rows the cut left (hybrid-layers-design.md).
With the linear blocks on a device the states are kept the same way
(read back to host memory) and no journal is written: a cut is that
forward pass, from the nearest kept state (on a
device). The paged cache's shared prefixes work the same way; the
swap stays off. An LFM2 context needs none of it: it keeps both and pays
the short replay.
The cache. The convolution blocks keep no key nor value row: a context reserves rows for the attention blocks only (6 of LFM2-350M's 16 blocks, 8 of the 2.6B's 30), on the CPU, in the paged cache and on a device. Cyber-Prime 1.1 2.6B at 32768 tokens takes 512 MiB of cache where it took 1.88 GiB (the figures).
Chat form. ChatML turns
(<|im_start|>role, <|im_end|>)
after <|startoftext|>; the pre-tokenizer
lfm2 is llama 3's (the tokenizer adds the BOS). LFM2-350M's
template is one line of ChatML and knows no tools. The 2.6B tunes carry
Liquid's full template: the tools as a JSON list in the system turn
(List of tools: [...]), a call written by the model as a
Python call list between two special tokens,
<|tool_call_start|>[get_weather(city='Paris')]<|tool_call_end|>,
results as tool turns, and the reply opened on
<think> (the reasoning of earlier turns is dropped
unless preserve_thinking is set). OKLM renders it (the
generation tag, macros, raise_exception); the
reasoning up to </think> goes to the reasoning field
as for every template that opens the reply on
<think>. The Python call list is not parsed into tool
calls yet.
On OpenCL and Vulkan. A context runs on a device,
every block or the last ones: the convolution blocks through the device
ops of the linear blocks, the experts of lfm2moe through
the llama family's. LFM2-350M, Cyber-Prime 1.1 2.6B and
Tini-Cybersec-8B-A1B run there against llama.cpp and the CPU, with a
paged cache, half of the blocks on the CPU and Tini's experts in host
memory (the figures).
gpt-oss
gpt-oss (gpt-oss-20b and 120b; llama.cpp's graph calls
it "openai-moe"; src/model/arch_gptoss.c for what the llama
family's forward pass lacks). Every block is attention, then routed
experts; there is no dense feed-forward. Audited block by block against
llama.cpp:
- RMSNorm times
attn_norm(attention.layer_norm_rms_epsilon). attn_q,attn_k,attn_vwith biases. The attention is wider than the residual stream: 64 query heads ofattention.key_length64 (4096 values for a width of 2880) over 8 key / value heads.- The NeoX rotation of whole heads (
rope.freq_base150 000) with YaRN (rope.scaling.factor32 overoriginal_context_length4096, boundaries 32 / 1, the magnitude 1 + 0.1 ln 32 on queries and keys, no log multiplier), the same for every block. - Attention with scores times 1 / sqrt(64). Even
blocks see the last
attention.sliding_window(128) positions, odd blocks every position (llama.cpp's pattern of 2 for this name; a file may giveattention.sliding_window_pattern). The key / value cache keeps every position for every block: the window is applied when reading. - Attention sinks:
attn_sinksholds one score per query head. It joins the softmax of every query of that head as one more term, as it is (not scaled by 1 / sqrt(64)), inside or outside the window, and brings no value: the weights areexp(s_j) / (sum of exp(s) + exp(sink)), so a head can attend to nothing. The CPU attention op takes the sinks (oklm_attention_params::sinks) on its three paths (prompt tiles, the split decode path, the scalar pairs); the OpenCL and Vulkan attention kernels take them from a device tensor (below). attn_outputwith its bias, added to the residual.- RMSNorm times the tensor stored as
post_attention_norm(the norm before the experts, under the name llama.cpp's converter gave it). - The router:
ffn_gate_inptimes the normalized input plusffn_gate_inp.bias. Theexpert_used_count(4) best of theexpert_count(32) scores choose the experts, and their weights are the softmax of those 4 scores (not of all 32, and no renormalization or scale after). - Each chosen expert: gate and up products plus the expert's
rows of
ffn_gate_exps.biasandffn_up_exps.bias, the activationmin(g, 7) x sigmoid(1.702 x min(g, 7)) x (clamp(u, -7, 7) + 1)(the sigmoid form of GELU on the gate, the up value plus one, both clamped), the down product plusffn_down_exps.bias; the four results summed with their weights, in rank order, and added to the residual.
Then RMSNorm times output_norm and an untied head
(output.weight).
Against OKLM's other mixtures of experts: qwen3moe,
granitemoe and olmoe take the softmax of all
the scores and pick the best probabilities (the first two then
renormalize the chosen ones, which gives the same weights as gpt-oss's
softmax of the chosen scores, the third does not);
deepseek2 and glm4moe use a sigmoid and a
selection bias. None has a router bias, expert biases or this
activation, so gpt-oss has its own function
(model_gptoss_ffn()): the products through the same ops
(oklm_matmul_id() reading the mapped experts), the biases
and the activation as loops over host memory for a CPU layer, as ops for
a device layer (below).
In the published Q4_K_M file the experts are MXFP4 (GGUF type 39): 32 values per block of 17 bytes, a shared power-of-two scale (one E8M0 byte: 2^(e - 127), stored halved because the value table is doubled) and 16 bytes of 4-bit codes, the low nibble of byte j for value j and the high one for value j + 16, each code one of {0, 0.5, 1, 1.5, 2, 3, 4, 6} with a sign bit (E2M1). OKLM has the type with its dot product against Q8_0 activations (core.md, blocks and dequantized values identical to ggml's); the rest of the file is Q5_0, Q8_0 and Q4_K.
Chat form (harmony). The template (17 kB: macros
that write the tools as a TypeScript namespace,
strftime_now, raise_exception) renders a
system turn (identity, knowledge cutoff, the date,
Reasoning: medium, the valid channels), an optional
developer turn (the caller's system prompt as instructions, the tools),
then <|start|>user<|message|>...<|end|>
turns and <|start|>assistant. The model answers in
channels, each a message of its own:
<|channel|>analysis<|message|> reasoning
<|end|>, then
<|start|>assistant<|channel|>final<|message|>
the answer <|return|>; a tool call is
<|channel|>commentary to=functions.NAME <|constrain|>json<|message|>{...}<|call|>,
its result a
<|start|>functions.NAME to=assistant<|channel|>commentary<|message|>
turn.
OKLM ends a reply at the first control token it samples. For a
vocabulary that has all the harmony tokens, the ones inside a reply
(<|start|>, <|end|>,
<|message|>, <|channel|>,
<|constrain|>) are therefore read as added tokens
that are not special (harmony_tokens() in
src/tok/tok_gguf.c): they are sampled like text and written
as their own text, and the reply ends at <|return|>
or <|call|>, which stay control tokens. The text of a
reply is then the channels' markup as the model wrote it, and two
readers take it apart:
- the reasoning splitter (
src/harness/think.c, the one of<think>) takes the analysis channel's message as reasoning and drops the final channel's header, so/v1/messagesanswers with athinkingblock and atextblock as for the other thinking models, and/v1/chat/completionswithreasoning_contentbesidecontent(server.md); - the tool-call parser (
src/model/tool_calls.c, formatharmony, recognized from the template) takes a message addressedto=functions.NAME, the address before or after the channel, and its JSON arguments, and removes that message from the text. With tools, the call grammar starts atto=functions.: a tool's name, the end of the header, the arguments its parameters describe.
oklm chat shows the reply split the same way: in the
full-screen UI (--tui) the analysis is folded or dimmed as
other models' reasoning is; on standard output it is dimmed on a
terminal and written between <think> and
</think> when the output is a file or a pipe, then
the answer, without the channels' markup. The conversation keeps the
answer alone (the template writes the final channel's header again, and
the format drops the analysis of earlier turns).
/v1/completions and oklm run, which complete a
prompt, give the text as the model wrote it:
<|channel|>analysis<|message|>...<|end|><|start|>assistant<|channel|>final<|message|>....
On OpenCL and Vulkan. A context runs on a device,
every block or the last ones (--gpu-layers), since
2026-11-14; a device layer reads nothing back. What the devices lacked,
and how each part is done:
- The sinks are an F32 tensor of one value per query
head, copied to the device with its layer and given to the attention op,
which reads them where it runs. On OpenCL the partial results (maximum,
sum, weighted values) of an op with sinks always go through
attention_merge, which adds each head's sink to its sum as one more term: the per-token kernel splits a query's keys in two ranges at least, the tiled prompt kernels write partials even for one range. On Vulkanattn_splitandattn_tileadd the sink where a row's output is normalized (one range, or where the ranges are combined). Both take the maximum over the scores and the sink, so a head whose sink dominates gives an output near zero, not a division by a small sum. Heads wider than 256 on Vulkan (the hybrid group's shader) have no sinks: there the op runs on the CPU, the sinks read back with its other operands (no model has both). - The router: the scores plus their bias (a broadcast
add), the expert choice and the chosen scores
(
oklm_moe_select(),oklm_moe_weights()without normalization), then a softmax over the 4 values of each token: existing ops, no kernel of its own. - The experts' biases: each (token, rank) pair's row
is gathered from the layer's table [width, 32] by the chosen expert
(
oklm_get_rows()on the pairs' indices) and added to the product's rows: two launches per bias instead of an indexed add, three biases a block. The gathered rows use the buffer of the shared expert's output, free in this architecture and as long as the model's width, so a model whose experts are wider inside than the model runs on the CPU (a notice; both published sizes have 2880 for both). - The activation is a new element-wise op,
oklm_sigglu()(OKLM_OP_SIGGLU:g x sigmoid(1.702 g) x (clamp(u, -c, c) + 1)withg = min(gate, c)), on the CPU, in OpenCL'sewkernel and in Vulkan'sewshader. The CPU layers keep their own loop (the same expression), so the CPU's results are unchanged. - The products are the MXFP4 kernels both backends had.
The routed experts are computed where their layer runs: a context
asked with n_cpu_moe, an expert cache or a device expert
tier is created without them, with a notice (the first batch failed
before). A file larger than the device therefore has the layer split
alone.
test_gptoss_shapes
(tests/core/test_opencl.c,
tests/core/test_vulkan_attention.c; RX 7900 XT and Intel
UHD 750) holds the operands to the CPU kernels at gpt-oss-20b's counts:
attention with sinks of 64 query heads of 64 on 8 key / value heads,
decode and prompt shapes, a window of 128 and none, a first token (one
key and the sink), a Q8_0 cache, heads of 128, 256 and 512, on Vulkan a
paged cache; the gated unit with values on both sides of its clamp; the
gathered biases (the sums bit for bit).
tests/core/test_ops.c holds the CPU's gated unit to a
float64 expression. The fixture (test_gptoss_device) runs
on OpenCL and on each Vulkan device with every block there and with the
last two, F32 and F16 caches, batches, several sequences, a paged cache,
and on OpenCL with the CPU layers streamed for prompts: 36 / 36 choices,
within 2.9e-5 of the float64 reference (0.039 with an F16 cache);
batches against single tokens bit for bit on OpenCL, within 1.6e-5 on
Vulkan (two attention kernels, see the test). The fixture's experts are
16 values wide inside, which the indexed-product kernels do not take (a
multiple of 32): its down product runs on the CPU for these operands on
both backends, and Vulkan does not replay its decode steps; the
published file's products are all on the device.
gpt-oss-20b Q4_K_M, every block on the RX 7900 XT
(tests/model/test_archs_public.c: the 160 steps of the
corpus on llama.cpp's tokens, in a device context and in a CPU one):
| backend | choices equal to llama.cpp's | mean / max logit difference from llama.cpp | choices equal to the CPU's | mean / max from the CPU |
|---|---|---|---|---|
| CPU | 153 / 160 | 0.081 / 3.96 | ||
| OpenCL | 158 / 160 | 0.081 / 4.05 | 155 / 160 | 0.067 / 2.88 |
| Vulkan | 157 / 160 | 0.081 / 4.05 | 156 / 160 | 0.067 / 2.69 |
OpenCL's row is with the experts' prompt products on the float matrix units, the default since 2026-11-14 (opencl.md); on the F32 tiles it was 0.082 / 4.03 and 0.066 / 2.50, the same choices.
Two sequences decoded together give the plain device context's choices on their first 8 steps with a paged cache (16 of 16 on both backends) and with the first 12 of the 24 blocks on the CPU (16 and 15 of 16). Vulkan replays the decode steps (172 of 179 in the run).
Tokens a second (bench_llama: a prompt of 512 tokens in
one batch, 128 tokens one at a time, the best of three runs; one run on
the CPU):
| CPU, prompt / decode | OpenCL | Vulkan | |
|---|---|---|---|
| gpt-oss-20b Q4_K_M (whole on the 20 GB device) | 7.08 / 5.62 | 3323 to 3346 / 129.9 to 130.6 | 3656 / 163.6 |
Vulkan's prompt rate was 844 until the grouped expert products of a prompt went to the prompt products' tiles, and its rates 2149 / 96.3 until the shaders stopped reading MXFP4's values from a constant array (vulkan.md); OpenCL's was 1681 until the experts' prompt products went to the float matrix units (opencl.md), and its decode rate 127 to 128 until the end of a residual branch read its elements in batches (opencl.md: what keeps it below Vulkan's is 25 launches a block). The device rows are of 2026-11-14.
The models without sinks kept their rates (Llama-3.2-1B Q4_K_M, before and after: OpenCL 15 062 / 409.5 and 14 700 to 14 910 / 399 to 409 over three runs, Vulkan 16 247 / 511.2 and 16 210 to 16 520 / 507 to 516; Qwen3-8B on Vulkan 2670 / 111.9 and 2688 / 112.1).
Not done on a device: Vulkan's rates against OpenCL's (half the prompt rate, three quarters of the decode rate: its routing and its gated unit do not fit the fused MoE dispatches, 6 more launches a block for the biases; not looked into), a fused indexed add, a published file on the Intel UHD 750, the 120b (larger than the device: the layer split alone).
gpt-oss: perplexity between the backends
A perplexity of gpt-oss-20b over a thousand tokens is known to about
2 % either way, on one backend as between two
(oklm ppl -c 512, the first 14 000 bytes of
harness.md, 1275 tokens scored, 2026-11-14):
| perplexity | |
|---|---|
llama.cpp (CPU, llama-perplexity -c 512 -b 512) |
62.6978 |
| CPU | 64.3276 |
| CPU, F32 cache | 64.1297 |
| OpenCL | 61.6072 |
OpenCL, float kernels
(OKLM_CL_Q8=0 OKLM_CL_WMMA=0) |
63.0249 |
| Vulkan | 63.2014 |
Vulkan, F32 tiles (OKLM_VK_Q8=0 OKLM_VK_COOPMAT=0) |
62.9504 |
The router keeps 4 experts of 32 on scores whose fourth and fifth are within 0.01 for 6 to 14 % of the tokens of a block, and the backends round a product's activations differently (the CPU to Q8_K for a K-quant, as llama.cpp and OpenCL's 8-bit kernels; the matrix units to F16): block 0's scores are 0.0004 to 0.0033 RMS apart between two backends, block 23's 0.07 to 0.10, and the experts chosen differ in 9 % of the (token, block) pairs, 8 % between the CPU's two cache types. A token's log-probability then differs by 0.08 to 0.11 on average (0.03 to 0.04 with the CPU's choices forced on a device), and over some windows the differences keep one sign. With both devices on float kernels the choices differ in 0.54 % of the pairs and the perplexities by 0.12 %: the kernels agree, the rounding is what parts the rows. The window of 128 is not involved (the differences do not step there, and the attention op handed to the CPU's kernel leaves them). Tini-Cybersec-8B-A1B has the same scatter per token and no drift (50.39 on the CPU, 50.24 on Vulkan, 50.12 on OpenCL). The measurements are in ROADMAP_DONE.md.
Spark-X2.6 and Muse Glimmer
spark2_5 (Spark-X2.6 1.7B and 4B) and
muse-glimmer (Muse Glimmer 30B) are the llama block with a
sliding window on three blocks out of four and a gate on the
attention output: before the output projection, the attention's
result is multiplied by the sigmoid of a projection of the block's
normalized input (attn_gate). The forward pass is the llama
family's, driven by the hyper-parameters;
src/model/arch_gate.c binds the gate and applies it.
Audited against llama.cpp's src/models/spark2-5.cpp and
muse-glimmer.cpp.
spark2_5, block by block:
- RMSNorm times
attn_norm(epsilon 1e-6). - One fused
attn_qkv, rows Q | K | V: 16 query heads ofattention.key_length256 over 4 key / value heads in the 4B (4096 query values for a width of 2560; 8 heads over 2 in the 1.7B). No biases, no Q/K norms. - The NeoX rotation, with a count of dimensions per kind of
block: a window block rotates
rope.dimension_count_swa(256, the whole head) with the baserope.freq_base_swa(10 000); a block without window rotates the firstrope.dimension_count(64: 32 pairs, value i with value i + 32) with the baserope.freq_base(5 000 000) and leaves the other 192 values of each head as they are. The blocks that see the whole context thus turn slowly, over a trained context of 1 048 576 positions. - Attention with scores times 1 / sqrt(256). The blocks named by
attention.sliding_window_pattern, one boolean per block (three window blocks, then a full one, repeated), see the lastattention.sliding_window(512) positions. OKLM reads the list as a period (4) and refuses a list that is not one pattern repeated. - The gate:
attn_gate[n_embd, n_head] times the normalized input of step 1, a sigmoid, and each head's 256 output values times its gate. attn_output, added to the residual.- RMSNorm times
ffn_norm, thenffn_down(gelu(ffn_gate x) * ffn_up x)(the tanh form of GELU, as Gemma), added to the residual.
Then RMSNorm times output_norm and the head
(output.weight; a file without one ties it to the
embeddings). In the published Q4_K_M files the token embeddings and the
gates are BF16 (the "one BF16 tensor a block" of the tensor list is the
gate, a 2560 x 16 matrix kept at full precision), the head and
attn_output Q8_0, attn_qkv and
ffn_down Q6_K, ffn_gate and
ffn_up Q4_K.
muse-glimmer differs from it as follows (its language
side; the vision projector of the published files is another
matter):
- the token embeddings go through an RMS norm without weight before the first block (no scale by sqrt(n_embd));
- separate
attn_q,attn_k,attn_v(32 query heads of 128 over 2 key / value heads), then per-head RMS norms of queries and keys (attn_q_norm,attn_k_norm; the converter folds the model's query scale into the first); - only the window blocks rotate queries and keys (adjacent pairs, one base); a block without window has no position signal at all;
attention.sliding_window_patternis a period (4; llama.cpp's default for this name when the key is absent);- the gate has one value per attention output value
(
attn_gate[n_embd, n_head x head_dim]); - norms after the attention's projection
(
post_attention_norm) and after the feed-forward (post_ffw_norm) as Gemma 2 and 3, with their own epsilon, 1e-8; - the feed-forward is SwiGLU;
- the logits are multiplied by
logit_scale(0.196 in the 30B), then soft-capped byfinal_logit_softcapping(20).
Muse Glimmer's replies. Its chat template (macros,
strftime_now, raise_exception) writes a system
turn when the conversation has none (identity, knowledge cutoff, the
date, Reasoning strength: high, the valid recipients), then
<|start|>user<|message|>...<|eot|> turns
and <|start|>assistant. The model answers in messages
that each name a recipient: to=self<|message|> its
reasoning <|eom|>, then
<|start|>assistant to=user<|message|> the
answer <|eot|>; a tool call is a message to the
tool's name. <|eom|> closes a message another
follows, <|eot|> the turn. The published file types
the four tokens as control tokens, and OKLM ends a reply at the first
control token it samples: the first run of oklm chat on the
file stopped after to=self, at
<|message|>. For a vocabulary that has the four, the
three written inside a reply (<|start|>,
<|message|>, <|eom|>) are
therefore read as added tokens that are not special
(recipient_tokens() in src/tok/tok_gguf.c, as
harmony's are): they are sampled like text and
written as their own text, and the reply ends at
<|eot|>, where llama.cpp ends it too. The reasoning
splitter (src/harness/think.c) takes the message to
self as reasoning and drops the header of the message to
user, so oklm chat shows a thought and an
answer as for the other thinking models, the conversation keeps the
answer alone (the template writes its header again), and the server
answers with reasoning_content or a thinking
block. A message to another recipient stays in the text as the model
wrote it: the template's tool calls
(<atem:function_calls> blocks) have no parser and no
grammar here yet.
On OpenCL and Vulkan. A context of either runs on a
device, every block or the last ones (--gpu-layers), since
2026-11-14. No kernel was missing: the pass is the llama family's
sequence of ops, and the device kernels take what these models add from
their operands.
- The gate:
attn_gateis copied to the device with its layer's other matrices; the product, the sigmoid and the multiplication run there (for one gate per head, a multiplication broadcast over the head's 256 values). - The rotation: its count of dimensions and its base are parameters of each rotation op, so a block without window turns 64 of 256 values with one base and a window block the whole head with another; the values past the count are copied as they are. Spark's Q and K heads are rotated as one tensor (the rows Q | K of the fused projection).
- The window is a parameter of each attention op (as Gemma's); heads of 256 on 4 key / value heads and of 128 on 2 are shapes both backends' attention kernels take, for prompts and for single tokens.
- Muse Glimmer's norms without weight, its per-head norms of queries and keys, its norms with their own epsilon, and the scaled then soft-capped logits are the existing norm and element-wise ops.
test_attn_gate_shapes
(tests/core/test_opencl.c,
tests/core/test_vulkan_attention.c) holds these operand
shapes to the CPU kernels at the published counts, on the RX 7900 XT and
the Intel UHD 750. The fixtures (test_attn_gate_models) run
on OpenCL and on each Vulkan device with every block there and with the
last two (a block without window and a window block), F32 and F16
caches, batches, several sequences, a paged cache, and on OpenCL with
the CPU layers streamed for prompts: 36 / 36 choices, within 1.6e-5
(tiny-spark) and 7.9e-6 (tiny-muse-glimmer) of
the float64 reference with every block on the device (0.011 and 0.006
with an F16 cache); batches against single tokens and several sequences
against one bit for bit on OpenCL, within 9.7e-6 on Vulkan (its prompts
and single tokens are two attention kernels).
The published files, every block on the RX 7900 XT
(tests/model/test_archs_public.c: the 160 steps of the
corpus on llama.cpp's tokens, in a device context and in a CPU one):
| file | backend | choices equal to llama.cpp's | mean / max logit difference from llama.cpp | choices equal to the CPU's | mean / max from the CPU |
|---|---|---|---|---|---|
| Spark-X2.6-1.7B Q4_K_M | OpenCL | 158 / 160 (the CPU 159) | 0.164 / 7.67 | 159 / 160 | 0.155 / 7.35 |
| Vulkan | 160 / 160 | 0.143 / 8.37 | 159 / 160 | 0.155 / 7.83 | |
| Spark-X2.6-4B Q4_K_M | OpenCL | 158 / 160 (the CPU 158) | 0.172 / 2.89 | 158 / 160 | 0.216 / 15.2 |
| Vulkan | 158 / 160 | 0.174 / 3.19 | 158 / 160 | 0.221 / 15.0 | |
| Muse Glimmer 30B UD-IQ2_XXS | OpenCL | 156 / 160 (the CPU 157) | 0.171 / 3.66 | 157 / 160 | 0.163 / 4.36 |
| Vulkan | 157 / 160 | 0.171 / 3.63 | 156 / 160 | 0.163 / 4.36 |
The devices are as far from llama.cpp as the CPU is (its means:
0.156, 0.268 and 0.194). The largest differences are single logits far
below a step's best one (the 1.7B at step 109: -21.1 on OpenCL where
llama.cpp has -28.7 under a best logit of -0.9). The 4B's sensitive
step, where the CPU is 16.2 from llama.cpp, is the step where the
devices are 15 from the CPU: they stay within 3.2 of llama.cpp there.
Neither side's kernels are off at that step (step 15 of the code prompt,
whose continuation is one token repeated): in the CPU's run the
feed-forward parts of blocks 20 to 24 write 1.9 to 12 times what they
write on a device (block 23: 73.0 against 6.0), the stream grows to a
norm of 149 against 38, and block 34 takes it back (a write of 107
against 18), from a stream that enters block 19 within 0.8 % of the
device's; the same excursion, smaller, at steps 12 to 14. A CPU context
with an F32 cache does not make it (its stream at step 15 has the
device's norms within 1 % in blocks 16 to 29), and with one rounding
changed the CPU is as far from llama.cpp at another step: the mean
difference of steps 14 and 15 is 1.43 and 7.12 as it is, 5.93 and 0.45
with OKLM_ATTN_F16=1, 5.63 and 4.90 with
OKLM_Q8_ROUND_EVEN=1. It is a state of the model on that
repeated token which a last place tips into, whoever computes (the CPU
against llama.cpp: below). Two sequences
decoded together give the plain device context's choices on their first
8 steps with a paged cache (16 of 16 for the three files on both
backends) and with the first half of the blocks on the CPU (16, 15 and
15 of 16; 14 of 28, 18 of 36 and 26 of 52 layers on the device), the
misses near ties.
Tokens a second (bench_llama: a prompt of 512 tokens in
one batch, 128 tokens one at a time, the best of three runs; one run for
the 30B on the CPU):
| file | CPU, prompt / decode | OpenCL | Vulkan |
|---|---|---|---|
| Spark-X2.6-1.7B Q4_K_M | 482.4 / 22.6 | 8030 / 215.7 | 8179 / 285.3 |
| Spark-X2.6-4B Q4_K_M | 166.4 / 9.80 | 3469 / 117.9 | 3614 / 144.9 |
| Muse Glimmer 30B UD-IQ2_XXS (whole on the 20 GB device) | 20.8 / 2.41 | 875 / 29.2 | 831 / 36.7 |
Muse Glimmer's row is of 2026-11-14, after OpenCL's prompt products of its 2- and 3-bit matrices went to the float matrix units (253.3 before: opencl.md); its OpenCL decode rate is its products' (26 of a step's 31 ms of kernel time) and 902 launches a step (opencl.md).
Not done on a device: a published file on the Intel UHD 750 (the op tests and the fixtures ran there), a file larger than the device, a comparison past the window of 512 positions.
Verified: the tiny models tiny-spark and
tiny-muse-glimmer (five blocks, a window of 4 positions, 8
and 16 rotated dimensions; see tools/gen_model_fixtures.py)
against the float64 reference (tools/infer_ref.py), 36 / 36
greedy choices, logits within 2.6e-5 and 4.4e-6; batches and several
sequences give the single tokens' logits bit for bit. llama.cpp run on
the same two fixtures (written without a vocabulary,
build/llama_ref) gives the reference's 36 / 36 tokens,
logits within 0.021 and 0.006 (its F16 key / value cache). The published
files are in the table of Against
llama.cpp.
Muse Glimmer 30B, the published file
(unsloth/Muse-Glimmer-30B-GGUF,
Muse-Glimmer-30B-UD-IQ2_XXS.gguf, 10.7 GB, the smallest: 52
blocks at width 6656, 731 tensors, IQ2_XXS, IQ2_S, IQ3_XXS, IQ3_S,
IQ4_XS, Q5_K and Q6_K among them; SHA-256
3e9bd0e6...0875ad9), run on the CPU on 2026-11-13 and
removed. The forward pass written from llama.cpp's graph and checked on
the tiny model needed no change: against llama.cpp at the pinned commit,
5 / 5 prompts tokenized alike (the llama4 pre-tokenizer, a
vocabulary of 202 048), 157 / 160 greedy choices on its tokens, logits
within 5.51, three of the five continuations identical and the others
parting where llama.cpp's two logits are at most 0.10 apart; over 100
greedy tokens, 100 / 100 choices after a plain prompt (logits within
6.88) and 96 / 100 after a chat turn, the text parting at step 14 where
llama.cpp's two logits are 0.10 apart (a file at about two bits a
weight: close choices are common). oklm plan gives 16.75
GiB at 4096 tokens (the 2- and 3-bit matrices repacked for the CPU); 0.5
tokens a second on 6 threads of a machine under a load of 14 to 17, no
figure for a quiet one. What the run showed was in the chat, not in the
layers: the reply stopped after two tokens (see Muse Glimmer's
replies above), and with that fixed oklm chat thinks,
then answers, its first 69 tokens llama.cpp's greedy ones for the same
rendered prompt; a second turn is rendered from the answer alone and
answered. The file was downloaded again on 2026-11-14, run on OpenCL and
Vulkan (the tables above) and removed. Not done: the perception encoder
(mmproj), the template's tool calls, a larger file of the
repository (Q4_K_XL is 15.9 GB).
MiMo-V2
mimo2 (Xiaomi's MiMo-V2 models; the published
MiMo-V2.6-Flash files were looked at) is the llama block with
values narrower than the keys, a sliding window and a
count of key heads set block by block, and routed experts after a first
dense block. The forward pass is the llama family's, driven by the
hyper-parameters; src/model/arch_mimo2.c binds the tensors
and holds the two steps of its own. Audited against llama.cpp's
src/models/mimo2.cpp and against the metadata and tensor
table of the published MiMo-V2.6-Flash-RL file.
Block by block:
xn = rms_norm(x) * attn_norm
q | k | v = attn_qkv xn n_head heads of dk, the block's key heads of dk, as many of dv
q, k = rope(q), rope(k) NeoX pairs on rope.dimension_count values of each head;
a window block with rope.freq_base_swa, the others with rope.freq_base
a = attention(q, K, V) scores / sqrt(dk); a window block sees the last
attention.sliding_window positions and has sinks (attn_sinks)
x += attn_output a * attention.value_scale a: n_head heads of dv
xn = rms_norm(x) * ffn_norm
x += ffn_down (silu(ffn_gate xn) * ffn_up xn) a block without router (the first)
x += sum of w_e * down_e (silu(gate_e xn) * up_e xn) a block with ffn_gate_inp
- dk is
attention.key_lengthand dvattention.value_length(192 and 128 in MiMo-V2.6-Flash). The attention op and the cache take one head size, so a block's values are written into heads of dk whose last dk - dv places are zero: the sums over those places are zero and the first dv places of each output head are the attention over values of dv, exactly; those are whatattn_outputreads. The cache keeps the padded values (192 places a head where 128 would do). attention.sliding_window_patternandattention.head_count_kvare lists, one entry a block. The published files open with a block without window, then repeat five window blocks and one without, and give 4 key heads to a block without window and 8 to a window block. They are read as leading blocks without window (oklm_hparams::n_swa_lead_full), a period, and one count of key heads per kind of block; a file laid out otherwise is refused. Each block's cache rows have its own count of heads, as gemma4's.- The router is as llama.cpp's graph calls it for this architecture,
whatever
expert_gating_funcandexpert_weights_normsay: sigmoid probabilities, the experts chosen on the probabilities plusexp_probs_b, their weights the probabilities divided by their sum. The count of leading dense blocks is not in the file: it is counted from the tensors; a dense block after a routed one is refused.
Not read: separate attn_q, attn_k and
attn_v (llama.cpp takes them too; the published files have
attn_qkv), the multi-token prediction blocks (three, in
files of their own: mtp-*.gguf, refused with a message),
the vision and audio projector. The model's tool calls,
<tool_call><function=NAME><parameter=KEY>VALUE</parameter></function></tool_call>,
are not one of the formats Tool calls
parses.
On a device: none yet. The two steps of its own move activations in host memory, so a context asked on OpenCL or Vulkan says so and runs on the CPU.
Verified: the tiny model tiny-mimo2 (seven blocks, heads
of 16 and values of 8, a window of 4 positions on five blocks, 1 and 2
key heads, sinks, a dense block then 8 experts;
tools/gen_model_fixtures.py) against the float64 reference
(tools/infer_ref.py): 36 / 36 greedy choices, logits within
1.4e-5; within 0.042 with an F16 cache; batches and several sequences on
the contiguous and the paged cache give the single tokens' logits bit
for bit; a sequence saved to a KV file and loaded gives the same logits.
llama.cpp run on the same model (build/llama_ref) gives the
reference's 36 / 36 tokens, logits within 0.062: its key / value cache
is F16 for this architecture whatever is asked, and the same pass with
queries, keys, values and attention weights rounded to F16 is within
1.2e-3 of it. The published MiMo-V2.6-Flash-RL-Q2_K set
(126.4 GB in two parts, the smallest of that repository;
ggml-org/MiMo-V2.6-Flash-RL-GGUF) was not run: the disk
does not take it. Its metadata and its table of 472 tensors, read from
the first megabytes of both parts, load and bind (48 blocks of 4096, 64
heads, 4 and 8 key heads, a window of 128, 256 experts of which 8 a
token, one dense block), oklm plan gives 117.7 GiB of
weights of which 111.6 GiB are experts (8.85 GiB resident with the
experts streamed from disk), and its chat template renders as jinja2
renders it on 12 conversations.
Gemma 4
gemma4 (Gemma 4 12B, 26B-A4B, 31B, and the E2B / E4B
models) is Gemma 3's block with two kinds of block that do not
have the same heads, values that are normalized, a scale on
each block's output, in the 26B-A4B routed experts beside the dense
feed-forward, and in the E2B and E4B an embedding per layer and blocks
that read earlier blocks' keys and values. Audited from the headers of
unsloth/gemma-4-12b-it-GGUF,
gemma-4-26B-A4B-it-GGUF and
gemma-4-E2B-it-GGUF (their first 40 MB by an HTTP range,
the files extended sparse to their size; the E2B and E4B files whole
since) and from llama.cpp's src/models/gemma4.cpp at the
pinned commit. Licence: Apache-2.0 (general.license of the
three files and the repositories' cards).
| 12B | 26B-A4B | E2B | |
|---|---|---|---|
| blocks, width | 48, 3840 | 30, 2816 | 35, 1536 |
| query heads | 16 | 16 | 8 |
| window blocks: head size, key / value heads | 256, 8 | 256, 8 | 256, 1 |
| other blocks: head size, key / value heads | 512, 1 | 512, 2 | 512, 1 |
| window, pattern | 1024, five then one | 1024, five then one | 512, four then one |
| feed-forward | 15360 | 2112, and 128 experts of 704, 8 a token | 6144 then 12288 (a list) |
attn_v in the blocks without window |
no | no | yes |
| embedding per layer, blocks sharing keys and values | no | no | 256 a layer, the last 20 |
| trained context | 262 144 | 262 144 | 131 072 |
What the three share, block by block:
- RMSNorm times
attn_norm(epsilon 1e-6). - Queries, keys and values from
attn_q,attn_k,attn_v. The blocks are named one by one byattention.sliding_window_pattern(a list of booleans). A window block has heads ofattention.key_length_swavalues (256); a block without window has heads ofattention.key_lengthvalues (512) and its own count of key / value heads (attention.head_count_kvis a list). The count of query heads is the same, soattn_qandattn_outputare twice as wide in a block without window (8192 against 4096 in the 12B). In the 12B and the 26B-A4B a block without window has noattn_v: its values are its key projection, taken before the key norm. - Queries and keys through a per-head RMSNorm with a weight
(
attn_q_norm,attn_k_norm, of the block's head size); the values through a per-head RMSNorm without weight. - Rotation in NeoX pairs: a window block rotates
rope.dimension_count_swa(256) dimensions with the baserope.freq_base_swa(10 000); a block without window rotatesrope.dimension_count(512) with the baserope.freq_base(1 000 000), each pair's angle divided byrope_freqs.weight(256 values, one tensor for the model). - Attention with no scaling of the scores (Gemma 3
divides by the square root of the head size) and no soft-cap; a window
block sees the last
attention.sliding_windowpositions. attn_output, RMSNorm timespost_attention_norm, added to the stream.- RMSNorm times
ffn_norm, the GELU-gated feed-forward, RMSNorm timespost_ffw_norm, added to the stream. - The stream multiplied by the block's
layer_output_scale(one value).
Before the first block the embeddings are multiplied by the square
root of the width; after the last, RMSNorm times
output_norm, the head tied to token_embd, and
the logits soft-capped by final_logit_softcapping (30).
The experts of the 26B-A4B. Every block has a router. The dense feed-forward stays and the experts run beside it, each side with its own norms:
- the dense side is step 7's feed-forward, its output through
post_ffw_norm_1; - the router does not read the normalized input: it reads the stream
through an RMSNorm without weight, times 1 / sqrt(width), times a
learned scale per dimension (
ffn_gate_inp.scale), thenffn_gate_inp; softmax over the 128 experts, the 8 best, their probabilities divided by their sum; - the experts read the stream through
pre_ffw_norm_2; each is a GELU-gated feed-forward of 704 whose gate and up rows are stored in one tensor,ffn_gate_up_exps[2816, 1408, 128] (gate rows first), and whose output is multiplied by a scale of its own (ffn_down_exps.scale, 128 values); the weighted sum goes throughpost_ffw_norm_2; - the two sides are added, then
post_ffw_norm, then the stream.
Of OKLM's expert forms this is the softmax router with renormalized weights of Qwen3-MoE, with three things of its own: the dense feed-forward as an unconditional second branch with separate norms (not a shared expert added to the experts' sum), the router's own input, and the joined gate | up tensor with a scale per expert. There is no selection bias, no group, no shared-expert gate.
The E2B and E4B models add three things the others do not have (the E4B: 42 blocks of width 2560, 8 query heads over 2 key / value heads, a window of 512 on five blocks out of six, a feed-forward of 10 240, the same 256 values a layer):
- an embedding per layer.
per_layer_token_embdholds, for each token, 256 values for every block (262 144 rows of 35 x 256 values in the E2B, 2.3 of the file's 4.6 billion parameters: they are looked up, never multiplied). Before the first block, each token's row is multiplied by sqrt(256) and added to a projection of its scaled token embedding (per_layer_model_proj, times 1 / sqrt(width), each block's 256 values through an RMSNorm timesper_layer_proj_norm), the sum times 1 / sqrt(2). Each block then ends, after its feed-forward and before its scale, withx += rms_norm(proj (gelu(inp_gate x) * e)) * post_norm,ebeing its 256 values; - blocks that compute no keys and values
(
attention.shared_kv_layers: the last 20 of the E2B's 35, the last 18 of the E4B's 42). They project their queries only and attend over an earlier block's cache: as llama.cpp, a window block over that of the block two before the first of them, a block without window over that of the block just before it (the last block of each kind that has a cache). The files still carryattn_k,attn_vandattn_k_normfor them, which nothing reads; - in the E2B a feed-forward width that is a list
(
feed_forward_length): 6144 in the first 15 blocks, 12 288 in the 20 that share. A block without window has itsattn_vin these files.
Images. With an image, llama.cpp lets the rows of
the window blocks see each other in both directions and keeps the blocks
without window causal. The projectors (mmproj, 175 MB for
the 12B) are not read.
The chat template (18 922 characters in the 12B and the 26B-A4B files, Unsloth's edition of Google's):
- turns are
<|turn>ROLE\n ... <turn|>\n, the assistant's role beingmodel; the prompt opens with the BOS token; - a
systemturn is written when there is a system message, tools, or thinking:<|think|>on its first line turns the thinking on (enable_thinking, off by default), then the system text, then each tool as<|tool>declaration:NAME{...}<tool|>; - the model's thinking is a channel:
<|channel>thought\n ... <channel|>before the answer. With thinking off the generation prompt ends with an empty one (<|turn>model\n<|channel>thought\n<channel|>), so the model answers directly; past turns' thinking is dropped unlesspreserve_thinkingkeeps it with tool calls; - a tool call is
<|tool_call>call:NAME{key:value,...}<tool_call|>(the arguments in the model's own notation, strings between<|"|>marks), its result<|tool_response>response:NAME{...}<tool_response|>in the same model turn; the template reads the OpenAI messages (an assistant turn'stool_calls, thentoolturns) and writes them so. The E2B and E4B files carry a later edition (18 808 characters): after a tool's result it writes no generation prompt, and with thinking on it opens the channel itself (<|channel>thought\n), so the reply starts inside its reasoning; an assistant turn'sreasoning_contentis written back as a channel before that turn's tool calls; - images, audio and video are placeholders
(
<|image|>,<|audio|>,<|video|>).
The tokenizer is the GGUF model gemma4 (BPE over raw
UTF-8 with 514 906 merges, 262 144 tokens, add_bos_token
true, EOS 106), which OKLM already reads; the 12B lists two tokens to
suppress (tokenizer.ggml.suppress_tokens, which llama.cpp
keeps out of the sampling: OKLM reads the list and its samplers never
choose those tokens, Sampling; the E2B and E4B
files carry none). The tokens a reply is made of
(<|channel>, <channel|>,
<|tool_call>, <tool_call|>,
<|"|>) are user-defined tokens in these files: they
are sampled and written as their own text, which the reasoning splitter
and the call parser read. Two tokens that end a reply are not marked so:
<|tool_response> (a user-defined token: the model
writes it after its calls, where the template then puts the results) and
<eos> (a plain token: the E2B ends a call with it
after its thinking; the file's end-of-sequence id is
<turn|>). Left as they are, a reply ran on into a
result the model made up, or into <eos> until the
token limit. Both are read as control tokens
(src/tok/tok_gguf.c), where generation ends, as llama.cpp
does.
In OKLM these forms run on the CPU and on the OpenCL
and Vulkan backends (below, "On OpenCL and Vulkan"). The forward pass is
the llama family's (src/model/arch_llama.c), driven by the
hyper-parameters src/io/hf_config.c reads
(head_dim_swa, n_head_kv_swa,
n_rot_swa, value_norm,
rope_freqs_full, moe_branch, and for the E2B
and E4B n_embd_per_layer, n_kv_shared,
n_ff_first; the list of window blocks must be one pattern
repeated, as it is in every file looked at, and the list of feed-forward
widths one width then a wider one). src/model/arch_gemma4.c
binds the tensors and holds what that pass calls for a block of this
architecture: the work views with the block's own heads, the projections
with the values taken from the keys where there is no
attn_v, the block scale, and the feed-forward of a block
with experts. Each layer's key / value cache has its block's shape (2048
values a token in a window block of the 12B, 512 in a block without
window), so the contiguous and the paged cache, the KV files and the
quantized cache types take them as they are. The joined gate | up tensor
is bound as the gate stack with twice the rows: one indexed product
gives both halves of each (token, expert) pair, in the buffer of the
gathered inputs, which holds them as long as an expert is at most half
as wide as the stream (704 against 2816; a file that is not so is
refused at load). An expert's output scale is applied to its router
weight. The expert cache and the device expert tier do not compute this
layout (the experts' own norm, the joined product, the scales): a
context asked with either is created without it, with a notice, and the
experts are computed from the mapped file or on their layer's
device.
For the E2B and E4B, the embeddings per layer of a batch are computed
once before the first block (model_gemma4_ple_inputs(): one
lookup in the quantized table, one product with
per_layer_model_proj) and kept beside the context's buffers
for the blocks' ends (model_gemma4_ple()). A block that
shares has no cache of its own: its entry in the
context's list of caches is the earlier block's
(model_kv_source()), it writes nothing there, and the
attention reads the rows that block wrote in the same pass. So the E2B
keeps 15 caches for 35 blocks and the E4B 24 for 42, on the contiguous
and on the paged cache, with any cache type. What walks every layer's
cache (KV files, a sequence copied or swapped out) goes over those rows
once per block that reads them: the result is right, a KV file of these
models is larger than it needs to be. oklm plan and the
context's own report count each layer's cache at its block's width and
none for a block that shares: for the E2B at 8192 tokens (F16) 144 MiB
of cache, where every layer at the widest block's width gave 560 MiB,
and 3.78 GiB resident in all (the batch's embeddings per layer, 36.7
MiB, are in its scratch line).
On OpenCL and Vulkan a context of these models runs
on the device, every block or the last ones. The pass is the same
sequence of ops; what a device needed is in the placement and in
src/model/arch_gemma4.c, no kernel or shader was added:
- the two head sizes: each block's work views and its
cache have the block's own heads, and the rotation and attention kernels
take the head size from their operands. OpenCL's attention takes heads
of 256 and of 512 alike. Vulkan's attention group takes heads up to 256
(the window blocks); the heads of 512 go on to the shaders written for
latent attention (
mla_attn, vulkan.md), which read F32 and F16 caches: with another cache type (BF16, a quantized one) the blocks without window run their attention on the CPU for these operands (the CPU's result, checked on a Q8_0 cache intest_vulkan_attention; no model was run that way); - the projections: a device's joined Q, K, V matrix
is not used for this architecture (the rows of the three differ from
block to block and a block may have no
attn_v): two or three products, the values copied from the keys where there is noattn_v, then normalized. The joined gate and up matrix is used, at the block's own width; - the block scale is one value: it stays in host memory and is read as a number;
- the experts (26B-A4B) run on their layer's device
through the indexed products: the joined product, two strided copies,
the GELU, the down product; each chosen expert's scale is looked up on
the device (a gather from the layer's 128 values) and multiplied into
its weight. With
n_cpu_moethe routing and the rows are read back, the experts and their sum computed on the CPU and the sum written to the device. Neither the expert cache nor the device expert tier computes this layout; - the embeddings per layer (E2B, E4B): the projection and its norm are placed with the head, the table is looked up in host memory as the token embeddings are, and the batch's embeddings live in a second set of two device buffers (17.5 MiB each for the E2B at a batch of 512); when the first blocks run on the CPU they are computed there and copied to the device with the stream;
- the shared caches (E2B, E4B): a block and the cache
it reads are in one memory. The device holds a cache for the blocks that
keep one only (15 of the E2B's 35), and the split never falls between a
block and its cache: a count of device layers that would is raised to
the blocks that share plus the two they read
(
--gpu-layers 17gives 22 for the E2B), and when that many do not fit the model stays on the CPU. Such a model is placed on one device (not spread over several GPUs) and its CPU layers are not streamed (stream_batchis not used).
Measured on the RX 7900 XT, every block on the device, an F16 cache,
the backends' default products
(tests/model/test_gemma4_public.c: the five prompts, 32
steps each on llama.cpp's tokens, in a device context and in a CPU one;
the CPU's own line is the comparison above):
| file | backend | choices equal to llama.cpp's | logit difference, largest / mean | choices equal to the CPU's | largest / mean |
|---|---|---|---|---|---|
| 12B Q4_K_M | CPU | 148 / 160 | 12.5 / 0.581 | ||
| OpenCL | 148 / 160 | 10.3 / 0.550 | 149 / 160 | 13.1 / 0.579 | |
| Vulkan | 152 / 160 | 8.55 / 0.582 | 151 / 160 | 12.6 / 0.622 | |
| E2B Q4_K_M | CPU | 160 / 160 | 8.08 / 0.380 | ||
| OpenCL | 158 / 160 | 6.52 / 0.486 | 158 / 160 | 5.83 / 0.446 | |
| Vulkan | 157 / 160 | 7.46 / 0.447 | 157 / 160 | 7.32 / 0.444 | |
| E4B Q4_K_M | CPU | 156 / 160 | 3.51 / 0.267 | ||
| OpenCL | 158 / 160 | 3.96 / 0.267 | 156 / 160 | 3.63 / 0.261 | |
| Vulkan | 157 / 160 | 3.60 / 0.242 | 159 / 160 | 3.75 / 0.243 | |
| 26B-A4B UD-Q4_K_M | CPU | 152 / 160 | 12.1 / 0.501 | ||
| OpenCL | 155 / 160 | 22.8 / 0.680 | 153 / 160 | 22.5 / 0.639 | |
| Vulkan | 152 / 160 | 20.7 / 0.631 | 154 / 160 | 20.4 / 0.616 |
A device is as far from the CPU as the CPU is from llama.cpp, as on
the other files of this kind. The 26B-A4B's largest differences are one
step of the 160 (step 28 of "The three primary colors are"): there the
whole vector of logits moves on a device, by 9.5 on average on OpenCL
and 7.1 on Vulkan against the CPU, the best token the same on every side
with a lower logit (11.2 and 14.7 against the CPU's 19.8 and llama.cpp's
19.1, the next token at 7 to 9), and the two devices 2.5 apart on
average; against llama.cpp OpenCL's float products
(OKLM_CL_Q8=0) give 157 / 160 choices and 14.2 at most, and
with the experts computed on the CPU (n_cpu_moe) that step
moves less (6.4 on average). Between the CPU and itself with one
rounding changed (OKLM_GELU_F16=1,
OKLM_Q8_ROUND_EVEN=1, OKLM_ATTN_F16=1) no step
of that prompt moves by more than 3.1 on average. The softmax of that
step is 0.028 (OpenCL) and 0.007 (Vulkan) of total variation from
llama.cpp's.
That step is a choice of the model's routing, not of a device kernel. Traced with the records of an observed run (observe.md: each block's writes and its experts, a CPU context and a device one on the same tokens):
- Where the two part. Up to block 3 the CPU and
either device agree (the stream after block 3 within 3.5e-3 in norm, the
same eight experts in blocks 0 to 3). In block 4 seven experts are the
same and the eighth is not: the CPU takes expert 95, OpenCL and Vulkan
take 107. The CPU's router gives the two logits of 3.370 and 3.353
(probabilities 0.02648 and 0.02605 of 128 experts): 0.017 apart, when
one rounding switch of the CPU alone moves expert 107's logit there by
up to 0.06 (3.354 with
OKLM_GELU_F16=1, 3.292 withOKLM_Q8_ROUND_EVEN=1, 3.300 withOKLM_ATTN_F16=1; none of the three happens to put it above the eighth). - What follows. Block 4's write differs by 3.6 % in norm, and from there 26 of the 30 blocks choose another set of experts (the stream's norm up to 29 % apart, at block 24). On the 31 other steps of the prompt 2 to 16 blocks choose another set, the same routing noise, and the logits move by 0.24 to 2.4 on average. The 9.5 of step 28 is one offset of 9.5 common to every logit; 2.2 is left without it, which is why the softmax hardly moves.
- The CPU does the same when it takes that expert. A CPU context whose router is made to exchange experts 95 and 107 at that one place (block 4 of that one token; a change made for the measure, not kept) is 9.6 on average from the plain CPU context at step 28 (22.2 at most, the best token unchanged at 10.2) and 0.66 on average from OpenCL (5.4 at most, an ordinary step), 2.6 from Vulkan.
- The CPU against itself. The prompt evaluated a token at a time gives the bits of one batch. With an F32 cache step 28 is the step of the prompt that moves most from the F16 cache's (1.58 on average, the median of the steps 0.47), block 4 keeping expert 95.
So the CPU and llama.cpp, which round alike, fall on one side of a near tie of block 4's router and the two devices on the other, and this token's logits depend on that one expert by a common offset. A device that chose expert 95 there would stay within the CPU's bound. What the test bounds on a device is therefore 30 for the largest difference and 1.0 for the mean.
Other layouts of a device context give the plain device context's choices on the first 8 steps of two sequences decoded together, but where two tokens are all but tied: a paged cache (64 blocks) 16, 16, 15 and 14 to 15 of 16 for the four files; the first half of the blocks on the CPU 15 of 16 (12B, 24 of 48 layers on the device), 16 (E2B, 22 of 35), 15 (E4B, 21 of 42) and 14 to 15 (26B-A4B, 15 of 30); the 26B-A4B's experts in host memory 14 and 15.
Tokens a second (bench_llama: a prompt of 512 tokens in
one batch, 128 tokens one at a time, the best of two runs):
| file | CPU, prompt / decode | OpenCL | Vulkan |
|---|---|---|---|
| 12B Q4_K_M | 64.8 / 3.90 | 1464 / 53.8 | 1616 / 68.5 |
| E2B Q4_K_M | 285.7 / 17.7 | 5029 / 120.8 | 6042 / 141.2 |
| E4B Q4_K_M | 154.1 / 9.07 | 3110 / 80.4 | 3598 / 115.0 |
| 26B-A4B UD-Q4_K_M (whole on the 20 GB device) | 120.8 / 7.94 | 2220 / 71.8 | 2716 / 106.4 |
The 26B-A4B's Vulkan prompt rate was 1302 until the grouped expert products of a prompt went to the prompt products' tiles (vulkan.md); its row is that session's (OpenCL 2224 / 72.2 beside it).
The fixtures run on OpenCL and on each Vulkan device with every block
there and with one layer asked (five for tiny-gemma4e), on
the contiguous and the paged cache, the expert fixtures also with their
experts in host memory: within 1.7e-5 of the float64 reference token by
token (0.011 with an F16 cache), batches and several sequences bit for
bit on OpenCL and within 8.9e-5 on Vulkan, whose prompts and single
tokens go through two attention kernels
(test_gemma4_models). The operand shapes of a block are
held to the CPU kernels in test_gemma4_heads and
test_gemma4_experts (tests/core/test_opencl.c,
test_vulkan_attention.c,
test_vulkan_moe.c).
Verified: the tiny models tiny-gemma4,
tiny-gemma4moe and tiny-gemma4moe-split (six
blocks, a window of 4 positions on two blocks out of three, heads of 8
over 2 key / value heads and of 16 over 1, one block without
attn_v, 8 experts of 8 with 2 a token, joined or apart;
tools/gen_model_fixtures.py) against the float64 reference
(tools/infer_ref.py): 36 / 36 greedy choices each, logits
within 8.0e-6 (0.011 with an F16 cache); batches and several sequences,
on the contiguous and on the paged cache, give the single tokens' logits
bit for bit (test_gemma4_models in
tests/model/test_model.c). llama.cpp run on the same three
fixtures (written without a vocabulary, build/llama_ref)
gives the reference's 36 / 36 tokens, logits within 0.012 (dense) and
0.020 (experts), its key / value cache being F16.
The E2B and E4B's form has its own tiny model,
tiny-gemma4e: the same six blocks with an embedding per
layer of 4 values, the last three blocks reading the keys and values of
the second (window) and third (no window) blocks, a feed-forward twice
as wide in them, and the first of the three still carrying key and value
tensors that nothing reads. Against the float64 reference: 36 / 36
greedy choices, logits within 7.3e-6 (0.0068 with an F16 cache), batches
and several sequences bit for bit on the contiguous and the paged cache
(the same test). llama.cpp on the same weights (in F16, without the
logit cap, tools/gguf_cut_blocks.py) gives the reference's
36 / 36 tokens, logits within 0.011.
Gemma 4 E2B and E4B on the published files
(unsloth/gemma-4-E2B-it-GGUF,
gemma-4-E2B-it-Q4_K_M.gguf, 3.1 GB;
unsloth/gemma-4-E4B-it-GGUF,
gemma-4-E4B-it-Q4_K_M.gguf, 5.0 GB; CPU;
test_llama_cpp_gemma4), on the five prompts of the other
models, the chat turn with its control tokens among them:
| tokenization | greedy choices on llama.cpp's tokens | largest logit difference | continuations identical | tokens before parting | llama.cpp's margin where they part | |
|---|---|---|---|---|---|---|
| E2B | 5 / 5 | 160 / 160 | 8.08 | 5 / 5 | 160 / 160 | - |
E2B, OKLM_GELU_F16=1 |
5 / 5 | 158 / 160 | 6.38 | 3 / 5 | 100 / 160 | 1.0 |
| E4B | 5 / 5 | 156 / 160 | 3.51 | 1 / 5 | 79 / 160 | 0.341 |
The logit differences are of the 12B's kind (over the 262 144 logits
of a step, in a model that turns the rounding of one activation function
into units of logit: the second line is the same model with ggml's
half-precision GELU table, which changes more than the two programs
differ). The E2B's chat template renders as jinja2 renders it on the
twelve conversations of tools/gen_jinja_ref.py (tools among
them). oklm chat answers on both (the E4B: "The capital of
Australia is Canberra."); the E4B's file was removed after its run
(disk), the E2B's is the one the tests read.
Gemma 4 12B on the published file
(unsloth/gemma-4-12b-it-GGUF,
gemma-4-12b-it-Q4_K_M.gguf, 7.1 GB, CPU): tokenization
identical to llama.cpp's on five prompts (the chat turn with its control
tokens among them); on llama.cpp's tokens 148 of 160 greedy choices
equal and logits within 12.5; one of the five 32-token continuations
identical, 66 of 160 tokens before the others part, the widest gap in
llama.cpp's own logits where they part being 1.64. Those figures are
wider than the other families' and were looked into rather than
accepted:
the model turns a rounding into units of logit. ggml computes the tanh GELU through a table of 65 536 half-precision values; OKLM computes it in single precision.
OKLM_GELU_F16=1makes OKLM round the same way (input and result to F16,src/core/cpu/ops_cpu.c): OKLM's own logits then move by up to 2.5 at a step of these prompts, and the figures against llama.cpp become 149 of 160, 15.4 and 2.09. So one relative error of 1e-3 in one activation function moves this model as much as the two programs differ, and the comparison of the whole Q4_K_M file cannot be tighter than that (in the K-quant products both programs round the activations to 8 bits, where a last-bit difference becomes a step);so the blocks were compared without that amplification:
tools/gguf_cut_blocks.pywrites a model of two of the 12B's blocks (4 and 5: a window block, then a block without window), their matrices dequantized to F16, the first 16 384 tokens, and a logit cap that does nothing. On it (logits of 250 to 650) OKLM and llama.cpp agree within 0.25, 96 of 96 choices equal, the three continuations identical (gemma4-12b-cutintest_llama_cpp_gemma4); blocks 0 and 5: within 0.42, 96 of 96. WithOKLM_GELU_F16=1the best logits are the same to 0.02;then the whole stack was compared block by block and step by step with llama.cpp's hidden states on the same tokens (testing.md:
tools/llama_dump.c,OKLM_DUMP,tools/dump_compare.py), on "The capital of France is" (6 tokens) and on 72 tokens of text. The figures are the norm of the difference over the norm of llama.cpp's tensor.Each program on its own stream: the embeddings are equal, the first block's normalized input, queries, keys and values agree within 2.7e-7, and the first step that departs is the first block's attention (the heads' outputs,
kqv_out-0: 1.5e-4), then its GELU (ffn_geglu-0). The stream is 3.8e-3 apart after block 0, 1.5e-2 after block 2, 5.6e-2 after block 6 and 0.12 after the last, the logits up to 6.4 apart.Each block of OKLM on llama.cpp's stream (
OKLM_DUMP_FEED): no block stands out. A block adds 7e-4 to 1.8e-2 (median 4.0e-3), the blocks without window (5, 11, ... 47) as the window blocks.Each step on llama.cpp's own operands (
OKLM_DUMP_FEED_OPS=1), over the 48 blocks:step smallest median largest queries, keys (projection, norm, rotation), normalized values 1.7e-7 3.3e-7 2.5e-6 attention (heads' outputs) 1.3e-4 2.5e-4 3.7e-4 the same with OKLM_ATTN_F16=12.0e-8 2.8e-8 3.9e-5 output projection and its norm 9.2e-8 5.5e-7 5.9e-6 feed-forward input, gate and up products 1.1e-7 2.8e-7 2.1e-6 GELU-gated product 2.0e-5 1.8e-4 4.6e-4 the same with OKLM_GELU_F16=14.9e-7 2.0e-6 2.3e-5 down product, its norm, the block's output 8.4e-8 6.4e-7 2.8e-6 and the output norm and the head within 8.8e-7 and 4.4e-7. On the 72 tokens the same: every step within 8.5e-5 with the two variables (rotation at 72 positions included), attention 7.6e-6 at the median. So two steps a block are computed differently, and they are llama.cpp's roundings, not a different formula:
- attention over the F16 cache. llama.cpp (its graph
of separate products, which the reference runs use) multiplies the F16
keys by the queries, and the F16 values by the probabilities, with
ggml's matrix product, which converts its other operand to the type of
the first: each query is rounded to half precision before the scores,
each probability after the softmax (one below 3e-8 becomes 0, one below
6e-5 keeps 5.96e-8 of absolute precision). OKLM keeps the queries and
the probabilities in single precision and reads only the cache in half
precision.
OKLM_ATTN_F16=1rounds as ggml does (the plain kernel, several times slower): the step then agrees within 2.8e-8 at the median, the largest (3.9e-5) being a probability on the edge of two half-precision values; - the GELU table, as above.
Everything else agrees to rounding noise on equal operands, and that noise does not stay small on each program's own stream: with both variables set, block 0 ends 3.5e-7 apart (3.8e-3 without) and the stream is again 1.6e-3 apart after block 1, 8.6e-3 after block 2 and 0.12 after the last. A K-quant product rounds its operand to 8 bits a block of 256 values and attention has no score scale, so a difference in the last place of a single-precision value moves a rounding, and each block multiplies what it receives by 5 to 20 (block 1: the heads' outputs 3e-5 apart, the output projection of them 4.1e-4, the gate and up products 1e-3). The variables are there to show what differs, not to make the logits equal: with both, the figures against llama.cpp are 151 of 160 choices, logits within 9.2 and a gap of 0.82 on the 12B, and on the E2B 155 of 160, 8.23 and 2.49, from 160 of 160 without them.
Each program against itself gives the floor (
tools/ref_compare.py), on the reference's 160 steps:the 12B's file, on the same tokens choices largest logit difference widest gap where they differ OKLM against llama.cpp (the reference: separate products, F16 cache) 148 / 160 12.5 2.2 llama.cpp with flash attention against the reference 153 / 160 11.7 1.76 llama.cpp with an F32 cache against the reference 150 / 160 12.3 1.76 llama.cpp, the same run twice 160 / 160 0 - OKLM's scalar kernels against its AVX kernels 157 / 160 14.3 0.87 Flash attention is llama.cpp's other attention kernel, the one its programs take when the choice is left to them; the F32 cache (
LLAMA_KV_F32=1intools/llama_ref.c) takes the half-precision rounding out of its attention. The last line has no llama.cpp in it: OKLM's scalar and AVX kernels compute the same formulas, and of the first block only the sums of attention come in another order (the heads' outputs 5.5e-7 apart, every projection bit for bit equal); after block 1 the two streams are 1.3e-3 apart, after block 3 1.6e-2, and the logits of that step 4.4 apart. A difference in the last place of one sum is enough for logits 14 apart somewhere in 160 steps; llama.cpp's own roundings, hundreds of times larger, move a few more choices. The distance between OKLM and llama.cpp on this file is the distance between two runs of either program: rounding, with no step computed by another formula. Nothing was changed in OKLM's forward pass, and the test keeps the family's bounds.- attention over the F16 cache. llama.cpp (its graph
of separate products, which the reference runs use) multiplies the F16
keys by the queries, and the F16 values by the probabilities, with
ggml's matrix product, which converts its other operand to the type of
the first: each query is rounded to half precision before the scores,
each probability after the softmax (one below 3e-8 becomes 0, one below
6e-5 keeps 5.96e-8 of absolute precision). OKLM keeps the queries and
the probabilities in single precision and reads only the cache in half
precision.
oklm chat renders the file's template and answers:
thinking off by default (the generation prompt closes an empty thinking
channel), the answer ended by <turn|>; with
--template-vars '{"enable_thinking": true}' the reply opens
with <|channel>thought, reasons, closes with
<channel|> and answers (17 x 23: three ways to 391,
then the answer). (That markup was shown as the model wrote it at the
time; it is told apart now, below.) oklm info gives the
verdict and oklm plan the memory: 6.62 GiB of weights,
10.96 GiB resident with 8192 tokens of context (the cache was counted,
and its arena sized, at the window blocks' width for every layer then;
each layer is counted at its own width since, which by the 12B's shapes
is 0.37 GiB less at that context). About 2 tokens/s on this machine's
CPU while other work ran; 2.8 to 3.2 tokens/s (a prompt at 21 to 23) at
six and eight threads on a calmer run.
Gemma 4 26B-A4B on the published file
(unsloth/gemma-4-26B-A4B-it-GGUF,
gemma-4-26B-A4B-it-UD-Q4_K_M.gguf, 16.9 GB, CPU; SHA-256
f2c28b3d...f63f293f). The file's matrices are Q8_0
(embeddings, projections, the dense feed-forward), its experts Q4_K
(ffn_gate_up_exps) and Q5_1 (ffn_down_exps,
rows of 704 values). Its window blocks have an attn_v; its
blocks without window take their values from the key projection, as the
12B's. It runs as the tiny models did, with no change to the forward
pass:
| tokenization | greedy choices on llama.cpp's tokens | largest logit difference | continuations identical | tokens before parting | llama.cpp's widest margin where the choices differ | |
|---|---|---|---|---|---|---|
| 26B-A4B | 5 / 5 | 152 / 160 | 12.1 | 0 / 5 | 72 / 160 | 5.2 |
26B-A4B, OKLM_ATTN_F16=1 OKLM_GELU_F16=1
OKLM_Q8_ROUND_EVEN=1 |
5 / 5 | 153 / 160 | 13.0 | 1 / 5 | 58 / 160 | 2.49 |
| llama.cpp with flash attention, on the reference's tokens | 155 / 160 | 10.1 | 2.77 | |||
| llama.cpp with an F32 cache, on the reference's tokens | 150 / 160 | 15.0 | 3.56 |
Its hidden states were compared with llama.cpp's as the 12B's were
(testing.md),
on 6 tokens and on the 29 tokens that end where the two programs differ
most. On llama.cpp's operands, over the 30 blocks: the router's logits
and probabilities within 4.2e-6 (1.7e-7 at the median), the joined gate
| up product of the eight experts within 3.4e-7 (the same experts in the
same order for every token), the experts' weighted sum after its norm
within 4.1e-7, the dense side, the projections and the head as on the
12B. The steps that differ are the 12B's two, attention (2.7e-4 at the
median; 3.1e-8 with OKLM_ATTN_F16=1) and the GELU of both
feed-forwards (1.8e-4; 1.2e-6 with OKLM_GELU_F16=1), and
one more that Q8_0 matrices bring: the first block's queries are
4.6e-4 from llama.cpp's on the same input (its keys 3.5e-4, its
values 8.4e-4), where every other block's are 2.3e-7. A Q8_0 product
quantizes its input to 8 bits, and where a value falls exactly on a half
OKLM rounds it away from zero (ggml's portable code) and llama.cpp on
x86 to the even step (Numerics, below); the input of the first block,
made of Q8_0 embeddings, has such values (35 of 16 896 on six tokens),
the others do not. OKLM_Q8_ROUND_EVEN=1 rounds as x86 does
(7.6e-5 left, a value one place from a half in one of the two
programs).
The widest gap is a choice both programs are sure of. After
def fibonacci(n):
if n <= 0:
return 0
elif n == 1:
llama.cpp, with either attention kernel, continues with twelve spaces
(logit 17.9 against 12.7 for eight) and OKLM with eight (21.8 against
9.2), the indentation the code calls for. With
OKLM_GELU_F16=1 alone OKLM's choice does not move (22.3
against 6.7), with OKLM_ATTN_F16=1 alone the margin narrows
(17.6 against 12.0), and with both OKLM writes llama.cpp's twelve
spaces: on this step the half-precision roundings decide the answer, and
the two programs differ by them and by nothing else. The test's bound on
that margin is 6.5 for this file (20 and 0.9 for the two others, as the
family).
oklm plan counts 15.77 GiB of weights and 26.90 GiB
resident at 8192 tokens of context (the routed experts repacked for the
CPU: 8.67 GiB on top of their mapping); oklm chat answers
"The capital of Australia is Canberra." and, with
enable_thinking, reasons in its channel then answers "17
times 23 is 391.". 7.1 to 7.2 tokens/s at six and eight
threads on this machine's CPU while other work ran (a prompt at 26 to
27). In the catalogue as gemma4-26b-a4b, marked run; the
file was removed after its run (disk).
Tool calls and the thinking channel (the
gemma4 tool-call format, Grammars).
A call
<|tool_call>call:get_weather{city:<|"|>Paris<|"|>,days:3}<tool_call|>
comes back as get_weather with
{"city":"Paris","days":3}; under the grammar a call names
an offered tool and gives its keys sorted by name, as the template
writes an earlier call's, the required ones always. The thinking channel
is reasoning for the one splitter of src/harness/think.c:
oklm chat shows it dimmed (between
<think> and </think> when the
output is no terminal) and the agent as a REASONING event;
/v1/chat/completions gives it as
reasoning_content and the rest as content,
/v1/messages as a thinking block. A prompt
that ends on the channel's opening tag (after a tool's result with
thinking on) makes the reply start as reasoning
(oklm_think_opened()), and the turn is kept with that tag
so the template tells the reasoning from the answer when it renders the
turn again. The chat template is handed an assistant turn's leading
reasoning (the channel, or a <think> block) as
reasoning_content when it reads that member and writes no
</think> itself, as this one does.
Run on the published E2B file through oklm serve (CPU,
temperature 0, two tools offered, "What is the weather in Paris, in
celsius?"): tool_calls get_weather
{"city":"Paris","unit":"celsius"} with
finish_reason tool_calls (172 prompt tokens,
21 generated), whole and streamed; the result sent back as a
tool message gives "The weather in Paris is 21 degrees
Celsius and it is sunny." (219 prompt tokens, 192 of them cached);
tool_choice required on "Hello!" gives a call
(get_time {"zone":"UTC"}); a request that
needs no tool gets a plain answer. With
chat_template_kwargs
{"enable_thinking": true}: "What is 17 times 23?" gives 542
tokens split into a reasoning_content and the content "17
times 23 is 391.", the same streamed; with the tools,
877 characters of reasoning then the same call, and after the result 681
characters of reasoning (the channel the prompt opened) then "The
weather in Paris is 21 degrees Celsius and sunny.", whole and streamed.
oklm chat --tools does the same three turns
(/tool RESULT), the reasoning apart each time.
The same on the published 12B file (oklm serve, CPU,
temperature 0, the same two tools; its template is the earlier edition
of the same format): get_weather
{"city":"Paris","unit":"celsius"} with
finish_reason tool_calls (137 prompt tokens,
21 generated), whole and streamed; the result sent back gives "The
current weather in Paris is 21°C and sunny." (187 prompt tokens, 128 of
them cached); tool_choice required on "Hello!"
gives get_time {"zone":"UTC"}; a question that
needs no tool gets "The capital of Australia is Canberra.". With
enable_thinking: "What is 17 times 23?" gives 278 tokens,
579 characters of reasoning_content and the content "17
times 23 is 391.", the same streamed; with the tools,
417 characters of reasoning then the same call, and after the result 146
characters of reasoning then the same sentence.
The 12B's suppressed tokens
(tokenizer.ggml.suppress_tokens: 258883
<audio|> and 258882 <image|>, the
marks that close a recording and an image) on the published file: after
<|turn>user and "Describe this picture:
<|image>" the best logit is
<image|> (24.8; the next,
<|image>, at 18.6) and the sampler takes the next
(test_gemma4_suppress in
tests/model/test_infer_public.c, skipped without the
file).
Not done: the 26B-A4B's tool calls through oklm serve
(its template is the 12B's), its expert cache and its smaller
quantizations; for the E2B and E4B, the rows of a shared cache written
once to a KV file, and their audio and image projectors; a comparison
past the window of 1024 positions on the published file (the tiny models
cross theirs); the devices; images (mmproj); a cap on the
thinking channel's length (the reasoning budget closes
<think> blocks only); the 31B; the HF conversion; the
trainer.
Numerics
Products with quantized matrices quantize the activation row first,
as llama.cpp does: Q8_0 for Q4_0/Q5_0/Q8_0 weights, Q8_1 for Q4_1/Q5_1,
Q8_K for K-quants; then the dot product is integer arithmetic times the
scales. F16/BF16 matrices are multiplied in F32. Every kernel is
deterministic and independent of the thread count, and batched
evaluation gives bit-identical logits to token-by-token evaluation
(checked by tests/model/test_model.c).
The Q8_0 and Q8_1 rows are quantized as ggml's portable code writes
them (the scale d = max / 127 in half precision, each value times 1 / d
rounded to the nearest step, a value exactly halfway going away from
zero), on every machine and at every SIMD level. ggml's x86 code is not
that code: it multiplies by 127 / max and rounds a half to the even
step, and its ARM code rounds to even with 1 / d. The results differ
where a value falls exactly on a half, which ordinary activations do not
and the first block's input does when the embeddings are Q8_0
themselves: on Gemma 4 26B-A4B (Q8_0 embeddings and projections) 35 of
the 16 896 values of a 6-token prompt, and the first block's queries are
4.6e-4 from llama.cpp's on x86 for it. OKLM_Q8_ROUND_EVEN=1
rounds as the x86 code does, for comparisons. The default was left as it
is on 2026-11-13: following the x86 code would move the bits of every
model with Q8_0 or Q8_1 activation products, the pinned references and
the devices' 8-bit kernels, for a difference seen on one block of such
files; to be looked at again if a file shows more than that. The Q8_K
rows agree with llama.cpp's to the last place (2e-7 on the queries of
Gemma 4 12B).
Sampling
oklm_sampler_sample
(include/oklm/model/sampling.h) applies, in order:
- repetition (CTRL style: positive logits divided, negative
multiplied), frequency and presence penalties over the last
penalty_last_naccepted tokens, each distinct token once; - temperature 0: the arg-max (greedy, lowest id on ties);
- top-k (bounded heap, then sorted), top-p over the temperature-scaled distribution, min-p relative to the best token;
- a draw from what remains with xoshiro256** seeded by splitmix64, so a seed reproduces a run on every platform.
Defaults: temperature 0.8, top-k 40, top-p 0.95, min-p 0.05, no penalty, seed 1234.
A model's file may list tokens that are never to be sampled
(tokenizer.ggml.suppress_tokens, an array of ids: Gemma 4
12B has one). The tokenizer keeps the list
(oklm_tokenizer_suppress(), ids outside the vocabulary
dropped, written back by oklm_tokenizer_to_gguf()) and a
sampler takes it with oklm_sampler_set_tokenizer(), as
oklm run, chat, serve and the
agent do: those tokens then count as minus infinity before step 1,
whatever the options and the grammar, as llama.cpp's logit bias does. A
sampler with such a list no longer reads the best logit where the logits
are (oklm_sampler_greedy() is 0) and asks for that many
more of the best ones (oklm_sampler_top_needed()).
Grammars
include/oklm/model/grammar.h constrains generation with
a GBNF grammar, the syntax of llama.cpp's --grammar files
(ADR-0009):
root ::= "- " item ("\n- " item){2} # three dashed lines
item ::= [a-z]+ (" " [a-z]+)?
Rules are name ::= alternatives; items are
"literals", classes [a-z_] /
[^"\\], ., rule names and
( groups ), each optionally followed by *,
+, ?, {m}, {m,} or
{m,n}. Characters are code points; a token that ends inside
a UTF-8 character is allowed when some completion of it matches. Control
and end-of-generation tokens are allowed only where the grammar can
end.
oklm_grammar* g;
oklm_grammar_new(gbnf_text, "root", tok, &g, err, sizeof(err));
oklm_sampler_set_grammar(sampler, g); /* the sampler only returns allowed tokens */
int32_t id = oklm_sampler_sample(sampler, logits);
oklm_grammar_accept(g, id); /* the caller advances the grammar */The sampler checks the token it chose first and masks the vocabulary
only when the grammar refuses it.
oklm_json_schema_to_grammar() turns a JSON schema into a
grammar (types, required and optional properties, arrays with item
counts, enum, const, anyOf / oneOf, local $ref, string
lengths, integer and number ranges (exact decimal bounds), date / time /
uuid formats, regular expression patterns);
oklm_grammar_json() is a grammar of any JSON value.
Speculative decoding
include/oklm/model/speculative.h: a draft model sharing
the target's vocabulary proposes up to n tokens greedily; the target
evaluates them, after the previous round's last token, in one batch, and
its sampler draws position by position while it agrees with the
proposals. Each emitted token is drawn from the target's logits after
the same prefix with one random draw, so the output is the one plain
generation gives with the same seed (greedy and sampled, grammars
included); the draft changes only the speed. Caches are cut back to the
accepted tokens with oklm_kv_truncate.
oklm_speculative* sp;
oklm_speculative_new(target_ctx, draft_ctx, 5, &sp); /* both evaluated the prompt */
oklm_speculative_step(sp, sampler, grammar_or_NULL, out, &n_out);The gain depends on the size ratio and the backend: Qwen2.5-1.5B with the 0.5B as draft, 200 greedy tokens, decodes 26.0 → 31.9 t/s on the CPU with 3 proposals (89 % kept) and does not gain on the OpenCL device, where the target is already fast compared with three draft steps.
Multi-token prediction draft
DeepSeek V3 files may carry multi-token prediction blocks
(llama.cpp's NextN layers,
<arch>.nextn_predict_layers) after the main stack;
they count in block_count but are not part of the main
stack, which runs without them. oklm_model_n_mtp() tells
whether a file has their weights (files converted with
--no-mtp, and DeepSeek V3 conversions older than
llama.cpp's V3.2 support, have none).
The blocks can serve as the draft. A context created with
oklm_context_params.mtp holds only their caches (one per
block). With a device backend it runs where the model's head is: the
device copy of the weights carries the blocks with the head (create the
main context first; otherwise the head and blocks are placed alone and
the main stack can no longer go to a device), its token table staying on
the host; on the CPU otherwise. The blocks may also come from a file of
their own, llama.cpp's --mtp export (the header of the full
model, the blocks, the embeddings, final norm and head, none of the main
stack): oklm_model_mtp_only() tells such a file, which only
a multi-token prediction context can use, as the draft of the full model
(oklm run --mtp-model FILE). oklm_eval_mtp()
pairs the token at position p with the main stack's normalized hidden
state of the token at p - 1, which oklm_get_hidden()
returns for a context created with
oklm_context_params.hidden. The block normalizes the
embedding and the hidden state, projects them together
(nextn.eh_proj), runs one MLA + MoE block, and applies its
own final norm (nextn.shared_head_norm) and the output
head. Its logits predict the token after the one it was given.
cp.hidden = 1; oklm_context_new(m, &cp, &target); /* the main stack */
cp.hidden = 0; cp.mtp = 1; oklm_context_new(m, &cp, &mtp);
oklm_eval(target, prompt, n_prompt, 0); /* one call */
oklm_speculative_new_mtp(target, mtp, prompt, n_prompt, 3, &sp);
oklm_speculative_step(sp, sampler, grammar_or_NULL, out, &n_out);In a round, the block's first proposal uses the target's hidden state
of the last verified token. Further proposals chain on the block's own
normalized output, as llama.cpp does. With several blocks (chained
heads, oklm_set_mtp_head() choosing which one
oklm_eval_mtp() runs), proposal s comes from block min(s, n
- 1): a block taking over first re-evaluates the round's tokens so far
into its own cells, each paired with the output of the evaluation that
proposed it. After verification, the cells of every block are recomputed
from the target's real hidden states of the tokens kept (the first block
past its first cell, which already paired the pending token with the
real hidden state; in a round without a pending token, the first one,
several blocks hand the first proposal straight to the second block, so
the first block's first cell is recomputed too;
test_mtp_files checks that round on the chained fixture:
every block's logits for one more token, from the draft the round left,
are those of a draft that evaluated the kept tokens in one call).
oklm_speculative_last_proposals() returns a round's
proposals. The output is that of plain generation, as with a draft
model. oklm run --mtp uses it. The tiny V3-like fixture
checks the block's logits against the float64 reference (within 5e-4),
and checks that generation, greedy and sampled, is unchanged with the
target and the block on the CPU and on OpenCL; its --mtp
export gives the block's logits bit for bit and the same proposals; with
two chained blocks generation is unchanged, and a second block copying
the first proposes, round by round, what the first proposes alone. No
real V3 file fits this machine, so the speed gain has not been
measured.
Prompt lookup, grammar-forced drafts and the guard
oklm run -m model.gguf -p TEXT [--spec-ngram K] [--spec-grammar] [--spec-guard RATE] ...
oklm serve ... [--spec-ngram K] [--spec-guard RATE]
Three more sources and one brake for the same draft-and-verify rounds; the verification is unchanged, so the output stays the one plain generation gives with the same seed (greedy and sampled).
- Prompt lookup without a draft model (
--spec-ngram K,oklm_speculative_new_lookup()): the last 3, 2 or 1 tokens found earlier in the prompt and the output propose the K tokens that followed them there, as the server's--spec-ngramdoes for greedy requests. - Grammar-forced drafts (
--spec-grammar,oklm_speculative_set_grammar_drafts(), oroklm_speculative_new_grammar()when the grammar is the only source): when every way forward in the grammar allows exactly one character (and it cannot end there), that text is forced: a JSON schema's keys after their opening quote, the rest of an enum value ortrue/falseonce its first letter is chosen, a tool call's fixed text, closing brackets (oklm_grammar_forced_text()). Its tokens (oklm_grammar_forced_tokens(): the tokenizer's own tokens of the text, then the longest token at each point) are the round's proposals instead of the draft's, without sampling; a draft model or block then only evaluates the pending token and catches up with the accepted ones after the round. The model may write the text with other tokens (the text's last character often merges with the next free one,"into":), which the verification turns down like any proposal. The grammars of--json-schemaallow one optional space between tokens, so their forced runs stop at every space; a tool call's template text is forced whole. - The guard (
--spec-guard RATE,oklm_speculative_set_guard(), the server'soklm_server_params::spec_guard): with an expert cache, a round makes no proposal (forced ones aside) while the cache's recent hit rate (oklm_expert_cache_recent_hit_rate(): the experts batches needed that were in memory over about the last 8 forward passes) is below RATE. Verifying k proposals runs k + 1 tokens through every MoE layer: their union of experts is read, and the rejected proposals' experts were read for nothing. The guard decides round by round, so it resumes once the rate climbs back. The server counts the steps it rested (spec_restedin/api/stats).
Grammar-forced drafts, Qwen2.5-0.5B-Instruct Q4_K_M on the CPU (4 threads, best of 3, the machine shared), a chat-formatted prompt; the output identical with and without, greedy and sampled (seed 11):
| constrained by | sampling | plain | forced drafts | forced / kept |
|---|---|---|---|---|
a tool call's grammar (hermes <tool_call> text, 4
arguments), 48 tokens |
greedy | 41.2 t/s | 141.5 t/s (3.4x) | 42 / 39 |
| same | temperature 0.7 | 46.4 t/s | 140.6 t/s (3.0x) | 42 / 39 |
| a JSON schema (5 keys, an enum, an array of objects), 160 tokens | greedy | 54.1 t/s | 46.2 t/s | 47 / 20 |
| same, 83 tokens | temperature 0.7 | 38.0 t/s | 42.1 t/s | 37 / 18 |
A tool call's fixed text is forced whole and nearly every forced token is kept. Under a JSON schema the forced runs are short (a key's letters after its quote, between optional spaces) and the small model writes the key's closing quote merged with what follows, so fewer than half are kept and the gain stays within the noise.
The guard, Granite-3.1-1B-A400M Q4_K_M with direct reads and 2 loaders, a copying prompt (117 tokens) then 96 greedy tokens, prompt lookup with 4 proposals (26 % kept by this model), best of 3:
| cache | plain | prompt lookup | lookup, guard 0.8 | lookup, guard 0.95 |
|---|---|---|---|---|
| 10 % (cold: 10 % hits) | 3.9 t/s, 15.3 GiB read | 3.5 t/s, 18.3 GiB read | 4.1 t/s (96 of 96 rounds rested) | 4.3 t/s (96 of 96 rested) |
| 100 % (warm: 96 % hits) | 69.3 t/s | 61.6 t/s | 72.0 t/s (10 of 64 rested) | 72.1 t/s (20 of 69 rested) |
With a cold cache the verification batches read 20 % more and decoding slows by 10 %; the guard rests every round and gives the plain speed back. With a warm cache the guard rests only the first rounds, while the cache fills.
Tool calls
include/oklm/model/tool_calls.h reads the calls a model
writes in its own syntax: Hermes
<tool_call>{...}</tool_call> (Qwen 2.5 / 3,
Hermes, SmolLM3), Llama 3 {"name", "parameters"} (after an
optional <|python_tag|>), Mistral
[TOOL_CALLS][...] and
[TOOL_CALLS]f[ARGS]{...}, GLM-4.5 / 4.6 / 4.7
(glm45) <tool_call>f then
<arg_key>k</arg_key><arg_value>v</arg_value>
pairs and </tool_call> (a newline between the parts
for GLM-4.5 / 4.6, none for GLM-4.7; a value is JSON when it parses as a
number, a boolean, null, an object or an array, else a raw string; given
the tools offered, oklm_tool_calls_parse_tools() keeps a
string property's value a string whatever it reads as,
<arg_value>42</arg_value> giving
"42", as oklm chat, the agent and
oklm serve do), and GLM-4-0414 (glm4): a turn
that is a call starts with the function's name, then a newline and its
arguments as a JSON object (<|assistant|>f\n{...}), a
turn of text with the newline; the call ends the turn on
<|observation|>, and the first line is a call only
when it is the name of a tool offered; given the tools, a turn of text
may hold calls too, a line naming a tool and its JSON object below it
(in a code fence or not, the fence going with the call). The format is
recognized from the chat template (<arg_key> beside
<tool_call> for GLM-4.5;
<|observation|> and an assistant turn's
metadata for GLM-4-0414; <|channel|> and
an address to= for gpt-oss's harmony, whose call is a
message
<|channel|>commentary to=functions.f <|constrain|>json<|message|>{...};
<|tool_call_start|> for Liquid's LFM2,
lfm2; <|tool_call>call: for Gemma 4,
gemma4) or set with --tool-format.
Gemma 4 writes each call as call: and the function's
name, then its arguments between braces, between two tokens:
<|tool_call>call:get_weather{city:<|"|>Paris<|"|>,days:3}<tool_call|>.
The keys are bare words; a string stands between two
<|"|> marks and is taken as it is (it has no escapes:
quotes, braces, backslashes and newlines are its text); numbers,
true, false, null, lists and
nested dicts are written as JSON writes them. JSON quotes and a bare
word are read as strings too, for a model that leaves the notation. The
reply ends on <|tool_response>, after which the
template writes the result in the same model turn; a result the model
writes itself is dropped. A reply that starts with call:
and the name of an offered tool is read without its opening token.
LFM2 and the models tuned from it (LFM2-350M, Cyber-Prime) write
their calls as a Python list between two special tokens,
<|tool_call_start|>[get_weather(city="Paris", days=3), now()]<|tool_call_end|>.
The arguments' Python literals come back as JSON: a string with either
quote and its escapes (\n, \',
\xHH, \uHHHH...), a number (.5,
1., 1_000 and 0x1F too),
True / False / None, a list or a
tuple as an array, a dict as an object; an argument written without its
keyword takes the name of the tool's property at that position, and a
reply that is nothing but such a list of calls to offered tools is read
as calls even without the two tokens.
<|tool_call_start|> is a control token that would end
the reply: with tools it opens the call instead,
<|tool_call_end|> ends the turn. Some LFM2 GGUF files
carry the plain ChatML template, which says nothing of tools: the format
is then recognized from the vocabulary's two tokens, and
oklm serve, oklm chat --tools and
oklm agent render the conversation with a template of their
own that writes what Liquid's does (the tools in the system turn between
<|tool_list_start|> and
<|tool_list_end|>, a result in a tool
turn between <|tool_response_start|> and
<|tool_response_end|>, an earlier call as the list; a
template given with --chat-template is rendered instead, as
it is). Calls come back as name + compact JSON arguments, and
oklm_tool_calls_to_json() writes them as the OpenAI style
tool_calls array chat templates expect in the assistant
message (oklm_chat_msg.tool_calls); tool results are
tool messages with tool_call_id and
name. GLM-4-0414's template knows neither:
oklm_chat_template_render() hands it the conversation in
its own form, the assistant's text then one turn per call (the
function's name in the turn's metadata, the arguments as
JSON text in its content) and each result as an observation
turn.
oklm_tool_call_grammar() builds the grammar of a call of
the declared tools and returns the marker that opens a call
(<tool_call>, [TOOL_CALLS],
<|tool_call_start|>,
<|tool_call>); generation is free until the model
writes the marker, then constrained until the call is closed. For
gemma4 the grammar is
call:name{key:value,...}<tool_call|>: a tool's keys
sorted by name (the order the template writes an earlier call in), the
required ones always, the others or not; a string property a text
between the marks that holds anything but the mark, an integer, number
or boolean property that literal, an enum one of its values, an array
with typed items a list of them. For lfm2 the grammar is
the list: each call a tool's name and key=value arguments
(the required keys in their order, then optional ones), a string
property a quoted string, an integer, number or boolean property that
Python literal, an enum one of its values. For glm45 the
grammar takes the tools' names, each tool's property names as keys (the
required ones first, in the schema's order), a string property's value
as raw text up to </arg_value> and another's as the
JSON its schema describes. glm4 has no marker, as Llama 3
when it writes no <|python_tag|>: its grammar covers
the whole answer (a tool's name, a newline, the JSON its parameters
describe), used from the start with --tool-choice required;
otherwise the answer runs free and is parsed at its end (the server
holds back the text of such an answer until then).
Chat templates
Models ship a Jinja template (tokenizer.chat_template).
OKLM renders it with its own Jinja interpreter
(include/oklm/model/jinja.h, ADR-0006) exactly as Hugging
Face's apply_chat_template() does: the variables are
messages, add_generation_prompt,
bos_token, eos_token, and tools /
documents set to none. When the tokenizer adds BOS itself,
a BOS text at the start of the rendering is removed (as llama.cpp does),
so it is not doubled. A template that raises an error (for example roles
that must alternate) stops the turn with its message.
oklm_chat_template* t;
oklm_chat_template_new(oklm_tokenizer_chat_template(tok), "<s>", "</s>", &t);
oklm_chat_template_render(t, msgs, n, 1 /* generation prompt */, oklm_tokenizer_add_bos(tok),
"{\"enable_thinking\": false}" /* extra variables, or NULL */, &text, err, sizeof(err));Extra variables are a JSON object whose members are added to (or
replace) the defaults: enable_thinking for Qwen3,
tools for templates with tool calls,
date_string for Llama 3...
oklm chat --template-vars JSON passes them.
Models without a template, or with one the interpreter rejects, use
built-in renderings of the common formats recognized from the template's
markers: ChatML (<|im_start|>), Llama 2
([INST] <<SYS>>), Llama 3
(<|start_header_id|>), Mistral ([INST]),
Gemma (<start_of_turn>) and Phi-3
(<|user|> ... <|end|>); ChatML
when nothing matches.
A file without a template
GGUF files of 2023, and some later conversions, carry no
tokenizer.chat_template, and nothing else in them names the
format: LLaVA 1.5's file calls itself "LLaMA v2" and was trained on
Vicuna's turns. Written as ChatML, the conversation reaches such a model
between markers it was never trained on.
--chat-template FILE|NAME, in oklm chat,
oklm run (where it renders a chat turn:
--rag), oklm serve, oklm agent
and oklm describe, puts a template in place of the
file's:
oklm chat -m model.gguf --chat-template vicuna # a built-in format
oklm serve -m model.gguf --chat-template turns.jinja # a Jinja file
A name of the list below is that format (write ./NAME
for a file of that name); anything else is a Jinja file, which has to
compile. A name that is neither ends the command with the list. The
option also replaces a template the file does carry, when it is a wrong
one.
The template asked for is the one rendered, wherever the command renders a chat turn:
oklm runrenders one with--rag(and--rag-critique): the question and its sources go through the option's template. Without--ragthe prompt is the text given, and the option changes nothing.- An LFM2 file whose template says nothing of tools is otherwise
rendered, with tools, by a template of OKLM's own that writes them (tool calls). Not when a template was asked for,
or named by the catalogue: that one is rendered as it is, with the tools
if it writes them and without them if it does not
(
oklm_tokenizer_chat_template_given(),oklm_tool_format_model_template(), inoklm chat --tools,oklm serveandoklm agentalike). - The agent's small model (
smallModel, which answers the asides) is another file than the one the option was written for: it keeps the template its file carries. When its file carries none, it gets the catalogue's, else the option's. When it is the main model's file, it gets what that one got.
| Name | The turns | Defined by |
|---|---|---|
vicuna |
the system text and an empty line; USER: ... and
ASSISTANT: ... </s> lines;
ASSISTANT: to answer |
llama.cpp's vicuna (Vicuna 1.1; LLaVA 1.5) |
chatml |
<|im_start|>role, the text,
<|im_end|> |
oklm_chat_apply() |
llama2 |
[INST] <<SYS>> system
<</SYS>> user [/INST] answer
</s><s> |
oklm_chat_apply() |
llama3 |
<|start_header_id|>role<|end_header_id|>,
an empty line, the text, <|eot_id|> |
oklm_chat_apply() |
mistral |
[INST] user [/INST] answer
</s><s>, the system text in front of the first
user turn |
oklm_chat_apply() |
mistral-v3 |
[INST] user [/INST] answer
</s>, the system text in front of the last user
turn |
the template of Codestral 22B and Mistral 7B v0.3 |
gemma |
<start_of_turn>user or model, the
text, <end_of_turn>, the system text in the first
user turn |
oklm_chat_apply() |
phi3 |
<|role|>, the text,
<|end|> |
oklm_chat_apply() |
alpaca |
a preamble (or the system text), ### Instruction: and
### Response: headings, an empty line between turns |
Stanford Alpaca's prompt without an input, repeated turn after turn |
deepseek-coder |
its system text, ### Instruction: and
### Response:, an answer closed by
<|EOT|> |
DeepSeek-Coder instruct's tokenizer_config.json |
foundation-sec |
<|system|>, <|user|> and
<|assistant|> lines, the text on the next, an answer
closed by the end of the sequence |
Foundation-Sec-1.1-8B-Instruct's
chat_template.jinja |
One table serves every command (oklm_chat_builtin(),
oklm_chat_builtin_names()): each entry is the Jinja
template that writes the format, so everything that reads the model's
template (the rendering, the tool-call format found from it, the server,
the agent) sees the one chosen. oklm_chat_template_source()
reads the argument, oklm_model_set_chat_template() puts it
in the model, and oklm_models_chat_template() is what the
commands and oklm_server_new() call.
tests/model/test_model.c renders a conversation of two
turns with a system prompt in each format and compares it with the
definition, and the six formats of oklm_chat_apply() with
what that function writes.
Two cases need no option. A DeepSeek-Coder instruct file is known by
its BOS and EOS tokens (tokenizer.md). A
catalogue model whose file carries no template names its format in the
catalogue (models.md):
oklm chat -m llava-1.5-7b chats in Vicuna's turns, and so
does the file given by its path in the model's own folder
(.../llava-1.5-7b/ggml-model-q4_k.gguf), which is how the
chat window's own server gets it.
When the file has no template, none is given and the catalogue names none, the command says so once on stderr, with the option and the names, and goes on in ChatML:
$ oklm chat -m plain.gguf
oklm: this model's file carries no chat template: the turns are written as ChatML, which it may not have been trained on.
--chat-template FILE|NAME gives it one (a Jinja file, or one of vicuna, chatml, llama2, llama3, mistral, mistral-v3, gemma, phi3, alpaca, deepseek-coder, foundation-sec)
Checked on LLaVA 1.5 7B (Q4_K, the catalogue's file) on the CPU, without its projector and without an option:
$ printf 'What is the capital of France?\n/exit\n' | oklm chat -m llava-1.5-7b --backend cpu --temp 0 -n 64 --stats
chat template: jinja. Commands: /system TEXT, /reset, /decide A | B | ..., /tui, /exit (/help lists them all)
> (prompt 16 tokens: 0 cached, 16 evaluated)
The capital of France is Paris.
The prompt is Vicuna's turn (USER: and the question on
a line, then ASSISTANT:), 16 tokens with the BOS, where
ChatML's markers make it 35 (--chat-template chatml); the
answer ends on </s>, well short of the limit of 64
tokens. oklm describe -m llava-1.5-7b --mmproj llava-1.5-7b
on a screenshot answers as before through the same template (576 image
rows in a prompt of 600 tokens, "A screenshot of a web page.").
Commands
oklm run -m model.gguf -p "The capital of France is" [-n 128] [--temp 0] [--top-k 40] [--top-p 0.95] [--min-p 0.05] [--repeat-penalty 1.1] [--seed 1234] [-c CTX]
[--grammar list.gbnf | --json-schema person.json | --json] [--draft small.gguf [--draft-n 5]]
oklm chat -m model.gguf [--system "Be brief."] [--chat-template FILE|NAME] [--template-vars '{"enable_thinking": false}'] [--temp 0.7] [-c CTX]
[--tools tools.json [--tool-choice auto|required] [--tool-format hermes|llama3|mistral|glm45|glm4|harmony|lfm2|gemma4]]
[--resume] [--kv-save] [--chat-file FILE] [--stats] [--tui]
[--mmproj FILE|NAME|none [--tiles SIDE] [--fps 1] [--frames 8] [--ffmpeg]]
[--auto-compact on|off] [--auto-continue on|off] [--continue-max 8]
oklm ppl -m model.gguf -f data/tinyshakespeare.txt [-c CTX] [--progress-json FILE|-]
run, chat and ppl take
-ctk TYPE / -ctv TYPE
(--cache-type-k, --cache-type-v): the KV cache
types, F16 by default; f32, bf16 and the block quants q8_0, q4_0, q4_1,
q5_0, q5_1, iq4_nl as llama.cpp. A quantized cache holds Walsh-Hadamard
rotated heads, as llama.cpp's (adr/0005-kv-cache-precision.md):
on Llama-3.2-1B, Q8_0 keeps the perplexity within 0.2 % of F16 and Q4_0
within 5 %, with half and a quarter of the memory.
runprints the prompt and the continuation as it is generated, then the prompt and generation speeds on stderr. Generation stops after-ntokens, at the end-of-sequence token or at any control token. With a grammar (a GBNF file, a JSON schema, or--jsonfor any JSON value) the output follows it and generation stops once it is complete and the model ends.chatis a REPL. Each turn re-renders the whole conversation with the model's chat template, evaluates only the tokens after the part already in the cache, and streams the answer./system TEXTsets the system prompt,/resetstarts over,/exit(or end of input) quits;/helplists the commands (The chat's commands), which also change the model (/model, a selection), the context's size, the sampling and what the model answers from (In the terminal chat). With--tools(a JSON array of OpenAI style tool definitions, passed to the template astools), calls are constrained by the tools' schemas once the model opens one (from the start with--tool-choice required), printed as a[tool calls]JSON array, and answered with/tool RESULT, which adds atoolmessage and lets the model continue./decide label1 | label2 | ...asks the model to choose among the labels for the conversation so far, by scoring rather than generating (the server's/v1/systemonerule, server.md): the conversation is the state, each label is scored as the answer, and each label's probability is printed, then the most probable and the confidence (0 flat, 1 certain). The cache then holds the decision's prompt; the next turn evaluates again only what differs. With--resume,--kv-saveor--chat-file, the conversation is written after each turn to.oklm/chat/last.json(in the current directory;--chat-file FILEanother; without them nothing is written), and--resumegoes on with it (its system prompt too, unless--systemis given).--kv-save(off by default) also keeps the cache's rows in a KV file (above) beside it (last.kv), appended after each turn (the prompt's and the reply's tokens);--resumeloads it when it is there and fits the model, so the first turn evaluates only the new message: on SmolLM2-135M, a third turn of 127 tokens evaluated 15, the 112 others coming from the file, with the answer of a cold continue (tools/check_chat_continue.sh).--statsprints each turn's prompt tokens, those found cached and those evaluated, what was written to the KV file and, for a model with recurrent blocks whose sequence was cut (/retry,/rewind, a turn whose prompt parts from the cached one), what bringing their state back cost: the tokens that went through the journal and through a forward pass (hybrid-layers-design.md). The reply's tokens are kept with the prompt's, so the next turn also finds the reply cached when the template renders it as generated. With a projector (--mmproj FILE, or the name of an installed catalogue model that carries one; a catalogue model given to-mgets its own without the option, and--mmproj noneloads none)/attachalso takes a picture, a sound or a video (A picture, a sound or a video). A turn's prompt is then rows: tokens, and for each picture or sound its embeddings, each row known by a hash of its item's content and its index within it. They are compared with the rows the context holds as token ids are, so a turn after a picture evaluates the new message only;--statsthen tells the prompt's rows, how many are media, how many were cached and evaluated, how many media items the encoder ran on, and the time the prompt and the evaluation took.--tiles,--fps,--framesand--ffmpegareoklm describe's (multimodal.md).--tui(or/tuityped in the chat, which also goes back to line mode) runs the chat in the agent's full-screen UI (agent-ui.md) on a terminal: the transcript, where each reply's Markdown is rendered as it streams and a thinking model's reasoning is folded into a[thought, N words]line (Ctrl-O shows it as it is written, and folds it again); the input, with the history (Up, Down, Ctrl-R) and multi-line messages (Alt-Enter, or a backslash before Enter); a status bar with the model, the context's use and the last reply's speed in tokens per second. Esc or Ctrl-C stops a reply (kept as written); keys typed meanwhile wait for the next prompt. The chat's commands work there as in line mode; Ctrl-D on an empty line quits. Without a terminal--tuisays so and stays in line mode.tools/check_chat_tui.shdrives it on a pseudo-terminal with SmolLM2-135M on the CPU.pplcomputes the perplexity of a text: windows of-ctokens starting with BOS when the tokenizer adds one (not GLM's, whose BOS[gMASK]only the chat template writes), only the second half of each window scored, so every scored token has at least half a window of context (llama.cpp's method). A text shorter than one window scores nothing: the command says so and fails, instead of printing a perplexity of 1 over 0 tokens.--progress-json FILE(-for standard output, the figure's line then going to standard error) writes the run as JSON lines:startwith the number of windows, awindowline after each with the perplexity so far,endwith the figure (training.md).
run, chat and serve check the
memory a model would take before its context is made (memory-plan.md); a MoE model's options
(--expert-cache, --n-cpu-moe,
--expert-gpu) are in moe.md, which
also has what each costs and gains on this machine (measured).
The chat's commands
A line that starts with / in a chat is a command, read
by one module (include/oklm/client/chat_commands.h) for the
terminal chat, the chat window and the server's dashboard, so that a
command has the same name, arguments and help everywhere. The rules are
the agent's (agent-ui.md): a command only at
the line's start, names without case; //text (or a blank
before the slash) sends a text that starts with /; the
models' own switches /think and /no_think, and
a first word that holds another slash (/usr/bin/cc), are
text too. Arguments are words; quotes ("...",
'...') hold a word with blanks and a backslash keeps the
next character; a text (/system, /decide,
/tool) is the rest of the line as it is, and so is a file's
name or an address when it is not within quotes.
A line that is not a command that runs is never sent to the model,
and the chat says why: an unknown name, with the names close to it
(unknown command /modle: did you mean /model?); wrong
arguments, with the usage; a command of other chats
(/connect in a terminal); a command not written in this
chat yet (not available here yet); a command that misses
something where the chat is (a server whose context is fixed, no
retrieval index, a model that cannot be changed from a remote page).
/help lists the commands of the chat it is typed in,
/help NAME tells one with its other names and the chats it
belongs to. In oklm chat, at the full-screen prompt and in
line mode on a terminal, Tab completes the command's name: whole when
one command starts with what was typed, up to what they share when
several do, which are then listed. After a name, Tab completes the
argument the same way: on or off,
markdown or json, for /model the
catalogue's names a chat can serve and the .gguf files (on
a server, oklm chat --url, the names the server lists,
asked for at the first Tab), for /attach,
/save and /rag the files of the directory
typed so far, and force after /context N. Line
mode on a terminal reads key by key with the line editor of the
full-screen prompt: the arrows move in the line and through the lines
sent before, Ctrl-C clears the line, and what Tab lists is written under
the line, which is drawn again after it. Input that is not a terminal (a
pipe, a file) is read a whole line at a time and completes nothing. In
the chat window a list above the message shows the matching commands
with their help while one is typed, then the values of its argument, and
a command's answer is a note in the conversation that the model is never
sent (gui.md).
The dashboard's chat lists the matching commands while a
/ word is typed (arrows, Tab or Enter, Escape) and shows a
command's answer as a note in the conversation; what each command does
in a page is in server.md.
The table below is written by the module
(oklm_chat_commands_markdown()) and compared with it by
tests/client/test_chat_commands.c: yes where a
command runs today, not yet where it belongs and is not
written yet, - where it does not belong. The window
attached to an agent passes the lines it does not take itself to the
agent, which has its own commands.
| Command | What it does | Terminal | Window, own server | Window, remote server | Dashboard | Window on an agent |
|---|---|---|---|---|---|---|
/help [NAME], /? |
List the commands, or tell one in full | yes | yes | yes | yes | yes |
/system [TEXT|-] |
Show the system prompt, or set it and start a new conversation (- for none) | yes | yes | yes | yes | - |
/reset, /clear, /new |
Start a new conversation | yes | yes | yes | yes | - |
/decide A | B | ... |
Have the model choose one of the options, with its confidence; the conversation is kept as it is | yes | yes | yes | yes | - |
/tool RESULT |
Answer the model's tool call with its result | yes | - | - | - | - |
/model [NAME] |
Show the model, choose another among those listed, or switch to the one named | yes | yes | yes | yes | - |
/models |
List the models by group, with their sizes and whether they are installed | yes | yes | yes | yes | - |
/context [TOKENS [force]], /ctx,
/context-size |
Show the context's size and its use, or change the size (the conversation is kept) | yes | yes | yes | yes | - |
/attach FILE |
Attach an image, a sound or a document to the next message | yes | yes | yes | yes | yes |
/detach [N] |
Remove what is attached, or the Nth file only | yes | yes | yes | yes | yes |
/rag INDEX|off |
Answer from a retrieval index's passages, or stop doing so | yes | yes | yes | yes | - |
/save FILE |
Write the conversation to a file, to be continued later | yes | yes | yes | yes | yes |
/export [markdown|json] |
Write the conversation as Markdown or JSON | yes | yes | yes | yes | yes |
/copy [N] |
Copy the Nth latest reply to the clipboard | yes | yes | yes | yes | yes |
/retry, /regenerate |
Drop the last reply and have it written again | yes | yes | yes | yes | - |
/rewind [N], /undo |
Go back to before the last message, or before the last N | yes | yes | yes | yes | - |
/continue |
Go on with the last reply where it was cut (by the context or by the token limit) | yes | yes | yes | yes | - |
/compact [FOCUS] |
Summarize the older messages into a note to free the context; the recent ones are kept as they are | yes | yes | yes | yes | - |
/autocompact [on|off] |
Show or set whether the conversation is compacted by itself when the context is nearly full | yes | yes | yes | yes | - |
/autocontinue [on|off|N] |
Show or set whether a cut reply is continued by itself, or how many times in a row at most | yes | yes | yes | yes | - |
/stop |
Stop the reply being written | - | yes | yes | yes | yes |
/stats, /usage |
The conversation's tokens, the context's use and the speed of the last reply | yes | yes | yes | yes | - |
/temp [T], /temperature |
Show or set the sampling temperature (0: the likeliest token) | yes | yes | yes | yes | - |
/top-k [K] |
Show or set top-k: sample among the K likeliest tokens (0: all) | yes | yes | yes | yes | - |
/top-p [P] |
Show or set top-p: sample among the likeliest tokens holding the probability P | yes | yes | yes | yes | - |
/min-p [P] |
Show or set min-p: leave out the tokens below P times the likeliest one's probability | yes | yes | yes | yes | - |
/seed [N] |
Show or set the sampling seed | yes | yes | yes | yes | - |
/max-tokens [N] |
Show or set the longest reply, in tokens | yes | yes | yes | yes | - |
/thinking [on|off] |
Have the model reason before it answers, or not | yes | yes | yes | yes | - |
/connect URL |
Chat with the server at an address in place of the current one | - | yes | yes | - | - |
/video [fps F] [frames N] | default |
Show or set how the videos attached next are sampled: frames a second and frames kept, at most | - | yes | yes | - | - |
/speak [on|off|stop|pause|resume|TEXT] |
Read the last reply aloud, every reply as it ends (on, off) or a text; stop ends what is being said, pause holds it and resume goes on | - | yes | yes | yes | yes |
/pause |
Pause the voice that reads aloud, or have it go on from where it was | - | yes | yes | yes | yes |
/listen [stop|send] |
Record the microphone and write what was said in the message | - | yes | yes | yes | yes |
/talk [on|off] |
Hold a spoken conversation: listen until a pause, send what was said, speak the reply, listen again | - | yes | yes | yes | not yet |
/tui |
Enter or leave the full-screen interface | yes | - | - | - | - |
/exit, /quit |
Leave | yes | - | - | - | - |
A server tells its clients the same table with
GET /api/commands (server.md), each command with
whether a chat on that server can run it.
In the terminal chat
oklm chat runs every command of its column, in line mode
and in the full-screen UI (with --url, on a server's model:
On a server says what each does there). A
command that takes a value shows the current one without it. What each
does there:
/systemshows the system prompt,/system TEXTsets it and starts a new conversation,/system -removes it./modeland/models: Choosing a model./contextshows the context's size, its use and the range the model takes;/context Nmakes a context of N tokens (256 to 1048576) for the same model, the conversation kept and evaluated again on the next turn. A size the conversation does not fit in is refused. A size the memory plan says does not fit (memory-plan.md) is warned of in red and not made unlessforcefollows (/context 65536 force); one beyond the context the model was trained with is made with a warning in red. The context in use is freed before the new one is made, so that a device holds one at a time; when the new one cannot be made the old size comes back./attach FILEreads a document as--docdoes (PDF, DOCX, HTML, JSON, text: rag.md) and keeps it for the next message, which holds it before the text typed (Document NAME:and its text between<document>tags), so that/save,/exportand a later model see it; its size in tokens is told, with a warning when it is more than the context has left. Several files can be attached;/detachremoves them all,/detach Nthe Nth (the documents are numbered first, then the pictures, sounds and videos, as/statslists them). With a projector the file may be a picture, a sound or a video (below); without one such a file is refused, with how the chat takes it (--mmproj FILE, or a catalogue model that carries a projector)./rag INDEXopens a retrieval index as--ragdoes at start, with the other--rag-*options the chat was given, in place of the one in use (which stays when the new one cannot be opened);/rag offcloses it./save FILEwrites the conversation as--chat-filekeeps it (oklm chat --resume --chat-file FILEgoes on with it);/exportwrites it as Markdown and/export jsonas the body of a chat request ({"model", "messages"}), toconversation-DATE-TIME.mdor.jsonin the current directory./copyputs the last reply, without its reasoning, in the clipboard (/copy 2the one before):wl-copy,xclip,xsel,pbcopyorclipwhen one is there, else the terminal's own clipboard (OSC 52)./retrydrops the last reply and has it written again (at temperature 0 it comes out the same, which the chat says);/rewindgoes back to before the last message typed,/rewind Nbefore the last N./statsprints the model, the conversation's messages and the context's use, the last turn (its prompt's tokens, those evaluated, the reply's tokens and speed), the sampling, what is attached and the retrieval index./temp,/top-k,/top-p,/min-p,/seedand/max-tokensshow or set what the options of the same names set at start, from the next reply on./thinking on|offsets the template'senable_thinkingvariable (Qwen3, GLM); the chat says so when the model's template has no such switch./compact [FOCUS]summarizes the older messages into a note, dwelling on the focus when one is given;/continuegoes on with a reply that was cut, or answers a message the full context left unanswered;/autocompact on|offand/autocontinue on|off|Nsay whether the chat does either by itself (A full context)./statstells where compaction starts, how a cut reply is continued and how many messages compactions replaced.
tests/harness/test_cli_chat.c types them into a chat on
SmolLM2-135M (419 checks with the media's below and a full context's)
and tools/check_chat_tui.sh drives the full-screen UI on a
pseudo-terminal.
A picture, a sound or a video
oklm chat -m MODEL --mmproj FILE (since 2026-11-15;
/attach took documents only) loads the model's projector as
oklm describe and oklm serve do: a vision
encoder, or an audio one. -m smolvlm-256m, or any installed
catalogue model that carries a projector, loads it without the option.
/attach FILE then reads a picture, a sound or a video (by
the file's extension, as oklm describe tells them) and
keeps it for the next message; a tool's result (/tool)
takes none, and what is attached waits for the message after it. A video
is its sound for an audio encoder, one item for Qwen2-VL and Qwen2.5-VL,
and for another vision encoder its frames as pictures, each announced by
its time in the message.
$ oklm chat -m smolvlm-256m --backend cpu --temp 0 --stats
(projector mmproj-SmolVLM-256M-Instruct-f16.gguf: with the next message, /attach FILE takes a picture, or a video as its frames)
> /attach front-page.ppm
(attached front-page.ppm: a picture of 640 x 488; it goes with the next message, /detach removes it)
> What is in this picture?
(prompt 348 rows, 320 of them media: 0 cached, 348 evaluated; 1 of 1 media items encoded, the prompt made in 4.32 s)
(348 rows evaluated in 0.22 s)
A newspaper with an old picture of an astronaut on the front.
> What is the headline?
(prompt 375 rows, 320 of them media: 361 cached, 14 evaluated; 0 of 1 media items encoded, the prompt made in 0.05 s)
(14 rows evaluated in 0.02 s)
Men walk on moon.
The conversation keeps a message's text and, beside it, the files
that went with it. Each turn the messages are written as the projector's
model writes a turn with media (oklm_mm_question(),
oklm_mm_format_chat(): the functions of
oklm describe and of the server's chat routes) and made
rows by oklm_mm_build(), the function the server builds a
request's rows with. The first time, an item's encoder runs; its
embeddings are then kept by the hash of its content, the item is named
by that hash and its pixels or samples are freed. A later turn's prompt
takes the embeddings kept, so the encoder does not run again, and its
rows are those the context already holds, so they are not evaluated
again either. The same picture attached twice is encoded once.
What the other commands do with media in the conversation:
| Command | With a picture, a sound or a video in the conversation |
|---|---|
/retry |
the reply is written again; the prompt's rows are in the context but for the last |
/rewind [N] |
the messages dropped take their media and embeddings with them; the rows before them stay cached |
/reset, /system TEXT |
the conversation's media go; what is attached to the next message stays |
/detach [N] |
removes what is attached to the next message, documents and media alike |
/context N |
the new context evaluates the whole conversation again from the embeddings kept: no encoder runs |
/decide A | B |
scores the labels on the conversation's text: its pictures and sounds are not in the decision's prompt (said); the next turn evaluates the rows again, without encoding |
/save, --chat-file,
--resume |
the file names each message's media files
("media": ["front-page.ppm"]); --resume reads
them again and encodes them once; with --kv-save their rows
come back from the KV file (version 2) and are not evaluated again. A
file that is gone, or a chat without the projector, starts a new
conversation and says so |
/export |
Markdown lists each message's files (Attached: NAME);
the JSON holds the text |
/model |
refused while the chat holds media: their rows are this model's projector's. Without media the model changes, and the projector with it: the catalogue's for the new model, or none |
--rag, --rag-critique |
the question is searched as text; its sources go in the prompt beside the message's media, and the critique's retry is rendered the same way |
/tool RESULT |
a tool message holds no media |
A conversation that no longer fits the context says so with its size
in rows (/rewind, /context N,
/reset); a file the encoder cannot read, or a template with
no place for a picture, drops the message with what was attached to it
and the chat goes on. A projector whose embeddings are not of the
model's width is refused at start
(it is another model's projector). The context is not sized
for the pictures as oklm describe's is:
/attach tells the rows a picture will take when the encoder
says so before it runs (Qwen2-VL, Qwen2.5-VL) and warns when they are
more than the context has left.
Measured on the CPU (2026-11-15, --temp 0, the machine
shared with other work): SmolVLM-256M (Q8_0), the front page of
test_mm_public's answer case at 640 x 488: the turn with
the picture 348 rows, 320 of them the picture's, encoded in 4.3 s and
evaluated in 0.22 s; the next question 375 rows, 361 cached, 14
evaluated in 0.02 s, nothing encoded; /retry 1 row;
/rewind and another question 11 rows. Ultravox v0.5 on
Llama 3.2 1B (Q8_0), jfk.wav (11 s): 228 rows, 188 the
sound's, encoded in 8.5 s; the next question 294 rows, 276 cached, 18
evaluated in 0.07 s; /retry 1 row. Qwen2-VL 2B (Q4_K_M),
the same front page: 418 rows, 391 the picture's (9.4 s to encode, 1.7 s
to evaluate); the next question 15 rows of 473; a second picture two
turns later 292 rows of 773, one item of two encoded;
--resume with --kv-save in a new process: 813
rows from the file (24.1 MB), 19 evaluated of 832, the two pictures
encoded again and not evaluated. The turn with the second picture gives
the same text when the whole conversation is evaluated from nothing
(/context 8192 before it) as when 481 of its rows come from
the cache. Gemma 3 4B (Q4_K_M), whose picture is 256 rows that see each
other: 272 rows for the picture's turn, the headline read; the next
question 22 rows of 312; /retry 1 row.
tests/harness/test_cli_chat.c (test_media)
runs this on a tiny model and tiny encoders made by the test, a vision
one and an audio one: the rows --stats tells for a
picture's turn, the next one, /retry, /rewind,
the same picture attached again (not encoded), /decide,
/context N, /reset; the conversation's file
and the Markdown export; --resume with and without the
projector; a sound; a retrieval turn with its critique; a tool's result;
a conversation longer than the context; a projector of another width;
the catalogue's projector and /model away from it.
tools/check_chat_tui.sh attaches a picture in the
full-screen UI on SmolVLM-256M.
A full context
oklm chat, the chat window (gui.md) and oklm agent
(agent-sessions.md) do two things by
themselves when a conversation reaches the end of the context. Each of
the two can be left to the person, in each program.
Compaction. Past a share of the context the older
messages are summarized by the model into a note, and the recent ones
stay as they are. The code is the agent's, in one place for the three
programs (src/client/compaction.c):
- When. The agent's threshold: a share of the context less the reply's reserve (the reply's token limit, a quarter of the context at most). The share follows the model's size: 0.60 under 2 billion parameters, 0.75 under 8 billion, 0.85 above or when the size is not known (a server at an address). With a context of 512 tokens, replies of 1024 and a small model: 230 tokens.
- What stays as it is. The last exchange (the person's latest message and what follows it), always; the exchanges before it, the latest first, while together they take a quarter of the context at most; a message that carries a picture, a sound, a video or an attached document, with its reply; the system prompt.
- The note. The summary of the other messages. The
model reads it after the system prompt, in the same
systemmessage, after the sentenceThis session continues an earlier conversation that grew too long for the context; it was summarized:. A later compaction summarizes the earlier note with the messages that go. - The summary. Asked after the conversation when the context has room for the request and the answer (the cache is reused), else from a transcript of the messages that go, read in parts the context holds, each part's summary carried into the next. An eighth of the context at most (48 to 1024 tokens), with reasoning off.
- Nothing to gain. When the messages that would go do
not hold an eighth of the context more than the note may take, nothing
is compacted: the note would free nothing worth a summary
(
oklm_compact_gains()). The agent holds its own compactions to the same rule (it did not: with a system prompt and tools that nearly filled a server's context, each compaction made the prompt longer, until the server refused it), and says so when its own part leaves no room for a conversation (agent-sessions.md).oklm chatandPOST /api/compactwere run on the same case, a system prompt of 535 tokens in a context of 512: nothing is compacted ("compacted":0), and the chat saysThe context is full (535 of 512 tokens): the message was not answered. - What was replaced is kept.
/exportwrites the replaced messages before the note, underCompacted, and the conversation's file keeps them ("compacted") beside the note ("summary").
Continuing. A reply that the context or the token
limit cut is gone on with: the conversation is given to the model again
with the cut reply as the open start of its turn, with no end of turn
after it, so the model writes on from the reply's last token. The rule
is one function for the three programs
(src/client/reply_continue.c):
- Room first. When the context is what cut the reply, room is made before it goes on: a larger context where one fits, else a compaction, which then keeps the last exchange alone. A note says which. When neither makes room (the reply alone fills the context) the reply stays cut, and a note says that a larger context holds more.
- The token limit. The limit a program starts with is
the size of one piece: 512 tokens in
oklm chat, 1024 in the window, 8192 in the agent. A reply goes on past it, to its own end. A limit the person gave (-n N,/max-tokens N, the window's Reply length moved, the agent's--max-tokens N) ends the reply for good, with no note: it is what was asked for./continuestill goes on with it, one piece. - A bound. A reply is continued 8 times in a row at
most (
--continue-max N,/autocontinue N). A reply that repeats itself (its last 160 bytes found twice more in the 4000 bytes before them, or a piece that added nothing) is not continued. Either way a note says so, and that/continuegoes on. - Reasoning. The open turn is the reply as the model
wrote it, its
<think>block included: cut inside its reasoning, the model goes on reasoning and then answers.oklm chatand the in-process agent write the reply's own tokens after the prompt, so the rows are those the cache holds. Over a server the text is sent back (server.md); for a model that writes its reasoning in a channel (harmony, Gemma 4, Muse Glimmer) the server writes the turn again in the model's channels and leaves it open in the one the reply was cut in. Inoklm chatsuch a reply cut inside its reasoning keeps its block open on the screen, and the next piece goes on in it: Gemma-4-E2B-it Q4_K_M on the CPU, thinking on, the question of the server's measure in a context of 128 tokens, is cut inside its reasoning (the context made 256), again inside its answer (made 512), and what is printed is byte for byte what a context of 4096 prints, 228 tokens. A reply of such a model cut inside the first header of its turn (<|channel|>,<|channel>,to=self: its first tokens) has written markup and nothing else: nothing of it is shown, kept or sent back as text, and the turn is opened afresh, the template writing the header, in the server,oklm chatand the agent alike (server.md). - Tool calls. A reply cut inside a tool call is gone
on with like any other, and the call is not parsed, shown as a call nor
run while it is half written. The calls' grammar reads the call's text
so far (
oklm_tool_call_open()) and the model goes on under it, so what comes out is a whole call, parsed with the whole reply.oklm chatand the in-process agent do it on the reply's own text; an agent on a server sends the call's text back astool_call_open(server.md). It needs a format whose calls open with a marker (not Llama 3's bare JSON nor GLM-4's name line) and the calls' grammar (notoklm agent --no-tool-grammar).oklm chat --tools FILE --tool-choice requiredon SmolLM2-135M with replies of 8 tokens (-n 2): the call is cut inside, shown as text, and after the/continues that end it the one[tool calls]line is the line of the reply written in one go (test_call_cut).
The dashboard's chat does the same in a page: /compact,
/continue, /autocompact,
/autocontinue, a Continue on a cut reply,
both by themselves unless set otherwise (kept in the browser), with the
server's requests: the open last message, and
POST /api/compact for the plan and the summary (server.md). Room for a reply
the context cut is made as the chat window makes it: a larger context
first, asked of the server's resize after its dry run, where the server
is not --fixed-context; else a compaction (server.md).
One reply on the screen. The pieces of a continued
reply are one reply wherever it is shown or kept. oklm chat
in line mode writes them one after the other on standard output, the
note about the room made between them on standard error. In the
full-screen UI (--tui, /tui) they are one
block: a piece that ends cut leaves the block open (the Markdown
renderer and the reasoning's fold as they were), the next piece is
written after its last word, and what was said meanwhile
((the context was full: it is now 256 tokens, and the reply goes on; ...),
a compaction's two lines) is shown after the block, not inside it. A
piece asked for later with /continue follows its own line
of the transcript, as reasoning when the reply was cut inside its
reasoning. /copy, /retry,
/rewind, /export and the conversation's file
take the conversation's message, which is one reply in both modes:
tools/check_chat_tui.sh runs the same question in a context
of 128 tokens in line mode and in the full-screen UI, where the words on
each side of the cut follow one another, /copy takes the
same 678 bytes, and the conversation's file and /export's
are the same files.
oklm chat |
The chat window | oklm agent |
|
|---|---|---|---|
| At start | --auto-compact on|off,
--auto-continue on|off, --continue-max N, over
the settings for the run |
the same options | the same options, over the settings for the run |
| Commands | /compact [FOCUS], /continue,
/autocompact [on|off],
/autocontinue [on|off|N] |
the same, a Continue on a cut reply, two boxes in the model settings | /compact [focus], /continue,
/autocompact [on|off|auto|TOKENS],
/autocontinue [on|off|N] |
| Kept | autoCompact, autoContinue,
continueMax in the user's settings, the agent's |
in the window's settings file | autoCompact, autoContinue,
continueMax in the user's settings |
| Room for a cut reply | a context twice as large, up to the one the model was trained with, when the memory plan holds it; else a compaction | the next context the Context slider offers, when the line under it has nothing to warn of; else a compaction | in process as oklm chat; on a server that allows
resizing, the same size asked of it after its dry run; else a compaction
(agent-sessions.md) |
| Room before a turn | the same larger context when the conversation leaves less than a turn's room (512 tokens), else a compaction when it frees room; when the system prompt and the tools alone leave no room, the message is not sent and the agent says so (agent-sessions.md) |
One setting for the two terminal programs.
oklm chat reads autoCompact,
autoContinue and continueMax where
oklm agent keeps them (agent-settings.md: the user's
~/.config/oklm/settings.json,
$XDG_CONFIG_HOME/oklm/settings.json when that is set, with
the settings of the directory it runs in over it, as the agent reads
them). /autocompact on|off and
/autocontinue on|off|N in either program write the user's
file and say so,
(saved in PATH, for oklm chat and oklm agent), so the other
program starts with what was chosen. The start options are over the file
for the run and write nothing. The chat window keeps its own three in
its settings file (gui.md), which
the terminal programs do not read.
Manual. With --auto-compact off a full
context says so and names the ways out, and nothing is summarized until
/compact:
(The context is full (271 of 256 tokens): the message was not answered. /compact summarizes the older messages, a larger context (/context N) holds more; /continue then answers the message.)
With --auto-continue off a cut reply says
(The reply was cut by the context: /continue goes on with it.),
or the same of the reply's token limit, and /continue goes
on with it. /continue also answers a message the full
context left unanswered, once there is room.
Measured on the CPU (tests/harness/test_cli_chat.c,
test_full_context, SmolLM2-135M Q8_0, temperature 0):
- "Write three sentences about the sea." in a context of 128 tokens: the reply is cut at the context's end, the context is made 256 tokens, and the reply ends at 138 tokens, the same text as in a context of 2048.
- With
--auto-continue offthe same reply stops at 128 tokens of context with its note;/continuemakes the context 256 and ends it at the same 138 tokens. - With
-n 4(16 tokens a reply) the reply stops at 16 tokens without a note;/continueadds 16, one token of the prompt evaluated, the rest in the cache. - "Write a long story about a lighthouse keeper and the sea." with
--continue-max 1: continued once, then(The reply was continued 1 time in a row and is still not ended: ...). Asked in a context of 256 tokens without the bound, the context is made 512, and there the reply is not continued: it repeats itself, which the note says. - Five short questions in a context of 256 tokens with replies of 48:
with compaction manual the fifth does not fit (271 tokens),
/compacttakes the conversation from 271 tokens to 131 (6 of 9 messages become the note) and/continueanswers the message; automatic, the conversation is compacted twice on the way (163 tokens to 42, 169 to 40) and every message is answered.
A long answer continued across a compaction, on Qwen3-0.6B Q8_0 on
the CPU (oklm serve -c 2048 --fixed-context, the chat
window linked to it, /thinking off, temperature 0): after
three questions about tides (444, 403 and 369 tokens of replies) an
essay of about 1200 words on the history of lighthouses was asked for.
Its prompt took 1339 tokens and the reply was cut by the context after
709. The six older messages became a note of 288 bytes, and the reply
went on from a prompt of 832 tokens for 521 more, to its own end: 1230
tokens. Read as exported, it is one essay: one title, its sections in
order from the ancient world to the future and a conclusion, no heading
twice and no restart where it was cut. What it says is a 0.6B model's
(it repeats its phrases, and its history is thin), before the cut as
after it.
Choosing a model
/model without a name is a selection among the
catalogue's models (models.md), listed under
the names of their groups as the model manager lists them: each with its
size, whether it is the model in use, installed or to download (and
whether its licence asks for agreement), and its description. A model of
two groups is in both. The models a chat cannot serve (images, sounds,
retrieval) are listed dimmed, with what they are for in brackets, and
cannot be chosen. In the full-screen UI the list is a panel: the arrows
move the selection (over the groups' names), typing searches it, Enter
chooses, Esc keeps the model in use.
+- Choose a model (arrows and Enter; type to search; Esc keeps the current one) ---------------+
| General chat |
| > smollm2-135m 138 MiB current the smallest chat model: a f |
| qwen3-0.6b 610 MiB installed a small chat model with a th |
| llama3.2-1b 770 MiB to download (licence) Meta's 1B chat model |
| gemma3-1b 769 MiB to download (licence) Google's 1B chat model (text |
| phi3.5-mini 2.2 GiB to download Microsoft's 3.8B chat model |
| ... |
| Coding |
| qwen3-8b 4.7 GiB to download the default general model fo |
| ... |
| Images |
| smolvlm-256m 348 MiB to download [images: oklm describe] desc |
| ... |
| Sounds |
| whisper-base 281 MiB to download [sounds: oklm transcribe, ok |
+----------------------------------------------------------------------------------------------+
In line mode the same list is printed with a number before each model
a chat can serve, and the chat asks for a number or a name (Enter keeps
the model in use). /models prints the list and asks
nothing. /model NAME takes a catalogue name, a number of
the list or a model file.
A model that is not installed is downloaded only once the selection
asked: the question names its size, its licence and the address of the
licence's text, and where the files go. For a model whose licence asks
for agreement the answer is the agreement (yes typed in
full in line mode), recorded as oklm models pull records
it. In the full-screen UI the question is a panel whose first entry is
"No":
+- llama3.2-1b is not installed: download 770 MiB? --------------------------------------------+
| > No: keep the current model |
| nothing is downloaded |
| Yes, I agree to its licence: download 770 MiB |
| licence llama3.2: https://www.llama.com/llama3_2/license/ |
+----------------------------------------------------------------------------------------------+
The download shows its progress; Ctrl-C (or Esc in the full-screen UI) stops it, what arrived is kept and the same choice goes on with it.
The model chosen is loaded in place of the one in use and the
conversation is kept: the chat tells its messages and its tokens for the
new model, which reads it again on the next turn. The new model's
tokenizer, template and tool-call grammar are made, and the conversation
is counted with them, before anything of the old model goes; the old
context is then freed before the new one is made, and made again if the
new one fails. Nothing is dropped from a conversation: when it is longer
than the new model's context would be, the model in use stays and the
chat says how to make room (/rewind, /reset,
/context N). A file without a tokenizer or with a template
that cannot render the conversation is refused the same way. With
--kv-save, the cache's file is removed (it held the old
model's rows). The agent's /model is the same selection (agent-ui.md).
On a server
oklm chat --url URL (since 2026-11-17) is the same chat,
in line mode and in the full-screen UI, with the model of an
OpenAI-compatible server in place of one in this process: an
oklm serve, llama.cpp's server, or another. Each turn is a
streamed POST /v1/chat/completions through the client
oklm agent --url uses (the agent's HTTP backend, without
tools; agent-loop.md):
transient failures are sent again as the agent sends them. The key is
--api-key KEY, else OKLM_API_KEY, else
OPENAI_API_KEY. -m and --url
exclude each other. With neither, the baseUrl of the
settings is the server, and their model the name sent, as
for the agent.
$ oklm chat --url http://127.0.0.1:8080
server: http://127.0.0.1:8080, model models/qwen3-0.6b/Qwen3-0.6B-Q8_0.gguf, a context of 32768 tokens (the default for a server that does not tell its own). Commands: /system TEXT, /reset, /model, /tui, /exit (/help lists them all)
The model. --remote-model NAME names
it. Without it the chat asks the server (GET /v1/models,
oklm_client_remote_models() in
src/client/remote_models.c, which reads OKLM's answer,
llama.cpp's and an OpenAI-style one): a server that lists one model gets
that one; one that lists several shows them in a list to choose from on
a terminal; one that lists none is sent default. When the
input is not typed (a pipe, a script) and several are listed, none is
taken: the chat ends with status 2, the list and the option's name,
because the first of a list is the server's order and not a choice.
/model lists the server's models with the one in use marked
and, on a terminal, chooses among them; /model NAME or
/model N asks for one from the next message, listed or not,
the conversation kept; /models lists them.
The window. An oklm serve tells its
context (GET /api/context) and the chat counts with it;
/context N asks the server for another size
(POST /api/context, force as in process).
Another server tells none: the window is -c N when given,
else the contextWindow setting, else 32768 as for the
agent, and /context N sets the size this chat counts with.
The conversation's size against the window is an estimate: the server's
count of the last turn (its usage) and four bytes a token
for what was added since, four bytes a token for everything when the
server tells no count. /stats, /context and
the status bar say "about".
A full context (as in
process). Automatic compaction starts where that estimate passes the
agent's threshold, with the share of a model whose size is unknown
(0.85); the summary is asked of the server's model, without reasoning
(enable_thinking false), and the figures the chat tells are
estimates. A prompt the server refuses for its length
(oklm serve's 400 with its counts, another server's 400
naming the context) is compacted once and sent again when the chat
compacts by itself, and said with the ways out otherwise. A reply the
token limit cut is gone on with as in process: the reply so far goes
back as the last message, left open ("prefix": true, which
oklm serve reads; llama.cpp's server goes on with a last
message of the model by itself). A server that refuses an open last
message (400 or 422) is told so once, and from then on a cut reply goes
back closed, followed by a request to go on with it exactly where it
stopped; the conversation keeps one reply either way. A reply that ends
for its length with fewer tokens than its limit was cut by the server's
context: an oklm serve that allows it is asked for a larger
one, by the rule the agent uses; otherwise the conversation is
compacted, or the chat says there is no room. A reply cut before its
answer began (inside its reasoning, which a server sends apart) leaves
nothing to go on with: the chat says so, the message stays, and
/continue asks again.
Reasoning. Reasoning deltas
(reasoning_content) are shown as a model in process writes
its reasoning, within <think> tags: printed as they
are in line mode, folded in the full-screen UI (Ctrl-O). The
conversation and its file keep the answer alone.
What is sent. The system prompt and the messages;
max_tokens (the reply's limit, -n and
/max-tokens); temperature always, as the agent
sends it; top_p, top_k, min_p and
seed once the person gave them (--top-p...,
/top-p...), because a server that does not know one of them
may refuse the request, and what is not sent is the server's default;
--template-vars and /thinking on|off as
chat_template_kwargs, which a server whose template has no
such switch ignores. --repeat-penalty is not sent. A server
that refuses one of these members by name (a 400 or a 422:
Unrecognized request argument supplied: top_k) is asked
again without it at once, the chat says so once,
(the server does not take top_k: --top-k has no effect on it, and is no longer sent),
and no later request carries it; the same holds for
stream_options (the tokens are then estimated) and
chat_template_kwargs (agent-loop.md).
A refusal that names no member the chat sent stays an error, shown as
the server worded it.
What each command and option does on a server:
| Command or option | On a server |
|---|---|
/help, /exit, /tui,
/system, /reset, --system |
as in process |
/save, /export, /copy,
/retry, /rewind, --chat-file,
--resume |
as in process: the conversation and its file are the chat's own, in the same format |
/compact, /continue,
/autocompact, /autocontinue,
--auto-compact, --auto-continue,
--continue-max |
as in process, by estimate (above) |
/temp, /top-k, /top-p,
/min-p, /seed, /max-tokens,
/thinking, --template-vars,
-n |
sent in each request (above) |
/model, /models,
--remote-model |
the server's models |
/context, -c |
the server's context, or the size the chat counts with |
/stats, --stats |
the server, its model, the estimated use, the last turn by the server's counts (the prompt, of it the tokens in the server's cache, the reply), the sampling sent |
/attach FILE (a document), /detach |
as in process, its size estimated |
/attach FILE (a picture, a sound, a video),
--mmproj |
needs the projector in this process: said in a line
(--mmproj: refused at start) |
/rag, --rag and its options |
need the model in this process, which searches and judges: said in a
line (--rag: refused at start) |
/decide |
needs the model in this process, whose logits score the options: said in a line |
/tool, --tools |
need the model in this process, which writes a call under a grammar:
said in a line (--tools: refused at start;
oklm agent --url gives a server's model tools) |
--kv-save, --chat-template,
--grammar, --json-schema, --json,
--lora, --draft, --mtp,
--observe |
need the model in this process: refused at start, status 2 |
--backend, --gpu-layers and the other
options of where a model computes |
nothing to place: without effect |
What a failure says: a key the server refuses, or wants and was not
given, with its status and its reason and the two ways to give one
(status 1: at start when the server keeps its list behind the key, at
the first message otherwise); a server that cannot be reached; a model
the server does not have, with /model to list them; any
other error status with the server's own reason. After a failed turn the
message stays in the conversation and /continue asks
again.
tests/harness/test_cli_chat_url.c runs all of this
against a server of its own that records each request (127 checks, no
model loaded); tests/client/test_remote_models.c reads the
lists (55 checks) and tests/client/test_request_members.c
the member a refusal names and the request without it (34 checks);
tools/check_chat_url.sh runs it on llama.cpp's server and
on oklm serve with Qwen3-0.6B on the CPU (testing.md).
Verification
| Test | Reference | Result |
|---|---|---|
tests/model/test_model.c (committed fixtures) |
numpy float64 forward pass (tools/infer_ref.py) on
three tiny random models: llama (GQA, Llama 3 rope factors), qwen3
(biases, Q/K norms, tied F16 embeddings), gpt2 |
max logit difference 8e-7 / 1.4e-5 / 3.9e-5 with an F32 KV cache, every greedy choice identical; F16 KV cache within 0.04 |
| same | token-by-token evaluation | batched, multi-batch and truncate-and-replay logits bit-identical |
tests/model/test_infer_public.c
(make test-public) |
tools/infer_ref.py on the published
SmolLM2-135M-Instruct GGUFs, 5 prompts x 32 greedy steps, replaying the
reference's tokens |
F16: max logit difference 2.3e-5, 160/160 greedy choices equal; Q8_0: 1.69, 158/160; Q4_K_M: 2.11, 156/160 |
| same | llama.cpp v0.8.0 (tools/gen_llama_ref.sh) on the 8
corpus files and the SmolLM2 checkpoint loaded directly, same 5 prompts
x 32 steps |
see the table below |
tests/model/test_model.c |
the GGUF conversion of each tiny Hugging Face checkpoint | logits within 1e-6 (llama) or identical (qwen2, gpt2) |
The reference is written from the model definitions, not from OKLM's code, and reads the weights through gguf-py's own dequantizers, in float64 with float activations.
For quantized files OKLM (like llama.cpp) rounds each activation row
to Q8 before the product, and the reference does not. How far apart two
equally valid evaluations of a quantized model are was measured with
tools/infer_ref.py --act-q8, which rounds the activations
the way ggml does: against the plain reference, on the steps where both
follow the same tokens, its largest logit difference is 1.31 (Q8_0) and
1.78 (Q4_K_M) with 97 % of greedy choices equal. That is the size of
OKLM's own difference, and it cannot be made smaller by emulating the
rounding: across 30 layers, float32 / float64 differences flip a few
roundings, and each flip changes the next layers' inputs. The test
bounds are therefore 2.5 (Q8_0) and 3.0 (Q4_K_M) with 95 % agreement;
the exact behaviour of each quantized kernel is checked separately, bit
for bit, in tests/core/.
Perplexity of the first 100 kB of Tiny Shakespeare
(oklm ppl -c 512, 15 300 scored tokens): F16 30.94, Q8_0
31.08, Q4_K_M 31.77 (the windows start with their own first token:
SmolLM2 has a BOS id but does not add it, as llama.cpp does);
llama-perplexity 30.94, 31.08, 31.79.
Against llama.cpp
llama.cpp is built from a pinned release by
tools/gen_llama_ref.sh and used through
tools/llama_ref.c, which records its tokenization of each
prompt, its greedy continuation (32 tokens) and its logits at every
step. The test checks three things: the prompt tokenizes to the same
ids; on llama.cpp's tokens, OKLM's logits agree; and OKLM's own
free-running greedy continuation is compared with llama.cpp's. Where the
two continuations part, everything before is identical, so llama.cpp's
logits for that step are known: the last column is the margin by which
llama.cpp itself preferred its token over OKLM's.
| file | prompt ids | choices equal on llama.cpp's tokens | max logit difference | identical continuations | tokens before parting | llama.cpp margin where they part |
|---|---|---|---|---|---|---|
| SmolLM2 F16 | 5/5 | 160/160 | 0.047 | 5/5 | 160/160 | - |
| SmolLM2 BF16 checkpoint, loaded directly | 5/5 | 160/160 | 0.039 | 5/5 | 160/160 | - |
| SmolLM2 Q8_0 | 5/5 | 156/160 | 1.71 | 2/5 | 129/160 | 0.15 |
| SmolLM2 Q4_K_M | 5/5 | 154/160 | 2.10 | 1/5 | 93/160 | 0.11 |
| Qwen2.5-0.5B Q4_K_M | 5/5 | 160/160 | 1.72 | 5/5 | 160/160 | - |
| Qwen3-0.6B Q8_0 | 5/5 | 158/160 | 0.82 | 3/5 | 104/160 | 0.08 |
| Llama-3.2-1B Q4_K_M | 5/5 | 157/160 | 1.06 | 2/5 | 143/160 | 0.09 |
| TinyLlama-1.1B Q4_K_M | 4/5 | 156/160 | 0.80 | 2/5 | 110/160 | 0.06 |
| GPT-2 Q4_K_M | 5/5 | 152/160 | 139 | 1/5 | 62/160 | 0.26 |
| Qwen3-8B Q8_0 (made by OKLM) | 5/5 | 159/160 | 1.07 | 4/5 | 145/160 | 0.13 |
| Qwen3-8B Q4_K_M (made by OKLM, importance matrix) | 5/5 | 155/160 | 2.21 | 2/5 | 104/160 | 0.43 |
| OLMoE-1B-7B-0125-Instruct Q4_K_M (64 experts, 8 per token) | 5/5 | 157/160 | 2.04 | 3/5 | 117/160 | 0.12 |
DeepSeek-Coder-1.3B-Instruct Q4_K_M
(llama.rope.scale_linear) |
5/5 | 159/160 | 7.26 | 4/5 | 143/160 | 0.16 |
| DeepSeek-Coder-6.7B-Instruct Q4_K_M (the same keys) | 5/5 | 159/160 | 2.81 | 4/5 | 157/160 | 0.08 |
Qwen3.5-9B Q4_K_M (qwen35, Gated DeltaNet blocks) |
5/5 | 159/160 | 13.8 | 4/5 | 154/160 | 0.44 |
Qwen3.8-27B UD-Q4_K_M (qwen35, 64 blocks) |
5/5 | 160/160 | 3.22 | 5/5 | 160/160 | - |
Cyber-Ornith 1.5 9B Q4_K_M (qwen35, a tune of
Qwen3.5-9B) |
5/5 | 158/160 | 3.58 | 3/5 | 131/160 | 0.25 |
MiMo-V2.6-Distill-Qwen-9B Q4_K_M (qwen35, a tune of
Qwen3.5-9B) |
5/5 | 158/160 | 2.85 | 3/5 | 120/160 | 0.08 |
BugTraceAI-CORE-Ultra 27B Q4_K_S (qwen35, a tune on
Qwen3.8-27B's layout) |
5/5 | 157/160 | 3.59 | 3/5 | 115/160 | 0.08 |
Uraion Cyber V1.0 Q4_K_M (qwen35moe, a tune of
Qwen3.6-35B-A3B) |
5/5 | 159/160 | 4.40 | 4/5 | 158/160 | 0.38 |
Imperum-CybersecurityLLM v1.0 Q4_K_M (qwen35moe, the
same base) |
5/5 | 154/160 | 2.59 | 2/5 | 97/160 | 0.26 |
CyberStrike-OffSec-35B Q4_K_M (qwen35moe, the same
base, five shards) |
5/5 | 154/160 | 2.30 | 1/5 | 94/160 | 0.14 |
Spark-X2.6-1.7B Q4_K_M (spark2_5) |
5/5 | 159/160 | 4.53 | 4/5 | 129/160 | 0.03 |
Spark-X2.6-4B Q4_K_M (spark2_5) |
5/5 | 158/160 | 16.2 | 3/5 | 122/160 | 0.20 |
Muse Glimmer 30B UD-IQ2_XXS (muse-glimmer) |
5/5 | 157/160 | 5.51 | 3/5 | 114/160 | 0.10 |
Gemma-4-12B Q4_K_M (gemma4; see Gemma 4 for why it is wide) |
5/5 | 148/160 | 12.5 | 1/5 | 66/160 | 1.64 |
Gemma-4-12B, blocks 4 and 5 alone in F16
(tools/gguf_cut_blocks.py; prompts given as ids) |
- | 96/96 | 0.25 | 3/3 | 96/96 | - |
Gemma-4-E2B Q4_K_M (gemma4 with an embedding per layer
and blocks sharing keys and values) |
5/5 | 160/160 | 8.08 | 5/5 | 160/160 | - |
| Gemma-4-E4B Q4_K_M (the same form) | 5/5 | 156/160 | 3.51 | 1/5 | 79/160 | 0.34 |
Reading it:
- With F16 weights the continuations are identical. The 0.047 of logit difference is llama.cpp rounding activations to F16 for F16 products; OKLM keeps them in F32 and matches the float64 reference to 2.3e-5.
- With quantized weights both engines round activations to 8 bits, then sum in different orders, which moves logits by 1 to 2. Continuations stay identical until two candidates are nearly tied: wherever they part, llama.cpp's own margin was at most 0.15 (0.26 for GPT-2), an order of magnitude below the noise between the engines. OKLM never disagrees with llama.cpp on a clear choice. The test bounds that margin at 0.5.
- Qwen3.5-9B's 13.8 is on single logits far below the best one, at
four steps of the prompt it answers by repeating one sentence, where the
next token is all but certain; every other step is within 0.4 to 0.7 and
the mean difference over all logits is 0.116. Those steps are that
sensitive to rounding in OKLM alone: with an F32 instead of an F16 cache
its own logits move by 0.121 on average and by up to 10 at the same
steps (
tests/model/test_qwen35_public.chas the other checks of that file). And in llama.cpp alone: with F32 keys and values, or its prompt given a token a pass, it moves its own logits by 0.19 to 0.23 on average over 100 steps and by 13.7 to 14.9 at those steps, and the hidden states of two of its runs part in the same blocks as OKLM's and its own (hybrid-layers-design.md). - Spark-X2.6-4B's 16.2 is one step of the code prompt (5.5 and 4.8 on
the steps beside it) and its 9.4 one step of the first prompt; every
other step is under 2.6. The same sensitivity: with an F32 cache OKLM
gives 2.76 and 0.69 at those two steps, 3.88 at most over the 160, and
159 choices. On the 1.7B's unquantized file (BF16 weights, 3.4 GB) the
two engines agree on 160 of 160 choices with logits within 0.35,
llama.cpp rounding the activations of a BF16 product to BF16 (8 bits of
mantissa against F16's 11, hence more than SmolLM2's 0.047). The five
prompts stay inside the window of 512 positions; past it (the licence's
first 3600 bytes, 769 tokens, then 16 tokens; a copy of
tools/llama_ref.cwith a context of 2048, not kept) the 1.7B gives 14 of 16 choices and logits within 1.01, the 4B 16 of 16 and 1.42. - GPT-2 has activation outliers of very large magnitude: 8-bit
rounding of activations moves single logits by up to ~140 in both
engines (the float reference with ggml's rounding,
tools/infer_ref.py --act-q8, gives the same greedy tokens as OKLM and llama.cpp; without the rounding it does not). - Qwen3-8B, the default model, is made from the official BF16 weights
by
tools/make_qwen3_8b.sh(OKLM's conversion, importance matrix and quantization) and its rows are skipped when the files are missing. On OpenCL: Q8_0 159/160 (1.12), Q4_K_M 158/160 (2.68). Its perplexity on the last 30 KB of Tiny Shakespeare (512-token windows, 4080 tokens): Q8_0 6.997 in llama.cpp, 6.972 on OKLM's CPU, 6.984 on OpenCL; Q4_K_M 7.830, 7.820 and 7.697. The OpenCL figure is the exact one for those weights: the same Q4_K weights unpacked to an F32 file give 7.697 on the CPU, where no activation is rounded. Both CPU engines round activations to 8 bits for K-quant products, which costs this model 1.6 % (Qwen3-4B 0.4 %). - OLMoE-1B-7B (allenai's GGUF,
tests/public_models_large.txt): the three choices that differ on llama.cpp's tokens and the two places where the free-running continuations part are near ties (llama.cpp's own margin at most 0.12). Perplexity on the last 30 KB of Tiny Shakespeare (512-token windows, CPU): 2.2015 in llama.cpp, 2.1963 in OKLM (the file has a BOS id but does not add it, so neither engine puts BOS at the window starts). On OpenCL (float kernels, astest_infer_publicruns them): 157/160 choices equal on llama.cpp's tokens, max difference 1.84, 3/5 continuations identical, llama.cpp's margin where they part 0.04. With the default 8-bit matrix units, against OKLM's CPU on the CPU's greedy tokens: all layers on the device 155/160 (max difference 2.08), 8 of 16 layers 157/160,--n-cpu-moe 8158/160,--n-cpu-moe all155/160 and the same with a 50 % expert cache,--n-cpu-moe all --expert-gpu 1.8154/160; every choice that differs is a near tie on the CPU (its margin at most 0.26). Perplexity on the same text: 2.1959 (all layers), 2.1997 (--gpu-layers 8), 2.1969 (--n-cpu-moe 8), 2.1958 (--n-cpu-moe all, and with--expert-cache 1.8), 2.2015 (--expert-gpu 1.8), against 2.1963 on the CPU. - The TinyLlama file stores merges with all vocabulary scores 0:
llama.cpp ignores the merges and merges by score, leftmost first on
ties; OKLM uses the merges, which gives Hugging Face's tokenization of
the model ("def fibonacci(n):" is
▁fibonacciin both OKLM and Hugging Face,▁fibonacciin llama.cpp). - Without
tokenizer.ggml.add_bos_token, OKLM adds BOS by the same rule as llama.cpp: for SentencePiece and for the Llama 3 family of pre-tokenizers (llama-bpe...).
Published files on a device
tests/model/test_archs_public.c runs the published files
of the architectures whose device paths needed no kernel of their own on
OpenCL and on Vulkan, every block on the RX 7900 XT: the 160 steps of
the corpus on llama.cpp's tokens in a device context and in a CPU one
(2026-11-14; the CPU's choices equal to llama.cpp's in brackets).
| file | backend | choices equal to llama.cpp's | mean / max logit difference from llama.cpp | choices equal to the CPU's | mean / max from the CPU |
|---|---|---|---|---|---|
LFM2-350M Q4_K_M (lfm2) |
OpenCL | 158 / 160 (157) | 0.137 / 2.13 | 157 / 160 | 0.134 / 1.67 |
| Vulkan | 158 / 160 | 0.127 / 1.82 | 157 / 160 | 0.124 / 1.46 | |
Cyber-Prime 1.1 2.6B Q4_K_M (lfm2) |
OpenCL | 152 / 160 (154) | 0.228 / 4.44 | 150 / 160 | 0.214 / 4.00 |
| Vulkan | 151 / 160 | 0.225 / 7.36 | 149 / 160 | 0.216 / 7.52 | |
Tini-Cybersec-8B-A1B Q4_K_M (lfm2moe) |
OpenCL | 155 / 160 (156) | 0.315 / 7.30 | 157 / 160 | 0.295 / 8.70 |
| Vulkan | 155 / 160 | 0.327 / 5.65 | 159 / 160 | 0.314 / 5.42 | |
StarCoder2-3B Q4_K_M (starcoder2) |
OpenCL | 159 / 160 (159) | 0.193 / 1.72 | 158 / 160 | 0.187 / 1.63 |
| Vulkan | 157 / 160 | 0.187 / 1.60 | 156 / 160 | 0.189 / 1.70 | |
Devstral Small 2 24B Q4_K_M (mistral3) |
OpenCL | 159 / 160 (158) | 0.095 / 4.01 | 157 / 160 | 0.093 / 2.46 |
| Vulkan | 158 / 160 | 0.090 / 4.57 | 158 / 160 | 0.085 / 2.89 | |
DeepSeek-Coder-1.3B-Instruct Q4_K_M (llama) |
OpenCL | 159 / 160 (159) | 0.581 / 6.13 | 158 / 160 | 0.626 / 7.42 |
| Vulkan | 157 / 160 | 0.534 / 6.04 | 156 / 160 | 0.601 / 7.33 | |
DeepSeek-Coder-6.7B-Instruct Q4_K_M (llama) |
OpenCL | 157 / 160 (159) | 0.136 / 3.73 | 156 / 160 | 0.129 / 1.55 |
| Vulkan | 159 / 160 | 0.087 / 1.28 | 158 / 160 | 0.087 / 1.91 |
The devices are as far from llama.cpp as the CPU is (its means on the seven files: 0.128, 0.190, 0.237, 0.186, 0.073, 0.557 and 0.089), and the choices that differ are the near ties the CPU's own differ on; the largest differences are single logits far below a step's best one (Tini on OpenCL at step 25: 36.6 where llama.cpp has 29.3 under a best logit of 47.4). Cyber-Prime is the widest: 149 to 152 of 160 on a device where the CPU has 154. Its records block by block (an observed CPU context beside a device one, the five prompts) show no block that parts: the stream's norms differ by 1.5 % at the median of the 160 steps, growing along the blocks. The choices that differ from the CPU's or from llama.cpp's (12 steps on Vulkan, 11 on OpenCL) are ties: the best logit is 0.001 to 0.78 above the second on every side at each of them, where the logits of those steps differ by 0.07 to 1.07 on average. Vulkan's 7.5 is step 16 of the chat prompt, after llama.cpp's token 124895 seven times in a row, where the three sides give three tokens with best logits of 0.39 to 0.55: the difference grows from 1 % at block 2 to 29 % at block 23 without a step between two blocks (OpenCL is 1.04 at most there). Two sequences decoded together give the plain device context's choices on their first 8 steps with a paged cache (15 or 16 of 16 for every file on both backends), with the first half of the blocks on the CPU (14 of 16 for Cyber-Prime, 15 or 16 for the others) and, for Tini, with its routed experts in host memory (15 and 16 of 16).
Prompt and decode, tokens a second (bench_llama, 512 and
128 tokens, the best of two; Devstral on the CPU 128 and 32 tokens, one
run):
| file | CPU | OpenCL | Vulkan |
|---|---|---|---|
| LFM2-350M | 2480 / 123.0 | 29134 / 631.2 | 38828 / 1007.9 |
| Cyber-Prime 1.1 2.6B | 310.3 / 16.9 | 6626 / 222.0 | 8156 / 271.3 |
| Tini-Cybersec-8B-A1B | 451.2 / 25.6 | 7265 / 250.9 | 7720 / 341.1 |
| StarCoder2-3B | 202.2 / 14.3 | 4838 / 139.4 | 4952 / 203.0 |
| Devstral Small 2 24B (14.3 GB, whole on the 20 GB device) | 29.2 / 2.08 | 1017 / 38.8 | 1026 / 44.2 |
| DeepSeek-Coder-1.3B | 538.5 / 33.2 | 10845 / 341.1 | 11479 / 446.0 |
| DeepSeek-Coder-6.7B | 109.1 / 7.10 | 3083 / 111.2 | 2961 / 135.3 |
Tini's CPU rates were measured while a file was being downloaded. Vulkan's prompt rate on Tini was 2523, a third of OpenCL's, until the grouped expert products of a prompt went to the prompt products' tiles (vulkan.md); its row is that session's (OpenCL 6895 to 6958 / 254.8 beside it).
Limits
- Architectures (per family details and what was checked in compatibility.md): llama,
mistral, mistral3 (YaRN with the log multiplier, an attention
temperature by position), qwen2, qwen3, gemma, gemma2, gemma3, phi3,
phi2, granite, granitemoe (embedding, attention, residual and logit
scales), qwen2moe, qwen3moe (MoE: softmax router, top-k experts,
optional renormalization and sigmoid-gated shared expert), deepseek
(dense leading layers, scaled routing, shared experts), deepseek2
(DeepSeek V2 / V3 and GLM-4.7-Flash: multi-head latent attention in
llama.cpp's absorbed form, YaRN, sigmoid routing with a selection bias,
expert groups), glm4 (GLM-4-0414: norms after attention and the FFN
too), glm4moe (GLM-4.5 to 4.7: DeepSeek V3's MoE with QKV biases and
partial rotation), olmoe (Q/K norms over the whole projections, 64
experts, weights not renormalized), gpt2, starcoder2 (GPT-2's block with
separate projections, grouped keys and values, rotary positions), lfm2
and lfm2moe (short convolution blocks with a state per sequence between
attention blocks; sigmoid routing with a selection bias for the MoE),
gpt-oss (attention sinks, a window on every second block, experts with
biases and a clamped activation), spark2_5 and muse-glimmer (a gate on
the attention output, a window on three blocks out of four, a count of
rotated dimensions per kind of block or no rotation outside the window),
mimo2 (values narrower than the keys, the window and the key heads block
by block; on the CPU, checked on a tiny model only). Mixtral loads as
llama. MoE layers run on either backend, routing included (moe.md). - Against llama.cpp on the same machine, the CPU's prompts are ahead (1.02 to 1.94 times its rate) and its decoding level or ahead from four threads (1.00 to 1.09); the GPU rows still behind are in benchmarks.md.
- Sequences of a context have fixed, equal shares of the cache (no cells shared between sequences: a common prefix is copied).
- KV cache types are fixed for the life of a context (ADR-0005).