Activations of a running model
What OKLM can record of what happens inside a model while it
evaluates tokens: how much each block writes to the residual stream,
where the attention of chosen blocks looks, which experts each router
picks, and which tokens the output head would give at each depth; and
what it can take out of a run to see what a block's write or a head does
to the output. It is read with oklm run --observe (a file),
from a server started with oklm serve --observe
(GET /api/activations) and in the model inspector (gui.md). How to read the
figures, and what they do not say, is in What the figures mean.
Status: maintained.
What is recorded
For the last token of each sequence a forward pass evaluates (the last token of a prompt, then each token decoded):
| Figure | What |
|---|---|
resid[l] |
the Euclidean norm of the residual stream after block
l |
attn[l] |
the norm of what the block's token mixing added to the stream: its attention, or the recurrent or convolution block that stands in its place (Gated DeltaNet, LFM2) |
ffn[l] |
the norm of what its feed-forward part added: the dense FFN, or the weighted sum of its routed experts and its shared expert |
embd |
the norm of the stream before the first block: the token's embedding |
| an attention row | for a block asked for and one of its heads, the weights the softmax
gave to the positions before the token, itself included: the last
n_pos of them |
| the experts | for each routed block, the experts its router picked for the token, in rank order, and the weights their outputs were summed with |
| the lens | when asked for (lens_top): for each block, the residual
stream after it given as it is to the model's final norm and output
head, and the most likely tokens of the result with their probabilities:
what the model "would say" if it stopped there. The last block's is the
model's own output |
The norms come from the stream itself: it is read at three points of
a block (before, after the attention's output was added, at the end) and
the differences are what each part added, whatever the block does around
them (norms after the parts, a scale on the residual branch, an
adapter). A block whose two parts are added in one step (Phi-2) gives
their sum as ffn and 0 as attn.
An attention row is computed again from the head's query and the keys
in the cache, with the model's scale, soft-cap, sliding window and
attention sinks, as the attention itself does. The cache holds what was
written to it: with keys kept as half floats (the default) the row is
that of the rounded keys with the unrounded query. A row adds up to 1
when the head saw nothing else; positions before the n_pos
recorded ones and a sink (gpt-oss) take their share without being
listed.
The experts are read where the server's expert map counts them (server.md): the routing the forward pass used.
Bounds: the last 256 positions, the first 64 heads of a block and 64
MiB for the whole recording by default
(oklm_observe_params); the positions are halved until the
recording fits, and a recording that does not fit with 16 positions is
refused.
Limits:
- Blocks on a device (OpenCL, Vulkan) are recorded as
the CPU's are, by reading the device back once a pass: On a device. A block is marked as not observed
(
nullin the arrays,n_observedbelown_layer) only when its read failed. - Multi-head latent attention (DeepSeek V2 and V3) is computed with
the key decompression absorbed into the queries, over one compressed key
a position. The rows are those of that product: for head
hthe score of a position is(W_kb[h]^T q_h) . c + q_rope_h . k_rope, which isq_h . (W_kb[h] c)plus the rotated part: the score of the head the model defines, written the other way round. So the weights are the heads' own, and a row reads as any other model's. - Only the last token of each sequence of a pass: a prompt evaluated
in one pass leaves one record, for its last token.
oklm run --observetherefore writes one line for the prompt and one for each generated token. - Decoding plainly:
oklm runandoklm chatrefuse--observewith a draft model,--mtp,--spec-ngramor--spec-grammar, whose passes evaluate several tokens. On a server those passes are recorded as they are: the record is that of the last token of a pass, which a rejected draft may have been. - Encoders (BERT) and multi-token prediction contexts are not recorded.
Cost. Off (the default), the forward pass tests one pointer per block (measured: Tests). On, the norms and the experts cost two passes over one row of the stream per block; an attention row costs, per block asked for, one dot product a head and a position, on one thread. On a device the rows are copied aside there and read back together when the pass ends: On a device.
On a device
The blocks a device runs (--backend opencl or
vulkan, all of them or the last ones with
--gpu-layers) leave the same record as the CPU's. Nothing
is computed on the device for it: what the recording needs is copied
aside on the device as the pass goes, read back once when the pass ends,
and the arithmetic is done on the host, as for a CPU block.
| Kept on the device until the pass ends | When | Size |
|---|---|---|
| the token's row of the residual stream | twice a block (after its attention, at its end) and once before the first block the device runs, a sequence | n_embd floats |
| the token's queries | once a block whose attention is asked for, a sequence | the recorded heads x head_dim floats |
| the block's attention sinks (gpt-oss) | with them | a float a head |
| the token's routing | once a routed block, a sequence | n_expert_used ids and weights |
Each is one copy (oklm_cpy(), an op queued like those of
the pass, which nothing waits for) into staging memory the recording
allocates on the context's device: room for what eight sequences leave
in a pass (allocated for three rows of the stream a block, the queries
and the sinks of each attention slot, a routing a block; 393 KB a
sequence for Llama 3.2 1B without attention rows), or the context's
sequences when it has fewer. The hook leaves a note of what it staged
and goes on.
When the pass ends, one round trip reads the staging memory back and,
for the attention rows, the keys of the positions the heads saw, from
the block's cache where the pass left them (in the cache's type; a paged
cache a run of cells at a time; sixteen reads are waited for together at
most). The notes are then worked out in the order they were left, with
the arithmetic a block on the CPU gets: the records are, bit for bit,
those of reading the device at each point of a block, which
unbatched in oklm_observe_params still does
and the tests compare (Tests).
oklm_observe_device_reads(ctx) counts the waits.
So the device runs a whole pass without the host, and on Vulkan the decode step of a recording context is recorded and replayed like any other (vulkan.md). What it costs is measured below. Off, a device context does exactly what it did: the only test is the pointer per block, on the host.
What is still read inside a pass, each read waiting for the ops queued before it (and, on Vulkan, the pass not marked as a step):
- the lens's logits, a block and a sequence
(
--observe-lens); - every row of the stream, twice a block, for the means
(
means), and the stream of a context with parts taken out; - what does not fit the staging memory: the ninth sequence of one batch and those after it;
- the blocks of a second device of a context spread over several: the staging memory is on the first one.
The figures are those of the device's arithmetic, which differs from the CPU's by the device ops' bounds (Tests): a norm of a device block is the norm of what the device wrote. The keys are read in the cache's type, as on the CPU.
The command
oklm run -m model.gguf --backend cpu -p "The capital of France is" -n 6 --temp 0 \
--observe run.jsonl --observe-attn 5,23
--observe FILE writes one JSON object a line: the
prompt's last token, then each generated token.
--observe-attn adds the attention rows of the blocks listed
(at most 64), or of every block with all;
--observe-lens K adds each block's lens, the K
likeliest tokens (1 to 16). oklm chat takes the same three
options and writes a line for each turn's prompt and each token of the
replies.
A line (Granite 3.1 1B A400M, a mixture of experts of 24 blocks; arrays cut):
{"seq":0,"pos":4,"token":438,"piece":" is","serial":1,"n_layer":24,"n_observed":24,
"embd":7.30355,
"resid":[3.72,2.12,...,25.18],
"attn":[4.15,2.27,...,12.92],
"ffn":[4.49,1.16,...,23.31],
"attention":{"n_head":16,"n_head_model":16,"pos0":0,"n_pos":5,
"tokens":[1318,18926,432,45600,438],
"pieces":["The"," capital"," of"," France"," is"],
"layers":[{"layer":5,"heads":[[0.296,0.099,0.424,0.022,0.16], ...]}]},
"experts":{"n_expert":32,"n_expert_used":8,
"layers":[{"layer":0,"ids":[...],"weights":[...]}, ...,
{"layer":5,"ids":[8,13,24,29,20,10,15,3],"weights":[0.363,0.15,0.137,0.136,0.094,0.064,0.029,0.027]}, ...]}}| Member | What |
|---|---|
seq, pos, token,
piece |
the sequence, the token's position in it, its id and its text |
serial |
the forward passes recorded for the sequence: it changes with each new record |
n_layer, n_observed |
the model's blocks, and those recorded (all of them, unless a device could not be read) |
embd, resid, attn,
ffn |
the norms above, an entry a block; null for a block not
observed |
attention.n_head, n_head_model |
heads recorded, and heads of the model |
attention.pos0, n_pos |
the first position of the rows and their length: entry
i of a row is position pos0 + i, the last is
the token itself |
attention.tokens, pieces |
the ids and the texts of the tokens at those positions
(-1 and "" for a position given as an
embedding, an image's for example, or evaluated before the recording
started) |
attention.layers |
one object a recorded block: layer and
heads, a row a head |
experts |
for a mixture of experts: n_expert,
n_expert_used and one object a routed block with its
ids and weights |
lens |
with --observe-lens: top and
layers, one object a block with its layer, the
ids of the tokens the output head finds likeliest for the
stream after the block, their texts pieces and their
probabilities p |
A token's text that is a part of a character (a byte-level piece) has its stray bytes replaced by U+FFFD so that the line stays valid JSON.
The lens of Llama 3.2 1B for the last token of "The capital of France
is" (--observe-lens 3; blocks 0, 9, 10, 11 and 15 of
16):
0 " is" 0.999 " was" 0.001 " has" 0.000
9 "olit" 0.052 "polit" 0.009 "ocation" 0.005
10 "olit" 0.067 " capitals" 0.015 "ocation" 0.011
11 " Paris" 0.706 " Brussels" 0.082 "Paris" 0.046
15 " Paris" 0.619 " a" 0.064 " not" 0.041
Taking parts out
oklm run -m model.gguf -p "The capital of France is" -n 8 --temp 0 --ablate 15:ffn
oklm run -m model.gguf -p "The capital of France is" -n 8 --temp 0 --ablate 3:attn,9:head:4 --ablate-mean
oklm run -m model.gguf -p "The capital of France is" --temp 0 --ablate-scan blocks
oklm run -m model.gguf -p "The capital of France is" --temp 0 --ablate-scan heads:11
--ablate takes parts separated by commas (64 at most):
L:attn (block L's attention write),
L:ffn (its feed-forward write) or L:head:H
(its head H). The prompt is evaluated plainly, then again
with the parts taken out; what that did to the next token's distribution
is said on stderr, and the generation goes on without them, so the text
shows the model without the parts. With --observe the lines
are those of the run without them.
ablation (15:ffn, zero): KL 0.70055 nats from the plain run; " Paris" p=0.6194 -> 0.9500; likeliest now " Paris" p=0.9500
--ablate-mean replaces the parts by their mean over the
prompt's rows in place of zeros (the plain evaluation of the prompt is
where the means are taken).
--ablate-scan blocks takes each block's attention and
feed-forward write out in turn, one at a time, and
--ablate-scan heads:L each head of block L: a
line each on stdout, nothing generated (the prompt is evaluated once
more for each part). KL is the divergence of the next
token's distribution from the plain run's, p(plain) what is
left of the plain run's likeliest token:
one part of a block taken out at a time, each zeroed; the plain run gives " Paris" p=0.6194
part KL (nats) p(plain) likeliest after
0:attn 1.25425 0.9550 " Paris" p=0.9550
0:ffn 8.45301 0.0000 " out" p=0.0463
1:attn 0.05054 0.6138 " Paris" p=0.6138
1:ffn 6.47125 0.0001 " a" p=0.3013
...
11:ffn 0.69408 0.1966 " Paris" p=0.1966
...
14:ffn 2.76169 0.0054 "..." p=0.1774
15:attn 0.24868 0.3153 " Paris" p=0.3153
15:ffn 0.70055 0.9500 " Paris" p=0.9500
(Llama 3.2 1B Q4_K_M on Vulkan.) A part a block does not have is
listed as such (Phi-2's attention part, a head of a block without
attention). The three options decode plainly, like
--observe: not with a draft model, --mtp,
--spec-ngram or --spec-grammar.
What the figures mean
They are measurements of the computation, exact for what they measure. What they are evidence of is narrower than what a picture of them suggests.
The norms. attn[l] and
ffn[l] say how large the vector each part added to the
stream is; resid[l] how large the stream has become.
- A large write is not an important one. The stream goes through a normalization before every read and before the output: its direction counts, not its length. A block may add a long vector along what the stream already holds and change no prediction, and a short one that flips the answer. The last blocks of most models write the most because the stream has grown, not because they decide the most.
- The norms compare along one model's depth and from token to token. They do not compare between models, nor between quantizations of one model beyond their noise.
- A write of 0 for
attnon Phi-2 is the block's form (its two parts are added together), not silence. - To learn what a block's write does, it has to be changed and the
output compared: What a write or a head
does to the output. (
oklm abliteratedoes it for one direction of the stream: how-it-works.md.)
The attention rows. A row says where a head read from, as weights that add up to 1.
- Attention is not explanation. A head that looks at a token moves that position's value vector, which after a few blocks no longer stands for the token alone: positions carry what earlier blocks gathered there. A weight on "France" is a weight on what position 3 holds at that depth.
- What a head moves depends on the values and on the output projection, which the row does not show: a head may put 0.9 on a position whose value is close to nothing, or spread its weights and still move one thing.
- Many heads put most of their weight on the first position (the beginning-of-text token) when they have nothing to do: it is a resting state, not an interest in the first word.
- The rows are those of the last token only. They say nothing of how the earlier positions were built.
- One head is one of several hundred readers; a pattern in one row is an observation to test, not a mechanism found.
The experts. The ids and weights are the router's choice, exactly.
- An expert picked often for a kind of text is correlated with it. What the expert computes is not known from the choice: experts are not trained to have separate subjects, and the same expert serves unrelated tokens.
- The weights are the router's shares of the routed experts' sum. A shared expert (Qwen2-MoE, DeepSeek) is added beside them and is not in the list.
- Which experts answer a prompt is useful for what it was first counted for: knowing what must be in memory (moe.md).
The lens. The tokens are what the model's own head reads in the stream at a depth.
- The head was trained to read the last block's stream. An earlier block's stream is not in that form: the early lenses are mostly noise (the token itself, word pieces of no relation), and a block whose lens already names the answer shows that the answer is readable there by this head, not that the model has "decided".
- The norm before the head hides the stream's length: a lens says nothing of how much a later block can still change.
- A token that appears and disappears between two depths was moved by the blocks between them; which one, the lens does not say (take them out and compare).
A part taken out. The figures are the output of another model: the one without the part.
- Zeroing a write is far from anything the model met in training: the blocks after it read a stream they never saw, and the output may move for that reason alone. The mean keeps the stream in a place the later blocks know; it is the mean over the rows of the run it was taken from, nothing more general.
- A part whose removal changes nothing is not useless: another may do its work when it is gone. One whose removal breaks the output is needed by this prompt, which does not say what it computes.
- The first blocks' writes carry everything after them: taking them out breaks any prompt, and says little.
- The Kullback-Leibler divergence is of the next token's distribution after the prompt, one position: a part may matter elsewhere in the text.
What a write or a head does to the output
Two ways of asking, both in the library
(include/oklm/model/observe.h).
The logits a block's write leads to
(lens_top of oklm_observe_params). After each
block the forward pass gives the last row of the stream of each sequence
to the final norm and the output head (the same code as the model's own
last step: the scales and the soft-cap included), in the pass's own
buffers, which are free between two blocks, and the record keeps the
lens_top most likely tokens (16 at most) with their
probabilities over the whole vocabulary. Cost: one more product with the
output head a block and a sequence, which for a vocabulary of 130 000
tokens is as much as the blocks themselves (Tests);
on a device, the logits are read back each time, inside the pass (On a device).
Parts taken out
(oklm_ablate_set(ctx, parts, n)). A part is
part |
What is taken out |
|---|---|
OKLM_ABLATE_ATTN |
what the block's token mixing wrote to the stream: the stream is put back as it was before it |
OKLM_ABLATE_FFN |
what its feed-forward part (or its experts) wrote: the stream is put back as it was after the attention |
OKLM_ABLATE_HEAD |
the output of one attention head, before the block's output projection reads it |
with mode OKLM_ABLATE_ZERO (nothing in its
place) or OKLM_ABLATE_MEAN (its mean in its place: the mean
write over every row a recording started with means has
seen on the context, or the head's mean output, copied when the call is
made). The parts hold for every forward pass until another call;
oklm_ablate_set(ctx, NULL, 0) puts the model back whole,
and its outputs are then the plain run's, bit for bit.
How it is done. The stream is copied to host memory at the point before the part and written back after it (with the mean added), every row of the pass: whatever the part does inside is undone, on any backend and for any kind of block (attention, latent attention, a recurrent or convolution block, routed experts), and the part still runs, so its cache rows and states are written as in a plain run. A head's output is rewritten in the buffer the output projection reads. These are the points the recording reads: a zeroed write is recorded with a norm of 0. A context with parts taken out does not replay decode steps on a device: the stream goes through host memory inside the pass.
What is refused: the attention part of a block that adds its two parts in one step (Phi-2: its whole write is the feed-forward part's), a head of a block that has no attention (a Gated DeltaNet or convolution block), an encoder, a mean when none was recorded.
oklm_logits_compare(first, second, n_vocab, &change)
compares two rows of logits as distributions (the softmax in double
precision): the Kullback-Leibler divergence
KL(first || second) in nats, each run's most likely token
and its probability in both.
The records kept (history of
oklm_observe_params). The records a sequence's passes
publish replace each other; with history the last ones (of
all the sequences together, at most history_bytes, 32 MiB
by default, the oldest dropped first) are kept, each at its own size,
and oklm_observe_read_at(ctx, seq, serial, &record)
reads one back by the serial it was published with;
oklm_observe_kept(ctx, seq, &first, &last) gives
the range kept for a sequence.
In a browser
A server started with --observe draws its records
itself: GET /activations (a link in the dashboard) is a
page with the token, each block's writes, the lens, the attention rows
of the blocks named there and the experts picked, following the token
decoded last or stepping through the kept records (server.md). It reads
GET /api/activations and nothing else; what its pictures
are evidence of is what the section above says, no more.
The interface
include/oklm/model/observe.h:
| Function | What |
|---|---|
oklm_observe_start(ctx, params) |
start recording on a context (while no thread evaluates with it):
attn_slots blocks whose attention can be recorded at once
(0 none, -1 every block), the first ones chosen, the bounds;
lens_top, means and history for
the lens, the means a part can be replaced by, and the records kept |
oklm_observe_select(ctx, layers, n) |
choose the blocks whose attention the next passes record; any thread |
oklm_observe_read(ctx, seq, &record) |
a copy of the last record published for a sequence (-1:
the one recorded last); any thread |
oklm_observe_read_at(ctx, seq, serial, &record) |
a copy of a record that was kept; any thread |
oklm_observe_kept(ctx, seq, &first, &last) |
how many records are kept for a sequence, and their serials |
oklm_ablate_set(ctx, parts, n),
oklm_ablate_count(ctx) |
take parts out of the passes that follow (while no thread evaluates), or put the model back whole |
oklm_logits_compare(a, b, n_vocab, &change) |
how two output distributions differ |
oklm_observation_json(&record, layer, tokenizer, extra) |
the record as the JSON object above |
oklm_observe_limits(ctx, ¶ms) |
the bounds as applied and the bytes allocated |
oklm_observe_device_reads(ctx) |
the reads of device memory the recording has waited for: one a pass, unless more is read inside it (On a device) |
oklm_observe_stop(ctx) |
stop and free |
The thread that evaluates fills a working record per sequence and publishes it under a lock once the forward pass is done; a reader copies the published record under the same lock, so it always gets a whole one. A selection made while a pass runs is taken when that pass ends. What the hooks are given may be in device memory: they copy what they need into staging memory on the device and leave a note, and the end of the batch reads the staging memory back and works the notes out (On a device).
The forward passes call these when the context records
(src/model/observe.c): the stream at the three points of a
block, the queries after a block's attention, the heads' outputs before
the output projection (for the means), the routing after a block's
experts were chosen, the lens's logits, the end of the batch. With parts
taken out they call two more at the same points: the stream kept and put
back, the heads' outputs rewritten. The Llama family's pass
(arch_llama.c: Llama, Qwen, Gemma, Phi-3, Granite, the MoE
families, GLM-4, LFM2, gpt-oss) and GPT-2's (arch_gpt2.c:
GPT-2, Phi-2, StarCoder2) have them; the encoders'
(arch_bert.c) does not.
Tests
tests/model/test_observe.c, on the committed tiny
models, against what the numpy float64 reference computes for the same
tokens (tools/infer_ref.py --observe, written from the
model definitions and independent of the C code):
| Model | Records | Attention rows | Routings | Lenses | Worst norm (relative) | Worst weight | Worst expert weight | Worst lens probability |
|---|---|---|---|---|---|---|---|---|
| tiny-llama (dense) | 8 | 64 | 0 | 16 | 2.1e-7 | 1.1e-7 | 4.6e-8 | |
| tiny-gpt2 | 8 | 64 | 0 | 16 | 6.1e-7 | 4.1e-6 | 2.2e-6 | |
| tiny-granitemoe (softmax router) | 8 | 64 | 16 | 16 | 9.0e-7 | 1.7e-6 | 6.5e-7 | 6.2e-8 |
| tiny-gptoss (sinks, its own router) | 8 | 96 | 24 | 24 | 2.9e-6 | 6.5e-6 | 1.7e-6 | 1.8e-7 |
| tiny-deepseek2 (latent attention) | 8 | 64 | 8 | 16 | 3.4e-6 | 4.2e-6 | 1.1e-6 | 4.9e-7 |
| tiny-deepseek3 (latent attention, sigmoid router) | 8 | 64 | 8 | 16 | 1.2e-6 | 3.7e-6 | 6.6e-7 | 2.3e-7 |
with an F32 cache, contiguous and paged alike; every expert picked is the reference's, in the same order, every row adds up to 1 within 1e-5 (to the reference's sum with sinks), and the four tokens of each lens are the reference's (the reference computes latent attention's rows from the heads' scores, and each lens from the stream after the block through its own final norm and head). With the default F16 cache the model itself computes with rounded keys and values, and the same comparison holds within 2 % on the norms, 0.03 on a weight and 0.01 on a lens probability. The same file checks that the logits with the recording on (the lens, the means and the records kept included) are bit for bit those with it off, the bounds and the selection, two sequences in one batch against each alone, 19 000 records read by a second thread while 501 passes run (each whole), and the JSON.
On each device backend found, test_devices runs
twenty-two tiny models (dense Llama, GPT-2, Phi-2, Gemma 3, the mixtures
of experts of Granite, GLM-4, OLMoE and Qwen2, DeepSeek V2's latent
attention; and the stacks whose device paths came after them: Qwen3.5's
linear blocks, its experts and its folded weights, Gemma 4 with its two
widths, its shared caches and its experts, gpt-oss's sinks, LFM2's
convolution blocks and its experts, the attention gates of Spark and
Muse Glimmer, StarCoder2, Mistral 3) with every block on the device and
with the last one alone, an F32 and an F16 cache, contiguous and paged,
beside the same model on the CPU: a prompt in one pass, six tokens one
by one, two sequences in one batch, then the prompt and three tokens
again with three parts taken out on either side (a block's attention,
the last block's feed-forward part, a head). Every block is recorded on
either side, a block without attention (a linear block, a convolution
block) included; the routings are the CPU's, id for id. Measured here
(Radeon RX 7900 XT, Mesa, 2026-11-14):
| Backend | Cache | Contexts | Records | Attention rows | Routings | Worst norm (relative) | Worst weight | Worst expert weight | Worst lens probability | Logits with the parts taken out | Logits, recording on against off |
|---|---|---|---|---|---|---|---|---|---|---|---|
| OpenCL | F32 | 66 | 792 | 8640 | 864 | 9.4e-6 | 3.3e-5 | 1.9e-5 | 2.2e-6 | 1.3e-5 | the same bits |
| OpenCL | F16 | 44 | 528 | 5760 | 576 | 3.9e-4 | 5.8e-4 | 8.7e-4 | 1.0e-4 | 8.9e-4 | the same bits |
| Vulkan | F32 | 66 | 792 | 8640 | 864 | 8.0e-6 | 1.2e-5 | 1.4e-5 | 3.3e-6 | 2.0e-5 | the same bits |
| Vulkan | F16 | 44 | 528 | 5760 | 576 | 3.8e-4 | 5.3e-4 | 4.7e-4 | 3.3e-5 | 9.9e-4 | the same bits |
The test's bounds are 2e-4 with an F32 cache (the device ops' logits
are held to 1e-4 against the reference in test_model.c) and
2e-3 with an F16 one: a key or a value that differs in its last place
may round to another half on either side, which four of the later stacks
amplify (Gemma 4 with experts 8.7e-4 on an expert's weight and 9.9e-4 on
the logits with parts taken out, Spark 3.9e-4 on a norm, gpt-oss 5.3e-4
on an attention weight, Muse Glimmer 6.1e-4 on the logits with parts
taken out; with an F32 cache each is within 3.3e-5, and the eighteen
other models stay within 2.1e-4 with an F16 one). The recording adds
nothing to it: these models' logits with an F16 cache are 6e-3 to 1.1e-2
from the float64 reference on any backend (inference.md).
test_device_batched checks that reading the device once
a pass changes no bit of a record. The same twenty-two models, each in
two contexts of the device, one reading once a pass and one at each
point of a block (unbatched): every block on the device
without attention rows and with those of every block, the last block
alone, an F16 cache, a paged one, and ten sequences (two more than the
staging memory holds, the rest read inside the pass); a prompt in one
pass, six tokens one by one, two sequences in one batch, then every
sequence at once. After each pass the two records are compared with
memcmp (norms, attention rows, routings, tokens) and so are
the logits. Measured here:
| Backend | Pairs of contexts | Records, the same bits | Attention rows | Routings | Passes | Reads, once a pass | Reads, at each point |
|---|---|---|---|---|---|---|---|
| OpenCL | 154 | 1606 | 13200 | 1752 | 1254 | 1331 | 20866 |
| Vulkan | 154 | 1606 | 13200 | 1752 | 1254 | 1331 | 20866 |
Without attention rows the reads are exactly the passes; the 77 more
are the keys read after a pass with attention rows and the batches of
ten sequences (the three Qwen3.5 stacks keep ten sequences in the room
of eight: most of their blocks have no queries to keep, and their pass
is still one read). On Vulkan a third context runs the same passes
without recording: it reads nothing back
(oklm_observe_device_reads() stays 0), the recording one
replays as many decode steps (411 of them) and no step of it is ended by
a host operation that does not end the other's (231, the models whose
pass holds an op on the CPU).
test_ablate takes parts out of six tiny models (Llama,
GPT-2, Phi-2, Granite's experts, DeepSeek V2, Gemma 3) on the CPU and on
each device:
- a write zeroed is recorded with a norm of exactly 0, and the output moves (a divergence above 1e-6 from the plain run);
- every head of a block zeroed gives the output of the block's attention zeroed, where the output projection has no bias (the same bits);
- a token evaluated alone, then every write of every block replaced by its mean, then every head's output by its mean: the mean of one row is the row, and the token's own logits come back (the same bits on the three backends);
- a mean in a write's place gives the same norm for two different prompts;
- the model put back whole gives the plain run's logits, bit for bit;
- what is refused is refused (a block or a head that does not exist, Phi-2's attention part, a mean before any row was seen).
test_kept checks the records kept: the last five of
seven, each equal to the record read when it was made (norms, rows,
tokens, experts, lens), a second sequence sharing the ring, the bound in
bytes (1 500 bytes keep 3 of 5 records of a tiny model; a bound smaller
than one record keeps none). test_logits_compare checks the
divergence on a case worked by hand (0.25 ln 2).
On whole models the device's arithmetic is what separates the two
records.
oklm run --temp 0 --observe FILE --observe-attn ... on the
CPU and on the device, the lines compared while the two runs have
generated the same tokens:
| Model | Backend | Records compared | Norms: median, worst (relative) | Worst attention weight | Routings |
|---|---|---|---|---|---|
| SmolLM2-135M F16 (30 blocks; blocks 3, 17, 29) | OpenCL | 61 of 61 | 2.3e-5, 1.5e-3 | 1.2e-3 | |
| same | Vulkan | 61 of 61 | 1.6e-5, 1.5e-3 | 7.8e-4 | |
| Llama 3.2 1B Q4_K_M (blocks 5, 10) | OpenCL | 4 | 2.8e-3, 1.7e-2 | 0.040 | |
| same | Vulkan | 4 | 3.0e-3, 2.1e-2 | 0.026 | |
| Granite 3.1 1B A400M Q4_K_M (blocks 5, 10) | OpenCL | 6 | 1.5e-2, 0.12 | 0.053 | 144: the same eight experts in 120, 1128 of 1152 experts shared |
| same | Vulkan | 6 | 1.5e-2, 0.12 | 0.053 | the same counts |
| LFM2-350M Q4_K_M (16 blocks, 10 of them convolutions; attention blocks 2, 8, 14) | OpenCL | 27 of 61 | 4.3e-3, 4.3e-2 | 0.065 | |
| same | Vulkan | 27 of 61 | 4.2e-3, 5.0e-2 | 0.064 |
LFM2-350M's lines (2026-11-14) record its sixteen blocks on either device, the convolution blocks among them; its two devices' records agree with each other over 43 lines at a median of 1.4e-3.
With F16 weights the device computes what the CPU does and the records agree over the whole run. With quantized weights the CPU rounds the activations to 8 bits before its products and the devices do so for some of theirs only (opencl.md): the two runs part after a few tokens (the known logit noise of the two paths), and until then the norms differ by what that rounding moves, more where a router picks between experts of close weights. The recording adds nothing to it: the two devices' records of Granite agree with each other at a median of 9e-5 over all 121 lines.
Cost of the lens.
oklm run --temp 0 -n 100 --observe FILE, Llama 3.2 1B
Q4_K_M (16 blocks, a vocabulary of 128 256 tokens), without and with
--observe-lens 4, three runs each on a busy machine: on the
CPU (4 threads) 7.0 to 8.4 tokens a second and 1.4 to 1.9; on OpenCL 37
to 46 and 6.6 to 7.3; on Vulkan 87 to 102 and 12 to 17. Sixteen more
products with the output head a token, and on a device sixteen reads of
the logits: a lens is for looking at a prompt, not for a server under
load. Over the records the CPU's run and a device's share (4, the two
runs part after them), the lens's first token is the CPU's in 60 of 64
blocks on OpenCL and 58 of 64 on Vulkan, its probability within 0.05:
the device arithmetic of quantized weights again.
Cost on a device. oklm run --temp 0 -n 120, every block
on the Radeon RX 7900 XT, tokens a second of the generation (the lines'
JSON written included): the command built before the reads were batched
(two blocking reads a block for the norms and one more a pass, two for a
routing, two for an attention row) and the one built after, run in turn,
five pairs each. The machine was busy with other work (a load of 17 to
25 on 16 cores), which the spreads show, so each cell is the lowest, the
median and the highest of its five runs; the two commands of a pair ran
within the same few seconds. The two files of each pair are the same
bytes (cmp), on both backends and both models, 121 lines
each.
| Model | Backend | Not recording | --observe |
--observe --observe-attn 5,10 |
|
|---|---|---|---|---|---|
| Llama 3.2 1B Q4_K_M | Vulkan | before | 284, 310, 360 | 96, 104, 124 | 79, 87, 90 |
| after | 318, 328, 348 | 227, 235, 287 | 139, 154, 170 | ||
| same | OpenCL | before | 56, 62, 211 | 8, 169, 185 | 65, 110, 121 |
| after | 54, 177, 243 | 168, 196, 217 | 113, 120, 130 | ||
| Granite 3.1 1B A400M Q4_K_M | Vulkan | before | 303, 345, 365 | 57, 60, 72 | 16, 54, 59 |
| after | 224, 256, 405 | 257, 284, 300 | 102, 130, 205 | ||
| same | OpenCL | before | 163, 194, 222 | 100, 118, 125 | 8, 107, 111 |
| after | 176, 216, 237 | 70, 129, 148 | 68, 110, 133 |
On Vulkan the recording cost two thirds of the speed of Llama and more than four fifths of Granite's (49 reads of the stream and 48 of the routing a token, and no replayed step); it now costs about a quarter of Llama's and a fifth of Granite's (284 against the 345 of the command before, whose runs were the quieter ones of that cell), by the medians: 2.3 and 4.7 times the tokens a second while recording. What is left is the copies (two a block, four for a routed block) and one more wait a token.
On OpenCL the gain is small: 169 to 196 tokens a second for Llama and
118 to 129 for Granite by the medians, inside what the machine's other
work moves (nine more pairs of --observe alone gave 118 to
144 for Granite, 112 to 124 before and 113 to 158 after, and for Llama a
round too disturbed to read: medians of 34 and 73 with best runs of 183
and 178). A copy queued on this driver costs the host about what a
blocking read of a row did (the device was never the slow part there: by
the medians, 97 blocking reads a token cost Granite 3.3 ms, 97 copies
and one read 3.1), so recording still takes a tenth of OpenCL's speed
for Llama and a third or more for Granite. Fewer commands would be the
gain there, not fewer waits: it was not done.
The attention rows of two blocks still take a third to a half of what is left on either backend. Their reads no longer count (the keys come back in the pass's one round trip): the cost is the host's arithmetic, one dot product a head and a position on one thread, each key row converted from the cache's type for each head that reads it.
Not recording, a device context runs what it ran before the reads
existed. The command built before and the one built after, in turn, five
times each, 200 tokens of Llama 3.2 1B: on Vulkan 340 to 463 tokens a
second before (median 359) and 339 to 468 after (median 434); on OpenCL
40 to 364 before (median 240) and 40 to 364 after (median 203); the
spread is the machine's other work (one pair of runs fell to 40 on both
sides), the best runs are equal. The kernels launched are the same with
the recording on or off (OKLM_CL_PROFILE=2, Llama 3.2 1B on
OpenCL: 1209 launches for a prompt and 8 tokens, 5337 for 40, 129 a
token, either way): what the recording adds on the device is buffer
copies and the read that ends a pass, and a context that does not record
queues neither.
tests/server/test_server.c
(test_activations) checks the route: server.md.
Cost with the recording off. Decoding SmolLM2-135M Q4_K_M on the CPU, 300 tokens at temperature 0, the command built before the hooks and the one built after run in turn, twelve times each, on one thread: 27.6 tokens a second before (22.9 to 34.3; 11.1 s of CPU a run) and 29.9 after (22.0 to 36.7; 10.3 s), medians. The difference is inside the runs' spread, which is wide: the machine was busy with other work (a load of 17 on 16 cores). Idle, the command before gave 235 to 240 tokens a second on its default threads; the one after was not measured idle, nor was the cost of the recording when it is on.