OKLMdocumentation v1.0.3
On this page

Activations of a running model

What OKLM can record of what happens inside a model while it evaluates tokens: how much each block writes to the residual stream, where the attention of chosen blocks looks, which experts each router picks, and which tokens the output head would give at each depth; and what it can take out of a run to see what a block's write or a head does to the output. It is read with oklm run --observe (a file), from a server started with oklm serve --observe (GET /api/activations) and in the model inspector (gui.md). How to read the figures, and what they do not say, is in What the figures mean.

Status: maintained.

What is recorded

For the last token of each sequence a forward pass evaluates (the last token of a prompt, then each token decoded):

Figure What
resid[l] the Euclidean norm of the residual stream after block l
attn[l] the norm of what the block's token mixing added to the stream: its attention, or the recurrent or convolution block that stands in its place (Gated DeltaNet, LFM2)
ffn[l] the norm of what its feed-forward part added: the dense FFN, or the weighted sum of its routed experts and its shared expert
embd the norm of the stream before the first block: the token's embedding
an attention row for a block asked for and one of its heads, the weights the softmax gave to the positions before the token, itself included: the last n_pos of them
the experts for each routed block, the experts its router picked for the token, in rank order, and the weights their outputs were summed with
the lens when asked for (lens_top): for each block, the residual stream after it given as it is to the model's final norm and output head, and the most likely tokens of the result with their probabilities: what the model "would say" if it stopped there. The last block's is the model's own output

The norms come from the stream itself: it is read at three points of a block (before, after the attention's output was added, at the end) and the differences are what each part added, whatever the block does around them (norms after the parts, a scale on the residual branch, an adapter). A block whose two parts are added in one step (Phi-2) gives their sum as ffn and 0 as attn.

An attention row is computed again from the head's query and the keys in the cache, with the model's scale, soft-cap, sliding window and attention sinks, as the attention itself does. The cache holds what was written to it: with keys kept as half floats (the default) the row is that of the rounded keys with the unrounded query. A row adds up to 1 when the head saw nothing else; positions before the n_pos recorded ones and a sink (gpt-oss) take their share without being listed.

The experts are read where the server's expert map counts them (server.md): the routing the forward pass used.

Bounds: the last 256 positions, the first 64 heads of a block and 64 MiB for the whole recording by default (oklm_observe_params); the positions are halved until the recording fits, and a recording that does not fit with 16 positions is refused.

Limits:

  • Blocks on a device (OpenCL, Vulkan) are recorded as the CPU's are, by reading the device back once a pass: On a device. A block is marked as not observed (null in the arrays, n_observed below n_layer) only when its read failed.
  • Multi-head latent attention (DeepSeek V2 and V3) is computed with the key decompression absorbed into the queries, over one compressed key a position. The rows are those of that product: for head h the score of a position is (W_kb[h]^T q_h) . c + q_rope_h . k_rope, which is q_h . (W_kb[h] c) plus the rotated part: the score of the head the model defines, written the other way round. So the weights are the heads' own, and a row reads as any other model's.
  • Only the last token of each sequence of a pass: a prompt evaluated in one pass leaves one record, for its last token. oklm run --observe therefore writes one line for the prompt and one for each generated token.
  • Decoding plainly: oklm run and oklm chat refuse --observe with a draft model, --mtp, --spec-ngram or --spec-grammar, whose passes evaluate several tokens. On a server those passes are recorded as they are: the record is that of the last token of a pass, which a rejected draft may have been.
  • Encoders (BERT) and multi-token prediction contexts are not recorded.

Cost. Off (the default), the forward pass tests one pointer per block (measured: Tests). On, the norms and the experts cost two passes over one row of the stream per block; an attention row costs, per block asked for, one dot product a head and a position, on one thread. On a device the rows are copied aside there and read back together when the pass ends: On a device.

On a device

The blocks a device runs (--backend opencl or vulkan, all of them or the last ones with --gpu-layers) leave the same record as the CPU's. Nothing is computed on the device for it: what the recording needs is copied aside on the device as the pass goes, read back once when the pass ends, and the arithmetic is done on the host, as for a CPU block.

Kept on the device until the pass ends When Size
the token's row of the residual stream twice a block (after its attention, at its end) and once before the first block the device runs, a sequence n_embd floats
the token's queries once a block whose attention is asked for, a sequence the recorded heads x head_dim floats
the block's attention sinks (gpt-oss) with them a float a head
the token's routing once a routed block, a sequence n_expert_used ids and weights

Each is one copy (oklm_cpy(), an op queued like those of the pass, which nothing waits for) into staging memory the recording allocates on the context's device: room for what eight sequences leave in a pass (allocated for three rows of the stream a block, the queries and the sinks of each attention slot, a routing a block; 393 KB a sequence for Llama 3.2 1B without attention rows), or the context's sequences when it has fewer. The hook leaves a note of what it staged and goes on.

When the pass ends, one round trip reads the staging memory back and, for the attention rows, the keys of the positions the heads saw, from the block's cache where the pass left them (in the cache's type; a paged cache a run of cells at a time; sixteen reads are waited for together at most). The notes are then worked out in the order they were left, with the arithmetic a block on the CPU gets: the records are, bit for bit, those of reading the device at each point of a block, which unbatched in oklm_observe_params still does and the tests compare (Tests). oklm_observe_device_reads(ctx) counts the waits.

So the device runs a whole pass without the host, and on Vulkan the decode step of a recording context is recorded and replayed like any other (vulkan.md). What it costs is measured below. Off, a device context does exactly what it did: the only test is the pointer per block, on the host.

What is still read inside a pass, each read waiting for the ops queued before it (and, on Vulkan, the pass not marked as a step):

  • the lens's logits, a block and a sequence (--observe-lens);
  • every row of the stream, twice a block, for the means (means), and the stream of a context with parts taken out;
  • what does not fit the staging memory: the ninth sequence of one batch and those after it;
  • the blocks of a second device of a context spread over several: the staging memory is on the first one.

The figures are those of the device's arithmetic, which differs from the CPU's by the device ops' bounds (Tests): a norm of a device block is the norm of what the device wrote. The keys are read in the cache's type, as on the CPU.

The command

oklm run -m model.gguf --backend cpu -p "The capital of France is" -n 6 --temp 0 \
    --observe run.jsonl --observe-attn 5,23

--observe FILE writes one JSON object a line: the prompt's last token, then each generated token. --observe-attn adds the attention rows of the blocks listed (at most 64), or of every block with all; --observe-lens K adds each block's lens, the K likeliest tokens (1 to 16). oklm chat takes the same three options and writes a line for each turn's prompt and each token of the replies.

A line (Granite 3.1 1B A400M, a mixture of experts of 24 blocks; arrays cut):

{"seq":0,"pos":4,"token":438,"piece":" is","serial":1,"n_layer":24,"n_observed":24,
 "embd":7.30355,
 "resid":[3.72,2.12,...,25.18],
 "attn":[4.15,2.27,...,12.92],
 "ffn":[4.49,1.16,...,23.31],
 "attention":{"n_head":16,"n_head_model":16,"pos0":0,"n_pos":5,
   "tokens":[1318,18926,432,45600,438],
   "pieces":["The"," capital"," of"," France"," is"],
   "layers":[{"layer":5,"heads":[[0.296,0.099,0.424,0.022,0.16], ...]}]},
 "experts":{"n_expert":32,"n_expert_used":8,
   "layers":[{"layer":0,"ids":[...],"weights":[...]}, ...,
             {"layer":5,"ids":[8,13,24,29,20,10,15,3],"weights":[0.363,0.15,0.137,0.136,0.094,0.064,0.029,0.027]}, ...]}}
Member What
seq, pos, token, piece the sequence, the token's position in it, its id and its text
serial the forward passes recorded for the sequence: it changes with each new record
n_layer, n_observed the model's blocks, and those recorded (all of them, unless a device could not be read)
embd, resid, attn, ffn the norms above, an entry a block; null for a block not observed
attention.n_head, n_head_model heads recorded, and heads of the model
attention.pos0, n_pos the first position of the rows and their length: entry i of a row is position pos0 + i, the last is the token itself
attention.tokens, pieces the ids and the texts of the tokens at those positions (-1 and "" for a position given as an embedding, an image's for example, or evaluated before the recording started)
attention.layers one object a recorded block: layer and heads, a row a head
experts for a mixture of experts: n_expert, n_expert_used and one object a routed block with its ids and weights
lens with --observe-lens: top and layers, one object a block with its layer, the ids of the tokens the output head finds likeliest for the stream after the block, their texts pieces and their probabilities p

A token's text that is a part of a character (a byte-level piece) has its stray bytes replaced by U+FFFD so that the line stays valid JSON.

The lens of Llama 3.2 1B for the last token of "The capital of France is" (--observe-lens 3; blocks 0, 9, 10, 11 and 15 of 16):

 0  " is" 0.999   " was" 0.001   " has" 0.000
 9  "olit" 0.052  "polit" 0.009  "ocation" 0.005
10  "olit" 0.067  " capitals" 0.015  "ocation" 0.011
11  " Paris" 0.706  " Brussels" 0.082  "Paris" 0.046
15  " Paris" 0.619  " a" 0.064  " not" 0.041

Taking parts out

oklm run -m model.gguf -p "The capital of France is" -n 8 --temp 0 --ablate 15:ffn
oklm run -m model.gguf -p "The capital of France is" -n 8 --temp 0 --ablate 3:attn,9:head:4 --ablate-mean
oklm run -m model.gguf -p "The capital of France is" --temp 0 --ablate-scan blocks
oklm run -m model.gguf -p "The capital of France is" --temp 0 --ablate-scan heads:11

--ablate takes parts separated by commas (64 at most): L:attn (block L's attention write), L:ffn (its feed-forward write) or L:head:H (its head H). The prompt is evaluated plainly, then again with the parts taken out; what that did to the next token's distribution is said on stderr, and the generation goes on without them, so the text shows the model without the parts. With --observe the lines are those of the run without them.

ablation (15:ffn, zero): KL 0.70055 nats from the plain run; " Paris" p=0.6194 -> 0.9500; likeliest now " Paris" p=0.9500

--ablate-mean replaces the parts by their mean over the prompt's rows in place of zeros (the plain evaluation of the prompt is where the means are taken).

--ablate-scan blocks takes each block's attention and feed-forward write out in turn, one at a time, and --ablate-scan heads:L each head of block L: a line each on stdout, nothing generated (the prompt is evaluated once more for each part). KL is the divergence of the next token's distribution from the plain run's, p(plain) what is left of the plain run's likeliest token:

one part of a block taken out at a time, each zeroed; the plain run gives " Paris" p=0.6194
part          KL (nats)   p(plain)  likeliest after
0:attn          1.25425     0.9550  " Paris" p=0.9550
0:ffn           8.45301     0.0000  " out" p=0.0463
1:attn          0.05054     0.6138  " Paris" p=0.6138
1:ffn           6.47125     0.0001  " a" p=0.3013
...
11:ffn          0.69408     0.1966  " Paris" p=0.1966
...
14:ffn          2.76169     0.0054  "..." p=0.1774
15:attn         0.24868     0.3153  " Paris" p=0.3153
15:ffn          0.70055     0.9500  " Paris" p=0.9500

(Llama 3.2 1B Q4_K_M on Vulkan.) A part a block does not have is listed as such (Phi-2's attention part, a head of a block without attention). The three options decode plainly, like --observe: not with a draft model, --mtp, --spec-ngram or --spec-grammar.

What the figures mean

They are measurements of the computation, exact for what they measure. What they are evidence of is narrower than what a picture of them suggests.

The norms. attn[l] and ffn[l] say how large the vector each part added to the stream is; resid[l] how large the stream has become.

  • A large write is not an important one. The stream goes through a normalization before every read and before the output: its direction counts, not its length. A block may add a long vector along what the stream already holds and change no prediction, and a short one that flips the answer. The last blocks of most models write the most because the stream has grown, not because they decide the most.
  • The norms compare along one model's depth and from token to token. They do not compare between models, nor between quantizations of one model beyond their noise.
  • A write of 0 for attn on Phi-2 is the block's form (its two parts are added together), not silence.
  • To learn what a block's write does, it has to be changed and the output compared: What a write or a head does to the output. (oklm abliterate does it for one direction of the stream: how-it-works.md.)

The attention rows. A row says where a head read from, as weights that add up to 1.

  • Attention is not explanation. A head that looks at a token moves that position's value vector, which after a few blocks no longer stands for the token alone: positions carry what earlier blocks gathered there. A weight on "France" is a weight on what position 3 holds at that depth.
  • What a head moves depends on the values and on the output projection, which the row does not show: a head may put 0.9 on a position whose value is close to nothing, or spread its weights and still move one thing.
  • Many heads put most of their weight on the first position (the beginning-of-text token) when they have nothing to do: it is a resting state, not an interest in the first word.
  • The rows are those of the last token only. They say nothing of how the earlier positions were built.
  • One head is one of several hundred readers; a pattern in one row is an observation to test, not a mechanism found.

The experts. The ids and weights are the router's choice, exactly.

  • An expert picked often for a kind of text is correlated with it. What the expert computes is not known from the choice: experts are not trained to have separate subjects, and the same expert serves unrelated tokens.
  • The weights are the router's shares of the routed experts' sum. A shared expert (Qwen2-MoE, DeepSeek) is added beside them and is not in the list.
  • Which experts answer a prompt is useful for what it was first counted for: knowing what must be in memory (moe.md).

The lens. The tokens are what the model's own head reads in the stream at a depth.

  • The head was trained to read the last block's stream. An earlier block's stream is not in that form: the early lenses are mostly noise (the token itself, word pieces of no relation), and a block whose lens already names the answer shows that the answer is readable there by this head, not that the model has "decided".
  • The norm before the head hides the stream's length: a lens says nothing of how much a later block can still change.
  • A token that appears and disappears between two depths was moved by the blocks between them; which one, the lens does not say (take them out and compare).

A part taken out. The figures are the output of another model: the one without the part.

  • Zeroing a write is far from anything the model met in training: the blocks after it read a stream they never saw, and the output may move for that reason alone. The mean keeps the stream in a place the later blocks know; it is the mean over the rows of the run it was taken from, nothing more general.
  • A part whose removal changes nothing is not useless: another may do its work when it is gone. One whose removal breaks the output is needed by this prompt, which does not say what it computes.
  • The first blocks' writes carry everything after them: taking them out breaks any prompt, and says little.
  • The Kullback-Leibler divergence is of the next token's distribution after the prompt, one position: a part may matter elsewhere in the text.

What a write or a head does to the output

Two ways of asking, both in the library (include/oklm/model/observe.h).

The logits a block's write leads to (lens_top of oklm_observe_params). After each block the forward pass gives the last row of the stream of each sequence to the final norm and the output head (the same code as the model's own last step: the scales and the soft-cap included), in the pass's own buffers, which are free between two blocks, and the record keeps the lens_top most likely tokens (16 at most) with their probabilities over the whole vocabulary. Cost: one more product with the output head a block and a sequence, which for a vocabulary of 130 000 tokens is as much as the blocks themselves (Tests); on a device, the logits are read back each time, inside the pass (On a device).

Parts taken out (oklm_ablate_set(ctx, parts, n)). A part is

part What is taken out
OKLM_ABLATE_ATTN what the block's token mixing wrote to the stream: the stream is put back as it was before it
OKLM_ABLATE_FFN what its feed-forward part (or its experts) wrote: the stream is put back as it was after the attention
OKLM_ABLATE_HEAD the output of one attention head, before the block's output projection reads it

with mode OKLM_ABLATE_ZERO (nothing in its place) or OKLM_ABLATE_MEAN (its mean in its place: the mean write over every row a recording started with means has seen on the context, or the head's mean output, copied when the call is made). The parts hold for every forward pass until another call; oklm_ablate_set(ctx, NULL, 0) puts the model back whole, and its outputs are then the plain run's, bit for bit.

How it is done. The stream is copied to host memory at the point before the part and written back after it (with the mean added), every row of the pass: whatever the part does inside is undone, on any backend and for any kind of block (attention, latent attention, a recurrent or convolution block, routed experts), and the part still runs, so its cache rows and states are written as in a plain run. A head's output is rewritten in the buffer the output projection reads. These are the points the recording reads: a zeroed write is recorded with a norm of 0. A context with parts taken out does not replay decode steps on a device: the stream goes through host memory inside the pass.

What is refused: the attention part of a block that adds its two parts in one step (Phi-2: its whole write is the feed-forward part's), a head of a block that has no attention (a Gated DeltaNet or convolution block), an encoder, a mean when none was recorded.

oklm_logits_compare(first, second, n_vocab, &change) compares two rows of logits as distributions (the softmax in double precision): the Kullback-Leibler divergence KL(first || second) in nats, each run's most likely token and its probability in both.

The records kept (history of oklm_observe_params). The records a sequence's passes publish replace each other; with history the last ones (of all the sequences together, at most history_bytes, 32 MiB by default, the oldest dropped first) are kept, each at its own size, and oklm_observe_read_at(ctx, seq, serial, &record) reads one back by the serial it was published with; oklm_observe_kept(ctx, seq, &first, &last) gives the range kept for a sequence.

In a browser

A server started with --observe draws its records itself: GET /activations (a link in the dashboard) is a page with the token, each block's writes, the lens, the attention rows of the blocks named there and the experts picked, following the token decoded last or stepping through the kept records (server.md). It reads GET /api/activations and nothing else; what its pictures are evidence of is what the section above says, no more.

The interface

include/oklm/model/observe.h:

Function What
oklm_observe_start(ctx, params) start recording on a context (while no thread evaluates with it): attn_slots blocks whose attention can be recorded at once (0 none, -1 every block), the first ones chosen, the bounds; lens_top, means and history for the lens, the means a part can be replaced by, and the records kept
oklm_observe_select(ctx, layers, n) choose the blocks whose attention the next passes record; any thread
oklm_observe_read(ctx, seq, &record) a copy of the last record published for a sequence (-1: the one recorded last); any thread
oklm_observe_read_at(ctx, seq, serial, &record) a copy of a record that was kept; any thread
oklm_observe_kept(ctx, seq, &first, &last) how many records are kept for a sequence, and their serials
oklm_ablate_set(ctx, parts, n), oklm_ablate_count(ctx) take parts out of the passes that follow (while no thread evaluates), or put the model back whole
oklm_logits_compare(a, b, n_vocab, &change) how two output distributions differ
oklm_observation_json(&record, layer, tokenizer, extra) the record as the JSON object above
oklm_observe_limits(ctx, &params) the bounds as applied and the bytes allocated
oklm_observe_device_reads(ctx) the reads of device memory the recording has waited for: one a pass, unless more is read inside it (On a device)
oklm_observe_stop(ctx) stop and free

The thread that evaluates fills a working record per sequence and publishes it under a lock once the forward pass is done; a reader copies the published record under the same lock, so it always gets a whole one. A selection made while a pass runs is taken when that pass ends. What the hooks are given may be in device memory: they copy what they need into staging memory on the device and leave a note, and the end of the batch reads the staging memory back and works the notes out (On a device).

The forward passes call these when the context records (src/model/observe.c): the stream at the three points of a block, the queries after a block's attention, the heads' outputs before the output projection (for the means), the routing after a block's experts were chosen, the lens's logits, the end of the batch. With parts taken out they call two more at the same points: the stream kept and put back, the heads' outputs rewritten. The Llama family's pass (arch_llama.c: Llama, Qwen, Gemma, Phi-3, Granite, the MoE families, GLM-4, LFM2, gpt-oss) and GPT-2's (arch_gpt2.c: GPT-2, Phi-2, StarCoder2) have them; the encoders' (arch_bert.c) does not.

Tests

tests/model/test_observe.c, on the committed tiny models, against what the numpy float64 reference computes for the same tokens (tools/infer_ref.py --observe, written from the model definitions and independent of the C code):

Model Records Attention rows Routings Lenses Worst norm (relative) Worst weight Worst expert weight Worst lens probability
tiny-llama (dense) 8 64 0 16 2.1e-7 1.1e-7 4.6e-8
tiny-gpt2 8 64 0 16 6.1e-7 4.1e-6 2.2e-6
tiny-granitemoe (softmax router) 8 64 16 16 9.0e-7 1.7e-6 6.5e-7 6.2e-8
tiny-gptoss (sinks, its own router) 8 96 24 24 2.9e-6 6.5e-6 1.7e-6 1.8e-7
tiny-deepseek2 (latent attention) 8 64 8 16 3.4e-6 4.2e-6 1.1e-6 4.9e-7
tiny-deepseek3 (latent attention, sigmoid router) 8 64 8 16 1.2e-6 3.7e-6 6.6e-7 2.3e-7

with an F32 cache, contiguous and paged alike; every expert picked is the reference's, in the same order, every row adds up to 1 within 1e-5 (to the reference's sum with sinks), and the four tokens of each lens are the reference's (the reference computes latent attention's rows from the heads' scores, and each lens from the stream after the block through its own final norm and head). With the default F16 cache the model itself computes with rounded keys and values, and the same comparison holds within 2 % on the norms, 0.03 on a weight and 0.01 on a lens probability. The same file checks that the logits with the recording on (the lens, the means and the records kept included) are bit for bit those with it off, the bounds and the selection, two sequences in one batch against each alone, 19 000 records read by a second thread while 501 passes run (each whole), and the JSON.

On each device backend found, test_devices runs twenty-two tiny models (dense Llama, GPT-2, Phi-2, Gemma 3, the mixtures of experts of Granite, GLM-4, OLMoE and Qwen2, DeepSeek V2's latent attention; and the stacks whose device paths came after them: Qwen3.5's linear blocks, its experts and its folded weights, Gemma 4 with its two widths, its shared caches and its experts, gpt-oss's sinks, LFM2's convolution blocks and its experts, the attention gates of Spark and Muse Glimmer, StarCoder2, Mistral 3) with every block on the device and with the last one alone, an F32 and an F16 cache, contiguous and paged, beside the same model on the CPU: a prompt in one pass, six tokens one by one, two sequences in one batch, then the prompt and three tokens again with three parts taken out on either side (a block's attention, the last block's feed-forward part, a head). Every block is recorded on either side, a block without attention (a linear block, a convolution block) included; the routings are the CPU's, id for id. Measured here (Radeon RX 7900 XT, Mesa, 2026-11-14):

Backend Cache Contexts Records Attention rows Routings Worst norm (relative) Worst weight Worst expert weight Worst lens probability Logits with the parts taken out Logits, recording on against off
OpenCL F32 66 792 8640 864 9.4e-6 3.3e-5 1.9e-5 2.2e-6 1.3e-5 the same bits
OpenCL F16 44 528 5760 576 3.9e-4 5.8e-4 8.7e-4 1.0e-4 8.9e-4 the same bits
Vulkan F32 66 792 8640 864 8.0e-6 1.2e-5 1.4e-5 3.3e-6 2.0e-5 the same bits
Vulkan F16 44 528 5760 576 3.8e-4 5.3e-4 4.7e-4 3.3e-5 9.9e-4 the same bits

The test's bounds are 2e-4 with an F32 cache (the device ops' logits are held to 1e-4 against the reference in test_model.c) and 2e-3 with an F16 one: a key or a value that differs in its last place may round to another half on either side, which four of the later stacks amplify (Gemma 4 with experts 8.7e-4 on an expert's weight and 9.9e-4 on the logits with parts taken out, Spark 3.9e-4 on a norm, gpt-oss 5.3e-4 on an attention weight, Muse Glimmer 6.1e-4 on the logits with parts taken out; with an F32 cache each is within 3.3e-5, and the eighteen other models stay within 2.1e-4 with an F16 one). The recording adds nothing to it: these models' logits with an F16 cache are 6e-3 to 1.1e-2 from the float64 reference on any backend (inference.md).

test_device_batched checks that reading the device once a pass changes no bit of a record. The same twenty-two models, each in two contexts of the device, one reading once a pass and one at each point of a block (unbatched): every block on the device without attention rows and with those of every block, the last block alone, an F16 cache, a paged one, and ten sequences (two more than the staging memory holds, the rest read inside the pass); a prompt in one pass, six tokens one by one, two sequences in one batch, then every sequence at once. After each pass the two records are compared with memcmp (norms, attention rows, routings, tokens) and so are the logits. Measured here:

Backend Pairs of contexts Records, the same bits Attention rows Routings Passes Reads, once a pass Reads, at each point
OpenCL 154 1606 13200 1752 1254 1331 20866
Vulkan 154 1606 13200 1752 1254 1331 20866

Without attention rows the reads are exactly the passes; the 77 more are the keys read after a pass with attention rows and the batches of ten sequences (the three Qwen3.5 stacks keep ten sequences in the room of eight: most of their blocks have no queries to keep, and their pass is still one read). On Vulkan a third context runs the same passes without recording: it reads nothing back (oklm_observe_device_reads() stays 0), the recording one replays as many decode steps (411 of them) and no step of it is ended by a host operation that does not end the other's (231, the models whose pass holds an op on the CPU).

test_ablate takes parts out of six tiny models (Llama, GPT-2, Phi-2, Granite's experts, DeepSeek V2, Gemma 3) on the CPU and on each device:

  • a write zeroed is recorded with a norm of exactly 0, and the output moves (a divergence above 1e-6 from the plain run);
  • every head of a block zeroed gives the output of the block's attention zeroed, where the output projection has no bias (the same bits);
  • a token evaluated alone, then every write of every block replaced by its mean, then every head's output by its mean: the mean of one row is the row, and the token's own logits come back (the same bits on the three backends);
  • a mean in a write's place gives the same norm for two different prompts;
  • the model put back whole gives the plain run's logits, bit for bit;
  • what is refused is refused (a block or a head that does not exist, Phi-2's attention part, a mean before any row was seen).

test_kept checks the records kept: the last five of seven, each equal to the record read when it was made (norms, rows, tokens, experts, lens), a second sequence sharing the ring, the bound in bytes (1 500 bytes keep 3 of 5 records of a tiny model; a bound smaller than one record keeps none). test_logits_compare checks the divergence on a case worked by hand (0.25 ln 2).

On whole models the device's arithmetic is what separates the two records. oklm run --temp 0 --observe FILE --observe-attn ... on the CPU and on the device, the lines compared while the two runs have generated the same tokens:

Model Backend Records compared Norms: median, worst (relative) Worst attention weight Routings
SmolLM2-135M F16 (30 blocks; blocks 3, 17, 29) OpenCL 61 of 61 2.3e-5, 1.5e-3 1.2e-3
same Vulkan 61 of 61 1.6e-5, 1.5e-3 7.8e-4
Llama 3.2 1B Q4_K_M (blocks 5, 10) OpenCL 4 2.8e-3, 1.7e-2 0.040
same Vulkan 4 3.0e-3, 2.1e-2 0.026
Granite 3.1 1B A400M Q4_K_M (blocks 5, 10) OpenCL 6 1.5e-2, 0.12 0.053 144: the same eight experts in 120, 1128 of 1152 experts shared
same Vulkan 6 1.5e-2, 0.12 0.053 the same counts
LFM2-350M Q4_K_M (16 blocks, 10 of them convolutions; attention blocks 2, 8, 14) OpenCL 27 of 61 4.3e-3, 4.3e-2 0.065
same Vulkan 27 of 61 4.2e-3, 5.0e-2 0.064

LFM2-350M's lines (2026-11-14) record its sixteen blocks on either device, the convolution blocks among them; its two devices' records agree with each other over 43 lines at a median of 1.4e-3.

With F16 weights the device computes what the CPU does and the records agree over the whole run. With quantized weights the CPU rounds the activations to 8 bits before its products and the devices do so for some of theirs only (opencl.md): the two runs part after a few tokens (the known logit noise of the two paths), and until then the norms differ by what that rounding moves, more where a router picks between experts of close weights. The recording adds nothing to it: the two devices' records of Granite agree with each other at a median of 9e-5 over all 121 lines.

Cost of the lens. oklm run --temp 0 -n 100 --observe FILE, Llama 3.2 1B Q4_K_M (16 blocks, a vocabulary of 128 256 tokens), without and with --observe-lens 4, three runs each on a busy machine: on the CPU (4 threads) 7.0 to 8.4 tokens a second and 1.4 to 1.9; on OpenCL 37 to 46 and 6.6 to 7.3; on Vulkan 87 to 102 and 12 to 17. Sixteen more products with the output head a token, and on a device sixteen reads of the logits: a lens is for looking at a prompt, not for a server under load. Over the records the CPU's run and a device's share (4, the two runs part after them), the lens's first token is the CPU's in 60 of 64 blocks on OpenCL and 58 of 64 on Vulkan, its probability within 0.05: the device arithmetic of quantized weights again.

Cost on a device. oklm run --temp 0 -n 120, every block on the Radeon RX 7900 XT, tokens a second of the generation (the lines' JSON written included): the command built before the reads were batched (two blocking reads a block for the norms and one more a pass, two for a routing, two for an attention row) and the one built after, run in turn, five pairs each. The machine was busy with other work (a load of 17 to 25 on 16 cores), which the spreads show, so each cell is the lowest, the median and the highest of its five runs; the two commands of a pair ran within the same few seconds. The two files of each pair are the same bytes (cmp), on both backends and both models, 121 lines each.

Model Backend Not recording --observe --observe --observe-attn 5,10
Llama 3.2 1B Q4_K_M Vulkan before 284, 310, 360 96, 104, 124 79, 87, 90
after 318, 328, 348 227, 235, 287 139, 154, 170
same OpenCL before 56, 62, 211 8, 169, 185 65, 110, 121
after 54, 177, 243 168, 196, 217 113, 120, 130
Granite 3.1 1B A400M Q4_K_M Vulkan before 303, 345, 365 57, 60, 72 16, 54, 59
after 224, 256, 405 257, 284, 300 102, 130, 205
same OpenCL before 163, 194, 222 100, 118, 125 8, 107, 111
after 176, 216, 237 70, 129, 148 68, 110, 133

On Vulkan the recording cost two thirds of the speed of Llama and more than four fifths of Granite's (49 reads of the stream and 48 of the routing a token, and no replayed step); it now costs about a quarter of Llama's and a fifth of Granite's (284 against the 345 of the command before, whose runs were the quieter ones of that cell), by the medians: 2.3 and 4.7 times the tokens a second while recording. What is left is the copies (two a block, four for a routed block) and one more wait a token.

On OpenCL the gain is small: 169 to 196 tokens a second for Llama and 118 to 129 for Granite by the medians, inside what the machine's other work moves (nine more pairs of --observe alone gave 118 to 144 for Granite, 112 to 124 before and 113 to 158 after, and for Llama a round too disturbed to read: medians of 34 and 73 with best runs of 183 and 178). A copy queued on this driver costs the host about what a blocking read of a row did (the device was never the slow part there: by the medians, 97 blocking reads a token cost Granite 3.3 ms, 97 copies and one read 3.1), so recording still takes a tenth of OpenCL's speed for Llama and a third or more for Granite. Fewer commands would be the gain there, not fewer waits: it was not done.

The attention rows of two blocks still take a third to a half of what is left on either backend. Their reads no longer count (the keys come back in the pass's one round trip): the cost is the host's arithmetic, one dot product a head and a position on one thread, each key row converted from the cache's type for each head that reads it.

Not recording, a device context runs what it ran before the reads existed. The command built before and the one built after, in turn, five times each, 200 tokens of Llama 3.2 1B: on Vulkan 340 to 463 tokens a second before (median 359) and 339 to 468 after (median 434); on OpenCL 40 to 364 before (median 240) and 40 to 364 after (median 203); the spread is the machine's other work (one pair of runs fell to 40 on both sides), the best runs are equal. The kernels launched are the same with the recording on or off (OKLM_CL_PROFILE=2, Llama 3.2 1B on OpenCL: 1209 launches for a prompt and 8 tokens, 5337 for 40, 129 a token, either way): what the recording adds on the device is buffer copies and the read that ends a pass, and a context that does not record queues neither.

tests/server/test_server.c (test_activations) checks the route: server.md.

Cost with the recording off. Decoding SmolLM2-135M Q4_K_M on the CPU, 300 tokens at temperature 0, the command built before the hooks and the one built after run in turn, twelve times each, on one thread: 27.6 tokens a second before (22.9 to 34.3; 11.1 s of CPU a run) and 29.9 after (22.0 to 36.7; 10.3 s), medians. The difference is inside the runs' spread, which is wide: the machine was busy with other work (a load of 17 on 16 cores). Idle, the command before gave 235 to 240 tokens a second on its default threads; the one after was not measured idle, nor was the cost of the recording when it is on.