Hybrid linear-attention layers (Gated DeltaNet)
How OKLM gains the linear-attention layers that newer bases (Qwen3-Next, and the roadmap's "Next" item 18 step 5) mix with full attention, so the inference engine runs them and the trainer trains them. Inference comes first; the trainer follows (its kernels check against the inference engine).
Status: proposed (2026-11-05). No code yet; this scopes the work.
The architecture
A hybrid model is a stack where most blocks are Gated
DeltaNet (linear attention with a recurrent state) and a
minority are the full attention OKLM already runs
(qwen3-style, with its KV cache). Qwen3-Next is the
reference shape: a repeating pattern (e.g. 3 linear : 1 full), the same
RMSNorm, SwiGLU FFN (often MoE) and residual stream as the full layers,
so only the token-mixing differs.
A Gated DeltaNet layer, per head, mixes tokens through a state matrix
S_t (d_v x d_k) updated by the gated
delta rule (Yang et al., Gated Delta Networks,
2024):
- Project the input to
q_t, k_t(dimd_k) andv_t(dimd_v) per head, a per-head decaya_t in (0,1)(from a sigmoid of a projection, often raised to a learned power) and a write strengthb_t in (0,1). - A short causal depthwise convolution (kernel ~4)
over
q, k, valong time, then L2-normalizeq_tandk_tper head. - The recurrence:
S_t = a_t * S_{t-1} + b_t * (v_t - a_t * (S_{t-1} k_t)) k_t^T(equivalentlyS_t = S_{t-1} (a_t I - b_t k_t k_t^T) + b_t v_t k_t^T): the state decays bya_t, then takes a rank-1 step that replaces the part of the prediction alongk_twithv_t. - Output
o_t = S_t q_t, then a per-head output gate (a sigmoid/SiLU of a projection, elementwise) and an RMSNorm, then the output projection back to the residual width.
The exact projections, the gate and the power on a_t are
read from the model's config; the design confirms them against
Qwen3-Next's reference (and llama.cpp if it runs the family) before the
kernels are written.
Two forms that must agree
The recurrence is sequential, but training and prefill run a whole sequence at once. As for full attention, OKLM keeps two forms and tests that they give the same result:
- Recurrent (decode, one token): carry
Sacross steps;O(d_k d_v)state per head,O(d_k d_v)work per token. This is the state the inference cache holds for a linear layer, in place of a KV cache. - Chunked scan (prefill, training): split the
sequence into chunks; within a chunk compute the contributions in
parallel (matrix products over the chunk, like attention), carry
Sbetween chunks. This is the standard chunk-parallel DeltaNet form; it is what the GPU runs and what the trainer differentiates. Its result equals the recurrent form to float rounding — the first self-consistency test.
The GGUF convention (from gguf-py / llama.cpp)
The reference names, so OKLM can eventually load a real Qwen3-Next GGUF, not only the synthetic fixture:
- Arch:
qwen3next. Hparams:qwen3next.ssm.{conv_kernel, inner_size, state_size, time_step_rank, group_count}beside the usualqwen3next.{embedding_length, block_count, attention.head_count, feed_forward_length, ...}. Which blocks are full attention vs linear is read from the tensors present (a block withssm_*is linear; one withattn_qis full), as llama.cpp does. - Linear (Gated DeltaNet) block tensors:
blk.N.ssm_in(the input projection to q, k, v and the output gate z),blk.N.ssm_conv1d(the short causal conv over q/k/v),blk.N.ssm_dt(the time-step / strength projection),blk.N.ssm_a(the decay parameter A),blk.N.ssm_ba(Qwen3-Next's packed beta/alpha gates),blk.N.ssm_norm(the gated output RMSNorm),blk.N.ssm_out(the output projection). The FFN is the usual (MoE) set. - Full-attention block tensors: the usual
attn_norm, attn_q, attn_k, attn_v, attn_output, attn_q_norm, attn_k_norm, plusblk.N.attn_gate(Qwen3-Next gates the attention output too).
The exact split of ssm_in into q/k/v/z, and how
ssm_dt, ssm_a and ssm_ba yield
the per-step decay a_t and strength b_t for
oklm_gated_delta_*, is the parity work of integration step
3 (checked against llama.cpp); the synthetic fixture fixes a
decomposition OKLM controls so the engine is tested before real
weights.
OKLM mapping
- Hparams (
include/oklm/io/hf.h): a per-layer kind (linear or attention) — a bit pattern or a count, asswa_patternalready marks sliding layers; the linear hparamsn_linear_head,head_dim_k_lin,head_dim_v_lin,conv_kernel, and the gate/decay form. The converter (src/io/convert.c,hf_config.c) reads them from config.json. - Weights (
src/model/model.cbind): per linear layer, the input projections (q,k,v, the decayaand strengthb), the short-conv weights (depthwise,conv_kernel x channels), the output gate and its norm, and the output projection. Named underblk.N.*as the GGUF writer lays them out. - Forward (
src/model/arch_llama.clin_block): dispatch per layer on its kind. Two ops ininclude/oklm/core/ops.h,OKLM_OP_SSM_CONVandOKLM_OP_GATED_DELTA(the L2 norms folded in), on the CPU (src/core/cpu/ssm_cpu.c) and OpenCL (kernels.cl), in the recurrent form; the gated output norm is the existing RMS norm and SwiGLU. - State cache (
src/model/model.c,kv_pool.c): a linear layer holds a fixed-size recurrent state per sequence (n_linear_head * d_k * d_vfloats), not a growing KV cache. The context allocates it beside the KV blocks of the full layers; a sequence's state is reset, copied and swapped with the sequence (truncate drops it,seq_copyduplicates it). This is the largest inference change — the cache is no longer KV-only. - Trainer (
src/train/layers_train.c,train_kernels.cl): the chunked forward (kept activations for the backward) and its gradient (through the scan, the gates, the conv, the norms), on the CPU and OpenCL, as the other blocks. LoRA targets gain the linear layer's projections.
Order and tests
Inference first (the trainer's kernels check against it):
- CPU recurrent forward of one Gated DeltaNet head
(done,
oklm_gated_delta_recurrent): checked by the delta rule's properties and by processing a sequence in one call versus in pieces. - CPU chunked forward (done,
oklm_gated_delta_chunked): checked equal to the recurrent form for every chunk size and decay regime (the self-consistency test). The short causal conv and the L2 norm are done too (oklm_gated_delta_short_conv,oklm_gated_delta_l2norm), each with its test. - Model integration: a tiny synthetic hybrid model (a
qwen3nextarch in the test fixtures, a couple of linear layers and one full), greedy output checked against the recurrent reference; then a real Qwen3-Next checked against llama.cpp if it runs the family, else against the recurrent form over the real weights. - OpenCL forward, checked equal to the CPU (as every
other op). Done: the block runs where its layer runs, on two ops,
OKLM_OP_SSM_CONV(the short conv with its ring) andOKLM_OP_GATED_DELTA(the L2 norms, the sigmoids and the recurrence), between the existing products, RMS norm and SwiGLU; on OpenCL the states of the device layers stay in device memory (read back only foroklm_kv_save). Both use the recurrent form, a token at a time per head: the chunked form is not on the device.test_openclchecks the ops against the CPU at 128-value heads,test_modelthe tiny model's OpenCL prefill and token-by-token decode against the CPU,test_kv_filea device state through a KV file. - Trainer forward + backward, the gradient checked by
finite differences (
test_train.c, as every block) and the forward checked equal to the inference engine's. On the device (done 2026-11-11): thetr_gd_*kernels oftrain_kernels.cl, the recurrence's backward from states kept every 16 tokens;test_train_clchecks the gradients against the CPU trainer over 7 and 37 tokens (training.md).
Risks
| Risk | Signal | Mitigation |
|---|---|---|
| The exact recurrence/gate differs from Qwen3-Next | output parts from the reference | confirm the math against the reference (and llama.cpp) in step 1 before the kernels |
| State cache reshapes the inference cache | the paged KV pool assumes KV-only | the linear state is a separate fixed allocation per sequence, not paged; the KV pool is untouched for full layers |
| Chunked ≠ recurrent | self-consistency test fails | the chunk form is derived from the recurrence and tested against it before integration |
| Backward through the scan is subtle | finite-difference check fails | build the CPU forward and its finite-diff'd backward before the device kernel |
| Scope | a large multi-part change | land it in the order above, each step tested, behind a new arch so nothing else moves |
The published layout:
qwen35
The sections above describe qwen3next as OKLM first
built it, against a fixture of its own design. The Qwen3.5 / 3.8 files
(general.architecture = qwen35; Qwen3.5-9B was the file
audited: 427 tensors, 32 blocks, width 4096; Qwen3.8-27B, 64 blocks at
width 5120 with 48 value heads, was run after it) are laid out
differently, and OKLM reads them as llama.cpp does. The reference is
llama.cpp at the commit tools/gen_llama_ref.sh pins
(8642541b: its src/models/qwen35.cpp and
delta-net-base.cpp). A tiny model of this layout
(tests/fixtures/model/tiny-qwen35.gguf) agrees with it to
2e-4 on every logit; the 9B file's figures are in compatibility.md.
What the file says
Key (qwen35.) |
Qwen3.5-9B | Meaning |
|---|---|---|
block_count, embedding_length,
feed_forward_length |
32, 4096, 12288 | the stack; the FFN is dense SwiGLU |
attention.head_count, head_count_kv,
key_length |
16, 4, 256 | the full-attention blocks: heads of 256, wider than width / heads |
rope.dimension_count, rope.freq_base,
rope.dimension_sections |
64, 1e7, [11, 11, 10, 0] | 64 of a head's 256 dimensions are rotated, pairs (i, i + 32); the sections split them between time, height and width for image patches, and a text token has the same position in all three, which is the plain rotation |
ssm.state_size, ssm.group_count |
128, 16 | a linear block's key heads: 16 of 128 |
ssm.time_step_rank, ssm.inner_size |
32, 4096 | its value heads: 32, of 4096 / 32 = 128 |
ssm.conv_kernel |
4 | taps of the short causal conv |
full_attention_interval |
4 | blocks 3, 7, ... are full attention, the others linear
(attention.recurrent_layers, when a file has it, lists them
instead) |
OKLM tells the two kinds of block apart by their tensors (a block
with ssm_conv1d is linear), as it does for the rest of a
model's layout.
A linear block
With xn the block's normalized input
(attn_norm, RMS, eps from
attention.layer_norm_rms_epsilon), G = 16 key heads, H = 32
value heads, d = 128:
attn_qkv[4096 -> 8192] gives q (G x d) | k (G x d) | v (H x d);attn_gate[4096 -> 4096] the output gate z;ssm_betaandssm_alpha[4096 -> 32] a write strength and a decay per value head.ssm_conv1d[4 taps x 8192 channels]: each channel of q | k | v is convolved with its own 4 taps over the last 4 tokens (zeros before the sequence's start), then SiLU. The file stores it taps first; OKLM keeps a copy channels first, the layout of its kernel.- q and k are L2-normalized per head
(
x / sqrt(sum x^2 + eps)), q is divided by sqrt(d). - Per value head h and token:
b = sigmoid(beta),a = exp(ssm_a[h] * softplus(alpha + ssm_dt.bias[h]))(ssm_aholds-exp(A_log), so a is in (0, 1)); with the head's state S (d x d) and the key headh mod G:S = a S + b (v - a S k) k^T,o = S q. ogoes through an RMS norm per value head (ssm_norm, 128 weights), is multiplied bySiLU(z), andssm_out[4096 -> 4096] projects it back onto the residual stream.
Then post_attention_norm and the FFN, as in the other
blocks: post_attention_norm is the norm before the FFN, not
a norm of the attention's output.
A full-attention block
attn_q [4096 -> 8192] gives each of the 16 heads 512
values: its query (256) then an output gate (256). Queries and keys go
through an RMS norm per head (attn_q_norm,
attn_k_norm) and the rotation above; after the attention (4
key/value heads shared by 16 query heads, scale 1/16) each head's output
is multiplied by the sigmoid of its gate, then
attn_output.
Where the test layout differed
qwen3next test layout |
published qwen35 |
|
|---|---|---|
| projections | one ssm_in to q, k, v and z |
attn_qkv and attn_gate |
| decay | sigmoid(ssm_a[h]), fixed per head |
per token, from ssm_alpha, ssm_dt.bias and
ssm_a |
| write strength | ssm_dt |
ssm_beta |
| key head of value head h | h / (H / G) |
h mod G |
| query | as normalized | divided by sqrt(d) |
| conv weights | channels first | taps first |
| full attention | plain queries | queries joined with an output gate |
| norm before the FFN | ffn_norm |
post_attention_norm |
| L2 and gated-norm eps | 1e-6 | the model's RMS eps |
The recurrence itself, the conv with its window and the gated norm
are the same operations: the published layout runs on
OKLM_OP_SSM_CONV and on OKLM_OP_GATED_DELTA
with a decay per token (oklm_gated_delta_tok()), on the
CPU, on OpenCL and on Vulkan (on a device).
qwen3next stays as it was, a layout the trainer and the
device kernels are tested on; it is not the layout of the published
Qwen3-Next files (their ssm_in, ssm_ba and
expert FFN are not read yet).
The state a sequence keeps
Per linear block and sequence: the H matrices S (32 x 128 x 128 floats) and the conv's window, the last 3 inputs of its 8192 channels: 548 864 floats, 2.1 MiB; 24 linear blocks make 50 MiB a sequence for Qwen3.5-9B, whatever the context length. The 8 full-attention blocks keep keys and values as any model does. What follows from a state that only moves forward:
- Several sequences: one state each; a forward pass reads and writes those of the sequences in the batch, and a sequence that starts again at position 0 starts from zeros.
- A cut (
oklm_seq_truncate()to a position in the sequence's middle: a chat whose next prompt shares only a prefix, a rejected draft): the state cannot go back, so the context keeps each sequence's token ids and, before the sequence's next use, brings the state to the cut from the nearest state it kept (below), or from the sequence's start when it kept none. The caller sees the cut honoured. A sequence that holds embedding rows (an image) cannot be computed again: the cut is refused and the sequence takes nothing more until it is emptied. - A copy (
oklm_seq_copy()): the whole sequence is copied with its state; a part of it takes the kept state of its tokens, which the source left. - The paged cache: the attention blocks' rows are paged as usual, the states are not (they have one size). The prefix cache lends a sequence the cached rows of another's prefix; the sequence takes the nearest kept state of those tokens and runs the tokens after it over the lent rows. Without kept states it is off, and swapping a sequence out is off in both cases.
- A file (
oklm_kv_save()/oklm_kv_load()): the states are saved after the rows, for the whole sequence only. - A context of another size: a new context starts empty; a sequence moves to it through a file or by evaluating its tokens.
Kept states
Without them a cut costs the evaluation of everything before it: a
rejected draft, a chat turn asked again, a request sharing a system
prompt with the previous one each paid the whole prefix. A context of
these models now keeps what gives a past state back
(src/model/lin_keep.c;
oklm_context_params::state_keep and
state_every, the environment's
OKLM_STATE_KEEP, 0 for none;
oklm_state_keep_stats() counts what they did). The states
are kept in host memory wherever the blocks run; the journal is written
only when every linear block's state is in host memory (on a device, a cut is a forward pass
from the nearest kept state).
What is kept.
| what | when | size (Qwen3.5-9B) | |
|---|---|---|---|
| kept state | every linear block's state after a prefix, in a store the context's sequences share (8 slots, allocated as they fill) | eight tokens before the end of a prompt (a batch of several tokens that keeps the last one's logits: the linear blocks copy their state as they pass it), with the journal of those eight; where a prompt begins after generated tokens; every 32 tokens while a sequence grows (two per sequence, the older one replaced); where a cut had to be paid by a forward pass | 50 MiB each, 6 MiB more with a prompt's journal |
| journal | per sequence, what each linear block's recurrence read at its last 96 positions: the q, k, v rows before the convolution, the write strengths and decays | every forward pass | 0.76 MiB a position, 73 MiB |
| state before a draft | per sequence, its state before the last batch that kept every logit, which is how a draft is verified | before such a batch | 50 MiB |
A prompt's state is kept eight tokens before its end, with the
journal of those eight, because its sequence is cut among them when the
prompt comes again: the callers (the chat's /retry, the
server given the same request) keep the common prefix less one token,
which they evaluate again for its logits; the next turn of a
conversation shares the prompt up to the few tokens a chat template ends
it with and does not write in the history (22 of a first prompt's 24
tokens with the Ternary Bonsai 2 template); a model whose reasoning is
dropped from the history shares all of it. Each of those cuts is then
paid by the recurrence over at most eight positions, whatever the
sequence generated since. A turn taken back and asked otherwise
(/rewind) is cut a few tokens after the place its prompt
began, where a state is kept as well: those few tokens go through a
forward pass, once.
A kept state is found by what it is the state of: two 64-bit hashes of the tokens before its position and the adapter, compared with those of the sequence asking. The states are a function of those tokens alone, so a sequence takes one another sequence computed, in the same call or long after that sequence was emptied. When the store is full the least recently used state goes (taking one, or finding it already there, counts as a use).
A cut to position n
(model_lin_keep_settle()), at the sequence's next use:
- the kept state with the largest position at or before n is copied back (the state before the last draft when it is nearer; zeros when there is none);
- if the sequence's journal, or the one kept with that state, holds every position between it and n, the convolution and the recurrence of each linear block run again over the journal, on the same two operations as the forward pass: no projection, no attention, no feed-forward. This is the case of a rejected draft, which costs the tokens accepted before the rejection, and of a cut of up to 32 tokens into what a sequence generated;
- otherwise the tokens between go through a forward pass with the
cache frozen (
model_replay()): the attention blocks read the rows the cut left instead of writing them, so nothing but the states moves, and a paged cache copies no block. The state reached is then kept: a place cut to once (the end of a shared system prompt) is reached for nothing the next time.
Exactness. The journal path gives the bits of a
context that never went past the cut: the recurrence reads the values it
read the first time, in the same order. The forward path gives what
evaluating those tokens in that batch shape gives, as any prompt
evaluation does. On the tiny models both are equal to a fresh
evaluation, bit for bit, in tests/model/test_state_keep.c
(contiguous and paged, F16 and F32, the folded and the expert variants,
the test layout), which holds the first to equality and the second to
1e-4.
Kept states on a device
A context whose linear blocks run on OpenCL or Vulkan holds their states in device memory. It keeps states as a CPU context does, in host memory: taking one reads every linear block's state back (50 MiB for the 9B), giving one writes it to the device again. It writes no journal: an entry is what a block read at a position, and reading those rows back from the device at each of the 24 blocks of every decode step would stop the device between blocks, which costs more than what the journal saves there. So on a device:
- the states are kept at the same places (eight tokens before a prompt's end, read back as the blocks pass it; where a prompt begins after generated tokens; every 32 tokens of a growing sequence; before a draft; where a cut was paid);
- a cut takes the nearest one and runs the tokens after it through a
forward pass with the cache frozen, one batch on the device: at most
eight tokens for a prompt asked again or a next turn, fewer than 32 for
a cut into the last 32 tokens a sequence generated, the accepted tokens
for a rejected draft.
journal_tokensofoklm_state_keep_stats()stays 0 andforward_tokenscounts them; - a context with layers on both sides (
--gpu-layers), with its CPU layers streamed to the device for prompts, or spread over several devices does the same: one block without a journal entry leaves the batch without one.
tests/model/test_state_keep.c runs its cuts, its shared
states and its random walk on OpenCL and on each Vulkan device as on the
CPU, with the forward pass where the CPU has the journal (36 tokens for
the 6 cuts of its first scene where the CPU pays 25 and 11 journaled),
within 2e-4 of a fresh evaluation (measured: 0 on OpenCL, 4e-5 at most
on Vulkan). Keeping states costs a decode step nothing that was measured
on Qwen3.5-9B: 83.0 against 83.3 tokens a second on OpenCL and 104.2
against 104.0 on Vulkan with OKLM_STATE_KEEP=0; a prompt of
512 tokens 1 to 2 % (the state read back before its end).
The memory is counted with the recurrent state by
oklm plan (every slot filled, a journal and a draft copy
per sequence): 640 MiB instead of 67 for Qwen3.5-9B and Ternary Bonsai 2
4B with one sequence, 267 MiB instead of 26 for the 2B, whatever the
context length. A context counts the same, and allocates each of them
when it is first used (oklm_state_keep_stats(),
state_keep.bytes of the server's
/api/model).
Measured (Ternary Bonsai 2 4B,
...-PQ2_0-MTP-Q8_0.gguf, one prompt of 13 tokens, greedy,
96 tokens generated, CPU, 4 threads; the text is the plain run's in
every case). The machine was shared and loaded (a load average of 14 to
23 on 16 threads), so the rounds were run one after the other, each a
plain run then the drafted ones, and differ by more than the runs of a
round do (another job started during the fourth round's
--mtp run):
oklm run |
drafts accepted | tokens a second, rounds 1 to 4 |
|---|---|---|
| plain | - | 3.5, 4.8, 2.0, 3.9 |
--mtp, kept states |
64 of 156 | 4.0, 4.4, 1.9, 2.2 |
--mtp --draft-n 1, kept states |
44 of 52 | 4.1, 2.0, 3.4, - |
--mtp, OKLM_STATE_KEEP=0 (as before) |
64 of 156 | 0.5 (0.7 on a calmer run) |
With kept states the 25 rejected drafts of the --mtp run
cost 57 tokens through the journal and none through a forward pass: 1.0
to 1.7 s of state copies and recurrence, and 0.4 s for the copies before
each draft, in a run of 44 to 59 s (3 %). Without them the same 25
rejections evaluated the prefix again each time: 132 to 181 s for the 96
tokens. What is left between --mtp and the plain run is the
drafting itself: on the CPU a verification batch costs about as much per
token as plain decoding, and 92 of the 156 drafted tokens were evaluated
for nothing; --mtp ran at 0.6 to 1.1 times the plain speed
of its round, which that machine's noise does not separate from 1, and
is not shown faster here. Prompt lookup on the 2B
(--spec-ngram 4, 15 of 118 proposals accepted, the plain
text): 7.7 tokens a second plain, 4.3 with kept states, 0.4 without
(13.2, 6.9 and 1.0 on a calmer run, made before the state before a draft
was kept apart).
In the chat (oklm chat --stats on
Ternary Bonsai 2 2B, which prints what a cut cost): /retry
cut the prompt of 20 tokens at 19 and paid 7 tokens through the journal
kept with the prompt's state; the next turn shared 18 tokens and paid 6
the same way; /rewind then another question shared 27 of 36
tokens and paid a forward pass over 15, from the first prompt's state. A
longer script (replies of 500 tokens, /retry, a second
turn, /rewind, a third) took 296 s with kept states and 412
s without, the same text.
Not kept. The state of a sequence holding embedding rows (an image: its positions have no token to hash); a kept state in a KV file (the file stores the state of its end, and loads whole or not at all); states on a device.
The sensitive steps
On a few steps of the recorded prompts of Qwen3.5-9B Q4_K_M, single logits differ by 2 to 13.8 between OKLM and llama.cpp, where every other step is within 0.4 to 0.7; the build for Windows chose another token than llama.cpp on one of them (step 38 of the continuation of "The capital of France is": llama.cpp's two logits 4.59 apart, OKLM's own 0.34 the other way). Measured on 2026-11-13, this is the model's own sensitivity on those steps: llama.cpp moves as far from itself when only its last places change, on the same steps and in the same blocks, and OKLM's gated delta blocks lose no precision. What was measured, on the CPU:
The logits. The 101 steps of that continuation (the model repeats "Paris.\nThe capital of France is"), every run on the recorded run's tokens, against the recorded run (llama.cpp at the pinned commit, F16 keys and values, the prompt in one batch, 4 threads):
| run | mean |logit difference| | largest | another choice |
|---|---|---|---|
| llama.cpp, 1 or 6 threads (46 steps) | 0 | 0 (the same bits) | |
| llama.cpp, keys and values as F32 | 0.186 | 14.9 (step 17) | |
| llama.cpp, the prompt one token a pass | 0.233 | 13.8 (step 13) | |
| llama.cpp, the prompt in batches of 3 tokens | 0.233 | 13.8 | |
| llama.cpp, both | 0.231 | 13.8 | |
llama.cpp, its other attention kernel (flash_attn) |
0.174 | 13.8 | |
| llama.cpp, that kernel and the prompt one token a pass | 0.210 | 13.7 | |
| OKLM, Linux | 0.222 | 13.8 (step 13) | |
| OKLM, Linux, F32 cache | 0.248 | 14.4 (step 17) | step 2, where llama.cpp's two logits are 0.09 apart |
| OKLM, the build for Windows under Wine | 0.217 | 13.8 | step 38 (4.59 apart) |
| OKLM, the build for Windows, F32 cache | 0.221 | 13.8 |
Between any two of these runs the mean difference over the 82 ordinary steps is 0.06 to 0.12, two of llama.cpp, two of OKLM or one of each (0.10 to 0.115 for the last). Over the 18 sensitive steps (13, 17, and from 37 on the two that predict " of" and " France" in each repetition of the sentence: 37, 38, 45, 46, ... 93, 94) it is 0.08 to 0.90 between two llama.cpp runs with single logits up to 14.9 apart, 0.31 to 0.64 and 10.1 between two OKLM runs, 0.55 to 0.89 and 14.4 between one of each. The 13.8 of step 13 is the recorded run alone against every other one: the six other llama.cpp runs and the four of OKLM are within 0.9 of each other there. At step 17 the runs fall in three groups, whichever engine they come from: the recorded run; llama.cpp with an F32 cache and OKLM on Linux with an F32 cache, 1.4 apart; the eight others, within 1.2 of each other, 4.2 to 5.0 from the first and 9.7 to 10.7 from the second. As a control, the same changes of llama.cpp on Qwen3-8B Q4_K_M (no linear block; 33 steps of the same prompt) move its logits by 0.14 on average, near what they do here, but by 1.5 at most: the mean is what a change of last places costs a Q4_K_M file of this size in llama.cpp, the steps where a logit moves by 10 are this model's.
The hidden state, block by block. The residual
stream after every block and, inside every linear block, the
projections, the conv's output, the recurrence's output and state, the
gated norm's output, read from llama.cpp through its
cb_eval callback and from OKLM through taps put in its
forward pass for the measure (not kept in the tree), at each of the
first 46 steps. Relative L2 difference of the last token's stream after
blocks 0 to 31, at an ordinary step and at three sensitive ones:
Step 12 (ordinary):
| 0 | 1 | 2 | 4 | 8 | 12 | 16 | 20 | 24 | 28 | 30 | 31 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OKLM (Linux) against llama.cpp | 4e-07 | 0.002 | 0.007 | 0.013 | 0.045 | 0.064 | 0.060 | 0.046 | 0.040 | 0.037 | 0.044 | 0.042 |
| OKLM (Windows) against llama.cpp | 4e-07 | 0.002 | 0.007 | 0.013 | 0.047 | 0.061 | 0.064 | 0.052 | 0.043 | 0.038 | 0.041 | 0.039 |
| OKLM (Windows) against OKLM (Linux) | 0 | 1e-08 | 8e-08 | 0.009 | 0.033 | 0.044 | 0.048 | 0.039 | 0.035 | 0.033 | 0.037 | 0.039 |
| llama.cpp, prompt a token a pass, against llama.cpp | 0 | 0.001 | 0.007 | 0.014 | 0.038 | 0.053 | 0.059 | 0.047 | 0.039 | 0.035 | 0.036 | 0.037 |
| llama.cpp, F32 cache, against llama.cpp | 0 | 0 | 0 | 0.008 | 0.034 | 0.046 | 0.052 | 0.048 | 0.044 | 0.042 | 0.050 | 0.047 |
Step 13 (13.8 on one logit):
| 0 | 1 | 2 | 4 | 8 | 12 | 16 | 20 | 24 | 28 | 30 | 31 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OKLM (Linux) against llama.cpp | 2e-07 | 0.002 | 0.007 | 0.012 | 0.034 | 0.041 | 0.043 | 0.041 | 0.037 | 0.044 | 1.09 | 1.68 |
| OKLM (Windows) against llama.cpp | 2e-07 | 0.002 | 0.007 | 0.012 | 0.036 | 0.045 | 0.048 | 0.045 | 0.041 | 0.078 | 1.10 | 1.72 |
| OKLM (Windows) against OKLM (Linux) | 2e-08 | 3e-08 | 6e-08 | 0.008 | 0.025 | 0.031 | 0.037 | 0.039 | 0.037 | 0.066 | 0.058 | 0.050 |
| llama.cpp, prompt a token a pass, against llama.cpp | 0 | 0.002 | 0.007 | 0.013 | 0.031 | 0.036 | 0.045 | 0.042 | 0.037 | 0.055 | 1.08 | 1.65 |
| llama.cpp, F32 cache, against llama.cpp | 0 | 0 | 0 | 0.007 | 0.030 | 0.036 | 0.044 | 0.045 | 0.039 | 0.11 | 1.11 | 1.74 |
Step 37:
| 0 | 1 | 2 | 4 | 8 | 12 | 16 | 20 | 24 | 28 | 30 | 31 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OKLM (Linux) against llama.cpp | 2e-07 | 0.003 | 0.007 | 0.012 | 0.035 | 0.13 | 0.30 | 0.61 | 0.78 | 0.78 | 0.80 | 0.54 |
| OKLM (Windows) against llama.cpp | 2e-07 | 0.003 | 0.006 | 0.013 | 0.053 | 0.23 | 0.47 | 0.81 | 0.94 | 0.91 | 0.96 | 0.63 |
| OKLM (Windows) against OKLM (Linux) | 9e-09 | 0.002 | 0.006 | 0.011 | 0.053 | 0.14 | 0.25 | 0.35 | 0.35 | 0.33 | 0.40 | 0.28 |
| llama.cpp, prompt a token a pass, against llama.cpp | 0 | 0.001 | 0.005 | 0.011 | 0.045 | 0.085 | 0.20 | 0.30 | 0.38 | 0.38 | 0.38 | 0.27 |
| llama.cpp, F32 cache, against llama.cpp, prompt a token a pass | 0 | 0.001 | 0.005 | 0.011 | 0.047 | 0.15 | 0.37 | 0.67 | 0.92 | 0.95 | 0.96 | 0.68 |
Step 38 (where the build for Windows chose another token):
| 0 | 1 | 2 | 4 | 8 | 12 | 16 | 20 | 24 | 28 | 30 | 31 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OKLM (Linux) against llama.cpp | 4e-07 | 0.003 | 0.009 | 0.018 | 0.057 | 0.14 | 0.31 | 0.36 | 0.30 | 0.24 | 0.30 | 0.23 |
| OKLM (Windows) against llama.cpp | 4e-07 | 0.003 | 0.009 | 0.020 | 0.044 | 0.053 | 0.15 | 0.29 | 0.29 | 0.25 | 0.34 | 0.29 |
| OKLM (Windows) against OKLM (Linux) | 0 | 0.002 | 0.008 | 0.013 | 0.052 | 0.13 | 0.39 | 0.47 | 0.38 | 0.30 | 0.36 | 0.34 |
| llama.cpp, prompt a token a pass, against llama.cpp | 0 | 0.002 | 0.008 | 0.012 | 0.056 | 0.086 | 0.19 | 0.21 | 0.17 | 0.14 | 0.18 | 0.18 |
| llama.cpp, F32 cache, against llama.cpp, prompt a token a pass | 0 | 0.002 | 0.008 | 0.012 | 0.046 | 0.068 | 0.19 | 0.20 | 0.17 | 0.16 | 0.24 | 0.23 |
What the rows say:
- The recurrence is exact. In block 0, whose input is the same in both engines, every quantity of OKLM is llama.cpp's to 1e-7 to 4e-7: the projections, the conv, the recurrence's output, the gated norm, and the state of the 32 heads after 43 tokens (2.8e-7). That is the rounding of single-precision sums taken in another order, not a loss: a recurrence that lost precision would show it here, where nothing else differs.
- The first differences are jumps, not growth. Block 0's output is llama.cpp's to 4e-7 at 37 of the 46 steps and 1e-5 to 6e-4 away at the 9 others, nothing between; the Windows build is the Linux build's to 1e-8 until a block where it is 1e-4 to 7e-3 away, at step 0 past block 16, from step 29 on in the first two blocks. Both engines round the activations to 8 bits before a product with quantized weights; one 8-bit step of one activation is 6e-4 to 1e-3 of the vector's norm at block 0's feed-forward, and the jumps there are that size. A last place that tips one rounding becomes a difference of 1e-3, in any engine: llama.cpp given its prompt a token a pass is itself to the bit in block 0 and 1e-3 to 2e-3 away after block 1. The states then carry it to every later step.
- From there the model amplifies it, llama.cpp's as OKLM's. At an ordinary step the difference grows from 2e-3 after block 1 to 4e-2 to 6e-2 at blocks 12 to 16 and stays there, the same curve whichever two runs are compared. Inside a linear block the recurrence is not what amplifies: its output differs less than what it reads (at step 38 between two llama.cpp runs, block 12: the projections 0.09 apart, the conv's keys and values 0.20 and 0.16, the recurrence's output 0.066, after the gated norm 0.15, after the feed-forward 0.16 of its output). The projections, the conv over the last four tokens, the gated norm and the feed-forward each widen the gap.
- A sensitive step is one where a block switches. At step 13 the runs are 4e-2 to 1e-1 apart until block 29 and 1.1 apart after block 30: one value channel of its head 6 reads -19 to -23 at every step around and, at that step, -3.8 in the recorded run, +8.4, +5.9 in the two other llama.cpp runs, +4.7, +6.7, +6.2 in OKLM's three. After the conv and SiLU that is -0.17 against 1.9 to 3.0, 99.9 % of the head's difference, and the recorded run stands alone on its side of the gate. At steps 37, 38 and 45 the growth is spread from block 8 to block 24, up to 1.0 between two llama.cpp runs.
- The choices. Over the 100 steps the gap between llama.cpp's two best tokens moves by up to 2.3 between its own seven runs (step 38: 4.59 to 6.91; step 62: 4.48 to 6.68) and by up to 5.2 in OKLM's four (step 38: -0.34 in the Windows build, 4.36 to 5.59 in the three others; step 61: 15.5 to 17.1 where the recorded run has 11.9 and llama.cpp's others up to 14.1). No llama.cpp run chose another token on any of the 100 steps, so the Windows build's choice at step 38 is not shown to be one llama.cpp could make; it is shown to come from the same amplification, on a step where two llama.cpp runs are 0.2 apart in the stream after block 20 and the Windows and Linux builds 0.47. The chat turn of the same test has no such step (the gap moves by 2.5 at most, in llama.cpp's runs as in OKLM's); there llama.cpp does part from itself, at steps 1, 64, 80 and 87 (its recorded gaps 0.20, 0.05, 0.13 and 0.03; step 64 is where the Windows build parts), and at step 26, where OKLM on Linux parts (0.44 recorded, -0.02 in OKLM), its eight runs have 0.05 to 0.46.
tools/gen_llama_ref.sh records the second llama.cpp run
of the table (LLAMA_REF_KV=f32, read by
tools/llama_ref.c, as LLAMA_REF_ONE=1 gives
the prompt a token a pass), and test_llama_itself() in
tests/model/test_qwen35_public.c compares the two
recordings and puts OKLM beside them: OKLM may be 1.5 times as far from
the recorded run as llama.cpp's own second run is, in the mean and in
the largest difference. The bound on a parted choice stays what was
measured on each platform (0.6 on Linux, 5.0 for the build for Windows):
the measure explains the step, it does not make the choice
llama.cpp's.
Not in the 9B file, or not done
- Multi-token prediction: optional blocks after the
stack (
blk.N.nextn.*, each a full-attention block as above, counted inblock_countand innextn_predict_layers). The 9B file has none; Qwen3.8-27B has one. OKLM binds it with the gated queries and runs it as the other families' (oklm run --mtp): on the 27B file, 20 of its 45 drafted tokens were accepted and the text is the plain greedy one. It was not compared with llama.cpp's draft pass. Each rejected draft is a cut of the sequence, paid from the state kept before the draft (kept states); the 27B's run was made before those and not timed again. - The vision projector (
mmproj) is a separate file, not read. - Devices: done, on OpenCL and on Vulkan (below).
- Training: the trainer knows the test layout only.
- The cache's size: done. A context reserves key and value rows for the attention blocks only (the section "The cache of a hybrid stack", below).
- The expert variant (
qwen35moe): done, on a fixture and on a published file of Qwen3.6-35B-A3B (below).
On a device
A qwen35 or qwen35moe context runs on
OpenCL and on Vulkan like the other families: every block on the device,
or the last ones (--gpu-layers), the linear blocks' states
in device memory. What it took:
- The decay per token in the
gated_deltakernel (OpenCL) and shader (Vulkan), which had the test layout's decay per head: the decays' projections beside the write strengths, their bias and factor per head,exp(factor * softplus(projection + bias))computed by the rows of a head for each token, the key headh mod G, the query divided bysqrt(dk). One kernel serves both layouts; it is checked against the CPU's intest_openclandtest_vulkan_ssm(outputs and states within 1e-5 absolute plus 1e-4 relative, the tolerance of the decay per head). - The gated queries: the joined query and gate
projection is one product, split by two strided copies; the sigmoid and
the product after attention are the element-wise kernels. A device joins
the query, key and value matrices of a block into one when their rows
have one length; the gated query matrix has twice the rows the output
projection reads and is left out of that
(
dev_place_layer(),src/model/model.c). - Nothing else: the conv, the RMS norms, the rotation, SwiGLU, the
routed experts and the matrix products are the kernels the other
families run on (
ssm_betaandssm_alphaare Q8_0 in the published files). The token lookup is the CPU's, as for every model. - Kept states work with the states on a device (above).
The fixtures (test_model, test_qwen35
and test_qwen35moe; OpenCL on the RX 7900 XT, Vulkan on it and on the
Intel UHD 750): llama.cpp's logits within 1.9e-4
(tiny-qwen35) and 2.9e-4 (tiny-qwen35moe), 36
/ 36 greedy choices, every block on the device and the first one left on
the CPU; a prompt in one batch, in batches of 3 and token by token
within 5e-7 of each other; two sequences in one pass; a sequence through
a copy, a file and a cut; a paged cache. For tiny-qwen35moe
also the routed experts of the first or of every layer on the CPU
(n_cpu_moe), a CPU layer streamed to the device for
prompts, the expert cache and the experts' device tier
(test_n_cpu_moe, test_expert_gpu), as the
other expert fixtures.
Qwen3.5-9B Q4_K_M
(tests/model/test_qwen35_public.c, every test of the CPU
run again on each device, 32 of 32 blocks there, an F16 cache; RX 7900
XT, 2026-11-13):
| CPU | OpenCL | Vulkan | |
|---|---|---|---|
| choices on llama.cpp's tokens (5 prompts, 32 steps) | 159 / 160 | 158 / 160 | 158 / 160 |
| logit difference from llama.cpp, mean and largest | 0.116, 13.8 | 0.122, 13.7 | 0.122, 13.7 |
| against the CPU on those steps: choices, mean, largest | 157 / 160, 0.113, 3.65 | 159 / 160, 0.119, 3.55 | |
| 100 greedy tokens, chat turn: equal before parting (llama.cpp's margin there), choices on its tokens | 26 (0.44), 97 | 80 (0.13), 98 | 26 (0.44), 97 |
| 100 greedy tokens, plain text | 100 | 100 | 100 |
| a prompt of 296 tokens in batches of 64: choices, largest difference | 16 / 16, 1.72 | 16 / 16, 2.37 | 16 / 16, 2.75 |
| two sequences in one pass: choices | 32 / 32 and 32 / 32 | 32 / 32 and 31 / 32 | 32 / 32 and 31 / 32 |
| a paged cache, two sequences, 8 steps: choices llama.cpp's | 16 / 16 | 16 / 16 | |
| 16 of 32 blocks on the CPU, the same | 16 / 16 | 16 / 16 | |
prompt of 512 tokens, tokens a second
(bench_llama) |
79.3 | 1839 | 2300 |
| 128 tokens one at a time, tokens a second | 5.46 | 83.0 | 104.2 |
A device is as far from the CPU as the CPU is from llama.cpp, and as
the CPU is from itself with an F32 cache (0.121): the model's
sensitivity to the last places (the
sensitive steps), not a kernel's error, which the op tests and the
fixtures bound at 1e-4. The state through a file, a copy and a cut gives
the same 8 tokens on each device. With the float kernels
test_infer_public asks of a device
(OKLM_CL_Q8=0, OKLM_VK_Q8=0,
OKLM_VK_COOPMAT=0) OpenCL and Vulkan give the same figures
as each other: 159 / 160 choices, logits within 14.3, four of five
continuations identical. Qwen3.8-27B (16.6 GB) was not run on a device:
the same code path, twice the blocks.
Qwen3.6-35B-A3B UD-IQ3_XXS (qwen35moe,
13.2 GB, downloaded for the run and removed; 40 of 40 blocks and the 256
experts of each on the RX 7900 XT, nothing handed back to the CPU but
the token lookup):
| CPU | OpenCL | Vulkan | |
|---|---|---|---|
| choices on llama.cpp's tokens (5 prompts, 32 steps) | 158 / 160 | 156 / 160 | 156 / 160 |
| largest logit difference from llama.cpp | 5.14 | 2.74 | 8.35 |
| llama.cpp's margin where a choice differs, at most | 0.08 | 0.16 | 0.16 |
| identical continuations | 4 / 5 | 2 / 5 | 2 / 5 |
| against the CPU on those steps: choices, mean, largest | 156 / 160, 0.146, 6.01 | 155 / 160, 0.159, 7.81 | |
| the float kernels: choices, largest difference from llama.cpp | 156 / 160, 4.4 | 156 / 160, 4.4 | |
prompt of 512 tokens, tokens a second
(bench_llama) |
97.0 | 1222 | 1536 |
| 128 tokens one at a time, tokens a second | 11.65 | 64.7 | 108.6 |
The choices that differ are near ties for llama.cpp (0.16 at most between its two logits); the file's 3-bit and 2-bit tensors leave it flat distributions, as its tunes showed on the CPU. The experts' device tier and the expert cache were run under the hybrid stack on the fixture only: this file fits the device whole.
The cache of a hybrid stack
A linear block keeps a recurrent state and never reads a key or a
value row; a context used to reserve them all the same, one cache per
block. It now reserves rows for the blocks that attend, and nothing for
the others: 8 of the 9B's 32 blocks, so 256 MiB of cache at 8192 tokens
instead of 1 GiB, and 1 GiB at 32768 instead of 4. The same rule serves
the short convolution blocks of lfm2 and
lfm2moe (inference.md),
which are bound the same way (a block with an ssm_in
projection has no attention: model_layer_cached() in
src/model/model_internal.h).
A block without rows has no cache tensor at all
(k_cache[l] and v_cache[l] are NULL), and each
place that walked the layers skips it:
| Where | What changes |
|---|---|
the allocation (oklm_context_new()) |
the host arena holds the attention blocks' rows
(ctx_kv_data_bytes(), ctx_kv_arena_bytes()); a
device's allocations hold its attention blocks' rows, counted and
indexed among those (ctx_kv_alloc()), and none at all for a
device whose layers are all linear |
the layer fit on a device (fit_layers()) |
a layer without attention costs its weights, no cache: more layers fit |
| the paged cache | a block of the pool is the attention blocks' rows
(oklm_kv_block_bytes()): a budget buys four times the
blocks for the 9B; the copy of a block (copy on write), the block tables
sent to a device and the largest allocation a pool may take count those
layers only |
| the copy of a sequence, the swap | the attention blocks' rows (a recurrent context still refuses the swap) |
the mirrors of --gpu-stream |
one per CPU attention block |
| the file | unchanged: a KV file never held rows for these blocks
(layer_rows() in src/model/kv_file.c), so a
file written before loads as it is |
| the plan | context.kv and device_kv count the
attention blocks; each layer's kv_bytes is its own, 0 for a
linear block (memory-plan.md); the text says
8 of 32 layers x 32.0 MiB |
Nothing computed changes: the rows that are gone were never written
nor read. Measured on the CPU with oklm run (24 tokens,
greedy, eight threads), the peak resident size and the cache
oklm plan gives, before and after:
| Model | Context | Cache before | Cache now | Peak RSS before | Peak RSS now |
|---|---|---|---|---|---|
| Qwen3.5-9B (Q4_K_M), 8 of 32 blocks attend | 8192 | 1.00 GiB | 256.0 MiB | 12.66 GiB | 11.90 GiB |
| 32768 | 4.00 GiB | 1.00 GiB | 15.66 GiB | 12.66 GiB | |
| Cyber-Prime 1.1 2.6B (Q4_K_M), 8 of 30 | 8192 | 480.0 MiB | 128.0 MiB | 4.25 GiB | 3.90 GiB |
| 32768 | 1.88 GiB | 512.0 MiB | 5.65 GiB | 4.28 GiB | |
| LFM2-350M (Q4_K_M), 6 of 16 | 8192 | 256.0 MiB | 96.0 MiB | 943.7 MiB | 784.4 MiB |
| 32768 | 1.00 GiB | 384.0 MiB | 1.67 GiB | 1.05 GiB |
The resident size falls by what the plan says, within 4 MiB (every
reserved row was resident). The text, and the --observe
record of every block's norms and of the five likeliest tokens after
each block, are the same bytes before and after on the three models at
both contexts, and on OpenCL for the two LFM2 files (every layer on the
device, whose cache goes from 1.88 GiB to 512 MiB for the 2.6B at 32768
tokens). test_rows_of_attention_blocks in
tests/model/test_mem_plan.c checks, on the five hybrid
fixtures, that the projection and the context agree block by block,
contiguous and paged, and on OpenCL with all, one, two and three layers
on the device, with the CPU's logits.
Not done: the recurrent state is still reserved for every block of
the stack, the attention blocks included (lin_state: one
state per layer and sequence, 2.1 MiB each for the 9B, so about 17 MiB a
sequence for its 8 attention blocks).
Weights in a rotated basis: Ternary Bonsai 2
The Ternary Bonsai 2 files
(prism-ml/Ternary-Bonsai-2-2B-gguf, -4B-gguf,
-9B-gguf) are qwen35 models whose matrices
hold ternary values, one scale per group of weights. Rounding a matrix
to three values costs less when no input carries much more than the
others, so each matrix was rotated before it was rounded, and the file
stores the rotated one: the prism.hadamard.* metadata says
which, and a reader has to undo it, or it computes something else
without any error (src/model/fold.c). The reference is
llama.cpp's load_hparams_hadamard() and
build_hadamard_input() at the commit
tools/gen_llama_ref.sh pins.
Key (prism.hadamard.) |
2B, 4B | 9B | Meaning |
|---|---|---|---|
version, tied_output |
2, true | 1, absent | version 2: the token embeddings are the output head as well, and
there is no output.weight |
block_size |
512 | 4096 | size of the Walsh-Hadamard blocks |
transform, axis |
normalized-sylvester-walsh-hadamard,
input-last-dimension |
the same | the only ones read |
sign_mode, sign_widths,
sign_values |
explicit; 2048 and 6144 (2B), 2560, 4096 and 9216
(4B) |
explicit; 4096 and 12288 |
one vector of +1 / -1 per input width, the values of all of them one after the other |
weight_names |
150 (2B), 200 (4B) | 201 | the folded matrices: attn_qkv, attn_gate,
ssm_out, attn_q, attn_k,
attn_v, attn_output, ffn_gate,
ffn_up, ffn_down of every block, and
output.weight in the 9B |
inverse_weight_names |
token_embd.weight |
the same | tables read by row whose rows are stored rotated |
gdn_v_grouped |
absent (2B), true (4B) | true | the input of ssm_out was folded with its value heads
grouped by key head |
With H the orthonormal block Walsh-Hadamard matrix (the Sylvester matrix of the block size over its square root, one block after the other along the input) and D the diagonal matrix of the signs of an input width:
- A folded matrix is
W D H.H H = IandD D = I, soW x = (W D H) (H (D x)): the product readsH (D x)in place ofx. OKLM computes it per token into a buffer of the context (the signs, then a fast transform,width x log2(block)additions) and gives the matrix product that buffer. The matrices that are not folded (ssm_alpha,ssm_beta, the conv, the norms: F32 in these files) readxitself. ssm_outwithgdn_v_grouped: the recurrence gives its value heads in the orderr * G + k(key head k, repeat r); the fold was done on the orderk * (H / G) + r. The heads are put in that order before the signs and the transform. The 2B has as many value heads as key heads and needs none of it.- The token embeddings store
z = H (D e)per row; a looked-up row is turned back,e = D (H z), before anything reads it (the embeddings an image's projector gives are not rotated and are left alone). - A tied head (version 2) is the table of the rotated
rows used as a folded matrix:
Z (H (D x)) = E x.
What OKLM refuses at load rather than misread: another version, transform, axis or sign mode, a block size that is not a power of two, a listed matrix that the forward pass would not give the transformed input (the experts' tensors, anything outside the list above), a width without its sign vector, a table other than the token embeddings to rotate back, and the metadata on an architecture other than the dense Llama-family blocks.
Two fixtures hold the algebra
(tools/gen_model_fixtures.py):
tiny-qwen35-fold is tiny-qwen35 with every
matrix folded (block 16, grouped value heads) and gives the logits of
tiny-qwen35's reference within 1.9e-4 (llama.cpp itself
gives the two files the same logits within 4.3e-5);
tiny-qwen35-fold-tied is the version 2 form, within 2.9e-4
of llama.cpp on it.
The three packings. Each repository holds the same
weights in three tensor types: Q2_0 (ggml's type 42: two
bits per weight, a scale per 64), and two types of PrismML's llama.cpp
fork alone, PQ2_0 (142: the same two bits, a scale per 128)
and PTQ1_0 (143: five base-3 digits per byte as in
TQ1_0, a scale per 128; 1.75 bits per weight). OKLM reads
the three (formats/quant.md);
the decoded tensors of the Q2_0 and PQ2_0 files of the 2B are equal
value for value. The figures against llama.cpp (the pinned build for
Q2_0, the fork at commit 6447906b for the two others) are
in compatibility.md.
Folded weights on a device
A context of these models runs on OpenCL and on Vulkan like the plain
qwen35 ones (on a device): every
block there or the last ones, a paged cache, several sequences. What it
took:
- The rotation as one device op.
oklm_hadamard_fold()isOKLM_OP_HADAMARDwith a sign per element, before the rotation (a product's input,H (D x)) or after it (an embedding row turned back,D (H z)), a row's value heads read in the order they were folded in, and groups up to 4096 (256 before, the quantized caches' rotation). OpenCL'shadamard_foldkernel and Vulkan'shadamard_foldshader compute it in one launch, a work-group per block with the block in local memory, the butterflies in the CPU's order; the plainhadamardkernel and shader are as they were. Against the CPU: 1e-6 absolute plus 1e-5 relative (test_opencl,test_vulkan_ssm, test_hadamard_fold). The CPU keeps its own loop infold.cfor host layers; its results are unchanged. - The signs on each device. The sign vector of each input width is copied with the weights to every device holding blocks (a few KiB, counted with the head's bytes in the memory plan).
- Which weight is folded. The list is kept by the weights' addresses in the mapped file; a device copy, or a streaming slot's, is recognized by its header's place among the context's weights, and the joined header a device makes of query, key and value (or gate and up) is read as its first part. A file that folded those parts unlike, or a block above 4096, is given the CPU with a notice; no published file is so.
- The two types.
PQ2_0andPTQ1_0are decoded 32 weights at a time in every matrix-product family of both backends, asQ2_0andTQ1_0with a scale per 128 (test_opencltest_matmul,test_vulkan_matmul,test_vulkan_gemm).
A device multiplies the ternary weights by the activations as they are; the CPU, as llama.cpp, rounds them to 8 bits first. So a device gives the three packings of a model the same logits to the bit, and stands as far from the CPU as the CPU from llama.cpp.
Measured
(tests/model/test_bonsai_public.c, RX 7900 XT, 8 CPU
threads; 5 prompts and 32 steps on llama.cpp's tokens, 160 choices; the
speed is that of those decode steps, one token a pass):
| File | Backend | Choices llama.cpp's | Logit difference from llama.cpp: largest (mean) | Against the CPU: choices, largest (mean) | Tokens a second (CPU) |
|---|---|---|---|---|---|
2B Q2_0 |
OpenCL | 160 / 160 | 0.21 (0.012) | 160 / 160, 0.48 (0.012) | 121 (27.6) |
2B Q2_0 |
Vulkan | 160 / 160 | 0.21 (0.012) | 160 / 160, 0.48 (0.012) | 140 (28.2) |
2B PQ2_0 |
OpenCL | 159 / 160 | 0.25 (0.012) | 160 / 160, 0.22 (0.012) | 123 (29.7) |
2B PQ2_0 |
Vulkan | 159 / 160 | 0.25 (0.012) | 160 / 160, 0.22 (0.012) | 140 (30.1) |
2B PTQ1_0 |
OpenCL | 159 / 160 | 0.21 (0.012) | 160 / 160, 0.22 (0.012) | 49 (15.2) |
2B PTQ1_0 |
Vulkan | 159 / 160 | 0.21 (0.012) | 160 / 160, 0.22 (0.012) | 58 (15.2) |
4B PQ2_0 |
OpenCL | 160 / 160 | 0.65 (0.012) | 159 / 160, 0.17 (0.011) | 77 (16.5) |
4B PQ2_0 |
Vulkan | 160 / 160 | 0.65 (0.012) | 159 / 160, 0.17 (0.011) | 82 (16.5) |
The 100 greedy tokens after a chat turn and after plain text are
llama.cpp's on each file and backend (the 2B PTQ1_0 file's
chat turn parts at a tie of 0.001 for llama.cpp, 99 of its 100 choices
equal, as on the CPU). The 4B folds its ssm_out on grouped
value heads: the reordering runs in the device op there. With a paged
cache, and with the first half of the blocks left on the CPU, two
sequences decoded together give llama.cpp's 16 first tokens on every
file and backend. The fixtures (test_model, test_fold) run
check_qwen35_of()'s scenes on OpenCL and on each Vulkan GPU (the RX 7900
XT and the UHD 750): within 4e-6 of the CPU.
Not done. On a device: the 9B (block 4096: the op is
checked at that size, the file was not run), the 4B and 9B
Q2_0 and PTQ1_0 files, a published file on the
Intel UHD 750 (the fixtures and the op tests ran there on Vulkan), the
multi-token prediction block, and the speed of PTQ1_0 (five
base-3 digits a byte are decoded by multiplications, about 2.5 times
slower than the two-bit types on the same device). The multi-token
prediction files (...-PQ2_0-MTP-Q8_0.gguf: the
PQ2_0 model with a plain Q8_0 block after it, which reads
the rotated embeddings and the tied head as the main stack does) beyond
runs of the 4B's (oklm run --mtp: 64 of 156 drafted tokens
accepted, the text of the plain greedy run, at 0.6 to 1.1 times the
plain run's speed on a loaded machine now that a rejected draft starts
again from a kept state, against a seventh of it before: kept states), not compared with llama.cpp's
draft pass; a LoRA adapter on a folded file
(oklm lora-merge refuses one: the delta is in the plain
basis and the weights are rotated; an adapter loaded beside the model is
given the plain input, as llama.cpp's is, which was not tried); the
vision projector.
The dflash draft
model (audit)
The 4B and 9B repositories also hold ...-DFlash2-*.gguf
(general.architecture = dflash), which llama.cpp uses with
--spec-type draft-dflash -md FILE. It is not a small
qwen35, and oklm run --draft cannot use
it:
- What it is: 6 blocks of the target's width (4096
for the 9B) with full attention (32 heads of 128 over 8 key/value heads,
Q/K norms, a sliding window of 4096 on five blocks of six, YaRN-scaled
rotation), a grouped conv of 2 taps before and after the attention and
the FFN of each block (
attn_conv_*,ffn_conv_*), dense SwiGLU. It has no token embeddings and no output head of its own. - Its input: the target's hidden states.
dflash.target_layersnames eight of the target's blocks (2, 6, 10, ..., 30); their inputs for every token the target evaluated are concatenated (8 x 4096), projected byfcto 4096 and written into the draft's key/value cache. The draft then reads a block ofblock_size= 16 positions, the last accepted token followed by 15 mask tokens, with attention that is not causal inside the block, and fills the 15 in one pass. - Its output (DFlash2): no logits. A selector
(
selector_hiddento a rank of 256,selector_predecessorandselector_successorof 256 x vocabulary) gives each position 16 candidates and scores pairs of neighbours; the draft is a greedy walk through that lattice, which the target then verifies as any draft. - What running it would take: the hidden states of
chosen blocks kept for every token of the target (the activation capture
of the roadmap's T10 is the same hook); the
dflashblocks (the conv modules, block attention over injected keys and values); the lattice decoding; a draft source insrc/model/speculative.cthat drafts a block from the target's states instead of asking a second causal model for tokens (the verification is the existing one). On a hybrid target each rejected draft is a cut of the sequence, paid from the state kept before it (kept states). The model card itself recommends the multi-token prediction file over it, and plain decoding for the 2B.
The expert variant:
qwen35moe
Qwen3.6-35B-A3B and its tunes
(general.architecture = qwen35moe): the blocks above with a
routed feed-forward in place of the dense one. Audited against
llama.cpp's src/models/qwen35moe.cpp at the pinned commit,
which differs from its qwen35.cpp in the feed-forward
only.
| Qwen3.6-35B-A3B | |
|---|---|
| blocks, width | 40 (30 linear, 10 full attention), 2048 |
| full attention | 16 heads of 256 over 2 key/value heads |
| linear blocks | 16 key heads of 128 repeated over 32 value heads |
expert_count, expert_used_count |
256, 8 |
expert_feed_forward_length,
expert_shared_feed_forward_length |
512, 512; no feed_forward_length key |
| router | ffn_gate_inp [2048 -> 256], F32; softmax over the
256, the 8 largest kept and their weights divided by their sum |
| experts | ffn_gate_exps, ffn_up_exps [2048 ->
512] x 256 and ffn_down_exps [512 -> 2048] x 256,
SwiGLU, as separate tensors (llama.cpp also accepts gate and up joined
in one ffn_gate_up_exps; none of the files looked at has
it, and OKLM does not read it) |
| shared expert | ffn_gate_shexp, ffn_up_shexp,
ffn_down_shexp (512 wide), its output times the sigmoid of
ffn_gate_inp_shexp [2048 -> 1], added to the routed
sum |
Against qwen3moe: the routing is the same (softmax,
renormalized, no scale, no bias), and there is a shared expert under a
sigmoid gate, which qwen3moe does not have and
qwen2moe does. OKLM runs it on the code of those two; what
the name needed was the renormalization chosen for it, and a file
without feed_forward_length accepted when it gives the
experts' size.
The fixture
tiny-qwen35moe(6 experts, 3 a token, a shared expert, nofeed_forward_length): every logit within 2.9e-4 of llama.cpp's, 36 / 36 greedy choices (test_model, test_qwen35moe); prompt batches through the indexed expert products and the expert cache (one slot to all, the loaders, the pack) give the same logits, as on the other expert fixtures.The published headers (the first 24 MB of each file, the rest left on the server):
unsloth/Qwen3.6-35B-A3B-GGUFUD-Q4_K_M,UraionLabs/Uraion-Cyber-V1.0-GGUFQ4_K_M,IMPERUM/Imperum-CybersecurityLLM-v1.0-GGUFQ4_K_M and the five shards ofahmedandaloes/CyberStrike-OffSec-35B-GGUFQ4_K_M:oklm infosays loadable, every tensor binds with the expected shape (733 tensors; 753 for CyberStrike, which carries a multi-token prediction block), andoklm plangives 41.6 GiB resident at 8192 tokens for the Q4_K_M tunes (the 256 experts of each block repacked for the CPU on top of their mapping), 18.2 GiB of it the routed experts, which can be streamed from disk instead.A published file:
unsloth/Qwen3.6-35B-A3B-GGUF,Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf(13.2 GB, SHA-2569c964e65...07b88fbe; Q6_K, IQ3_XXS, IQ2_S and IQ4_XS tensors), the smallest of the base model's files that left the disk its margin, loaded, compared and removed. Against llama.cpp at the pinned commit (test_infer_public): 5 / 5 prompts tokenized alike, 158 / 160 greedy choices on its tokens, logits within 5.14, four of the five continuations identical and the fifth parting where llama.cpp's two logits are 0.08 apart.oklm plangives 31.5 GiB at 8192 tokens; decoding at 11.7 tokens a second on 8 threads of a busy machine (llama.cpp recorded its 160 tokens in 34 s); with--expert-cache 3the same text at 6.7;oklm chatwith the file's template thinks, then answers.The tunes' files, each 21.2 GB at Q4_K_M, loaded, compared with llama.cpp at the pinned commit and removed on 2026-11-13, one at a time, with no change to the runtime (
test_infer_publicwhen a file is present in its folder undermodels/):file (SHA-256) prompt ids choices on llama.cpp's tokens max logit difference identical continuations llama.cpp margin where they part 100 tokens after a chat turn / plain text Uraion-Cyber-V1.0-Q4_K_M.gguf(47375dc8...8f9b8a88)5 / 5 159 / 160 4.40 4 / 5 0.38 100 / 100, 100 / 100 Imperum-CybersecurityLLM-v1.0.Q4_K_M.gguf(a8d54f31...a0844569)5 / 5 154 / 160 2.59 2 / 5 0.26 97 / 100, 95 / 100 CyberStrike-OffSec-35B-Q4_K_M-0000N-of-00005.gguf(first shard4637db35...bd89e1be)5 / 5 154 / 160 2.30 1 / 5 0.14 97 / 100, 98 / 100 Imperum and CyberStrike agree with llama.cpp on fewer choices than Uraion and the base while their logits are closer to its own than any other file's: their distributions are flatter (llama.cpp's two best logits 0.01 to 0.26 apart at the steps that part), and a rounding decides more of their choices. Decoding at 12.3 to 12.5 tokens a second on 6 threads (a load of 5 to 9; Uraion 6.4 under a load of 15).
oklm chatwith each file's own template (Uraion and Imperum open a thought, CyberStrike carries Qwen2.5's template with its default system turn and answers at once) gives the start of llama.cpp's greedy reply to the same rendered prompt: 57, 38 and 10 tokens before a parting.CyberStrike's multi-token prediction block (a full-attention block with the routed feed-forward, 20 tensors more than the base) runs as the draft of
oklm run --mtp: over 100 tokens after a plain prompt, 195 drafts proposed and 64 accepted (32.8 %), the text the same as without it, at 8.8 tokens a second against 12.5 (a rejected draft costs a hybrid model its prefix, as on the 27B).The devices: done (on a device): the fixture on OpenCL and Vulkan with the experts on the device, on the CPU (
n_cpu_moe), in the expert cache and in the device tier; the UD-IQ3_XXS file whole on the RX 7900 XT.Left: the base model above a 3-bit file, the block's drafts against llama.cpp's own draft pass,
ffn_gate_up_exps; on a device, a published file larger than the device (the experts' tier and the expert cache under a hybrid stack, checked on the fixture only), the tunes' Q4_K_M files and the multi-token prediction block.