OKLMdocumentation v1.0.3
On this page

Hybrid linear-attention layers (Gated DeltaNet)

How OKLM gains the linear-attention layers that newer bases (Qwen3-Next, and the roadmap's "Next" item 18 step 5) mix with full attention, so the inference engine runs them and the trainer trains them. Inference comes first; the trainer follows (its kernels check against the inference engine).

Status: proposed (2026-11-05). No code yet; this scopes the work.

The architecture

A hybrid model is a stack where most blocks are Gated DeltaNet (linear attention with a recurrent state) and a minority are the full attention OKLM already runs (qwen3-style, with its KV cache). Qwen3-Next is the reference shape: a repeating pattern (e.g. 3 linear : 1 full), the same RMSNorm, SwiGLU FFN (often MoE) and residual stream as the full layers, so only the token-mixing differs.

A Gated DeltaNet layer, per head, mixes tokens through a state matrix S_t (d_v x d_k) updated by the gated delta rule (Yang et al., Gated Delta Networks, 2024):

  • Project the input to q_t, k_t (dim d_k) and v_t (dim d_v) per head, a per-head decay a_t in (0,1) (from a sigmoid of a projection, often raised to a learned power) and a write strength b_t in (0,1).
  • A short causal depthwise convolution (kernel ~4) over q, k, v along time, then L2-normalize q_t and k_t per head.
  • The recurrence: S_t = a_t * S_{t-1} + b_t * (v_t - a_t * (S_{t-1} k_t)) k_t^T (equivalently S_t = S_{t-1} (a_t I - b_t k_t k_t^T) + b_t v_t k_t^T): the state decays by a_t, then takes a rank-1 step that replaces the part of the prediction along k_t with v_t.
  • Output o_t = S_t q_t, then a per-head output gate (a sigmoid/SiLU of a projection, elementwise) and an RMSNorm, then the output projection back to the residual width.

The exact projections, the gate and the power on a_t are read from the model's config; the design confirms them against Qwen3-Next's reference (and llama.cpp if it runs the family) before the kernels are written.

Two forms that must agree

The recurrence is sequential, but training and prefill run a whole sequence at once. As for full attention, OKLM keeps two forms and tests that they give the same result:

  • Recurrent (decode, one token): carry S across steps; O(d_k d_v) state per head, O(d_k d_v) work per token. This is the state the inference cache holds for a linear layer, in place of a KV cache.
  • Chunked scan (prefill, training): split the sequence into chunks; within a chunk compute the contributions in parallel (matrix products over the chunk, like attention), carry S between chunks. This is the standard chunk-parallel DeltaNet form; it is what the GPU runs and what the trainer differentiates. Its result equals the recurrent form to float rounding — the first self-consistency test.

The GGUF convention (from gguf-py / llama.cpp)

The reference names, so OKLM can eventually load a real Qwen3-Next GGUF, not only the synthetic fixture:

  • Arch: qwen3next. Hparams: qwen3next.ssm.{conv_kernel, inner_size, state_size, time_step_rank, group_count} beside the usual qwen3next.{embedding_length, block_count, attention.head_count, feed_forward_length, ...}. Which blocks are full attention vs linear is read from the tensors present (a block with ssm_* is linear; one with attn_q is full), as llama.cpp does.
  • Linear (Gated DeltaNet) block tensors: blk.N.ssm_in (the input projection to q, k, v and the output gate z), blk.N.ssm_conv1d (the short causal conv over q/k/v), blk.N.ssm_dt (the time-step / strength projection), blk.N.ssm_a (the decay parameter A), blk.N.ssm_ba (Qwen3-Next's packed beta/alpha gates), blk.N.ssm_norm (the gated output RMSNorm), blk.N.ssm_out (the output projection). The FFN is the usual (MoE) set.
  • Full-attention block tensors: the usual attn_norm, attn_q, attn_k, attn_v, attn_output, attn_q_norm, attn_k_norm, plus blk.N.attn_gate (Qwen3-Next gates the attention output too).

The exact split of ssm_in into q/k/v/z, and how ssm_dt, ssm_a and ssm_ba yield the per-step decay a_t and strength b_t for oklm_gated_delta_*, is the parity work of integration step 3 (checked against llama.cpp); the synthetic fixture fixes a decomposition OKLM controls so the engine is tested before real weights.

OKLM mapping

  • Hparams (include/oklm/io/hf.h): a per-layer kind (linear or attention) — a bit pattern or a count, as swa_pattern already marks sliding layers; the linear hparams n_linear_head, head_dim_k_lin, head_dim_v_lin, conv_kernel, and the gate/decay form. The converter (src/io/convert.c, hf_config.c) reads them from config.json.
  • Weights (src/model/model.c bind): per linear layer, the input projections (q,k,v, the decay a and strength b), the short-conv weights (depthwise, conv_kernel x channels), the output gate and its norm, and the output projection. Named under blk.N.* as the GGUF writer lays them out.
  • Forward (src/model/arch_llama.c lin_block): dispatch per layer on its kind. Two ops in include/oklm/core/ops.h, OKLM_OP_SSM_CONV and OKLM_OP_GATED_DELTA (the L2 norms folded in), on the CPU (src/core/cpu/ssm_cpu.c) and OpenCL (kernels.cl), in the recurrent form; the gated output norm is the existing RMS norm and SwiGLU.
  • State cache (src/model/model.c, kv_pool.c): a linear layer holds a fixed-size recurrent state per sequence (n_linear_head * d_k * d_v floats), not a growing KV cache. The context allocates it beside the KV blocks of the full layers; a sequence's state is reset, copied and swapped with the sequence (truncate drops it, seq_copy duplicates it). This is the largest inference change — the cache is no longer KV-only.
  • Trainer (src/train/layers_train.c, train_kernels.cl): the chunked forward (kept activations for the backward) and its gradient (through the scan, the gates, the conv, the norms), on the CPU and OpenCL, as the other blocks. LoRA targets gain the linear layer's projections.

Order and tests

Inference first (the trainer's kernels check against it):

  1. CPU recurrent forward of one Gated DeltaNet head (done, oklm_gated_delta_recurrent): checked by the delta rule's properties and by processing a sequence in one call versus in pieces.
  2. CPU chunked forward (done, oklm_gated_delta_chunked): checked equal to the recurrent form for every chunk size and decay regime (the self-consistency test). The short causal conv and the L2 norm are done too (oklm_gated_delta_short_conv, oklm_gated_delta_l2norm), each with its test.
  3. Model integration: a tiny synthetic hybrid model (a qwen3next arch in the test fixtures, a couple of linear layers and one full), greedy output checked against the recurrent reference; then a real Qwen3-Next checked against llama.cpp if it runs the family, else against the recurrent form over the real weights.
  4. OpenCL forward, checked equal to the CPU (as every other op). Done: the block runs where its layer runs, on two ops, OKLM_OP_SSM_CONV (the short conv with its ring) and OKLM_OP_GATED_DELTA (the L2 norms, the sigmoids and the recurrence), between the existing products, RMS norm and SwiGLU; on OpenCL the states of the device layers stay in device memory (read back only for oklm_kv_save). Both use the recurrent form, a token at a time per head: the chunked form is not on the device. test_opencl checks the ops against the CPU at 128-value heads, test_model the tiny model's OpenCL prefill and token-by-token decode against the CPU, test_kv_file a device state through a KV file.
  5. Trainer forward + backward, the gradient checked by finite differences (test_train.c, as every block) and the forward checked equal to the inference engine's. On the device (done 2026-11-11): the tr_gd_* kernels of train_kernels.cl, the recurrence's backward from states kept every 16 tokens; test_train_cl checks the gradients against the CPU trainer over 7 and 37 tokens (training.md).

Risks

Risk Signal Mitigation
The exact recurrence/gate differs from Qwen3-Next output parts from the reference confirm the math against the reference (and llama.cpp) in step 1 before the kernels
State cache reshapes the inference cache the paged KV pool assumes KV-only the linear state is a separate fixed allocation per sequence, not paged; the KV pool is untouched for full layers
Chunked ≠ recurrent self-consistency test fails the chunk form is derived from the recurrence and tested against it before integration
Backward through the scan is subtle finite-difference check fails build the CPU forward and its finite-diff'd backward before the device kernel
Scope a large multi-part change land it in the order above, each step tested, behind a new arch so nothing else moves

The published layout: qwen35

The sections above describe qwen3next as OKLM first built it, against a fixture of its own design. The Qwen3.5 / 3.8 files (general.architecture = qwen35; Qwen3.5-9B was the file audited: 427 tensors, 32 blocks, width 4096; Qwen3.8-27B, 64 blocks at width 5120 with 48 value heads, was run after it) are laid out differently, and OKLM reads them as llama.cpp does. The reference is llama.cpp at the commit tools/gen_llama_ref.sh pins (8642541b: its src/models/qwen35.cpp and delta-net-base.cpp). A tiny model of this layout (tests/fixtures/model/tiny-qwen35.gguf) agrees with it to 2e-4 on every logit; the 9B file's figures are in compatibility.md.

What the file says

Key (qwen35.) Qwen3.5-9B Meaning
block_count, embedding_length, feed_forward_length 32, 4096, 12288 the stack; the FFN is dense SwiGLU
attention.head_count, head_count_kv, key_length 16, 4, 256 the full-attention blocks: heads of 256, wider than width / heads
rope.dimension_count, rope.freq_base, rope.dimension_sections 64, 1e7, [11, 11, 10, 0] 64 of a head's 256 dimensions are rotated, pairs (i, i + 32); the sections split them between time, height and width for image patches, and a text token has the same position in all three, which is the plain rotation
ssm.state_size, ssm.group_count 128, 16 a linear block's key heads: 16 of 128
ssm.time_step_rank, ssm.inner_size 32, 4096 its value heads: 32, of 4096 / 32 = 128
ssm.conv_kernel 4 taps of the short causal conv
full_attention_interval 4 blocks 3, 7, ... are full attention, the others linear (attention.recurrent_layers, when a file has it, lists them instead)

OKLM tells the two kinds of block apart by their tensors (a block with ssm_conv1d is linear), as it does for the rest of a model's layout.

A linear block

With xn the block's normalized input (attn_norm, RMS, eps from attention.layer_norm_rms_epsilon), G = 16 key heads, H = 32 value heads, d = 128:

  1. attn_qkv [4096 -> 8192] gives q (G x d) | k (G x d) | v (H x d); attn_gate [4096 -> 4096] the output gate z; ssm_beta and ssm_alpha [4096 -> 32] a write strength and a decay per value head.
  2. ssm_conv1d [4 taps x 8192 channels]: each channel of q | k | v is convolved with its own 4 taps over the last 4 tokens (zeros before the sequence's start), then SiLU. The file stores it taps first; OKLM keeps a copy channels first, the layout of its kernel.
  3. q and k are L2-normalized per head (x / sqrt(sum x^2 + eps)), q is divided by sqrt(d).
  4. Per value head h and token: b = sigmoid(beta), a = exp(ssm_a[h] * softplus(alpha + ssm_dt.bias[h])) (ssm_a holds -exp(A_log), so a is in (0, 1)); with the head's state S (d x d) and the key head h mod G: S = a S + b (v - a S k) k^T, o = S q.
  5. o goes through an RMS norm per value head (ssm_norm, 128 weights), is multiplied by SiLU(z), and ssm_out [4096 -> 4096] projects it back onto the residual stream.

Then post_attention_norm and the FFN, as in the other blocks: post_attention_norm is the norm before the FFN, not a norm of the attention's output.

A full-attention block

attn_q [4096 -> 8192] gives each of the 16 heads 512 values: its query (256) then an output gate (256). Queries and keys go through an RMS norm per head (attn_q_norm, attn_k_norm) and the rotation above; after the attention (4 key/value heads shared by 16 query heads, scale 1/16) each head's output is multiplied by the sigmoid of its gate, then attn_output.

Where the test layout differed

qwen3next test layout published qwen35
projections one ssm_in to q, k, v and z attn_qkv and attn_gate
decay sigmoid(ssm_a[h]), fixed per head per token, from ssm_alpha, ssm_dt.bias and ssm_a
write strength ssm_dt ssm_beta
key head of value head h h / (H / G) h mod G
query as normalized divided by sqrt(d)
conv weights channels first taps first
full attention plain queries queries joined with an output gate
norm before the FFN ffn_norm post_attention_norm
L2 and gated-norm eps 1e-6 the model's RMS eps

The recurrence itself, the conv with its window and the gated norm are the same operations: the published layout runs on OKLM_OP_SSM_CONV and on OKLM_OP_GATED_DELTA with a decay per token (oklm_gated_delta_tok()), on the CPU, on OpenCL and on Vulkan (on a device). qwen3next stays as it was, a layout the trainer and the device kernels are tested on; it is not the layout of the published Qwen3-Next files (their ssm_in, ssm_ba and expert FFN are not read yet).

The state a sequence keeps

Per linear block and sequence: the H matrices S (32 x 128 x 128 floats) and the conv's window, the last 3 inputs of its 8192 channels: 548 864 floats, 2.1 MiB; 24 linear blocks make 50 MiB a sequence for Qwen3.5-9B, whatever the context length. The 8 full-attention blocks keep keys and values as any model does. What follows from a state that only moves forward:

  • Several sequences: one state each; a forward pass reads and writes those of the sequences in the batch, and a sequence that starts again at position 0 starts from zeros.
  • A cut (oklm_seq_truncate() to a position in the sequence's middle: a chat whose next prompt shares only a prefix, a rejected draft): the state cannot go back, so the context keeps each sequence's token ids and, before the sequence's next use, brings the state to the cut from the nearest state it kept (below), or from the sequence's start when it kept none. The caller sees the cut honoured. A sequence that holds embedding rows (an image) cannot be computed again: the cut is refused and the sequence takes nothing more until it is emptied.
  • A copy (oklm_seq_copy()): the whole sequence is copied with its state; a part of it takes the kept state of its tokens, which the source left.
  • The paged cache: the attention blocks' rows are paged as usual, the states are not (they have one size). The prefix cache lends a sequence the cached rows of another's prefix; the sequence takes the nearest kept state of those tokens and runs the tokens after it over the lent rows. Without kept states it is off, and swapping a sequence out is off in both cases.
  • A file (oklm_kv_save() / oklm_kv_load()): the states are saved after the rows, for the whole sequence only.
  • A context of another size: a new context starts empty; a sequence moves to it through a file or by evaluating its tokens.

Kept states

Without them a cut costs the evaluation of everything before it: a rejected draft, a chat turn asked again, a request sharing a system prompt with the previous one each paid the whole prefix. A context of these models now keeps what gives a past state back (src/model/lin_keep.c; oklm_context_params::state_keep and state_every, the environment's OKLM_STATE_KEEP, 0 for none; oklm_state_keep_stats() counts what they did). The states are kept in host memory wherever the blocks run; the journal is written only when every linear block's state is in host memory (on a device, a cut is a forward pass from the nearest kept state).

What is kept.

what when size (Qwen3.5-9B)
kept state every linear block's state after a prefix, in a store the context's sequences share (8 slots, allocated as they fill) eight tokens before the end of a prompt (a batch of several tokens that keeps the last one's logits: the linear blocks copy their state as they pass it), with the journal of those eight; where a prompt begins after generated tokens; every 32 tokens while a sequence grows (two per sequence, the older one replaced); where a cut had to be paid by a forward pass 50 MiB each, 6 MiB more with a prompt's journal
journal per sequence, what each linear block's recurrence read at its last 96 positions: the q, k, v rows before the convolution, the write strengths and decays every forward pass 0.76 MiB a position, 73 MiB
state before a draft per sequence, its state before the last batch that kept every logit, which is how a draft is verified before such a batch 50 MiB

A prompt's state is kept eight tokens before its end, with the journal of those eight, because its sequence is cut among them when the prompt comes again: the callers (the chat's /retry, the server given the same request) keep the common prefix less one token, which they evaluate again for its logits; the next turn of a conversation shares the prompt up to the few tokens a chat template ends it with and does not write in the history (22 of a first prompt's 24 tokens with the Ternary Bonsai 2 template); a model whose reasoning is dropped from the history shares all of it. Each of those cuts is then paid by the recurrence over at most eight positions, whatever the sequence generated since. A turn taken back and asked otherwise (/rewind) is cut a few tokens after the place its prompt began, where a state is kept as well: those few tokens go through a forward pass, once.

A kept state is found by what it is the state of: two 64-bit hashes of the tokens before its position and the adapter, compared with those of the sequence asking. The states are a function of those tokens alone, so a sequence takes one another sequence computed, in the same call or long after that sequence was emptied. When the store is full the least recently used state goes (taking one, or finding it already there, counts as a use).

A cut to position n (model_lin_keep_settle()), at the sequence's next use:

  1. the kept state with the largest position at or before n is copied back (the state before the last draft when it is nearer; zeros when there is none);
  2. if the sequence's journal, or the one kept with that state, holds every position between it and n, the convolution and the recurrence of each linear block run again over the journal, on the same two operations as the forward pass: no projection, no attention, no feed-forward. This is the case of a rejected draft, which costs the tokens accepted before the rejection, and of a cut of up to 32 tokens into what a sequence generated;
  3. otherwise the tokens between go through a forward pass with the cache frozen (model_replay()): the attention blocks read the rows the cut left instead of writing them, so nothing but the states moves, and a paged cache copies no block. The state reached is then kept: a place cut to once (the end of a shared system prompt) is reached for nothing the next time.

Exactness. The journal path gives the bits of a context that never went past the cut: the recurrence reads the values it read the first time, in the same order. The forward path gives what evaluating those tokens in that batch shape gives, as any prompt evaluation does. On the tiny models both are equal to a fresh evaluation, bit for bit, in tests/model/test_state_keep.c (contiguous and paged, F16 and F32, the folded and the expert variants, the test layout), which holds the first to equality and the second to 1e-4.

Kept states on a device

A context whose linear blocks run on OpenCL or Vulkan holds their states in device memory. It keeps states as a CPU context does, in host memory: taking one reads every linear block's state back (50 MiB for the 9B), giving one writes it to the device again. It writes no journal: an entry is what a block read at a position, and reading those rows back from the device at each of the 24 blocks of every decode step would stop the device between blocks, which costs more than what the journal saves there. So on a device:

  • the states are kept at the same places (eight tokens before a prompt's end, read back as the blocks pass it; where a prompt begins after generated tokens; every 32 tokens of a growing sequence; before a draft; where a cut was paid);
  • a cut takes the nearest one and runs the tokens after it through a forward pass with the cache frozen, one batch on the device: at most eight tokens for a prompt asked again or a next turn, fewer than 32 for a cut into the last 32 tokens a sequence generated, the accepted tokens for a rejected draft. journal_tokens of oklm_state_keep_stats() stays 0 and forward_tokens counts them;
  • a context with layers on both sides (--gpu-layers), with its CPU layers streamed to the device for prompts, or spread over several devices does the same: one block without a journal entry leaves the batch without one.

tests/model/test_state_keep.c runs its cuts, its shared states and its random walk on OpenCL and on each Vulkan device as on the CPU, with the forward pass where the CPU has the journal (36 tokens for the 6 cuts of its first scene where the CPU pays 25 and 11 journaled), within 2e-4 of a fresh evaluation (measured: 0 on OpenCL, 4e-5 at most on Vulkan). Keeping states costs a decode step nothing that was measured on Qwen3.5-9B: 83.0 against 83.3 tokens a second on OpenCL and 104.2 against 104.0 on Vulkan with OKLM_STATE_KEEP=0; a prompt of 512 tokens 1 to 2 % (the state read back before its end).

The memory is counted with the recurrent state by oklm plan (every slot filled, a journal and a draft copy per sequence): 640 MiB instead of 67 for Qwen3.5-9B and Ternary Bonsai 2 4B with one sequence, 267 MiB instead of 26 for the 2B, whatever the context length. A context counts the same, and allocates each of them when it is first used (oklm_state_keep_stats(), state_keep.bytes of the server's /api/model).

Measured (Ternary Bonsai 2 4B, ...-PQ2_0-MTP-Q8_0.gguf, one prompt of 13 tokens, greedy, 96 tokens generated, CPU, 4 threads; the text is the plain run's in every case). The machine was shared and loaded (a load average of 14 to 23 on 16 threads), so the rounds were run one after the other, each a plain run then the drafted ones, and differ by more than the runs of a round do (another job started during the fourth round's --mtp run):

oklm run drafts accepted tokens a second, rounds 1 to 4
plain - 3.5, 4.8, 2.0, 3.9
--mtp, kept states 64 of 156 4.0, 4.4, 1.9, 2.2
--mtp --draft-n 1, kept states 44 of 52 4.1, 2.0, 3.4, -
--mtp, OKLM_STATE_KEEP=0 (as before) 64 of 156 0.5 (0.7 on a calmer run)

With kept states the 25 rejected drafts of the --mtp run cost 57 tokens through the journal and none through a forward pass: 1.0 to 1.7 s of state copies and recurrence, and 0.4 s for the copies before each draft, in a run of 44 to 59 s (3 %). Without them the same 25 rejections evaluated the prefix again each time: 132 to 181 s for the 96 tokens. What is left between --mtp and the plain run is the drafting itself: on the CPU a verification batch costs about as much per token as plain decoding, and 92 of the 156 drafted tokens were evaluated for nothing; --mtp ran at 0.6 to 1.1 times the plain speed of its round, which that machine's noise does not separate from 1, and is not shown faster here. Prompt lookup on the 2B (--spec-ngram 4, 15 of 118 proposals accepted, the plain text): 7.7 tokens a second plain, 4.3 with kept states, 0.4 without (13.2, 6.9 and 1.0 on a calmer run, made before the state before a draft was kept apart).

In the chat (oklm chat --stats on Ternary Bonsai 2 2B, which prints what a cut cost): /retry cut the prompt of 20 tokens at 19 and paid 7 tokens through the journal kept with the prompt's state; the next turn shared 18 tokens and paid 6 the same way; /rewind then another question shared 27 of 36 tokens and paid a forward pass over 15, from the first prompt's state. A longer script (replies of 500 tokens, /retry, a second turn, /rewind, a third) took 296 s with kept states and 412 s without, the same text.

Not kept. The state of a sequence holding embedding rows (an image: its positions have no token to hash); a kept state in a KV file (the file stores the state of its end, and loads whole or not at all); states on a device.

The sensitive steps

On a few steps of the recorded prompts of Qwen3.5-9B Q4_K_M, single logits differ by 2 to 13.8 between OKLM and llama.cpp, where every other step is within 0.4 to 0.7; the build for Windows chose another token than llama.cpp on one of them (step 38 of the continuation of "The capital of France is": llama.cpp's two logits 4.59 apart, OKLM's own 0.34 the other way). Measured on 2026-11-13, this is the model's own sensitivity on those steps: llama.cpp moves as far from itself when only its last places change, on the same steps and in the same blocks, and OKLM's gated delta blocks lose no precision. What was measured, on the CPU:

The logits. The 101 steps of that continuation (the model repeats "Paris.\nThe capital of France is"), every run on the recorded run's tokens, against the recorded run (llama.cpp at the pinned commit, F16 keys and values, the prompt in one batch, 4 threads):

run mean |logit difference| largest another choice
llama.cpp, 1 or 6 threads (46 steps) 0 0 (the same bits)
llama.cpp, keys and values as F32 0.186 14.9 (step 17)
llama.cpp, the prompt one token a pass 0.233 13.8 (step 13)
llama.cpp, the prompt in batches of 3 tokens 0.233 13.8
llama.cpp, both 0.231 13.8
llama.cpp, its other attention kernel (flash_attn) 0.174 13.8
llama.cpp, that kernel and the prompt one token a pass 0.210 13.7
OKLM, Linux 0.222 13.8 (step 13)
OKLM, Linux, F32 cache 0.248 14.4 (step 17) step 2, where llama.cpp's two logits are 0.09 apart
OKLM, the build for Windows under Wine 0.217 13.8 step 38 (4.59 apart)
OKLM, the build for Windows, F32 cache 0.221 13.8

Between any two of these runs the mean difference over the 82 ordinary steps is 0.06 to 0.12, two of llama.cpp, two of OKLM or one of each (0.10 to 0.115 for the last). Over the 18 sensitive steps (13, 17, and from 37 on the two that predict " of" and " France" in each repetition of the sentence: 37, 38, 45, 46, ... 93, 94) it is 0.08 to 0.90 between two llama.cpp runs with single logits up to 14.9 apart, 0.31 to 0.64 and 10.1 between two OKLM runs, 0.55 to 0.89 and 14.4 between one of each. The 13.8 of step 13 is the recorded run alone against every other one: the six other llama.cpp runs and the four of OKLM are within 0.9 of each other there. At step 17 the runs fall in three groups, whichever engine they come from: the recorded run; llama.cpp with an F32 cache and OKLM on Linux with an F32 cache, 1.4 apart; the eight others, within 1.2 of each other, 4.2 to 5.0 from the first and 9.7 to 10.7 from the second. As a control, the same changes of llama.cpp on Qwen3-8B Q4_K_M (no linear block; 33 steps of the same prompt) move its logits by 0.14 on average, near what they do here, but by 1.5 at most: the mean is what a change of last places costs a Q4_K_M file of this size in llama.cpp, the steps where a logit moves by 10 are this model's.

The hidden state, block by block. The residual stream after every block and, inside every linear block, the projections, the conv's output, the recurrence's output and state, the gated norm's output, read from llama.cpp through its cb_eval callback and from OKLM through taps put in its forward pass for the measure (not kept in the tree), at each of the first 46 steps. Relative L2 difference of the last token's stream after blocks 0 to 31, at an ordinary step and at three sensitive ones:

Step 12 (ordinary):

0 1 2 4 8 12 16 20 24 28 30 31
OKLM (Linux) against llama.cpp 4e-07 0.002 0.007 0.013 0.045 0.064 0.060 0.046 0.040 0.037 0.044 0.042
OKLM (Windows) against llama.cpp 4e-07 0.002 0.007 0.013 0.047 0.061 0.064 0.052 0.043 0.038 0.041 0.039
OKLM (Windows) against OKLM (Linux) 0 1e-08 8e-08 0.009 0.033 0.044 0.048 0.039 0.035 0.033 0.037 0.039
llama.cpp, prompt a token a pass, against llama.cpp 0 0.001 0.007 0.014 0.038 0.053 0.059 0.047 0.039 0.035 0.036 0.037
llama.cpp, F32 cache, against llama.cpp 0 0 0 0.008 0.034 0.046 0.052 0.048 0.044 0.042 0.050 0.047

Step 13 (13.8 on one logit):

0 1 2 4 8 12 16 20 24 28 30 31
OKLM (Linux) against llama.cpp 2e-07 0.002 0.007 0.012 0.034 0.041 0.043 0.041 0.037 0.044 1.09 1.68
OKLM (Windows) against llama.cpp 2e-07 0.002 0.007 0.012 0.036 0.045 0.048 0.045 0.041 0.078 1.10 1.72
OKLM (Windows) against OKLM (Linux) 2e-08 3e-08 6e-08 0.008 0.025 0.031 0.037 0.039 0.037 0.066 0.058 0.050
llama.cpp, prompt a token a pass, against llama.cpp 0 0.002 0.007 0.013 0.031 0.036 0.045 0.042 0.037 0.055 1.08 1.65
llama.cpp, F32 cache, against llama.cpp 0 0 0 0.007 0.030 0.036 0.044 0.045 0.039 0.11 1.11 1.74

Step 37:

0 1 2 4 8 12 16 20 24 28 30 31
OKLM (Linux) against llama.cpp 2e-07 0.003 0.007 0.012 0.035 0.13 0.30 0.61 0.78 0.78 0.80 0.54
OKLM (Windows) against llama.cpp 2e-07 0.003 0.006 0.013 0.053 0.23 0.47 0.81 0.94 0.91 0.96 0.63
OKLM (Windows) against OKLM (Linux) 9e-09 0.002 0.006 0.011 0.053 0.14 0.25 0.35 0.35 0.33 0.40 0.28
llama.cpp, prompt a token a pass, against llama.cpp 0 0.001 0.005 0.011 0.045 0.085 0.20 0.30 0.38 0.38 0.38 0.27
llama.cpp, F32 cache, against llama.cpp, prompt a token a pass 0 0.001 0.005 0.011 0.047 0.15 0.37 0.67 0.92 0.95 0.96 0.68

Step 38 (where the build for Windows chose another token):

0 1 2 4 8 12 16 20 24 28 30 31
OKLM (Linux) against llama.cpp 4e-07 0.003 0.009 0.018 0.057 0.14 0.31 0.36 0.30 0.24 0.30 0.23
OKLM (Windows) against llama.cpp 4e-07 0.003 0.009 0.020 0.044 0.053 0.15 0.29 0.29 0.25 0.34 0.29
OKLM (Windows) against OKLM (Linux) 0 0.002 0.008 0.013 0.052 0.13 0.39 0.47 0.38 0.30 0.36 0.34
llama.cpp, prompt a token a pass, against llama.cpp 0 0.002 0.008 0.012 0.056 0.086 0.19 0.21 0.17 0.14 0.18 0.18
llama.cpp, F32 cache, against llama.cpp, prompt a token a pass 0 0.002 0.008 0.012 0.046 0.068 0.19 0.20 0.17 0.16 0.24 0.23

What the rows say:

  1. The recurrence is exact. In block 0, whose input is the same in both engines, every quantity of OKLM is llama.cpp's to 1e-7 to 4e-7: the projections, the conv, the recurrence's output, the gated norm, and the state of the 32 heads after 43 tokens (2.8e-7). That is the rounding of single-precision sums taken in another order, not a loss: a recurrence that lost precision would show it here, where nothing else differs.
  2. The first differences are jumps, not growth. Block 0's output is llama.cpp's to 4e-7 at 37 of the 46 steps and 1e-5 to 6e-4 away at the 9 others, nothing between; the Windows build is the Linux build's to 1e-8 until a block where it is 1e-4 to 7e-3 away, at step 0 past block 16, from step 29 on in the first two blocks. Both engines round the activations to 8 bits before a product with quantized weights; one 8-bit step of one activation is 6e-4 to 1e-3 of the vector's norm at block 0's feed-forward, and the jumps there are that size. A last place that tips one rounding becomes a difference of 1e-3, in any engine: llama.cpp given its prompt a token a pass is itself to the bit in block 0 and 1e-3 to 2e-3 away after block 1. The states then carry it to every later step.
  3. From there the model amplifies it, llama.cpp's as OKLM's. At an ordinary step the difference grows from 2e-3 after block 1 to 4e-2 to 6e-2 at blocks 12 to 16 and stays there, the same curve whichever two runs are compared. Inside a linear block the recurrence is not what amplifies: its output differs less than what it reads (at step 38 between two llama.cpp runs, block 12: the projections 0.09 apart, the conv's keys and values 0.20 and 0.16, the recurrence's output 0.066, after the gated norm 0.15, after the feed-forward 0.16 of its output). The projections, the conv over the last four tokens, the gated norm and the feed-forward each widen the gap.
  4. A sensitive step is one where a block switches. At step 13 the runs are 4e-2 to 1e-1 apart until block 29 and 1.1 apart after block 30: one value channel of its head 6 reads -19 to -23 at every step around and, at that step, -3.8 in the recorded run, +8.4, +5.9 in the two other llama.cpp runs, +4.7, +6.7, +6.2 in OKLM's three. After the conv and SiLU that is -0.17 against 1.9 to 3.0, 99.9 % of the head's difference, and the recorded run stands alone on its side of the gate. At steps 37, 38 and 45 the growth is spread from block 8 to block 24, up to 1.0 between two llama.cpp runs.
  5. The choices. Over the 100 steps the gap between llama.cpp's two best tokens moves by up to 2.3 between its own seven runs (step 38: 4.59 to 6.91; step 62: 4.48 to 6.68) and by up to 5.2 in OKLM's four (step 38: -0.34 in the Windows build, 4.36 to 5.59 in the three others; step 61: 15.5 to 17.1 where the recorded run has 11.9 and llama.cpp's others up to 14.1). No llama.cpp run chose another token on any of the 100 steps, so the Windows build's choice at step 38 is not shown to be one llama.cpp could make; it is shown to come from the same amplification, on a step where two llama.cpp runs are 0.2 apart in the stream after block 20 and the Windows and Linux builds 0.47. The chat turn of the same test has no such step (the gap moves by 2.5 at most, in llama.cpp's runs as in OKLM's); there llama.cpp does part from itself, at steps 1, 64, 80 and 87 (its recorded gaps 0.20, 0.05, 0.13 and 0.03; step 64 is where the Windows build parts), and at step 26, where OKLM on Linux parts (0.44 recorded, -0.02 in OKLM), its eight runs have 0.05 to 0.46.

tools/gen_llama_ref.sh records the second llama.cpp run of the table (LLAMA_REF_KV=f32, read by tools/llama_ref.c, as LLAMA_REF_ONE=1 gives the prompt a token a pass), and test_llama_itself() in tests/model/test_qwen35_public.c compares the two recordings and puts OKLM beside them: OKLM may be 1.5 times as far from the recorded run as llama.cpp's own second run is, in the mean and in the largest difference. The bound on a parted choice stays what was measured on each platform (0.6 on Linux, 5.0 for the build for Windows): the measure explains the step, it does not make the choice llama.cpp's.

Not in the 9B file, or not done

  • Multi-token prediction: optional blocks after the stack (blk.N.nextn.*, each a full-attention block as above, counted in block_count and in nextn_predict_layers). The 9B file has none; Qwen3.8-27B has one. OKLM binds it with the gated queries and runs it as the other families' (oklm run --mtp): on the 27B file, 20 of its 45 drafted tokens were accepted and the text is the plain greedy one. It was not compared with llama.cpp's draft pass. Each rejected draft is a cut of the sequence, paid from the state kept before the draft (kept states); the 27B's run was made before those and not timed again.
  • The vision projector (mmproj) is a separate file, not read.
  • Devices: done, on OpenCL and on Vulkan (below).
  • Training: the trainer knows the test layout only.
  • The cache's size: done. A context reserves key and value rows for the attention blocks only (the section "The cache of a hybrid stack", below).
  • The expert variant (qwen35moe): done, on a fixture and on a published file of Qwen3.6-35B-A3B (below).

On a device

A qwen35 or qwen35moe context runs on OpenCL and on Vulkan like the other families: every block on the device, or the last ones (--gpu-layers), the linear blocks' states in device memory. What it took:

  • The decay per token in the gated_delta kernel (OpenCL) and shader (Vulkan), which had the test layout's decay per head: the decays' projections beside the write strengths, their bias and factor per head, exp(factor * softplus(projection + bias)) computed by the rows of a head for each token, the key head h mod G, the query divided by sqrt(dk). One kernel serves both layouts; it is checked against the CPU's in test_opencl and test_vulkan_ssm (outputs and states within 1e-5 absolute plus 1e-4 relative, the tolerance of the decay per head).
  • The gated queries: the joined query and gate projection is one product, split by two strided copies; the sigmoid and the product after attention are the element-wise kernels. A device joins the query, key and value matrices of a block into one when their rows have one length; the gated query matrix has twice the rows the output projection reads and is left out of that (dev_place_layer(), src/model/model.c).
  • Nothing else: the conv, the RMS norms, the rotation, SwiGLU, the routed experts and the matrix products are the kernels the other families run on (ssm_beta and ssm_alpha are Q8_0 in the published files). The token lookup is the CPU's, as for every model.
  • Kept states work with the states on a device (above).

The fixtures (test_model, test_qwen35 and test_qwen35moe; OpenCL on the RX 7900 XT, Vulkan on it and on the Intel UHD 750): llama.cpp's logits within 1.9e-4 (tiny-qwen35) and 2.9e-4 (tiny-qwen35moe), 36 / 36 greedy choices, every block on the device and the first one left on the CPU; a prompt in one batch, in batches of 3 and token by token within 5e-7 of each other; two sequences in one pass; a sequence through a copy, a file and a cut; a paged cache. For tiny-qwen35moe also the routed experts of the first or of every layer on the CPU (n_cpu_moe), a CPU layer streamed to the device for prompts, the expert cache and the experts' device tier (test_n_cpu_moe, test_expert_gpu), as the other expert fixtures.

Qwen3.5-9B Q4_K_M (tests/model/test_qwen35_public.c, every test of the CPU run again on each device, 32 of 32 blocks there, an F16 cache; RX 7900 XT, 2026-11-13):

CPU OpenCL Vulkan
choices on llama.cpp's tokens (5 prompts, 32 steps) 159 / 160 158 / 160 158 / 160
logit difference from llama.cpp, mean and largest 0.116, 13.8 0.122, 13.7 0.122, 13.7
against the CPU on those steps: choices, mean, largest 157 / 160, 0.113, 3.65 159 / 160, 0.119, 3.55
100 greedy tokens, chat turn: equal before parting (llama.cpp's margin there), choices on its tokens 26 (0.44), 97 80 (0.13), 98 26 (0.44), 97
100 greedy tokens, plain text 100 100 100
a prompt of 296 tokens in batches of 64: choices, largest difference 16 / 16, 1.72 16 / 16, 2.37 16 / 16, 2.75
two sequences in one pass: choices 32 / 32 and 32 / 32 32 / 32 and 31 / 32 32 / 32 and 31 / 32
a paged cache, two sequences, 8 steps: choices llama.cpp's 16 / 16 16 / 16
16 of 32 blocks on the CPU, the same 16 / 16 16 / 16
prompt of 512 tokens, tokens a second (bench_llama) 79.3 1839 2300
128 tokens one at a time, tokens a second 5.46 83.0 104.2

A device is as far from the CPU as the CPU is from llama.cpp, and as the CPU is from itself with an F32 cache (0.121): the model's sensitivity to the last places (the sensitive steps), not a kernel's error, which the op tests and the fixtures bound at 1e-4. The state through a file, a copy and a cut gives the same 8 tokens on each device. With the float kernels test_infer_public asks of a device (OKLM_CL_Q8=0, OKLM_VK_Q8=0, OKLM_VK_COOPMAT=0) OpenCL and Vulkan give the same figures as each other: 159 / 160 choices, logits within 14.3, four of five continuations identical. Qwen3.8-27B (16.6 GB) was not run on a device: the same code path, twice the blocks.

Qwen3.6-35B-A3B UD-IQ3_XXS (qwen35moe, 13.2 GB, downloaded for the run and removed; 40 of 40 blocks and the 256 experts of each on the RX 7900 XT, nothing handed back to the CPU but the token lookup):

CPU OpenCL Vulkan
choices on llama.cpp's tokens (5 prompts, 32 steps) 158 / 160 156 / 160 156 / 160
largest logit difference from llama.cpp 5.14 2.74 8.35
llama.cpp's margin where a choice differs, at most 0.08 0.16 0.16
identical continuations 4 / 5 2 / 5 2 / 5
against the CPU on those steps: choices, mean, largest 156 / 160, 0.146, 6.01 155 / 160, 0.159, 7.81
the float kernels: choices, largest difference from llama.cpp 156 / 160, 4.4 156 / 160, 4.4
prompt of 512 tokens, tokens a second (bench_llama) 97.0 1222 1536
128 tokens one at a time, tokens a second 11.65 64.7 108.6

The choices that differ are near ties for llama.cpp (0.16 at most between its two logits); the file's 3-bit and 2-bit tensors leave it flat distributions, as its tunes showed on the CPU. The experts' device tier and the expert cache were run under the hybrid stack on the fixture only: this file fits the device whole.

The cache of a hybrid stack

A linear block keeps a recurrent state and never reads a key or a value row; a context used to reserve them all the same, one cache per block. It now reserves rows for the blocks that attend, and nothing for the others: 8 of the 9B's 32 blocks, so 256 MiB of cache at 8192 tokens instead of 1 GiB, and 1 GiB at 32768 instead of 4. The same rule serves the short convolution blocks of lfm2 and lfm2moe (inference.md), which are bound the same way (a block with an ssm_in projection has no attention: model_layer_cached() in src/model/model_internal.h).

A block without rows has no cache tensor at all (k_cache[l] and v_cache[l] are NULL), and each place that walked the layers skips it:

Where What changes
the allocation (oklm_context_new()) the host arena holds the attention blocks' rows (ctx_kv_data_bytes(), ctx_kv_arena_bytes()); a device's allocations hold its attention blocks' rows, counted and indexed among those (ctx_kv_alloc()), and none at all for a device whose layers are all linear
the layer fit on a device (fit_layers()) a layer without attention costs its weights, no cache: more layers fit
the paged cache a block of the pool is the attention blocks' rows (oklm_kv_block_bytes()): a budget buys four times the blocks for the 9B; the copy of a block (copy on write), the block tables sent to a device and the largest allocation a pool may take count those layers only
the copy of a sequence, the swap the attention blocks' rows (a recurrent context still refuses the swap)
the mirrors of --gpu-stream one per CPU attention block
the file unchanged: a KV file never held rows for these blocks (layer_rows() in src/model/kv_file.c), so a file written before loads as it is
the plan context.kv and device_kv count the attention blocks; each layer's kv_bytes is its own, 0 for a linear block (memory-plan.md); the text says 8 of 32 layers x 32.0 MiB

Nothing computed changes: the rows that are gone were never written nor read. Measured on the CPU with oklm run (24 tokens, greedy, eight threads), the peak resident size and the cache oklm plan gives, before and after:

Model Context Cache before Cache now Peak RSS before Peak RSS now
Qwen3.5-9B (Q4_K_M), 8 of 32 blocks attend 8192 1.00 GiB 256.0 MiB 12.66 GiB 11.90 GiB
32768 4.00 GiB 1.00 GiB 15.66 GiB 12.66 GiB
Cyber-Prime 1.1 2.6B (Q4_K_M), 8 of 30 8192 480.0 MiB 128.0 MiB 4.25 GiB 3.90 GiB
32768 1.88 GiB 512.0 MiB 5.65 GiB 4.28 GiB
LFM2-350M (Q4_K_M), 6 of 16 8192 256.0 MiB 96.0 MiB 943.7 MiB 784.4 MiB
32768 1.00 GiB 384.0 MiB 1.67 GiB 1.05 GiB

The resident size falls by what the plan says, within 4 MiB (every reserved row was resident). The text, and the --observe record of every block's norms and of the five likeliest tokens after each block, are the same bytes before and after on the three models at both contexts, and on OpenCL for the two LFM2 files (every layer on the device, whose cache goes from 1.88 GiB to 512 MiB for the 2.6B at 32768 tokens). test_rows_of_attention_blocks in tests/model/test_mem_plan.c checks, on the five hybrid fixtures, that the projection and the context agree block by block, contiguous and paged, and on OpenCL with all, one, two and three layers on the device, with the CPU's logits.

Not done: the recurrent state is still reserved for every block of the stack, the attention blocks included (lin_state: one state per layer and sequence, 2.1 MiB each for the 9B, so about 17 MiB a sequence for its 8 attention blocks).

Weights in a rotated basis: Ternary Bonsai 2

The Ternary Bonsai 2 files (prism-ml/Ternary-Bonsai-2-2B-gguf, -4B-gguf, -9B-gguf) are qwen35 models whose matrices hold ternary values, one scale per group of weights. Rounding a matrix to three values costs less when no input carries much more than the others, so each matrix was rotated before it was rounded, and the file stores the rotated one: the prism.hadamard.* metadata says which, and a reader has to undo it, or it computes something else without any error (src/model/fold.c). The reference is llama.cpp's load_hparams_hadamard() and build_hadamard_input() at the commit tools/gen_llama_ref.sh pins.

Key (prism.hadamard.) 2B, 4B 9B Meaning
version, tied_output 2, true 1, absent version 2: the token embeddings are the output head as well, and there is no output.weight
block_size 512 4096 size of the Walsh-Hadamard blocks
transform, axis normalized-sylvester-walsh-hadamard, input-last-dimension the same the only ones read
sign_mode, sign_widths, sign_values explicit; 2048 and 6144 (2B), 2560, 4096 and 9216 (4B) explicit; 4096 and 12288 one vector of +1 / -1 per input width, the values of all of them one after the other
weight_names 150 (2B), 200 (4B) 201 the folded matrices: attn_qkv, attn_gate, ssm_out, attn_q, attn_k, attn_v, attn_output, ffn_gate, ffn_up, ffn_down of every block, and output.weight in the 9B
inverse_weight_names token_embd.weight the same tables read by row whose rows are stored rotated
gdn_v_grouped absent (2B), true (4B) true the input of ssm_out was folded with its value heads grouped by key head

With H the orthonormal block Walsh-Hadamard matrix (the Sylvester matrix of the block size over its square root, one block after the other along the input) and D the diagonal matrix of the signs of an input width:

  • A folded matrix is W D H. H H = I and D D = I, so W x = (W D H) (H (D x)): the product reads H (D x) in place of x. OKLM computes it per token into a buffer of the context (the signs, then a fast transform, width x log2(block) additions) and gives the matrix product that buffer. The matrices that are not folded (ssm_alpha, ssm_beta, the conv, the norms: F32 in these files) read x itself.
  • ssm_out with gdn_v_grouped: the recurrence gives its value heads in the order r * G + k (key head k, repeat r); the fold was done on the order k * (H / G) + r. The heads are put in that order before the signs and the transform. The 2B has as many value heads as key heads and needs none of it.
  • The token embeddings store z = H (D e) per row; a looked-up row is turned back, e = D (H z), before anything reads it (the embeddings an image's projector gives are not rotated and are left alone).
  • A tied head (version 2) is the table of the rotated rows used as a folded matrix: Z (H (D x)) = E x.

What OKLM refuses at load rather than misread: another version, transform, axis or sign mode, a block size that is not a power of two, a listed matrix that the forward pass would not give the transformed input (the experts' tensors, anything outside the list above), a width without its sign vector, a table other than the token embeddings to rotate back, and the metadata on an architecture other than the dense Llama-family blocks.

Two fixtures hold the algebra (tools/gen_model_fixtures.py): tiny-qwen35-fold is tiny-qwen35 with every matrix folded (block 16, grouped value heads) and gives the logits of tiny-qwen35's reference within 1.9e-4 (llama.cpp itself gives the two files the same logits within 4.3e-5); tiny-qwen35-fold-tied is the version 2 form, within 2.9e-4 of llama.cpp on it.

The three packings. Each repository holds the same weights in three tensor types: Q2_0 (ggml's type 42: two bits per weight, a scale per 64), and two types of PrismML's llama.cpp fork alone, PQ2_0 (142: the same two bits, a scale per 128) and PTQ1_0 (143: five base-3 digits per byte as in TQ1_0, a scale per 128; 1.75 bits per weight). OKLM reads the three (formats/quant.md); the decoded tensors of the Q2_0 and PQ2_0 files of the 2B are equal value for value. The figures against llama.cpp (the pinned build for Q2_0, the fork at commit 6447906b for the two others) are in compatibility.md.

Folded weights on a device

A context of these models runs on OpenCL and on Vulkan like the plain qwen35 ones (on a device): every block there or the last ones, a paged cache, several sequences. What it took:

  • The rotation as one device op. oklm_hadamard_fold() is OKLM_OP_HADAMARD with a sign per element, before the rotation (a product's input, H (D x)) or after it (an embedding row turned back, D (H z)), a row's value heads read in the order they were folded in, and groups up to 4096 (256 before, the quantized caches' rotation). OpenCL's hadamard_fold kernel and Vulkan's hadamard_fold shader compute it in one launch, a work-group per block with the block in local memory, the butterflies in the CPU's order; the plain hadamard kernel and shader are as they were. Against the CPU: 1e-6 absolute plus 1e-5 relative (test_opencl, test_vulkan_ssm, test_hadamard_fold). The CPU keeps its own loop in fold.c for host layers; its results are unchanged.
  • The signs on each device. The sign vector of each input width is copied with the weights to every device holding blocks (a few KiB, counted with the head's bytes in the memory plan).
  • Which weight is folded. The list is kept by the weights' addresses in the mapped file; a device copy, or a streaming slot's, is recognized by its header's place among the context's weights, and the joined header a device makes of query, key and value (or gate and up) is read as its first part. A file that folded those parts unlike, or a block above 4096, is given the CPU with a notice; no published file is so.
  • The two types. PQ2_0 and PTQ1_0 are decoded 32 weights at a time in every matrix-product family of both backends, as Q2_0 and TQ1_0 with a scale per 128 (test_opencl test_matmul, test_vulkan_matmul, test_vulkan_gemm).

A device multiplies the ternary weights by the activations as they are; the CPU, as llama.cpp, rounds them to 8 bits first. So a device gives the three packings of a model the same logits to the bit, and stands as far from the CPU as the CPU from llama.cpp.

Measured (tests/model/test_bonsai_public.c, RX 7900 XT, 8 CPU threads; 5 prompts and 32 steps on llama.cpp's tokens, 160 choices; the speed is that of those decode steps, one token a pass):

File Backend Choices llama.cpp's Logit difference from llama.cpp: largest (mean) Against the CPU: choices, largest (mean) Tokens a second (CPU)
2B Q2_0 OpenCL 160 / 160 0.21 (0.012) 160 / 160, 0.48 (0.012) 121 (27.6)
2B Q2_0 Vulkan 160 / 160 0.21 (0.012) 160 / 160, 0.48 (0.012) 140 (28.2)
2B PQ2_0 OpenCL 159 / 160 0.25 (0.012) 160 / 160, 0.22 (0.012) 123 (29.7)
2B PQ2_0 Vulkan 159 / 160 0.25 (0.012) 160 / 160, 0.22 (0.012) 140 (30.1)
2B PTQ1_0 OpenCL 159 / 160 0.21 (0.012) 160 / 160, 0.22 (0.012) 49 (15.2)
2B PTQ1_0 Vulkan 159 / 160 0.21 (0.012) 160 / 160, 0.22 (0.012) 58 (15.2)
4B PQ2_0 OpenCL 160 / 160 0.65 (0.012) 159 / 160, 0.17 (0.011) 77 (16.5)
4B PQ2_0 Vulkan 160 / 160 0.65 (0.012) 159 / 160, 0.17 (0.011) 82 (16.5)

The 100 greedy tokens after a chat turn and after plain text are llama.cpp's on each file and backend (the 2B PTQ1_0 file's chat turn parts at a tie of 0.001 for llama.cpp, 99 of its 100 choices equal, as on the CPU). The 4B folds its ssm_out on grouped value heads: the reordering runs in the device op there. With a paged cache, and with the first half of the blocks left on the CPU, two sequences decoded together give llama.cpp's 16 first tokens on every file and backend. The fixtures (test_model, test_fold) run check_qwen35_of()'s scenes on OpenCL and on each Vulkan GPU (the RX 7900 XT and the UHD 750): within 4e-6 of the CPU.

Not done. On a device: the 9B (block 4096: the op is checked at that size, the file was not run), the 4B and 9B Q2_0 and PTQ1_0 files, a published file on the Intel UHD 750 (the fixtures and the op tests ran there on Vulkan), the multi-token prediction block, and the speed of PTQ1_0 (five base-3 digits a byte are decoded by multiplications, about 2.5 times slower than the two-bit types on the same device). The multi-token prediction files (...-PQ2_0-MTP-Q8_0.gguf: the PQ2_0 model with a plain Q8_0 block after it, which reads the rotated embeddings and the tied head as the main stack does) beyond runs of the 4B's (oklm run --mtp: 64 of 156 drafted tokens accepted, the text of the plain greedy run, at 0.6 to 1.1 times the plain run's speed on a loaded machine now that a rejected draft starts again from a kept state, against a seventh of it before: kept states), not compared with llama.cpp's draft pass; a LoRA adapter on a folded file (oklm lora-merge refuses one: the delta is in the plain basis and the weights are rotated; an adapter loaded beside the model is given the plain input, as llama.cpp's is, which was not tried); the vision projector.

The dflash draft model (audit)

The 4B and 9B repositories also hold ...-DFlash2-*.gguf (general.architecture = dflash), which llama.cpp uses with --spec-type draft-dflash -md FILE. It is not a small qwen35, and oklm run --draft cannot use it:

  • What it is: 6 blocks of the target's width (4096 for the 9B) with full attention (32 heads of 128 over 8 key/value heads, Q/K norms, a sliding window of 4096 on five blocks of six, YaRN-scaled rotation), a grouped conv of 2 taps before and after the attention and the FFN of each block (attn_conv_*, ffn_conv_*), dense SwiGLU. It has no token embeddings and no output head of its own.
  • Its input: the target's hidden states. dflash.target_layers names eight of the target's blocks (2, 6, 10, ..., 30); their inputs for every token the target evaluated are concatenated (8 x 4096), projected by fc to 4096 and written into the draft's key/value cache. The draft then reads a block of block_size = 16 positions, the last accepted token followed by 15 mask tokens, with attention that is not causal inside the block, and fills the 15 in one pass.
  • Its output (DFlash2): no logits. A selector (selector_hidden to a rank of 256, selector_predecessor and selector_successor of 256 x vocabulary) gives each position 16 candidates and scores pairs of neighbours; the draft is a greedy walk through that lattice, which the target then verifies as any draft.
  • What running it would take: the hidden states of chosen blocks kept for every token of the target (the activation capture of the roadmap's T10 is the same hook); the dflash blocks (the conv modules, block attention over injected keys and values); the lattice decoding; a draft source in src/model/speculative.c that drafts a block from the target's states instead of asking a second causal model for tokens (the verification is the existing one). On a hybrid target each rejected draft is a cut of the sequence, paid from the state kept before it (kept states). The model card itself recommends the multi-token prediction file over it, and plain decoding for the 2B.

The expert variant: qwen35moe

Qwen3.6-35B-A3B and its tunes (general.architecture = qwen35moe): the blocks above with a routed feed-forward in place of the dense one. Audited against llama.cpp's src/models/qwen35moe.cpp at the pinned commit, which differs from its qwen35.cpp in the feed-forward only.

Qwen3.6-35B-A3B
blocks, width 40 (30 linear, 10 full attention), 2048
full attention 16 heads of 256 over 2 key/value heads
linear blocks 16 key heads of 128 repeated over 32 value heads
expert_count, expert_used_count 256, 8
expert_feed_forward_length, expert_shared_feed_forward_length 512, 512; no feed_forward_length key
router ffn_gate_inp [2048 -> 256], F32; softmax over the 256, the 8 largest kept and their weights divided by their sum
experts ffn_gate_exps, ffn_up_exps [2048 -> 512] x 256 and ffn_down_exps [512 -> 2048] x 256, SwiGLU, as separate tensors (llama.cpp also accepts gate and up joined in one ffn_gate_up_exps; none of the files looked at has it, and OKLM does not read it)
shared expert ffn_gate_shexp, ffn_up_shexp, ffn_down_shexp (512 wide), its output times the sigmoid of ffn_gate_inp_shexp [2048 -> 1], added to the routed sum

Against qwen3moe: the routing is the same (softmax, renormalized, no scale, no bias), and there is a shared expert under a sigmoid gate, which qwen3moe does not have and qwen2moe does. OKLM runs it on the code of those two; what the name needed was the renormalization chosen for it, and a file without feed_forward_length accepted when it gives the experts' size.

  • The fixture tiny-qwen35moe (6 experts, 3 a token, a shared expert, no feed_forward_length): every logit within 2.9e-4 of llama.cpp's, 36 / 36 greedy choices (test_model, test_qwen35moe); prompt batches through the indexed expert products and the expert cache (one slot to all, the loaders, the pack) give the same logits, as on the other expert fixtures.

  • The published headers (the first 24 MB of each file, the rest left on the server): unsloth/Qwen3.6-35B-A3B-GGUF UD-Q4_K_M, UraionLabs/Uraion-Cyber-V1.0-GGUF Q4_K_M, IMPERUM/Imperum-CybersecurityLLM-v1.0-GGUF Q4_K_M and the five shards of ahmedandaloes/CyberStrike-OffSec-35B-GGUF Q4_K_M: oklm info says loadable, every tensor binds with the expected shape (733 tensors; 753 for CyberStrike, which carries a multi-token prediction block), and oklm plan gives 41.6 GiB resident at 8192 tokens for the Q4_K_M tunes (the 256 experts of each block repacked for the CPU on top of their mapping), 18.2 GiB of it the routed experts, which can be streamed from disk instead.

  • A published file: unsloth/Qwen3.6-35B-A3B-GGUF, Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf (13.2 GB, SHA-256 9c964e65...07b88fbe; Q6_K, IQ3_XXS, IQ2_S and IQ4_XS tensors), the smallest of the base model's files that left the disk its margin, loaded, compared and removed. Against llama.cpp at the pinned commit (test_infer_public): 5 / 5 prompts tokenized alike, 158 / 160 greedy choices on its tokens, logits within 5.14, four of the five continuations identical and the fifth parting where llama.cpp's two logits are 0.08 apart. oklm plan gives 31.5 GiB at 8192 tokens; decoding at 11.7 tokens a second on 8 threads of a busy machine (llama.cpp recorded its 160 tokens in 34 s); with --expert-cache 3 the same text at 6.7; oklm chat with the file's template thinks, then answers.

  • The tunes' files, each 21.2 GB at Q4_K_M, loaded, compared with llama.cpp at the pinned commit and removed on 2026-11-13, one at a time, with no change to the runtime (test_infer_public when a file is present in its folder under models/):

    file (SHA-256) prompt ids choices on llama.cpp's tokens max logit difference identical continuations llama.cpp margin where they part 100 tokens after a chat turn / plain text
    Uraion-Cyber-V1.0-Q4_K_M.gguf (47375dc8...8f9b8a88) 5 / 5 159 / 160 4.40 4 / 5 0.38 100 / 100, 100 / 100
    Imperum-CybersecurityLLM-v1.0.Q4_K_M.gguf (a8d54f31...a0844569) 5 / 5 154 / 160 2.59 2 / 5 0.26 97 / 100, 95 / 100
    CyberStrike-OffSec-35B-Q4_K_M-0000N-of-00005.gguf (first shard 4637db35...bd89e1be) 5 / 5 154 / 160 2.30 1 / 5 0.14 97 / 100, 98 / 100

    Imperum and CyberStrike agree with llama.cpp on fewer choices than Uraion and the base while their logits are closer to its own than any other file's: their distributions are flatter (llama.cpp's two best logits 0.01 to 0.26 apart at the steps that part), and a rounding decides more of their choices. Decoding at 12.3 to 12.5 tokens a second on 6 threads (a load of 5 to 9; Uraion 6.4 under a load of 15). oklm chat with each file's own template (Uraion and Imperum open a thought, CyberStrike carries Qwen2.5's template with its default system turn and answers at once) gives the start of llama.cpp's greedy reply to the same rendered prompt: 57, 38 and 10 tokens before a parting.

  • CyberStrike's multi-token prediction block (a full-attention block with the routed feed-forward, 20 tensors more than the base) runs as the draft of oklm run --mtp: over 100 tokens after a plain prompt, 195 drafts proposed and 64 accepted (32.8 %), the text the same as without it, at 8.8 tokens a second against 12.5 (a rejected draft costs a hybrid model its prefix, as on the 27B).

  • The devices: done (on a device): the fixture on OpenCL and Vulkan with the experts on the device, on the CPU (n_cpu_moe), in the expert cache and in the device tier; the UD-IQ3_XXS file whole on the RX 7900 XT.

  • Left: the base model above a 3-bit file, the block's drafts against llama.cpp's own draft pass, ffn_gate_up_exps; on a device, a published file larger than the device (the experts' tier and the expert cache under a hybrid stack, checked on the fixture only), the tunes' Q4_K_M files and the multi-token prediction block.