Compatibility
The live matrix of open-weight model families, tensor types and
tokenizer pre-tokenizers OKLM can load and run. "Verified" means checked
in CI against a published model file; until the GGUF reader exists, only
tensor types can be marked. Verified files are listed in
tests/public_models.txt (and the larger ones in
tests/public_models_large.txt).
make test-public and make test check the
canary set of them at every run (tests/canary_models.txt);
the others are checked when their code changes (testing.md).
Status: maintained.
Tensor types
| Type | Dequantize | Compute (matmul) | Quantize | Verified against a public file |
|---|---|---|---|---|
| F32, F16 | yes | yes | yes | yes: SmolLM2 F16 (dequantization identical to gguf-py; F16 conversion identical to llama.cpp) |
| BF16 | yes | yes (AVX2) | yes | the SmolLM2 safetensors checkpoint, run directly |
| Q5_0 | yes | yes | yes | yes: SmolLM2 Q4_K_M |
| Q8_0 | yes | yes | yes | yes: SmolLM2 Q8_0 and Q4_K_M; quantization identical to llama.cpp |
| Q4_0, Q4_1, Q5_1 | yes | yes | yes | no |
| Q4_K, Q6_K | yes | yes | yes | yes: SmolLM2 Q4_K_M; quantization identical to llama-quantize |
| Q2_K, Q3_K, Q5_K | yes | yes | yes | yes: SmolLM2 Q2_K, Q3_K_M, Q5_K_M; quantization identical to llama-quantize |
| Q4_1, Q5_1 | yes | yes | yes | yes: SmolLM2 Q4_0, Q5_K_M (their fallback types) |
| IQ4_NL, IQ4_XS | yes | yes (AVX2) | yes | yes: SmolLM2 IQ4_XS; quantization identical to llama-quantize |
| TQ1_0, TQ2_0, MXFP4, Q1_0, Q2_0 | yes | yes (Q2_0: AVX2) | yes | blocks identical to ggml's; files identical to llama-quantize's; Q2_0: yes, Ternary Bonsai 2 2B, 4B and 9B |
| PQ2_0, PTQ1_0 (types 142 and 143 of PrismML's llama.cpp fork, not ggml's: formats/quant.md) | yes | yes (AVX2) | yes | yes: Ternary Bonsai 2 (2B in both, 4B and 9B in PQ2_0); blocks identical to the fork's |
| NVFP4 | yes | yes | yes | blocks identical to ggml's (llama-quantize's command line does not offer it) |
| IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S | yes | yes (own codebook search; llama.cpp reads the files) | yes | yes: SmolLM2 IQ3_M, and SmolLM2 in every grid type made by
llama-quantize (tools/gen_iq_models.sh); dequantization
identical to ggml's |
The OpenCL and Vulkan backends have matrix-product kernels for every type in the "Compute" column (opencl.md, vulkan.md).
Model families
Family (general.architecture) |
Hugging Face class | Target | State |
|---|---|---|---|
llama |
LlamaForCausalLM | inference milestone | inference verified against llama.cpp on SmolLM2-135M (F16, Q8_0,
Q4_K_M), Llama-3.2-1B-Instruct and TinyLlama-1.1B-Chat (Q4_K_M), and
against a float64 reference (docs/inference.md).
DeepSeek-Coder-1.3B-Instruct (TheBloke's Q4_K_M of 2023, CPU): its
linear RoPE scaling is under the older key
llama.rope.scale_linear (4), read as llama.cpp reads it;
tokenization identical on seven prompts, 159 of 160 greedy choices equal
on llama.cpp's tokens (logits within 7.3, a shift common to a step's
logits; 1.15 once it is removed), 100 greedy tokens identical on a code
prompt and on a chat turn; the file has no chat template and gets
DeepSeek-Coder's published one. DeepSeek-Coder-6.7B-Instruct (TheBloke's
Q4_K_M, the same keys, CPU; run and removed): tokenization identical on
seven prompts, 159 of 160 choices (logits within 2.81), four of five
continuations identical and the fifth parting where llama.cpp's two
tokens are 0.08 apart, 100 greedy tokens identical on a code prompt and
on a chat turn in its template; against a float64 pass of the model
(tools/infer_ref.py, activations not rounded) 23 of 24
choices and logits within 6.99, llama.cpp 21 of the 22 steps its tokens
share with the pass and within 6.98. Both on OpenCL and Vulkan (inference.md): the
1.3B 159 and 157 of 160 choices equal to llama.cpp's (logits within 6.13
and 6.04), the 6.7B 157 and 159 (3.73 and 1.28) |
mistral |
MistralForCausalLM | inference milestone | converts (as llama); runs through the llama forward
pass, no public file in the corpus yet; its chat template renders like
jinja2 |
mistral3 |
Mistral3ForConditionalGeneration (text part) | inference milestone | GGUF inference (Mistral Small 3.x, Devstral Small 2): the llama
block and its tensors, plus what llama.cpp reads for this name: YaRN
(rope.scaling.factor over
original_context_length, boundaries 32 / 1) with the
magnitude correction divided by yarn_log_multiplier, and an
attention temperature (attention.temperature_scale: queries
times 1 + scale x ln(floor(position / original context) + 1), so 1 below
the original context). Tiny random model with temperature steps at
positions 12 and 24 within 1e-6 of a float64 reference on the CPU,
OpenCL and Vulkan, llama.cpp within 9e-4 of the same reference (F16
cache). Devstral-Small-2-24B-Instruct-2512 (Unsloth's Q4_K_M, CPU; same
tensor names, shapes and types as Precog-24B, published as
llama): tokenization identical to llama.cpp's on five
prompts (tekken), 158 of 160 greedy choices equal on
llama.cpp's tokens, logits within 4.05, four of five 32-token
continuations identical, the fifth parting where llama.cpp's top two are
0.08 apart; its 13.7 kB chat template (with strftime_now)
renders and oklm chat answers. On OpenCL and Vulkan (RX
7900 XT, every block on the device, inference.md): 159
and 158 of 160 choices equal to llama.cpp's, logits within 4.01 and
4.57, a paged cache and half of the blocks on the CPU. Not checked on
the published file: a context beyond 8 192 tokens (where the temperature
and the YaRN ramp act); no HF conversion; the realtime speech variant
(time-conditioned norm) is refused; the vision tower is a separate
projector file, not read |
qwen2, qwen3 |
Qwen2/Qwen3ForCausalLM | inference milestone | inference verified against llama.cpp on Qwen2.5-0.5B-Instruct Q4_K_M and Qwen3-0.6B Q8_0, and on a tiny random qwen3 model against a float64 reference |
gemma, gemma2, gemma3 |
Gemma*ForCausalLM | inference milestone | GGUF inference: Gemma-1.1-2B-it, Gemma-2-2B-it (Q4_K_M),
Gemma-3-270M-it (Q8_0, also over a prompt past its 512-position window)
and Gemma-3-1B-it (Q4_K_M) match a float64 reference within 3e-4 on
OpenCL (CPU: within the activation quantization, 90 % of greedy choices
or more), and llama.cpp's output to its GELU approximation; tiny gemma2
/ gemma3 models in make test. Hugging Face checkpoints load
directly and convert (norm weights + 1): checkpoints made from the tiny
GGUF fixtures give their logits exactly |
phi3 |
Phi3ForCausalLM | inference milestone | GGUF inference: Phi-3-mini-4k-instruct (q4) and
Phi-3.5-mini-instruct (Q4_K_M, LongRoPE factors and attention factor)
match a float64 reference within 1e-4 on OpenCL; on the CPU, Phi-3.5 has
steps where quantized activations move the logits by up to 65
(llama.cpp's own CPU logits too); tiny phi3 model in
make test. Hugging Face checkpoints load directly and
convert (fused projections, LongRoPE factors from
config.json) |
phi2 |
PhiForCausalLM | inference milestone | GGUF inference: phi-2 (Q4_K_M; parallel block, partial NeoX
rotation, output bias) within 2e-3 of a float64 reference on OpenCL and
within the activation quantization on the CPU and against llama.cpp;
tiny phi2 model in make test |
granitemoe, granite |
GraniteMoeForCausalLM, GraniteForCausalLM | inference milestone | GGUF inference: Granite-3.1-1B-A400M-instruct (Q4_K_M; 32 experts, 8
per token, embedding / attention / residual / logit scales) within the
activation quantization of a float64 reference and of llama.cpp; tiny
granitemoe model in make test. MoE layers run on the CPU or
the OpenCL device, routing included (about 230 t/s decode there, 88 on
the CPU); dense granite loads with the same scales (also
from a Hugging Face checkpoint), no public file checked; GraniteMoE
checkpoints load directly and convert (input_linear split
into gate and up experts), tiny checkpoint in
make test |
gpt2 |
GPT2LMHeadModel | inference milestone | inference verified against llama.cpp on GPT-2 (124M) Q4_K_M, and on a tiny random model against a float64 reference |
starcoder2 |
Starcoder2ForCausalLM | inference milestone | GGUF inference (StarCoder2; llama.cpp also runs Granite code models
under this name): the GPT-2 block (LayerNorm with biases, GELU MLP with
biases, a final LayerNorm, the head tied to the embeddings) with
separate query, key and value projections, grouped keys and values (24
query heads over 2) and NeoX rotary positions instead of the position
table. Tiny random model within 8e-5 of a float64 reference on the CPU,
OpenCL (all layers, and one) and Vulkan. StarCoder2-3B (second-state's
Q4_K_M, CPU): tokenization identical to llama.cpp's on five prompts, 159
of 160 greedy choices equal on llama.cpp's tokens, logits within 1.64,
four of five 32-token continuations identical, the fifth parting where
llama.cpp's top two are 0.23 apart; 100 greedy tokens identical on
def quicksort(arr):; over 200 steps of two 100-token
recordings 199 choices equal, logits within 2.9. The file names no
pre-tokenizer and gets starcoder (tokenizer.md), identical to Hugging Face's on
400 lines; llama.cpp's own ids differ where a space precedes
punctuation. A completion model: no chat template (fill-in-the-middle
tokens <fim_prefix>, <fim_suffix>,
<fim_middle> are in the vocabulary). On OpenCL and
Vulkan (inference.md): 159
and 157 of 160 choices equal to llama.cpp's, logits within 1.72 and
1.60. Not checked: the float64 reference on the published file, the 7B
and 15B, the HF conversion (none) |
qwen2vl |
Qwen2VLForConditionalGeneration, Qwen2_5_VLForConditionalGeneration | multimodal milestone | GGUF inference on the CPU, OpenCL and Vulkan: qwen2
with rotary positions of three parts (time, height, width:
rope.dimension_sections), a text rotated exactly as
qwen2 (bit for bit on a tiny model), an image's rows by
their place in its grid (oklm_eval_embd_grid()), a video's
by their place in their pair's grid and the pair's time
(oklm_eval_embd_video(); on OpenCL and Vulkan since
2026-11-14, 46 to 48 of the CPU's 48 choices). Qwen2.5-VL 3B and 32B
(Q4_K_M) and 72B (IQ1_M) were run on 2026-11-14 on the CPU and both
devices: 47 of 48, 16 of 16 and 15 of 16 choices llama.cpp's, the 32B
with 61 of its 64 layers on a 20 GiB card and the 72B with 62 of 80.
Several pictures in one prompt and pictures up to 4 096 embeddings agree
with llama.cpp on the 2B (the 7B's language model further from it as the
image rows grow, on its BF16 file too: unexplained).
Qwen2-VL-2B-Instruct and Qwen2.5-VL-7B-Instruct (Q4_K_M) against
llama.cpp: on a text 48 and 47 of 48 greedy choices equal; with an
image's embeddings 46 of 48, parting where llama.cpp's two best logits
are 0.09 and 0.40 apart; tiny model within 3e-5 of a float64 pass (6e-5
on a device). An image's rows are rotated where their block runs
(oklm_rope_parts()): on OpenCL and Vulkan both files give
46 to 48 of the CPU's 48 choices, logits within 2.5 of the CPU's (0.09
to 0.18 on average); no Hugging Face checkpoint path (multimodal.md) |
lfm2 |
Lfm2ForCausalLM | inference milestone | GGUF inference (Liquid's LFM2 and its tunes; inference.md has the audit against llama.cpp's graph): short convolution blocks (a gate |
lfm2moe |
Lfm2MoeForCausalLM | inference milestone | GGUF inference (LFM2.5 8B-A1B and its tunes): the lfm2
stack with routed experts after leading_dense_block_count
dense blocks: a sigmoid router (expert_gating_func 2),
experts chosen on the probabilities plus exp_probs_b,
weights renormalized, SwiGLU experts without biases (DeepSeek V3's
routing without groups or shared experts). Tiny random model (8 experts,
2 a token) within 1.8e-5 of a float64 reference on the CPU, OpenCL and
Vulkan. Tini-Cybersec-8B-A1B (bartowski's Q4_K_M, 32 experts, 4 a token,
CPU): tokenization identical to llama.cpp's on five prompts, 156 of 160
greedy choices equal on llama.cpp's tokens, logits within 4.46,
continuations parting where llama.cpp's top two are within 0.1;
oklm chat answers. On OpenCL and Vulkan (inference.md): 155
and 155 of 160 choices equal to llama.cpp's, logits within 7.30 and
5.65, the experts on the device or in host memory. Not done: as
lfm2; the expert cache and the device expert tier not run
on this file; the 24B-A2B |
gpt-oss |
GptOssForCausalLM | inference milestone | GGUF inference on the CPU, OpenCL and Vulkan (gpt-oss-20b; inference.md has the audit against
llama.cpp's graph): attention wider than the residual stream (64 heads
of 64 over 8 key / value heads) with biases, NeoX rotation with YaRN, a
sliding window of 128 positions on the even blocks, attention sinks (one
score per head in every softmax), then 32 experts, 4 per token, with a
router bias, weights that are the softmax of the chosen scores, expert
biases and the activation
min(g, 7) sigmoid(1.702 min(g, 7)) (clamp(u, 7) + 1). Tiny
random model (3 blocks, sinks, a window of 4, YaRN, 8 experts with
biases and values past the clamp) within 1.6e-5 of a float64 reference;
batches equal to single tokens. gpt-oss-20b (Unsloth's Q4_K_M, 11.6 GB,
experts in MXFP4, CPU): tokenization identical to llama.cpp's on five
prompts (gpt-4o), 153 of 160 greedy choices equal on
llama.cpp's tokens, logits within 3.96, two of five 32-token
continuations identical, the others parting where llama.cpp's top two
are within 0.11; 3.3 tokens/s on this machine's CPU. On OpenCL and
Vulkan (the file whole on the RX 7900 XT, the sinks in the attention
kernels, the expert biases gathered and added and the activation an op
of its own, oklm_sigglu()): 158 and 157 of 160 choices
equal to llama.cpp's, mean logit difference 0.08, 126 and 164 tokens/s
decoding, prompts at 3313 and 3738 (5.6 and 7.1 on the CPU). The harmony
chat form: the 17 kB template renders, a reply runs through its channels
to `< |
spark2_5 |
(none: GGUF only) | inference milestone | GGUF inference on the CPU, OpenCL and Vulkan (Spark-X2.6; inference.md has
the audit against llama.cpp's graph): the llama block with one fused
attn_qkv (16 query heads of 256 over 4 key / value heads in
the 4B, wider than the residual stream), a sliding window of 512
positions on the blocks a list of booleans names (three out of four),
NeoX rotation of the whole head with one base in the window blocks and
of 64 dimensions with another in the others, the attention output of
each head times a sigmoid gate (attn_gate, a projection of
the block's normalized input), GELU gating in the feed-forward. Tiny
random model (5 blocks, a window of 4, 16 and 8 rotated dimensions)
within 2.6e-5 of a float64 reference, 36 / 36 of llama.cpp's own tokens
on it; batches and several sequences equal to single tokens.
Spark-X2.6-1.7B and 4B (XHToken's Q4_K_M, CPU): tokenization identical
to llama.cpp's on five prompts (pre-tokenizer spark2_5, the
chat turn with its control tokens among them), 159 and 158 of 160 greedy
choices equal on llama.cpp's tokens, logits within 4.53 and 16.2 (two
sensitive steps of the 4B; 3.88 with an F32 cache), four and three of
five 32-token continuations identical, the others parting where
llama.cpp's top two are within 0.03 and 0.20; the 1.7B's BF16 file 160
of 160, logits within 0.35; past the window, after a prompt of 769
tokens, 14 and 16 of 16 choices, logits within 1.01 and 1.42. The
template renders (a system block with the reasoning effort,
<think> opened by the generation prompt) and
oklm chat answers, about 8 tokens/s for the 4B on this
machine's CPU. On OpenCL and Vulkan (RX 7900 XT, every block on the
device): 158 and 160 of 160 choices equal to llama.cpp's for the 1.7B,
158 and 158 for the 4B, mean logit difference 0.14 to 0.17, a paged
cache and half of the blocks on the CPU giving the same choices but for
near ties; 216 and 285 tokens/s decoding the 1.7B, 118 and 145 the 4B
(22.6 and 9.8 on the CPU). Not done: a published file on the Intel UHD
750, a comparison beyond 800 tokens, the template's tool calls
(<tool_call>NAME<arg_key>..., the form of
GLM-4.5: not tried on this model), the HF conversion, the trainer
(refused: a sliding window) |
gemma4 |
(none: GGUF only) | inference milestone | GGUF inference on the CPU, OpenCL and Vulkan of Gemma 4 12B,
26B-A4B, E2B and E4B (inference.md
has the audit against llama.cpp's graph): window blocks with heads of
256 values and blocks without window with heads of 512 and their own key
/ value head count, each layer's cache in its block's shape; values
taken from the key projection where a block has no attn_v
and normalized without weight; unscaled scores; frequency factors on the
blocks without window; a scale on each block's output; routed experts
beside the dense feed-forward (the router's own input, gate and up
joined in ffn_gate_up_exps, a scale per expert). Tiny
random models (6 blocks, a window of 4, heads of 8 and 16; 8 experts, 2
a token, joined or apart) within 8.0e-6 of a float64 reference, 36 / 36
of llama.cpp's own tokens on them; batches and several sequences equal
to single tokens on the contiguous and the paged cache. Gemma-4-12B-it
(Unsloth's Q4_K_M, CPU): tokenization identical to llama.cpp's on five
prompts, 148 of 160 greedy choices equal on llama.cpp's tokens, logits
within 12.5, one of five continuations identical. Those figures are the
distance between two runs of one program: llama.cpp with flash attention
against llama.cpp without gives 153 of 160 and 11.7, OKLM's scalar
kernels against its AVX kernels 157 of 160 and 14.3; compared step by
step on llama.cpp's own hidden states, every step of the 48 blocks
agrees within 6e-6 but attention and the GELU, which llama.cpp rounds to
half precision (OKLM_ATTN_F16=1,
OKLM_GELU_F16=1); two of its blocks alone in F16
(tools/gguf_cut_blocks.py) within 0.25 of llama.cpp on
logits of 250 to 650, 96 of 96 choices. The E2B and E4B (an embedding
per layer added at each block's end, the last 20 or 18 blocks reading
the keys and values of an earlier block and keeping no cache, a
feed-forward width per block): a tiny model of that form within 7.3e-6
of the float64 reference; Gemma-4-E2B-it Q4_K_M 160 of 160 choices on
llama.cpp's tokens, logits within 8.08, five of five continuations
identical; Gemma-4-E4B-it Q4_K_M 156 of 160, within 3.51, one of five.
The template renders and oklm chat answers, with and
without thinking. Gemma-4-26B-A4B-it (Unsloth's UD-Q4_K_M, 16.9 GB: Q8_0
matrices, Q4_K and Q5_1 experts; about 27 GiB resident): tokenization 5
of 5, 152 of 160 choices, logits within 12.1 (llama.cpp against itself
with flash attention: 155 of 160 and 10.1); on llama.cpp's hidden states
the router, the eight experts of every token and their sum agree within
4.2e-6 over the 30 blocks. On OpenCL and Vulkan (RX 7900 XT, every block
on the device; tests/model/test_gemma4_public.c): on
llama.cpp's tokens 148 and 152 of 160 choices for the 12B, 158 and 157
for the E2B, 158 and 157 for the E4B, 155 and 152 for the 26B-A4B, the
logits as far from the CPU's as the CPU's are from llama.cpp's (0.24 to
0.68 on average; one step of the 26B-A4B moves by up to 22.8, its best
token unchanged); a paged cache, half of the blocks on the CPU and the
experts in host memory give the same choices but for near ties; decoding
at 53.8 and 68.5 tokens a second for the 12B (3.90 on the CPU), 120.8
and 141.2 for the E2B (17.7), 71.8 and 106.9 for the 26B-A4B (7.94). The
blocks of the E2B and E4B that share a cache go to a device together
with the blocks they read. Not done: past the 1024-position window on
the published file, a quantized cache under the 512-wide heads on Vulkan
(those blocks' attention then runs on the CPU), the expert cache and the
device expert tier for the 26B-A4B, the image projector, the HF
conversion, the trainer. The template's tool calls (`< |
muse-glimmer |
(none: GGUF only) | inference milestone | the language side of Muse Glimmer 30B as llama.cpp's
muse-glimmer.cpp has it (inference.md):
normalized embeddings, per-head Q/K norms, a window of 2048 positions on
three blocks out of four, rotation in the window blocks only, one
sigmoid gate per attention output value, norms after the attention and
the feed-forward (epsilon 1e-8), logits scaled then soft-capped. Tiny
random model within 4.4e-6 of a float64 reference, 36 / 36 of
llama.cpp's own tokens on it. unsloth/Muse-Glimmer-30B-GGUF
UD-IQ2_XXS (10.7 GB, the smallest; SHA-256
3e9bd0e6...0875ad9; test_infer_public when
present in models/muse-glimmer-30b/) against llama.cpp: 5 /
5 prompts tokenized alike, 157 / 160 greedy choices on its tokens,
logits within 5.51, three of five continuations identical and the others
parting where llama.cpp's two logits are at most 0.10 apart, 100 / 100
and 96 / 100 choices over 100 greedy tokens after a plain prompt and a
chat turn. Its replies are messages with a recipient (a thought to
self closed by `< |
mimo2 |
(none: GGUF only) | inference milestone, on a tiny model only | Xiaomi's MiMo-V2 models as llama.cpp's mimo2.cpp has
them (inference.md), on the CPU:
one fused attn_qkv with queries and keys of
attention.key_length and values of
attention.value_length, the window and the count of key
heads of each block from two lists (a first block without window, then
five window blocks out of six), the window blocks with their own
rotation base and attention sinks, the attention output scaled by
attention.value_scale, a dense block then experts chosen by
sigmoid with a selection bias. Tiny random model within 1.4e-5 of a
float64 reference; llama.cpp gives the reference's 36 / 36 tokens on it,
logits within 0.062 (its F16 cache). No published file was run: the
smallest, MiMo-V2.6-Flash-MOPD Q2_K_S, is 105.9 GB, and the disk does
not take it; the metadata and the 472 tensors of the MiMo-V2.6-Flash-RL
Q2_K set load and bind, and its chat template renders as jinja2 renders
it. Not read: separate attn_q / attn_k /
attn_v, the multi-token prediction blocks
(mtp-*.gguf, refused), the projector; the model's tool-call
form is not parsed; a context asked on OpenCL or Vulkan runs on the CPU.
The model's distillation into Qwen3.5-9B is a qwen35 file
and runs (the qwen35 row) |
clip (mmproj-*.gguf) |
CLIPVisionModel, SiglipVisionModel | multimodal milestone | the SigLIP encoder with the idefics3 and
gemma3 projectors, the encoders of Qwen2-VL and Qwen2.5-VL
at the image's own size with their mergers (qwen2vl_merger,
qwen2.5vl_merger: embeddings within 1.3e-5 and 2.9e-6 of a
float64 pass of the models' definition, multimodal.md) and the
CLIP encoder with mlp (LLaVA 1.5 7B: its 576 embeddings
within 7e-6 of a float64 pass of the model's definition, which the
pinned llama.cpp's encoder departs from; 47 of 48 greedy choices those
of llama.cpp's language model given the same embeddings, multimodal.md; LLaVA 1.6 Mistral 7B:
an overview and the tiles of the resolution its file lists that suits
the image, cut as the model's own processor cuts them, each tile's 576
embeddings within 8e-6 of the float64 pass, the image's rows those the
model's code places from them; 48 of 48 greedy choices those of
llama.cpp's language model given the same rows, multimodal.md) (multimodal.md):
SmolVLM-256M-Instruct's tile within 0.1 of llama.cpp's embeddings
(values up to 109); Gemma 3 4B's 256 embeddings within 2e-5 of a float64
pass and 7e-3 of llama.cpp's, its answer 47 of 48 greedy choices
llama.cpp's (multimodal.md);
SmolVLM-500M-Instruct and SmolVLM2-2.2B-Instruct (Q8_0, F16 projectors;
tiles of 384 pixels in patches of 14) caption the figure set of
tests/rag/figures_set (rag.md); tiny random
encoders in make test. Other projectors are refused, and an
mlp file with chosen feature layers. With the language
model on OpenCL or Vulkan, oklm describe and
oklm serve compute the blocks of every encoder on the
device (multimodal.md), those of
Qwen2-VL and Qwen2.5-VL included (multimodal.md) |
| whisper | WhisperForConditionalGeneration | multimodal milestone | not started |
qwen2moe, qwen3moe |
Qwen2MoeForCausalLM, Qwen3MoeForCausalLM | inference milestone | GGUF inference: tiny random models (experts with their own hidden
size; Qwen2-MoE's sigmoid-gated shared expert and QKV biases;
Qwen3-MoE's Q/K norms and renormalized weights) within 2e-5 of a float64
reference on the CPU and OpenCL; Hugging Face checkpoints load directly
and convert (per-expert matrices stacked), tiny checkpoints in
make test; no public file checked (the smallest are 8 GB
and more) |
deepseek |
DeepseekForCausalLM | inference milestone | GGUF inference (DeepSeek-MoE: dense leading layers, softmax routing
scaled by expert_weights_scale, shared experts run as one
ungated FFN): tiny random model within 2e-5 of a float64 reference on
the CPU and OpenCL; no public file checked (16B) |
deepseek2 |
DeepseekV2ForCausalLM, DeepseekV3ForCausalLM, Glm4MoeLiteForCausalLM | inference milestone | GGUF inference: multi-head latent attention in llama.cpp's absorbed
form (one shared cache head of kv_lora + rope, values read from it;
compressed or direct queries), files with attn_k_b /
attn_v_b and older ones with attn_kv_b (split
at load), YaRN with the score multiplier, softmax or sigmoid routing,
the V3 selection bias and expert groups, the multi-token prediction
block (left out of the main stack; usable as the speculative draft,
oklm run --mtp); tiny V2-Lite-like and V3-like models
within 2e-5 of a float64 reference on the CPU and OpenCL, the V3-like
one within llama.cpp's MoE precision of llama.cpp itself.
DeepSeek-Coder-V2-Lite-Instruct IQ4_XS (bartowski, an older
attn_kv_b file): greedy choices identical to llama.cpp's on
short prompts (32 / 32 steps, CPU and OpenCL) and on a 422-token prompt
(OpenCL 8 / 8, CPU 6 / 8: the two others where llama.cpp's top two are
within 0.5); 102 t/s decode on OpenCL, 16 on the CPU. GLM-4.7-Flash
(Glm4MoeLiteForCausalLM, written as deepseek2
with the GLM vocabulary; unsloth Q4_K_M, CPU): tokenization identical to
llama.cpp's on 435 strings (glm4 pre-tokenizer, the role,
thinking and tool tokens), its template the jinja2-checked fixture;
greedy output identical to llama.cpp's on four prompts (32 / 32), a
fifth parting at step 0 where llama.cpp's top two are 0.11 apart;
perplexity 44.42 against llama-perplexity's 43.28
(512-token windows, 1530 tokens; chunks 1 to 5 within 0.7 %, the last
one a window where the model, without [gMASK]<sop>,
writes logits near 150); a glm45 tool call through
oklm chat --tools. Files without
expert_gating_func (ggml-org's) route by sigmoid, as
llama.cpp decides from the shape. No published file carries the MTP
block, so --mtp is unchecked on it. 18.8 GB resident; with
8 threads on a loaded machine (load about 33 on 16 cores) 28 t/s prompt
(653 tokens) against llama.cpp's 43 (pp512), and decode collapsed to 1.2
t/s against llama.cpp's 9.5 (2026-11-07, before the CPU's own worker
team and the free-core cap). Re-measured 2026-11-11, decode with 4 / 8
threads: 11.2 / 11.1 t/s on a quiet machine (llama.cpp 11.3 / 11.1), 5.5
/ 6.0 beside 16 spinning processes (llama.cpp 2.6 / 0.24). The agent
evals through oklm serve: 21 of 25 (agent-loop.md), its 163
glm45 calls all parsed |
glm4moe |
Glm4MoeForCausalLM | inference milestone | GGUF inference (GLM-4.5 / 4.5-Air / 4.6 / 4.7): grouped-query
attention with QKV biases, optional Q/K norms, NeoX rotation of half of
each head, DeepSeek V3's MoE (sigmoid routing with the selection bias,
dense leading layers, shared experts, renormalized and scaled weights),
the norm before the FFN stored as post_attention_norm, the
nextn block left out of the main stack and usable as the
speculative draft (oklm run --mtp); tiny random models
(with Q/K norms and a block, and with neither) within 3e-5 of a float64
reference on the CPU, llama.cpp (CPU, F32 cache) within 1.4e-5 of the
same reference with the same greedy choices. Hugging Face checkpoints
load directly and convert (per-expert matrices stacked, the
nextn layer kept), tiny checkpoint in
make test, its conversion loaded by llama.cpp. No public
file checked: the smallest, GLM-4.5-Air (106B-A12B), is about 65 GB at
Q4 |
glm4 |
Glm4ForCausalLM | inference milestone | GGUF inference (GLM-4-0414 9B / 32B, GLM-Z1): norms after attention
and after the FFN as well as before, gate and up in one
ffn_up, QKV biases, normal-pair rotation of half of each
head; a tiny random model within 6e-6 of a float64 reference on the CPU,
llama.cpp (CPU, F32 cache) within 1e-5 of the same reference with the
same greedy choices. Hugging Face checkpoints load directly and convert,
tiny checkpoint in make test. GLM-4-9B-0414 Q4_K_M
(bartowski, sha256 2b95f0b8...216a;
test_infer_public when present in
models/glm4-9b/): tokenization identical to llama.cpp's on
its five prompts (a [gMASK]<sop><|user|> chat
turn among them), 157 / 160 greedy choices on llama.cpp's tokens on the
CPU and 158 / 160 on OpenCL, logits within 0.97 / 0.85; run freely,
three of five continuations identical, the two others parting where
llama.cpp's top two are within 0.16; perplexity 23.78 against
llama-perplexity's 23.81 (Tiny Shakespeare's first 30 KB,
512-token windows). Speed (2026-11-11, quiet machine, 8 threads): 84 t/s
prompt (497 tokens) and 4.7 decode on the CPU against llama.cpp's 60
(pp512) and 4.8; 2 074 and 61 on OpenCL. Its tools follow GLM-4's own
convention, the glm4 format (the function's name opening
the turn after <|assistant|>, then its arguments as
JSON, results in an <|observation|> turn): two calls
and their results through oklm chat --tools; the agent
evals through oklm serve 3 of 25 (agent-loop.md), its 26 calls all
parsed, most turns ending on a plan in prose |
olmoe |
OlmoeForCausalLM | inference milestone | GGUF inference (Q/K RMS norms over the whole projections, one weight
as wide as all the heads; 64 experts, 8 per token, softmax weights not
renormalized; no shared expert; NeoX rotation), through the Llama
forward pass and the MoE path of the other families (indexed expert
products, expert cache): tiny random model within 7.8e-6 of a float64
reference on the CPU and within llama.cpp's own distance from that
reference (0.017, its F16 KV cache; the tiny qwen2moe 0.014) of
llama.cpp. OLMoE-1B-7B-0125-Instruct Q4_K_M (allenai): tokenizer
identical to llama.cpp's on 400 strings, chat template identical to
jinja2, on llama.cpp's tokens 157 / 160 greedy choices equal (max logit
difference 2.04), the free-running continuations parting only where
llama.cpp's top two are within 0.12; perplexity on the last 30 KB of
Tiny Shakespeare 2.1963 (llama.cpp 2.2015); CPU, 4 threads on a shared
machine: 97 t/s prompt (682 tokens), 13.3 t/s decode (llama-bench beside
it: 110 and 20.4); the expert cache at 50 % and 25 % gives the resident
run's tokens (moe.md). Hugging Face
checkpoints load directly and convert (per-expert matrices stacked),
tiny checkpoint in make test. OpenCL (2026-11-08): the tiny
model within 6.6e-6 of the float64 reference with every layer on the
device, one layer on the CPU, and the routed experts of the first or of
every layer on the CPU (make test); OLMoE-1B-7B against
llama.cpp 157 / 160 choices (max logit difference 1.84, parting where
llama.cpp's margin is 0.04), against OKLM's CPU 154 to 158 / 160 over
all layers, --gpu-layers 8, --n-cpu-moe 8 and
all, with the expert cache and with
--expert-gpu (every differing choice a near tie, the CPU's
margin at most 0.26), perplexity 2.1958 to 2.2015 against the CPU's
2.1963 (inference.md) |
qwen35 |
(GGUF files only) | inference milestone | Qwen3.5-9B (unsloth/Qwen3.5-9B-GGUF, Q4_K_M): three
Gated DeltaNet blocks (a short conv, a state per head updated with a
decay and a write strength per token, a gated norm) then a
full-attention block whose queries carry an output gate, heads of 256
with 64 dimensions rotated, dense SwiGLU, the qwen35
pre-tokenizer and a 248 320-token vocabulary (hybrid-layers-design.md).
Against llama.cpp on the published file: 5 / 5 prompts tokenized alike,
159 / 160 greedy choices on its tokens, the mean logit difference 0.116
and the largest 13.8 (single low logits at four steps of a repeated
sentence; 0.4 to 0.7 a step otherwise, and the same file moves its own
logits as much between an F16 and an F32 cache, in llama.cpp as in OKLM:
hybrid-layers-design.md),
100 / 100 greedy tokens after a plain prompt and 26 after a chat turn
before a choice llama.cpp held by 0.44 (97 / 100 on its tokens), a
prompt of 296 tokens in batches of 64, two sequences in one pass, a
sequence through a file, a copy and a cut
(tests/model/test_qwen35_public.c); a tiny random model of
the layout within 2e-4 of llama.cpp in make test. On OpenCL
and Vulkan (RX 7900 XT, the same tests, every block on the device, the
linear blocks' states in device memory): 158 / 160 choices on
llama.cpp's tokens on each, the mean logit difference 0.122 and the
largest 13.7, 157 and 159 of 160 choices the CPU's, a paged cache and
half of the blocks on the CPU giving llama.cpp's tokens; 83 and 104
tokens a second against 5.5 on the CPU (hybrid-layers-design.md).
The trainer does not know the layout, the vision projector is not read;
Qwen3.8-27B (unsloth/Qwen3.8-27B-GGUF, UD-Q4_K_M: 64 blocks
at width 5120, 48 value heads over 16 key heads, IQ4_XS and IQ4_NL among
its types) the same way: 5 / 5 prompts, 160 / 160 choices, 3.22 at most,
the five continuations identical; its multi-token prediction block
(oklm run --mtp) drafts 45 tokens of which 20 are accepted,
for the same text. Cyber-Ornith 1.5 9B
(DuoNeural/Cyber-Ornith-1.5-9B-OBLITERATED-GGUF, Q4_K_M: a
security tune with the 9B's 427 tensors) the same way: 5 / 5 prompts,
158 / 160 choices, the mean logit difference 0.089 and the largest 3.58
(one low logit), three of the five continuations identical and the
others parting where llama.cpp's two logits are at most 0.25 apart, 100
/ 100 greedy tokens after a plain prompt and after a chat turn, a prompt
of 296 tokens in batches of 64 (16 / 16 choices, 1.03 at most);
oklm chat with the file's template, which opens a thought
by default, gives the 56 tokens of llama.cpp's greedy answer to the same
rendered prompt, and a direct answer with
--template-vars '{"enable_thinking": false}'.
BugTraceAI-CORE-Ultra 27B
(BugTraceAI/BugTraceAI-CORE-Ultra-27B-Q4, Q4_K_S with Q5_K
among its types: a security tune of 64 blocks at width 5120, 851
tensors, no multi-token prediction block; SHA-256
50170a34...563ff6e2) the same way: 5 / 5 prompts, 157 / 160
choices, 3.59 at most, three of the five continuations identical and the
others parting where llama.cpp's two logits are at most 0.08 apart; over
100 greedy tokens after a chat turn and after a plain prompt, 99 / 100
choices each on llama.cpp's tokens (logits within 5.57), the texts
parting at steps 44 and 72 where its two logits are 0.008 and 0.047
apart; oklm chat with the file's template opens a thought
whose first 80 tokens are llama.cpp's greedy ones for the same rendered
prompt, then answers. Ternary Bonsai 2
(prism-ml/Ternary-Bonsai-2-2B-gguf, -4B-gguf,
-9B-gguf: Qwen3.5 with ternary weights stored in a
Walsh-Hadamard rotated basis, the prism.hadamard.*
metadata, which OKLM undoes on the input of each product as llama.cpp
does, hybrid-layers-design.md),
in its three packings, on the CPU
(tests/model/test_bonsai_public.c): every file's five
prompts tokenized alike. Q2_0 against the pinned llama.cpp:
2B 160 / 160 greedy choices on its tokens, logits within 0.30 (0.016 on
average), the five continuations and 100 greedy tokens after a chat turn
and after plain text identical; 4B 160 / 160, 0.32 (0.016), the same
continuations; 9B 159 / 160, 0.29 (0.015), 100 / 100 and 99 / 100
choices over the 100 tokens. PQ2_0 and PTQ1_0
against PrismML's llama.cpp fork (commit 6447906b, the only
reader of these two types beside OKLM): 2B PQ2_0 159 / 160,
0.32 (0.017), 100 / 100 twice; 2B PTQ1_0 159 / 160, 0.29
(0.016), 97 / 100 and 100 / 100; 4B PQ2_0 159 / 160, 0.75
(0.016), 100 / 100 and 99 / 100; 9B PQ2_0 159 / 160, 0.44
(0.015), 100 / 100 and 98 / 100; where a choice differs the fork's own
two logits are 0.001 to 0.06 apart. The PQ2_0 and
PTQ1_0 files of a model give OKLM the same logits to the
bit, and the Q2_0 file logits within 0.12 of them (the
decoded weights are equal; the activations are rounded to 8 bits before
each product, by both engines). Decoding on 8 threads of a busy machine:
16 t/s (2B), 8.3 (4B), 5.2 (9B). A tiny model of the layout with folded
weights within 1.9e-4 of the plain one's llama.cpp reference in
make test. The 4B's multi-token prediction file
(oklm run --mtp) drafts 92 tokens of which 30 are accepted,
for the plain run's text; the vision projectors are not read, and the
dflash draft model of the 4B and 9B is another
architecture, not read. MiMo-V2.6-Distill-Qwen-9B
(bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF, Q4_K_M: Xiaomi's
fine-tune of Qwen3.5-9B on text written by MiMo-V2.6, the 9B's 427
tensors and a chat template of its own; the MiMo-V2.6 models themselves
are mimo2, see roadmap.md) against
llama.cpp on the CPU: 5 / 5 prompts, 158 / 160 choices, logits within
2.85, three of the five continuations identical and the others parting
where llama.cpp's two logits are at most 0.08 apart; its template
renders as jinja2 renders it on 12 conversations; oklm chat
answers a question and stops, and with --tools writes a
call in the hermes format that is parsed; 5.3 tokens/s on a
shared machine. |
qwen35moe |
(GGUF files only) | inference milestone | Qwen3.6-35B-A3B and its tunes: the qwen35 blocks with a
routed feed-forward (256 experts, 8 a token, softmax weights
renormalized over the chosen ones) and a shared expert under a sigmoid
gate, on the expert code of the other families (indexed products, expert
cache); a file without feed_forward_length is read from its
experts' sizes (hybrid-layers-design.md).
unsloth/Qwen3.6-35B-A3B-GGUF UD-IQ3_XXS (13.2 GB; Q6_K,
IQ3_XXS, IQ2_S, IQ4_XS) against llama.cpp
(test_infer_public when present in
models/qwen3.6-35b-a3b/): 5 / 5 prompts tokenized alike,
158 / 160 greedy choices on its tokens, logits within 5.14, four of five
continuations identical (the fifth parting where llama.cpp's top two are
0.08 apart); 11.7 t/s decoding on 8 threads of a busy machine, the same
text through a 3 GB expert cache. A tiny random model within 2.9e-4 of
llama.cpp in make test. The Q4_K_M files of the three tunes
in the catalogue (21.2 GB each), run the same way on 2026-11-13 and
removed, with no change to the runtime: Uraion Cyber V1.0 5 / 5 prompts,
159 / 160 choices, logits within 4.40, four of five continuations
identical (the fifth parting where llama.cpp's two logits are 0.38
apart), 100 / 100 greedy tokens after a chat turn and after a plain
prompt; Imperum-CybersecurityLLM v1.0 5 / 5, 154 / 160, 2.59, two of
five (0.26 at most), 97 / 100 and 95 / 100; CyberStrike-OffSec-35B (five
shards) 5 / 5, 154 / 160, 2.30, one of five (0.14 at most), 97 / 100 and
98 / 100; the last two have the closest logits of the table and the
flattest choices. oklm chat with each file's template gives
the start of llama.cpp's greedy reply to the same rendered prompt (57,
38 and 10 tokens before a parting). 12.5 t/s on 6 threads. CyberStrike's
multi-token prediction block runs (oklm run --mtp: 195
drafts of which 64 are accepted, 32.8 %, the same 100 tokens, at 8.8 t/s
against 12.5 without). On OpenCL and Vulkan the UD-IQ3_XXS file runs
whole on a 20 GB device (40 of 40 blocks): 156 / 160 choices on
llama.cpp's tokens on each (llama.cpp's two logits at most 0.16 apart
where one differs), logits within 2.74 and 8.35, 65 and 109 tokens a
second against 11.7 on the CPU; the tunes' Q4_K_M files were not run
there (hybrid-layers-design.md).
ffn_gate_up_exps (gate and up in one tensor) is not
read |
bert |
BertModel, RobertaModel, XLMRobertaModel | embeddings (RAG K1) | text encoder, GGUF and Hugging Face checkpoints, CPU and OpenCL
(attention on the CPU); oklm embed,
/v1/embeddings (rag.md).
BAAI/bge-small-en-v1.5: the checkpoint and CompendiumLabs' F32 GGUF
within 8e-14 (1 - cosine) of a float64 numpy encoder over 1000 strings;
the F32 GGUF within 1.0e-5 of llama-embedding (8e-8 with
llama.cpp's tanh GELU), the Q8_0 GGUF within 1.9e-4 (activation
quantization). Conversion (oklm convert) gives the same
vectors; tiny random model in make test. RoBERTa / XLM-R
checkpoints load (positions from pad + 1), no public file checked
yet |
nomic-bert |
NomicBertModel | embeddings (RAG K1) | rope on whole heads, SwiGLU; nomic-embed-text-v1.5: the checkpoint
and nomic-ai's F32 GGUF within 4.5e-13 of the numpy encoder, the GGUF
within 2.5e-7 of llama-embedding; tiny random model in
make test |
| Cross-encoders (rerankers) | BertForSequenceClassification (RobertaForSequenceClassification and XLMRobertaForSequenceClassification accepted, their head names mapped, not checked on a public file) | reranking (RAG K4) | the bert encoder with its classification head
(cls: BERT's pooler or RoBERTa's classifier.dense, then
tanh; cls.output), RANK pooling, the passage of token type
1; oklm rerank, oklm rag search --rerank,
/v1/rerank (rag.md).
cross-encoder/ms-marco-MiniLM-L-6-v2: the checkpoint and
oklm convert's F32 GGUF within 4.0e-6 (logits) of a float64
numpy reference over 990 pairs; that GGUF within 1.3e-3 of
llama-embedding --pooling rank with llama.cpp's tanh GELU
and type 0 (ggml's F16 GELU table: a numpy reference rounding the same
way comes within 4.0e-4); GGUF-my-repo's F16 GGUF (llama.cpp's
converter, which drops the pooler) within 2.5e-4 of
llama-embedding on the same file. Tiny random model in
make test |
| Decoder embedding models | Qwen3ForCausalLM (Qwen3-Embedding) | embeddings (RAG K1) | the Llama forward pass without its head, last-token pooling, `< |
| MoE families | Mixtral (as llama) |
later | Mixtral's GGUFs use the llama MoE path (renormalized
top-2), not checked on a public file |
Tokenizer pre-tokenizers
tokenizer.ggml.pre |
State | Verified on |
|---|---|---|
gpt-2 / default |
supported | GPT-2 tokenizer.json, identical to Hugging Face |
llama-bpe |
supported | Llama 3.2 tokenizer.json |
qwen2 |
supported (with NFC) | Qwen2.5 tokenizer.json |
smollm |
supported | SmolLM2 tokenizer.json and llama.cpp's published GGUF |
SentencePiece (llama model) |
supported | Llama 2 (TinyLlama) tokenizer.json |
olmo |
supported | OLMoE-1B-7B-0125-Instruct GGUF: 400 varied strings (prose, code, numbers, whitespace runs, eleven scripts, emoji, special tokens; 10 735 tokens) identical to llama.cpp's |
every other name llama.cpp knows (88: deepseek-llm,
deepseek-coder, deepseek-v3,
falcon, command-r, starcoder,
...) |
supported, with llama.cpp's steps (tokenizer.md) | llama.cpp's vocabulary tests for falcon, starcoder, refact,
command-r, deepseek-llm, deepseek-coder, phi-3, mpt, qwen35
(tests/tok/test_llama_vocabs.c); the others by their
pattern only |
| a name llama.cpp does not know | falls back to gpt-2 with a warning |
Tokenizers read from tokenizer.json use their own
patterns, so any pattern within the engine's syntax works there.
Tokenization of the inference corpus prompts is identical to
llama.cpp's, with one known difference: TinyLlama's GGUF (TheBloke)
stores its merges with every vocabulary score 0; llama.cpp merges by
score (leftmost first when tied) and OKLM by the merges, as Hugging
Face's tokenizer of the model does, so rare words such as "fibonacci"
split differently there. Without
tokenizer.ggml.add_bos_token, BOS is added for
SentencePiece and for the Llama 3 family of pre-tokenizers, as llama.cpp
does.
Chat templates
Rendered with OKLM's Jinja interpreter, byte-identical to jinja2 in
Hugging Face's environment on six conversations each (with and without
the generation prompt, make test-public):
| Template source | Result |
|---|---|
| SmolLM2, Qwen2.5, Qwen3, Llama 3.2, TinyLlama, OLMoE-1B-7B-0125-Instruct (GGUF files of the corpus) | identical (OLMoE's also in make test,
tests/fixtures/jinja/cases.json) |
Mistral 7B v0.3, Phi-3.5-mini, DeepSeek-R1-Distill-Qwen, Gemma 3,
Granite 3.3, Hermes 3 (pinned tokenizer_config.json) |
identical (Mistral's and Gemma's role checks raise on the same conversations as jinja2) |
DeepSeek-Coder instruct (compiled in: GGUF files of 2023 carry no
template; given to a file without one whose BOS is the sentence start in
full-width bars and whose EOS is <|EOT|>) |
identical to jinja2 on two conversations
(tests/tok/test_tok_units.c) |
The formats --chat-template NAME knows
(vicuna, chatml, llama2,
llama3, mistral, mistral-v3,
gemma, phi3, alpaca,
deepseek-coder, foundation-sec), compiled
in |
each as its definition writes a two-turn conversation with a system
prompt (tests/model/test_model.c); LLaVA 1.5 7B answers in
vicuna and stops on </s> |