OKLMdocumentation v1.0.3
On this page

Compatibility

The live matrix of open-weight model families, tensor types and tokenizer pre-tokenizers OKLM can load and run. "Verified" means checked in CI against a published model file; until the GGUF reader exists, only tensor types can be marked. Verified files are listed in tests/public_models.txt (and the larger ones in tests/public_models_large.txt). make test-public and make test check the canary set of them at every run (tests/canary_models.txt); the others are checked when their code changes (testing.md).

Status: maintained.

Tensor types

Type Dequantize Compute (matmul) Quantize Verified against a public file
F32, F16 yes yes yes yes: SmolLM2 F16 (dequantization identical to gguf-py; F16 conversion identical to llama.cpp)
BF16 yes yes (AVX2) yes the SmolLM2 safetensors checkpoint, run directly
Q5_0 yes yes yes yes: SmolLM2 Q4_K_M
Q8_0 yes yes yes yes: SmolLM2 Q8_0 and Q4_K_M; quantization identical to llama.cpp
Q4_0, Q4_1, Q5_1 yes yes yes no
Q4_K, Q6_K yes yes yes yes: SmolLM2 Q4_K_M; quantization identical to llama-quantize
Q2_K, Q3_K, Q5_K yes yes yes yes: SmolLM2 Q2_K, Q3_K_M, Q5_K_M; quantization identical to llama-quantize
Q4_1, Q5_1 yes yes yes yes: SmolLM2 Q4_0, Q5_K_M (their fallback types)
IQ4_NL, IQ4_XS yes yes (AVX2) yes yes: SmolLM2 IQ4_XS; quantization identical to llama-quantize
TQ1_0, TQ2_0, MXFP4, Q1_0, Q2_0 yes yes (Q2_0: AVX2) yes blocks identical to ggml's; files identical to llama-quantize's; Q2_0: yes, Ternary Bonsai 2 2B, 4B and 9B
PQ2_0, PTQ1_0 (types 142 and 143 of PrismML's llama.cpp fork, not ggml's: formats/quant.md) yes yes (AVX2) yes yes: Ternary Bonsai 2 (2B in both, 4B and 9B in PQ2_0); blocks identical to the fork's
NVFP4 yes yes yes blocks identical to ggml's (llama-quantize's command line does not offer it)
IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S yes yes (own codebook search; llama.cpp reads the files) yes yes: SmolLM2 IQ3_M, and SmolLM2 in every grid type made by llama-quantize (tools/gen_iq_models.sh); dequantization identical to ggml's

The OpenCL and Vulkan backends have matrix-product kernels for every type in the "Compute" column (opencl.md, vulkan.md).

Model families

Family (general.architecture) Hugging Face class Target State
llama LlamaForCausalLM inference milestone inference verified against llama.cpp on SmolLM2-135M (F16, Q8_0, Q4_K_M), Llama-3.2-1B-Instruct and TinyLlama-1.1B-Chat (Q4_K_M), and against a float64 reference (docs/inference.md). DeepSeek-Coder-1.3B-Instruct (TheBloke's Q4_K_M of 2023, CPU): its linear RoPE scaling is under the older key llama.rope.scale_linear (4), read as llama.cpp reads it; tokenization identical on seven prompts, 159 of 160 greedy choices equal on llama.cpp's tokens (logits within 7.3, a shift common to a step's logits; 1.15 once it is removed), 100 greedy tokens identical on a code prompt and on a chat turn; the file has no chat template and gets DeepSeek-Coder's published one. DeepSeek-Coder-6.7B-Instruct (TheBloke's Q4_K_M, the same keys, CPU; run and removed): tokenization identical on seven prompts, 159 of 160 choices (logits within 2.81), four of five continuations identical and the fifth parting where llama.cpp's two tokens are 0.08 apart, 100 greedy tokens identical on a code prompt and on a chat turn in its template; against a float64 pass of the model (tools/infer_ref.py, activations not rounded) 23 of 24 choices and logits within 6.99, llama.cpp 21 of the 22 steps its tokens share with the pass and within 6.98. Both on OpenCL and Vulkan (inference.md): the 1.3B 159 and 157 of 160 choices equal to llama.cpp's (logits within 6.13 and 6.04), the 6.7B 157 and 159 (3.73 and 1.28)
mistral MistralForCausalLM inference milestone converts (as llama); runs through the llama forward pass, no public file in the corpus yet; its chat template renders like jinja2
mistral3 Mistral3ForConditionalGeneration (text part) inference milestone GGUF inference (Mistral Small 3.x, Devstral Small 2): the llama block and its tensors, plus what llama.cpp reads for this name: YaRN (rope.scaling.factor over original_context_length, boundaries 32 / 1) with the magnitude correction divided by yarn_log_multiplier, and an attention temperature (attention.temperature_scale: queries times 1 + scale x ln(floor(position / original context) + 1), so 1 below the original context). Tiny random model with temperature steps at positions 12 and 24 within 1e-6 of a float64 reference on the CPU, OpenCL and Vulkan, llama.cpp within 9e-4 of the same reference (F16 cache). Devstral-Small-2-24B-Instruct-2512 (Unsloth's Q4_K_M, CPU; same tensor names, shapes and types as Precog-24B, published as llama): tokenization identical to llama.cpp's on five prompts (tekken), 158 of 160 greedy choices equal on llama.cpp's tokens, logits within 4.05, four of five 32-token continuations identical, the fifth parting where llama.cpp's top two are 0.08 apart; its 13.7 kB chat template (with strftime_now) renders and oklm chat answers. On OpenCL and Vulkan (RX 7900 XT, every block on the device, inference.md): 159 and 158 of 160 choices equal to llama.cpp's, logits within 4.01 and 4.57, a paged cache and half of the blocks on the CPU. Not checked on the published file: a context beyond 8 192 tokens (where the temperature and the YaRN ramp act); no HF conversion; the realtime speech variant (time-conditioned norm) is refused; the vision tower is a separate projector file, not read
qwen2, qwen3 Qwen2/Qwen3ForCausalLM inference milestone inference verified against llama.cpp on Qwen2.5-0.5B-Instruct Q4_K_M and Qwen3-0.6B Q8_0, and on a tiny random qwen3 model against a float64 reference
gemma, gemma2, gemma3 Gemma*ForCausalLM inference milestone GGUF inference: Gemma-1.1-2B-it, Gemma-2-2B-it (Q4_K_M), Gemma-3-270M-it (Q8_0, also over a prompt past its 512-position window) and Gemma-3-1B-it (Q4_K_M) match a float64 reference within 3e-4 on OpenCL (CPU: within the activation quantization, 90 % of greedy choices or more), and llama.cpp's output to its GELU approximation; tiny gemma2 / gemma3 models in make test. Hugging Face checkpoints load directly and convert (norm weights + 1): checkpoints made from the tiny GGUF fixtures give their logits exactly
phi3 Phi3ForCausalLM inference milestone GGUF inference: Phi-3-mini-4k-instruct (q4) and Phi-3.5-mini-instruct (Q4_K_M, LongRoPE factors and attention factor) match a float64 reference within 1e-4 on OpenCL; on the CPU, Phi-3.5 has steps where quantized activations move the logits by up to 65 (llama.cpp's own CPU logits too); tiny phi3 model in make test. Hugging Face checkpoints load directly and convert (fused projections, LongRoPE factors from config.json)
phi2 PhiForCausalLM inference milestone GGUF inference: phi-2 (Q4_K_M; parallel block, partial NeoX rotation, output bias) within 2e-3 of a float64 reference on OpenCL and within the activation quantization on the CPU and against llama.cpp; tiny phi2 model in make test
granitemoe, granite GraniteMoeForCausalLM, GraniteForCausalLM inference milestone GGUF inference: Granite-3.1-1B-A400M-instruct (Q4_K_M; 32 experts, 8 per token, embedding / attention / residual / logit scales) within the activation quantization of a float64 reference and of llama.cpp; tiny granitemoe model in make test. MoE layers run on the CPU or the OpenCL device, routing included (about 230 t/s decode there, 88 on the CPU); dense granite loads with the same scales (also from a Hugging Face checkpoint), no public file checked; GraniteMoE checkpoints load directly and convert (input_linear split into gate and up experts), tiny checkpoint in make test
gpt2 GPT2LMHeadModel inference milestone inference verified against llama.cpp on GPT-2 (124M) Q4_K_M, and on a tiny random model against a float64 reference
starcoder2 Starcoder2ForCausalLM inference milestone GGUF inference (StarCoder2; llama.cpp also runs Granite code models under this name): the GPT-2 block (LayerNorm with biases, GELU MLP with biases, a final LayerNorm, the head tied to the embeddings) with separate query, key and value projections, grouped keys and values (24 query heads over 2) and NeoX rotary positions instead of the position table. Tiny random model within 8e-5 of a float64 reference on the CPU, OpenCL (all layers, and one) and Vulkan. StarCoder2-3B (second-state's Q4_K_M, CPU): tokenization identical to llama.cpp's on five prompts, 159 of 160 greedy choices equal on llama.cpp's tokens, logits within 1.64, four of five 32-token continuations identical, the fifth parting where llama.cpp's top two are 0.23 apart; 100 greedy tokens identical on def quicksort(arr):; over 200 steps of two 100-token recordings 199 choices equal, logits within 2.9. The file names no pre-tokenizer and gets starcoder (tokenizer.md), identical to Hugging Face's on 400 lines; llama.cpp's own ids differ where a space precedes punctuation. A completion model: no chat template (fill-in-the-middle tokens <fim_prefix>, <fim_suffix>, <fim_middle> are in the vocabulary). On OpenCL and Vulkan (inference.md): 159 and 157 of 160 choices equal to llama.cpp's, logits within 1.72 and 1.60. Not checked: the float64 reference on the published file, the 7B and 15B, the HF conversion (none)
qwen2vl Qwen2VLForConditionalGeneration, Qwen2_5_VLForConditionalGeneration multimodal milestone GGUF inference on the CPU, OpenCL and Vulkan: qwen2 with rotary positions of three parts (time, height, width: rope.dimension_sections), a text rotated exactly as qwen2 (bit for bit on a tiny model), an image's rows by their place in its grid (oklm_eval_embd_grid()), a video's by their place in their pair's grid and the pair's time (oklm_eval_embd_video(); on OpenCL and Vulkan since 2026-11-14, 46 to 48 of the CPU's 48 choices). Qwen2.5-VL 3B and 32B (Q4_K_M) and 72B (IQ1_M) were run on 2026-11-14 on the CPU and both devices: 47 of 48, 16 of 16 and 15 of 16 choices llama.cpp's, the 32B with 61 of its 64 layers on a 20 GiB card and the 72B with 62 of 80. Several pictures in one prompt and pictures up to 4 096 embeddings agree with llama.cpp on the 2B (the 7B's language model further from it as the image rows grow, on its BF16 file too: unexplained). Qwen2-VL-2B-Instruct and Qwen2.5-VL-7B-Instruct (Q4_K_M) against llama.cpp: on a text 48 and 47 of 48 greedy choices equal; with an image's embeddings 46 of 48, parting where llama.cpp's two best logits are 0.09 and 0.40 apart; tiny model within 3e-5 of a float64 pass (6e-5 on a device). An image's rows are rotated where their block runs (oklm_rope_parts()): on OpenCL and Vulkan both files give 46 to 48 of the CPU's 48 choices, logits within 2.5 of the CPU's (0.09 to 0.18 on average); no Hugging Face checkpoint path (multimodal.md)
lfm2 Lfm2ForCausalLM inference milestone GGUF inference (Liquid's LFM2 and its tunes; inference.md has the audit against llama.cpp's graph): short convolution blocks (a gate
lfm2moe Lfm2MoeForCausalLM inference milestone GGUF inference (LFM2.5 8B-A1B and its tunes): the lfm2 stack with routed experts after leading_dense_block_count dense blocks: a sigmoid router (expert_gating_func 2), experts chosen on the probabilities plus exp_probs_b, weights renormalized, SwiGLU experts without biases (DeepSeek V3's routing without groups or shared experts). Tiny random model (8 experts, 2 a token) within 1.8e-5 of a float64 reference on the CPU, OpenCL and Vulkan. Tini-Cybersec-8B-A1B (bartowski's Q4_K_M, 32 experts, 4 a token, CPU): tokenization identical to llama.cpp's on five prompts, 156 of 160 greedy choices equal on llama.cpp's tokens, logits within 4.46, continuations parting where llama.cpp's top two are within 0.1; oklm chat answers. On OpenCL and Vulkan (inference.md): 155 and 155 of 160 choices equal to llama.cpp's, logits within 7.30 and 5.65, the experts on the device or in host memory. Not done: as lfm2; the expert cache and the device expert tier not run on this file; the 24B-A2B
gpt-oss GptOssForCausalLM inference milestone GGUF inference on the CPU, OpenCL and Vulkan (gpt-oss-20b; inference.md has the audit against llama.cpp's graph): attention wider than the residual stream (64 heads of 64 over 8 key / value heads) with biases, NeoX rotation with YaRN, a sliding window of 128 positions on the even blocks, attention sinks (one score per head in every softmax), then 32 experts, 4 per token, with a router bias, weights that are the softmax of the chosen scores, expert biases and the activation min(g, 7) sigmoid(1.702 min(g, 7)) (clamp(u, 7) + 1). Tiny random model (3 blocks, sinks, a window of 4, YaRN, 8 experts with biases and values past the clamp) within 1.6e-5 of a float64 reference; batches equal to single tokens. gpt-oss-20b (Unsloth's Q4_K_M, 11.6 GB, experts in MXFP4, CPU): tokenization identical to llama.cpp's on five prompts (gpt-4o), 153 of 160 greedy choices equal on llama.cpp's tokens, logits within 3.96, two of five 32-token continuations identical, the others parting where llama.cpp's top two are within 0.11; 3.3 tokens/s on this machine's CPU. On OpenCL and Vulkan (the file whole on the RX 7900 XT, the sinks in the attention kernels, the expert biases gathered and added and the activation an op of its own, oklm_sigglu()): 158 and 157 of 160 choices equal to llama.cpp's, mean logit difference 0.08, 126 and 164 tokens/s decoding, prompts at 3313 and 3738 (5.6 and 7.1 on the CPU). The harmony chat form: the 17 kB template renders, a reply runs through its channels to `<
spark2_5 (none: GGUF only) inference milestone GGUF inference on the CPU, OpenCL and Vulkan (Spark-X2.6; inference.md has the audit against llama.cpp's graph): the llama block with one fused attn_qkv (16 query heads of 256 over 4 key / value heads in the 4B, wider than the residual stream), a sliding window of 512 positions on the blocks a list of booleans names (three out of four), NeoX rotation of the whole head with one base in the window blocks and of 64 dimensions with another in the others, the attention output of each head times a sigmoid gate (attn_gate, a projection of the block's normalized input), GELU gating in the feed-forward. Tiny random model (5 blocks, a window of 4, 16 and 8 rotated dimensions) within 2.6e-5 of a float64 reference, 36 / 36 of llama.cpp's own tokens on it; batches and several sequences equal to single tokens. Spark-X2.6-1.7B and 4B (XHToken's Q4_K_M, CPU): tokenization identical to llama.cpp's on five prompts (pre-tokenizer spark2_5, the chat turn with its control tokens among them), 159 and 158 of 160 greedy choices equal on llama.cpp's tokens, logits within 4.53 and 16.2 (two sensitive steps of the 4B; 3.88 with an F32 cache), four and three of five 32-token continuations identical, the others parting where llama.cpp's top two are within 0.03 and 0.20; the 1.7B's BF16 file 160 of 160, logits within 0.35; past the window, after a prompt of 769 tokens, 14 and 16 of 16 choices, logits within 1.01 and 1.42. The template renders (a system block with the reasoning effort, <think> opened by the generation prompt) and oklm chat answers, about 8 tokens/s for the 4B on this machine's CPU. On OpenCL and Vulkan (RX 7900 XT, every block on the device): 158 and 160 of 160 choices equal to llama.cpp's for the 1.7B, 158 and 158 for the 4B, mean logit difference 0.14 to 0.17, a paged cache and half of the blocks on the CPU giving the same choices but for near ties; 216 and 285 tokens/s decoding the 1.7B, 118 and 145 the 4B (22.6 and 9.8 on the CPU). Not done: a published file on the Intel UHD 750, a comparison beyond 800 tokens, the template's tool calls (<tool_call>NAME<arg_key>..., the form of GLM-4.5: not tried on this model), the HF conversion, the trainer (refused: a sliding window)
gemma4 (none: GGUF only) inference milestone GGUF inference on the CPU, OpenCL and Vulkan of Gemma 4 12B, 26B-A4B, E2B and E4B (inference.md has the audit against llama.cpp's graph): window blocks with heads of 256 values and blocks without window with heads of 512 and their own key / value head count, each layer's cache in its block's shape; values taken from the key projection where a block has no attn_v and normalized without weight; unscaled scores; frequency factors on the blocks without window; a scale on each block's output; routed experts beside the dense feed-forward (the router's own input, gate and up joined in ffn_gate_up_exps, a scale per expert). Tiny random models (6 blocks, a window of 4, heads of 8 and 16; 8 experts, 2 a token, joined or apart) within 8.0e-6 of a float64 reference, 36 / 36 of llama.cpp's own tokens on them; batches and several sequences equal to single tokens on the contiguous and the paged cache. Gemma-4-12B-it (Unsloth's Q4_K_M, CPU): tokenization identical to llama.cpp's on five prompts, 148 of 160 greedy choices equal on llama.cpp's tokens, logits within 12.5, one of five continuations identical. Those figures are the distance between two runs of one program: llama.cpp with flash attention against llama.cpp without gives 153 of 160 and 11.7, OKLM's scalar kernels against its AVX kernels 157 of 160 and 14.3; compared step by step on llama.cpp's own hidden states, every step of the 48 blocks agrees within 6e-6 but attention and the GELU, which llama.cpp rounds to half precision (OKLM_ATTN_F16=1, OKLM_GELU_F16=1); two of its blocks alone in F16 (tools/gguf_cut_blocks.py) within 0.25 of llama.cpp on logits of 250 to 650, 96 of 96 choices. The E2B and E4B (an embedding per layer added at each block's end, the last 20 or 18 blocks reading the keys and values of an earlier block and keeping no cache, a feed-forward width per block): a tiny model of that form within 7.3e-6 of the float64 reference; Gemma-4-E2B-it Q4_K_M 160 of 160 choices on llama.cpp's tokens, logits within 8.08, five of five continuations identical; Gemma-4-E4B-it Q4_K_M 156 of 160, within 3.51, one of five. The template renders and oklm chat answers, with and without thinking. Gemma-4-26B-A4B-it (Unsloth's UD-Q4_K_M, 16.9 GB: Q8_0 matrices, Q4_K and Q5_1 experts; about 27 GiB resident): tokenization 5 of 5, 152 of 160 choices, logits within 12.1 (llama.cpp against itself with flash attention: 155 of 160 and 10.1); on llama.cpp's hidden states the router, the eight experts of every token and their sum agree within 4.2e-6 over the 30 blocks. On OpenCL and Vulkan (RX 7900 XT, every block on the device; tests/model/test_gemma4_public.c): on llama.cpp's tokens 148 and 152 of 160 choices for the 12B, 158 and 157 for the E2B, 158 and 157 for the E4B, 155 and 152 for the 26B-A4B, the logits as far from the CPU's as the CPU's are from llama.cpp's (0.24 to 0.68 on average; one step of the 26B-A4B moves by up to 22.8, its best token unchanged); a paged cache, half of the blocks on the CPU and the experts in host memory give the same choices but for near ties; decoding at 53.8 and 68.5 tokens a second for the 12B (3.90 on the CPU), 120.8 and 141.2 for the E2B (17.7), 71.8 and 106.9 for the 26B-A4B (7.94). The blocks of the E2B and E4B that share a cache go to a device together with the blocks they read. Not done: past the 1024-position window on the published file, a quantized cache under the 512-wide heads on Vulkan (those blocks' attention then runs on the CPU), the expert cache and the device expert tier for the 26B-A4B, the image projector, the HF conversion, the trainer. The template's tool calls (`<
muse-glimmer (none: GGUF only) inference milestone the language side of Muse Glimmer 30B as llama.cpp's muse-glimmer.cpp has it (inference.md): normalized embeddings, per-head Q/K norms, a window of 2048 positions on three blocks out of four, rotation in the window blocks only, one sigmoid gate per attention output value, norms after the attention and the feed-forward (epsilon 1e-8), logits scaled then soft-capped. Tiny random model within 4.4e-6 of a float64 reference, 36 / 36 of llama.cpp's own tokens on it. unsloth/Muse-Glimmer-30B-GGUF UD-IQ2_XXS (10.7 GB, the smallest; SHA-256 3e9bd0e6...0875ad9; test_infer_public when present in models/muse-glimmer-30b/) against llama.cpp: 5 / 5 prompts tokenized alike, 157 / 160 greedy choices on its tokens, logits within 5.51, three of five continuations identical and the others parting where llama.cpp's two logits are at most 0.10 apart, 100 / 100 and 96 / 100 choices over 100 greedy tokens after a plain prompt and a chat turn. Its replies are messages with a recipient (a thought to self closed by `<
mimo2 (none: GGUF only) inference milestone, on a tiny model only Xiaomi's MiMo-V2 models as llama.cpp's mimo2.cpp has them (inference.md), on the CPU: one fused attn_qkv with queries and keys of attention.key_length and values of attention.value_length, the window and the count of key heads of each block from two lists (a first block without window, then five window blocks out of six), the window blocks with their own rotation base and attention sinks, the attention output scaled by attention.value_scale, a dense block then experts chosen by sigmoid with a selection bias. Tiny random model within 1.4e-5 of a float64 reference; llama.cpp gives the reference's 36 / 36 tokens on it, logits within 0.062 (its F16 cache). No published file was run: the smallest, MiMo-V2.6-Flash-MOPD Q2_K_S, is 105.9 GB, and the disk does not take it; the metadata and the 472 tensors of the MiMo-V2.6-Flash-RL Q2_K set load and bind, and its chat template renders as jinja2 renders it. Not read: separate attn_q / attn_k / attn_v, the multi-token prediction blocks (mtp-*.gguf, refused), the projector; the model's tool-call form is not parsed; a context asked on OpenCL or Vulkan runs on the CPU. The model's distillation into Qwen3.5-9B is a qwen35 file and runs (the qwen35 row)
clip (mmproj-*.gguf) CLIPVisionModel, SiglipVisionModel multimodal milestone the SigLIP encoder with the idefics3 and gemma3 projectors, the encoders of Qwen2-VL and Qwen2.5-VL at the image's own size with their mergers (qwen2vl_merger, qwen2.5vl_merger: embeddings within 1.3e-5 and 2.9e-6 of a float64 pass of the models' definition, multimodal.md) and the CLIP encoder with mlp (LLaVA 1.5 7B: its 576 embeddings within 7e-6 of a float64 pass of the model's definition, which the pinned llama.cpp's encoder departs from; 47 of 48 greedy choices those of llama.cpp's language model given the same embeddings, multimodal.md; LLaVA 1.6 Mistral 7B: an overview and the tiles of the resolution its file lists that suits the image, cut as the model's own processor cuts them, each tile's 576 embeddings within 8e-6 of the float64 pass, the image's rows those the model's code places from them; 48 of 48 greedy choices those of llama.cpp's language model given the same rows, multimodal.md) (multimodal.md): SmolVLM-256M-Instruct's tile within 0.1 of llama.cpp's embeddings (values up to 109); Gemma 3 4B's 256 embeddings within 2e-5 of a float64 pass and 7e-3 of llama.cpp's, its answer 47 of 48 greedy choices llama.cpp's (multimodal.md); SmolVLM-500M-Instruct and SmolVLM2-2.2B-Instruct (Q8_0, F16 projectors; tiles of 384 pixels in patches of 14) caption the figure set of tests/rag/figures_set (rag.md); tiny random encoders in make test. Other projectors are refused, and an mlp file with chosen feature layers. With the language model on OpenCL or Vulkan, oklm describe and oklm serve compute the blocks of every encoder on the device (multimodal.md), those of Qwen2-VL and Qwen2.5-VL included (multimodal.md)
whisper WhisperForConditionalGeneration multimodal milestone not started
qwen2moe, qwen3moe Qwen2MoeForCausalLM, Qwen3MoeForCausalLM inference milestone GGUF inference: tiny random models (experts with their own hidden size; Qwen2-MoE's sigmoid-gated shared expert and QKV biases; Qwen3-MoE's Q/K norms and renormalized weights) within 2e-5 of a float64 reference on the CPU and OpenCL; Hugging Face checkpoints load directly and convert (per-expert matrices stacked), tiny checkpoints in make test; no public file checked (the smallest are 8 GB and more)
deepseek DeepseekForCausalLM inference milestone GGUF inference (DeepSeek-MoE: dense leading layers, softmax routing scaled by expert_weights_scale, shared experts run as one ungated FFN): tiny random model within 2e-5 of a float64 reference on the CPU and OpenCL; no public file checked (16B)
deepseek2 DeepseekV2ForCausalLM, DeepseekV3ForCausalLM, Glm4MoeLiteForCausalLM inference milestone GGUF inference: multi-head latent attention in llama.cpp's absorbed form (one shared cache head of kv_lora + rope, values read from it; compressed or direct queries), files with attn_k_b / attn_v_b and older ones with attn_kv_b (split at load), YaRN with the score multiplier, softmax or sigmoid routing, the V3 selection bias and expert groups, the multi-token prediction block (left out of the main stack; usable as the speculative draft, oklm run --mtp); tiny V2-Lite-like and V3-like models within 2e-5 of a float64 reference on the CPU and OpenCL, the V3-like one within llama.cpp's MoE precision of llama.cpp itself. DeepSeek-Coder-V2-Lite-Instruct IQ4_XS (bartowski, an older attn_kv_b file): greedy choices identical to llama.cpp's on short prompts (32 / 32 steps, CPU and OpenCL) and on a 422-token prompt (OpenCL 8 / 8, CPU 6 / 8: the two others where llama.cpp's top two are within 0.5); 102 t/s decode on OpenCL, 16 on the CPU. GLM-4.7-Flash (Glm4MoeLiteForCausalLM, written as deepseek2 with the GLM vocabulary; unsloth Q4_K_M, CPU): tokenization identical to llama.cpp's on 435 strings (glm4 pre-tokenizer, the role, thinking and tool tokens), its template the jinja2-checked fixture; greedy output identical to llama.cpp's on four prompts (32 / 32), a fifth parting at step 0 where llama.cpp's top two are 0.11 apart; perplexity 44.42 against llama-perplexity's 43.28 (512-token windows, 1530 tokens; chunks 1 to 5 within 0.7 %, the last one a window where the model, without [gMASK]<sop>, writes logits near 150); a glm45 tool call through oklm chat --tools. Files without expert_gating_func (ggml-org's) route by sigmoid, as llama.cpp decides from the shape. No published file carries the MTP block, so --mtp is unchecked on it. 18.8 GB resident; with 8 threads on a loaded machine (load about 33 on 16 cores) 28 t/s prompt (653 tokens) against llama.cpp's 43 (pp512), and decode collapsed to 1.2 t/s against llama.cpp's 9.5 (2026-11-07, before the CPU's own worker team and the free-core cap). Re-measured 2026-11-11, decode with 4 / 8 threads: 11.2 / 11.1 t/s on a quiet machine (llama.cpp 11.3 / 11.1), 5.5 / 6.0 beside 16 spinning processes (llama.cpp 2.6 / 0.24). The agent evals through oklm serve: 21 of 25 (agent-loop.md), its 163 glm45 calls all parsed
glm4moe Glm4MoeForCausalLM inference milestone GGUF inference (GLM-4.5 / 4.5-Air / 4.6 / 4.7): grouped-query attention with QKV biases, optional Q/K norms, NeoX rotation of half of each head, DeepSeek V3's MoE (sigmoid routing with the selection bias, dense leading layers, shared experts, renormalized and scaled weights), the norm before the FFN stored as post_attention_norm, the nextn block left out of the main stack and usable as the speculative draft (oklm run --mtp); tiny random models (with Q/K norms and a block, and with neither) within 3e-5 of a float64 reference on the CPU, llama.cpp (CPU, F32 cache) within 1.4e-5 of the same reference with the same greedy choices. Hugging Face checkpoints load directly and convert (per-expert matrices stacked, the nextn layer kept), tiny checkpoint in make test, its conversion loaded by llama.cpp. No public file checked: the smallest, GLM-4.5-Air (106B-A12B), is about 65 GB at Q4
glm4 Glm4ForCausalLM inference milestone GGUF inference (GLM-4-0414 9B / 32B, GLM-Z1): norms after attention and after the FFN as well as before, gate and up in one ffn_up, QKV biases, normal-pair rotation of half of each head; a tiny random model within 6e-6 of a float64 reference on the CPU, llama.cpp (CPU, F32 cache) within 1e-5 of the same reference with the same greedy choices. Hugging Face checkpoints load directly and convert, tiny checkpoint in make test. GLM-4-9B-0414 Q4_K_M (bartowski, sha256 2b95f0b8...216a; test_infer_public when present in models/glm4-9b/): tokenization identical to llama.cpp's on its five prompts (a [gMASK]<sop><|user|> chat turn among them), 157 / 160 greedy choices on llama.cpp's tokens on the CPU and 158 / 160 on OpenCL, logits within 0.97 / 0.85; run freely, three of five continuations identical, the two others parting where llama.cpp's top two are within 0.16; perplexity 23.78 against llama-perplexity's 23.81 (Tiny Shakespeare's first 30 KB, 512-token windows). Speed (2026-11-11, quiet machine, 8 threads): 84 t/s prompt (497 tokens) and 4.7 decode on the CPU against llama.cpp's 60 (pp512) and 4.8; 2 074 and 61 on OpenCL. Its tools follow GLM-4's own convention, the glm4 format (the function's name opening the turn after <|assistant|>, then its arguments as JSON, results in an <|observation|> turn): two calls and their results through oklm chat --tools; the agent evals through oklm serve 3 of 25 (agent-loop.md), its 26 calls all parsed, most turns ending on a plan in prose
olmoe OlmoeForCausalLM inference milestone GGUF inference (Q/K RMS norms over the whole projections, one weight as wide as all the heads; 64 experts, 8 per token, softmax weights not renormalized; no shared expert; NeoX rotation), through the Llama forward pass and the MoE path of the other families (indexed expert products, expert cache): tiny random model within 7.8e-6 of a float64 reference on the CPU and within llama.cpp's own distance from that reference (0.017, its F16 KV cache; the tiny qwen2moe 0.014) of llama.cpp. OLMoE-1B-7B-0125-Instruct Q4_K_M (allenai): tokenizer identical to llama.cpp's on 400 strings, chat template identical to jinja2, on llama.cpp's tokens 157 / 160 greedy choices equal (max logit difference 2.04), the free-running continuations parting only where llama.cpp's top two are within 0.12; perplexity on the last 30 KB of Tiny Shakespeare 2.1963 (llama.cpp 2.2015); CPU, 4 threads on a shared machine: 97 t/s prompt (682 tokens), 13.3 t/s decode (llama-bench beside it: 110 and 20.4); the expert cache at 50 % and 25 % gives the resident run's tokens (moe.md). Hugging Face checkpoints load directly and convert (per-expert matrices stacked), tiny checkpoint in make test. OpenCL (2026-11-08): the tiny model within 6.6e-6 of the float64 reference with every layer on the device, one layer on the CPU, and the routed experts of the first or of every layer on the CPU (make test); OLMoE-1B-7B against llama.cpp 157 / 160 choices (max logit difference 1.84, parting where llama.cpp's margin is 0.04), against OKLM's CPU 154 to 158 / 160 over all layers, --gpu-layers 8, --n-cpu-moe 8 and all, with the expert cache and with --expert-gpu (every differing choice a near tie, the CPU's margin at most 0.26), perplexity 2.1958 to 2.2015 against the CPU's 2.1963 (inference.md)
qwen35 (GGUF files only) inference milestone Qwen3.5-9B (unsloth/Qwen3.5-9B-GGUF, Q4_K_M): three Gated DeltaNet blocks (a short conv, a state per head updated with a decay and a write strength per token, a gated norm) then a full-attention block whose queries carry an output gate, heads of 256 with 64 dimensions rotated, dense SwiGLU, the qwen35 pre-tokenizer and a 248 320-token vocabulary (hybrid-layers-design.md). Against llama.cpp on the published file: 5 / 5 prompts tokenized alike, 159 / 160 greedy choices on its tokens, the mean logit difference 0.116 and the largest 13.8 (single low logits at four steps of a repeated sentence; 0.4 to 0.7 a step otherwise, and the same file moves its own logits as much between an F16 and an F32 cache, in llama.cpp as in OKLM: hybrid-layers-design.md), 100 / 100 greedy tokens after a plain prompt and 26 after a chat turn before a choice llama.cpp held by 0.44 (97 / 100 on its tokens), a prompt of 296 tokens in batches of 64, two sequences in one pass, a sequence through a file, a copy and a cut (tests/model/test_qwen35_public.c); a tiny random model of the layout within 2e-4 of llama.cpp in make test. On OpenCL and Vulkan (RX 7900 XT, the same tests, every block on the device, the linear blocks' states in device memory): 158 / 160 choices on llama.cpp's tokens on each, the mean logit difference 0.122 and the largest 13.7, 157 and 159 of 160 choices the CPU's, a paged cache and half of the blocks on the CPU giving llama.cpp's tokens; 83 and 104 tokens a second against 5.5 on the CPU (hybrid-layers-design.md). The trainer does not know the layout, the vision projector is not read; Qwen3.8-27B (unsloth/Qwen3.8-27B-GGUF, UD-Q4_K_M: 64 blocks at width 5120, 48 value heads over 16 key heads, IQ4_XS and IQ4_NL among its types) the same way: 5 / 5 prompts, 160 / 160 choices, 3.22 at most, the five continuations identical; its multi-token prediction block (oklm run --mtp) drafts 45 tokens of which 20 are accepted, for the same text. Cyber-Ornith 1.5 9B (DuoNeural/Cyber-Ornith-1.5-9B-OBLITERATED-GGUF, Q4_K_M: a security tune with the 9B's 427 tensors) the same way: 5 / 5 prompts, 158 / 160 choices, the mean logit difference 0.089 and the largest 3.58 (one low logit), three of the five continuations identical and the others parting where llama.cpp's two logits are at most 0.25 apart, 100 / 100 greedy tokens after a plain prompt and after a chat turn, a prompt of 296 tokens in batches of 64 (16 / 16 choices, 1.03 at most); oklm chat with the file's template, which opens a thought by default, gives the 56 tokens of llama.cpp's greedy answer to the same rendered prompt, and a direct answer with --template-vars '{"enable_thinking": false}'. BugTraceAI-CORE-Ultra 27B (BugTraceAI/BugTraceAI-CORE-Ultra-27B-Q4, Q4_K_S with Q5_K among its types: a security tune of 64 blocks at width 5120, 851 tensors, no multi-token prediction block; SHA-256 50170a34...563ff6e2) the same way: 5 / 5 prompts, 157 / 160 choices, 3.59 at most, three of the five continuations identical and the others parting where llama.cpp's two logits are at most 0.08 apart; over 100 greedy tokens after a chat turn and after a plain prompt, 99 / 100 choices each on llama.cpp's tokens (logits within 5.57), the texts parting at steps 44 and 72 where its two logits are 0.008 and 0.047 apart; oklm chat with the file's template opens a thought whose first 80 tokens are llama.cpp's greedy ones for the same rendered prompt, then answers. Ternary Bonsai 2 (prism-ml/Ternary-Bonsai-2-2B-gguf, -4B-gguf, -9B-gguf: Qwen3.5 with ternary weights stored in a Walsh-Hadamard rotated basis, the prism.hadamard.* metadata, which OKLM undoes on the input of each product as llama.cpp does, hybrid-layers-design.md), in its three packings, on the CPU (tests/model/test_bonsai_public.c): every file's five prompts tokenized alike. Q2_0 against the pinned llama.cpp: 2B 160 / 160 greedy choices on its tokens, logits within 0.30 (0.016 on average), the five continuations and 100 greedy tokens after a chat turn and after plain text identical; 4B 160 / 160, 0.32 (0.016), the same continuations; 9B 159 / 160, 0.29 (0.015), 100 / 100 and 99 / 100 choices over the 100 tokens. PQ2_0 and PTQ1_0 against PrismML's llama.cpp fork (commit 6447906b, the only reader of these two types beside OKLM): 2B PQ2_0 159 / 160, 0.32 (0.017), 100 / 100 twice; 2B PTQ1_0 159 / 160, 0.29 (0.016), 97 / 100 and 100 / 100; 4B PQ2_0 159 / 160, 0.75 (0.016), 100 / 100 and 99 / 100; 9B PQ2_0 159 / 160, 0.44 (0.015), 100 / 100 and 98 / 100; where a choice differs the fork's own two logits are 0.001 to 0.06 apart. The PQ2_0 and PTQ1_0 files of a model give OKLM the same logits to the bit, and the Q2_0 file logits within 0.12 of them (the decoded weights are equal; the activations are rounded to 8 bits before each product, by both engines). Decoding on 8 threads of a busy machine: 16 t/s (2B), 8.3 (4B), 5.2 (9B). A tiny model of the layout with folded weights within 1.9e-4 of the plain one's llama.cpp reference in make test. The 4B's multi-token prediction file (oklm run --mtp) drafts 92 tokens of which 30 are accepted, for the plain run's text; the vision projectors are not read, and the dflash draft model of the 4B and 9B is another architecture, not read. MiMo-V2.6-Distill-Qwen-9B (bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF, Q4_K_M: Xiaomi's fine-tune of Qwen3.5-9B on text written by MiMo-V2.6, the 9B's 427 tensors and a chat template of its own; the MiMo-V2.6 models themselves are mimo2, see roadmap.md) against llama.cpp on the CPU: 5 / 5 prompts, 158 / 160 choices, logits within 2.85, three of the five continuations identical and the others parting where llama.cpp's two logits are at most 0.08 apart; its template renders as jinja2 renders it on 12 conversations; oklm chat answers a question and stops, and with --tools writes a call in the hermes format that is parsed; 5.3 tokens/s on a shared machine.
qwen35moe (GGUF files only) inference milestone Qwen3.6-35B-A3B and its tunes: the qwen35 blocks with a routed feed-forward (256 experts, 8 a token, softmax weights renormalized over the chosen ones) and a shared expert under a sigmoid gate, on the expert code of the other families (indexed products, expert cache); a file without feed_forward_length is read from its experts' sizes (hybrid-layers-design.md). unsloth/Qwen3.6-35B-A3B-GGUF UD-IQ3_XXS (13.2 GB; Q6_K, IQ3_XXS, IQ2_S, IQ4_XS) against llama.cpp (test_infer_public when present in models/qwen3.6-35b-a3b/): 5 / 5 prompts tokenized alike, 158 / 160 greedy choices on its tokens, logits within 5.14, four of five continuations identical (the fifth parting where llama.cpp's top two are 0.08 apart); 11.7 t/s decoding on 8 threads of a busy machine, the same text through a 3 GB expert cache. A tiny random model within 2.9e-4 of llama.cpp in make test. The Q4_K_M files of the three tunes in the catalogue (21.2 GB each), run the same way on 2026-11-13 and removed, with no change to the runtime: Uraion Cyber V1.0 5 / 5 prompts, 159 / 160 choices, logits within 4.40, four of five continuations identical (the fifth parting where llama.cpp's two logits are 0.38 apart), 100 / 100 greedy tokens after a chat turn and after a plain prompt; Imperum-CybersecurityLLM v1.0 5 / 5, 154 / 160, 2.59, two of five (0.26 at most), 97 / 100 and 95 / 100; CyberStrike-OffSec-35B (five shards) 5 / 5, 154 / 160, 2.30, one of five (0.14 at most), 97 / 100 and 98 / 100; the last two have the closest logits of the table and the flattest choices. oklm chat with each file's template gives the start of llama.cpp's greedy reply to the same rendered prompt (57, 38 and 10 tokens before a parting). 12.5 t/s on 6 threads. CyberStrike's multi-token prediction block runs (oklm run --mtp: 195 drafts of which 64 are accepted, 32.8 %, the same 100 tokens, at 8.8 t/s against 12.5 without). On OpenCL and Vulkan the UD-IQ3_XXS file runs whole on a 20 GB device (40 of 40 blocks): 156 / 160 choices on llama.cpp's tokens on each (llama.cpp's two logits at most 0.16 apart where one differs), logits within 2.74 and 8.35, 65 and 109 tokens a second against 11.7 on the CPU; the tunes' Q4_K_M files were not run there (hybrid-layers-design.md). ffn_gate_up_exps (gate and up in one tensor) is not read
bert BertModel, RobertaModel, XLMRobertaModel embeddings (RAG K1) text encoder, GGUF and Hugging Face checkpoints, CPU and OpenCL (attention on the CPU); oklm embed, /v1/embeddings (rag.md). BAAI/bge-small-en-v1.5: the checkpoint and CompendiumLabs' F32 GGUF within 8e-14 (1 - cosine) of a float64 numpy encoder over 1000 strings; the F32 GGUF within 1.0e-5 of llama-embedding (8e-8 with llama.cpp's tanh GELU), the Q8_0 GGUF within 1.9e-4 (activation quantization). Conversion (oklm convert) gives the same vectors; tiny random model in make test. RoBERTa / XLM-R checkpoints load (positions from pad + 1), no public file checked yet
nomic-bert NomicBertModel embeddings (RAG K1) rope on whole heads, SwiGLU; nomic-embed-text-v1.5: the checkpoint and nomic-ai's F32 GGUF within 4.5e-13 of the numpy encoder, the GGUF within 2.5e-7 of llama-embedding; tiny random model in make test
Cross-encoders (rerankers) BertForSequenceClassification (RobertaForSequenceClassification and XLMRobertaForSequenceClassification accepted, their head names mapped, not checked on a public file) reranking (RAG K4) the bert encoder with its classification head (cls: BERT's pooler or RoBERTa's classifier.dense, then tanh; cls.output), RANK pooling, the passage of token type 1; oklm rerank, oklm rag search --rerank, /v1/rerank (rag.md). cross-encoder/ms-marco-MiniLM-L-6-v2: the checkpoint and oklm convert's F32 GGUF within 4.0e-6 (logits) of a float64 numpy reference over 990 pairs; that GGUF within 1.3e-3 of llama-embedding --pooling rank with llama.cpp's tanh GELU and type 0 (ggml's F16 GELU table: a numpy reference rounding the same way comes within 4.0e-4); GGUF-my-repo's F16 GGUF (llama.cpp's converter, which drops the pooler) within 2.5e-4 of llama-embedding on the same file. Tiny random model in make test
Decoder embedding models Qwen3ForCausalLM (Qwen3-Embedding) embeddings (RAG K1) the Llama forward pass without its head, last-token pooling, `<
MoE families Mixtral (as llama) later Mixtral's GGUFs use the llama MoE path (renormalized top-2), not checked on a public file

Tokenizer pre-tokenizers

tokenizer.ggml.pre State Verified on
gpt-2 / default supported GPT-2 tokenizer.json, identical to Hugging Face
llama-bpe supported Llama 3.2 tokenizer.json
qwen2 supported (with NFC) Qwen2.5 tokenizer.json
smollm supported SmolLM2 tokenizer.json and llama.cpp's published GGUF
SentencePiece (llama model) supported Llama 2 (TinyLlama) tokenizer.json
olmo supported OLMoE-1B-7B-0125-Instruct GGUF: 400 varied strings (prose, code, numbers, whitespace runs, eleven scripts, emoji, special tokens; 10 735 tokens) identical to llama.cpp's
every other name llama.cpp knows (88: deepseek-llm, deepseek-coder, deepseek-v3, falcon, command-r, starcoder, ...) supported, with llama.cpp's steps (tokenizer.md) llama.cpp's vocabulary tests for falcon, starcoder, refact, command-r, deepseek-llm, deepseek-coder, phi-3, mpt, qwen35 (tests/tok/test_llama_vocabs.c); the others by their pattern only
a name llama.cpp does not know falls back to gpt-2 with a warning

Tokenizers read from tokenizer.json use their own patterns, so any pattern within the engine's syntax works there.

Tokenization of the inference corpus prompts is identical to llama.cpp's, with one known difference: TinyLlama's GGUF (TheBloke) stores its merges with every vocabulary score 0; llama.cpp merges by score (leftmost first when tied) and OKLM by the merges, as Hugging Face's tokenizer of the model does, so rare words such as "fibonacci" split differently there. Without tokenizer.ggml.add_bos_token, BOS is added for SentencePiece and for the Llama 3 family of pre-tokenizers, as llama.cpp does.

Chat templates

Rendered with OKLM's Jinja interpreter, byte-identical to jinja2 in Hugging Face's environment on six conversations each (with and without the generation prompt, make test-public):

Template source Result
SmolLM2, Qwen2.5, Qwen3, Llama 3.2, TinyLlama, OLMoE-1B-7B-0125-Instruct (GGUF files of the corpus) identical (OLMoE's also in make test, tests/fixtures/jinja/cases.json)
Mistral 7B v0.3, Phi-3.5-mini, DeepSeek-R1-Distill-Qwen, Gemma 3, Granite 3.3, Hermes 3 (pinned tokenizer_config.json) identical (Mistral's and Gemma's role checks raise on the same conversations as jinja2)
DeepSeek-Coder instruct (compiled in: GGUF files of 2023 carry no template; given to a file without one whose BOS is the sentence start in full-width bars and whose EOS is <|EOT|>) identical to jinja2 on two conversations (tests/tok/test_tok_units.c)
The formats --chat-template NAME knows (vicuna, chatml, llama2, llama3, mistral, mistral-v3, gemma, phi3, alpaca, deepseek-coder, foundation-sec), compiled in each as its definition writes a two-turn conversation with a system prompt (tests/model/test_model.c); LLaVA 1.5 7B answers in vicuna and stops on </s>