OKLMdocumentation v1.0.3
On this page

Benchmarks

Measured performance of OKLM's kernels, with the machine, the command and the date of each run, so regressions and improvements can be tracked. Numbers are informational; nothing gates on them.

Status: maintained.

Against llama.cpp

sh tools/bench_vs_llama.sh measures OKLM and llama.cpp on the same GGUF files the way llama-bench does: pp512, 512 random tokens in one batch from an empty cache, and tg128, 128 tokens decoded one at a time from an empty cache, each the best of three runs after one warm-up. OKLM runs tests/model/bench_llama.c (make build/release/bin/tests/model/bench_llama, threads from OKLM_THREADS), llama.cpp its llama-bench (-o json, the best of the per-run samples), one process at a time, each run waiting until at most 1.5 hardware threads are busy (/proc/stat over two seconds). The CPU rows use the CPU build of the pinned llama.cpp (build/llama.cpp/build, GGML_NATIVE, AVX-512, its repacked x86 weights on); the GPU rows OKLM's OpenCL backend (its defaults: since round 4 the 8-bit matrix units for the quantized prompt products, the F32 tile kernels before; since round 7 the K-quants' larger decode products on 8-bit dot products; then OKLM_CL_WMMA=1) against llama.cpp's HIP build with every layer on the GPU (-ngl 99), built for this comparison:

cmake -B build-hip -DGGML_HIP=ON -DGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF \
  -Dhip_DIR=/usr/lib/x86_64-linux-gnu/cmake/hip -Dhip-lang_DIR=/usr/lib/x86_64-linux-gnu/cmake/hip-lang \
  -Dhipblas_DIR=/usr/lib/x86_64-linux-gnu/cmake/hipblas -Drocblas_DIR=/usr/lib/x86_64-linux-gnu/cmake/rocblas \
  -DCMAKE_PREFIX_PATH=/usr -DCMAKE_HIP_FLAGS=-DHIP_ENABLE_WARP_SYNC_BUILTINS
cmake --build build-hip --target llama-bench -j16      # 215 s

(the Debian ROCm 6.4 packages; /opt/rocm there holds no hipcc, and HIP 6.4 hides __ballot_sync behind that define). The script's variables: THREADS (default "1 4 8 16"), GPU, REPEATS, PP, TG, LLAMA, GPU_LLAMA, LOAD_MAX; arguments: GGUF files (default: the models below that exist).

Current standing (after round 11, 2026-11-12)

The block closed after round 9 (the roadmap's 4); round 10 took the two CPU decode rows left. Ratios are OKLM / llama.cpp, t/s; the tables to read are after rounds 7 and 8 (CPU), then round 9 (CPU) for Llama-3.2-1B, round 10 (CPU) for SmolLM2 and GLM-4.7-Flash decoding, round 9 (GPU), and round 11 (GPU) for the prompts of Llama-3.2-1B and Qwen3-8B Q4_K_M.

  • CPU prompts: ahead on every row (1.02 to 1.94; Qwen3-8B Q8_0 1.71 to 1.80, the MoE models 1.65 to 1.94).
  • CPU decoding: level or ahead from four threads on every model (1.00 to 1.09; SmolLM2 1.07 to 1.09 and GLM-4.7-Flash 1.02 at four and eight threads since round 10, the other models' rows not measured again); one thread 1.09 to 1.56.
  • GPU prompts: ahead on six of eight models (SmolLM2 1.15 to 1.37, Qwen3-8B Q8_0 1.65, GLM-4.7-Flash 1.04, Granite 1.06); behind on Llama-3.2-1B (0.84 since round 11, 0.77 before), Qwen3-8B Q4_K_M (0.86, 0.80 before) and OLMoE (0.85).
  • GPU decoding: SmolLM2 ahead (1.04 to 1.17); the dense models 0.90 to 0.95; the MoE models 0.77 (Granite) to 0.83 (GLM-4.7-Flash). Since round 10 (2026-11-12): Granite 1.09, OLMoE 0.88, GLM-4.7-Flash 0.89.
  • Under load: with spinning processes beside both, OKLM now degrades gently (decoding on a busy machine).

The gaps left and why they are not pursued are in the roadmap's known speed gaps.

First measure (2026-11-08)

i7-11700K (8 cores, 16 hardware threads, AVX-512 VNNI), 128 GB DDR4, Radeon gfx1100 (20 GB), nothing else running (load average 3.3 from the earlier build when it started, 0.04 hardware threads busy). OKLM at 665e6ae, llama.cpp v0.8.0. t/s, OKLM / llama.cpp; ratio above 1: OKLM faster.

model backend threads pp512 OKLM / llama.cpp ratio tg128 OKLM / llama.cpp ratio
SmolLM2-135M Q8_0 cpu 1 485 / 468 1.04 82.4 / 124 0.66
cpu 4 1798 / 1680 1.07 164 / 190 0.86
cpu 8 3234 / 2950 1.10 166 / 187 0.89
cpu 16 3719 / 3043 1.22 154 / 177 0.87
opencl - 23637 / 29159 0.81 417 / 381 1.09
opencl wmma - 31296 / 29159 1.07 408 / 381 1.07
SmolLM2-135M Q4_K_M cpu 1 507 / 449 1.13 85.3 / 128 0.67
cpu 4 1886 / 1580 1.19 212 / 253 0.84
cpu 8 3359 / 2753 1.22 217 / 252 0.86
cpu 16 3774 / 2815 1.34 197 / 234 0.84
opencl - 22737 / 31372 0.72 371 / 384 0.97
opencl wmma - 29501 / 31372 0.94 369 / 384 0.96
Llama-3.2-1B Q4_K_M cpu 1 98.4 / 111 0.88 18.8 / 22.3 0.85
cpu 4 344 / 404 0.85 35.4 / 36.0 0.98
cpu 8 650 / 713 0.91 35.1 / 35.1 1.00
cpu 16 671 / 679 0.99 33.0 / 33.1 1.00
opencl - 5483 / 17839 0.31 348 / 412 0.84
opencl wmma - 9447 / 17839 0.53 356 / 412 0.86
Qwen3-8B Q4_K_M cpu 1 15.1 / 15.6 0.97 3.31 / 3.91 0.85
cpu 4 53.8 / 56.5 0.95 6.19 / 6.24 0.99
cpu 8 86.6 / 91.5 0.95 6.11 / 6.11 1.00
cpu 16 88.9 / 89.1 1.00 5.78 / 5.86 0.99
opencl - 855 / 3253 0.26 83.3 / 106 0.78
opencl wmma - 1386 / 3253 0.43 83.3 / 106 0.78
Qwen3-8B Q8_0 cpu 1 16.3 / 9.47 1.72 1.69 / 2.66 0.64
cpu 4 58.8 / 33.2 1.77 3.56 / 3.73 0.95
cpu 8 94.0 / 56.0 1.68 3.57 / 3.60 0.99
cpu 16 101 / 57.4 1.77 3.39 / 3.43 0.99
opencl - 876 / 1633 0.54 65.7 / 74.3 0.88
opencl wmma - 1504 / 1633 0.92 65.2 / 74.3 0.88
Granite-3.1-1B-A400M Q4_K_M cpu 1 134 / 139 0.96 50.8 / 59.6 0.85
cpu 4 503 / 498 1.01 96.4 / 97.7 0.99
cpu 8 908 / 834 1.09 97.5 / 98.5 0.99
cpu 16 830 / 854 0.97 89.6 / 94.2 0.95
opencl - 4872 / 15421 0.32 242 / 365 0.66
opencl wmma - 5480 / 15421 0.36 241 / 365 0.66
OLMoE-1B-7B Q4_K_M cpu 1 56.3 / 56.2 1.00 19.8 / 23.0 0.86
cpu 4 201 / 201 1.00 37.4 / 38.1 0.98
cpu 8 372 / 318 1.17 37.5 / 37.9 0.99
cpu 16 391 / 347 1.13 35.4 / 36.3 0.98
opencl - 2920 / 10299 0.28 265 / 362 0.73
opencl wmma - 3895 / 10299 0.38 259 / 362 0.71
GLM-4.7-Flash Q4_K_M cpu 1 15.3 / 17.3 0.89 5.85 / 6.86 0.85
cpu 4 54.9 / 59.2 0.93 11.2 / 11.5 0.97
cpu 8 88.2 / 92.7 0.95 11.2 / 11.5 0.97
cpu 16 94.6 / 88.7 1.07 10.6 / 10.8 0.98
opencl - 785 / 2090 0.38 67.9 / 88.2 0.77
opencl wmma - 986 / 2090 0.47 68.2 / 88.2 0.77

Reading it:

  • CPU prompts: level or ahead on most rows (Q8_0 1.7 times llama.cpp, SmolLM2 up to 1.34, OLMoE 1.13 to 1.17 from 8 threads); behind on Llama-3.2-1B Q4_K_M (0.85 to 0.91 below 16 threads), GLM-4.7-Flash (0.89 to 0.95 below 16) and Qwen3-8B Q4_K_M (0.95 to 0.97), at every thread count alike: the kernels, not the thread team.
  • CPU decoding: behind at one thread on every model (0.64 to 0.86), level from four threads (0.95 to 1.00) on everything but SmolLM2 (0.84 to 0.89), where both are bound by memory. The one-thread gap is the matrix-vector kernel: on the file layout, each output row is one dot product whose blocks add into one float in block order (the order that keeps every kernel's bits equal to the scalar one), so one thread waits on that chain; the packed copies the prompt kernels read (eight rows side by side, the same bits) were not used for one-row products.
  • GPU: prompts are the largest gap, 0.26 to 0.54 with the F32 tile kernels and 0.36 to 0.94 with the matrix units (0.92 on Qwen3-8B Q8_0, 1.07 on SmolLM2 Q8_0); decoding 0.66 (Granite, MoE) to 0.97, the MoE models worst (0.66 to 0.77).

Round 1: decoding on the packed copy with one or two threads

One-row products read the packed copy (eight rows side by side) with one thread, and with two when it is at most 10 % larger than the file layout (Q8_0 1.06 times, Q4_K 1.06; Q6_K 1.31 and Q5_0 1.64 stay on the file layout). The same bits (test_matmul checks it at one and two threads; the logits of 100 prompt and 16 decoded tokens hash the same). tg128 at one thread, before / after (llama.cpp): SmolLM2 Q8_0 82.4 / 133 (123), SmolLM2 Q4_K_M 85.3 / 142 (128), Llama-3.2-1B 18.8 / 24.5 (22.4), Qwen3-8B Q4_K_M 3.31 / 4.35 (3.92), Qwen3-8B Q8_0 1.69 / 3.11 (2.66), Granite 50.8 / 52.1 (59.6), OLMoE 19.8 / 21.1 (23.1), GLM-4.7-Flash 5.85 / 6.45 (6.90); prompts unchanged. Two threads, before / after: Llama-3.2-1B 31.4 / 32.2, SmolLM2 Q8_0 136 / 162, SmolLM2 Q4_K_M 150 / 155. Measured and left out: every one-row product on the packed copy at three threads and more (Llama-3.2-1B 35.6 to 31.6 at three, 35.3 to 30.1 at eight: the larger Q6_K copy, bound by memory), and a four-row kernel on the file layout with the four sums side by side (Q8_0 +6 % at one thread, Q4_K 18.9 to 12.8). Four threads and more run the code they ran before. The MoE models' expert products have no packed copy, which is what is left of their one-thread gap.

Round 2: GPU prompts on larger matrix-unit tiles, tiled attention from 128 keys, one-column products

Three changes to the OpenCL backend (opencl.md): w2_<type> replaces wm_<type> (OKLM_CL_WMMA) for prompts of 64 columns and more (activations rounded to F16 once per product, tiles of 64 rows by 128 columns, 64 values of k a step); prompts with a float cache take the tiled attention from 128 keys instead of 1024 (OKLM_CL_ATT_KEYS); one-column products (decode, MoE indexed products) run mv_<type>, the mm_<type> kernel without the reduction of seven absent columns (the same bits). Greedy tokens after a 190-token prompt: the same before and after on the seven models, and the same with and without the matrix units (top logit within 0.0023 of the 1024-key attention, Granite 0.064). GPU rows, before / after:

model pp512 F32 tiles pp512 matrix units tg128
SmolLM2-135M Q8_0 23637 / 26925 31296 / 37474 417 / 430
SmolLM2-135M Q4_K_M 22737 / 25557 29501 / 35484 371 / 387
Llama-3.2-1B Q4_K_M 5483 / 5910 9447 / 14157 348 / 369
Qwen3-8B Q4_K_M 855 / 895 1386 / 2624 83.3 / 81.8
Qwen3-8B Q8_0 876 / 914 1504 / 2775 65.7 / 65.2
Granite-3.1-1B-A400M Q4_K_M 4872 / 5162 5480 / 5585 242 / 251
OLMoE-1B-7B Q4_K_M 2920 / 2989 3895 / 4175 265 / 267
GLM-4.7-Flash Q4_K_M 785 / 774 986 / 1018 67.9 / 71.5

llama.cpp (HIP) in the same run: 29337, 31259, 17530, 3195, 1651, 15416, 10292, 2066 t/s prompts; 380, 385, 414, 106, 74.3, 365, 361, 88.3 decoding. With the matrix units the dense prompts are now 0.81 to 1.68 times llama.cpp's; the F32 tile kernels stay at 0.28 to 0.92 (they cannot match a matrix-unit build). Qwen3-8B Q4_K_M decoding is within the noise (83.3 / 81.8). Profile of a Llama-3.2-1B prompt (OKLM_CL_PROFILE=1) before the attention change: 22 ms of matrix products (about 45 TFLOPS), 12.7 ms of attention on the per-token kernel, 2.7 ms of element-wise ops, for 43.5 ms of wall time against llama.cpp's 28.7.

Round 3: MoE prompts grouped by expert on the GPU

Device batches of more rows than experts used to read the chosen experts back and run one product per expert (four launches per expert and layer, and a split-K reduction each). The indexed product now groups the rows on the device (moe_group) and runs every expert's tiles in one launch (gi_<type>, wi_<type>; opencl.md, moe.md). A first version sorted the rows with one item of the work-group (in a fixed order): Granite 5585 to 7555 t/s; with the rows placed by every item (local atomics) 15341. The same greedy tokens as the row kernels on the three models, and the same bits from run to run. GPU rows, before / after (llama.cpp):

model pp512 F32 tiles pp512 matrix units tg128
Granite-3.1-1B-A400M Q4_K_M 5162 / 9730 5585 / 15341 (15345) 251 / 254 (364)
OLMoE-1B-7B Q4_K_M 2989 / 4060 4175 / 8614 (10284) 267 / 271 (358)
GLM-4.7-Flash Q4_K_M 774 / 807 1018 / 1304 (2067) 71.5 / 71.8 (88.1)

After the three rounds (2026-11-08)

The same script, machine and llama.cpp builds, every row again (load average 0.74 when it started, 0.06 hardware threads busy). OKLM at f353351.

model backend threads pp512 OKLM / llama.cpp ratio tg128 OKLM / llama.cpp ratio
SmolLM2-135M Q8_0 cpu 1 487 / 469 1.04 133 / 124 1.07
cpu 4 1804 / 1683 1.07 166 / 194 0.86
cpu 8 3229 / 2974 1.09 166 / 188 0.88
cpu 16 3719 / 3076 1.21 154 / 178 0.87
opencl - 27060 / 29500 0.92 427 / 381 1.12
opencl wmma - 37231 / 29500 1.26 433 / 381 1.14
SmolLM2-135M Q4_K_M cpu 1 509 / 450 1.13 143 / 128 1.11
cpu 4 1885 / 1580 1.19 213 / 254 0.84
cpu 8 3351 / 2783 1.20 217 / 254 0.86
cpu 16 3762 / 2854 1.32 197 / 239 0.82
opencl - 25895 / 31323 0.83 385 / 383 1.01
opencl wmma - 35153 / 31323 1.12 380 / 383 0.99
Llama-3.2-1B Q4_K_M cpu 1 99.2 / 111 0.89 24.4 / 22.4 1.09
cpu 4 345 / 397 0.87 35.6 / 36.1 0.99
cpu 8 658 / 718 0.92 35.2 / 35.5 0.99
cpu 16 671 / 689 0.97 33.1 / 33.8 0.98
opencl - 6010 / 17809 0.34 360 / 414 0.87
opencl wmma - 14510 / 17809 0.81 339 / 414 0.82
Qwen3-8B Q4_K_M cpu 1 15.2 / 15.6 0.98 4.32 / 3.92 1.10
cpu 4 54.1 / 56.5 0.96 6.23 / 6.29 0.99
cpu 8 87.2 / 92.4 0.94 6.12 / 6.15 1.00
cpu 16 89.6 / 89.3 1.00 5.78 / 5.85 0.99
opencl - 905 / 3236 0.28 82.5 / 106 0.78
opencl wmma - 2645 / 3236 0.82 82.0 / 106 0.77
Qwen3-8B Q8_0 cpu 1 16.4 / 9.46 1.74 3.13 / 2.66 1.18
cpu 4 58.8 / 33.5 1.76 3.55 / 3.73 0.95
cpu 8 94.9 / 56.1 1.69 3.57 / 3.60 0.99
cpu 16 101 / 57.2 1.77 3.39 / 3.43 0.99
opencl - 927 / 1629 0.57 65.4 / 74.4 0.88
opencl wmma - 2791 / 1629 1.71 65.6 / 74.4 0.88
Granite-3.1-1B-A400M Q4_K_M cpu 1 137 / 139 0.98 52.1 / 59.4 0.88
cpu 4 501 / 496 1.01 96.4 / 99.0 0.97
cpu 8 891 / 833 1.07 97.0 / 98.0 0.99
cpu 16 993 / 854 1.16 89.7 / 93.6 0.96
opencl - 9978 / 14356 0.70 247 / 321 0.77
opencl wmma - 15401 / 14356 1.07 250 / 321 0.78
OLMoE-1B-7B Q4_K_M cpu 1 56.2 / 56.2 1.00 21.1 / 23.2 0.91
cpu 4 201 / 203 0.99 37.3 / 38.1 0.98
cpu 8 374 / 318 1.18 37.5 / 37.9 0.99
cpu 16 392 / 325 1.21 35.5 / 36.3 0.98
opencl - 4154 / 10449 0.40 260 / 361 0.72
opencl wmma - 8740 / 10449 0.84 256 / 361 0.71
GLM-4.7-Flash Q4_K_M cpu 1 15.3 / 17.3 0.89 6.45 / 6.87 0.94
cpu 4 55.0 / 59.7 0.92 11.2 / 11.6 0.97
cpu 8 88.3 / 92.9 0.95 11.2 / 11.5 0.97
cpu 16 94.4 / 92.5 1.02 10.6 / 11.0 0.96
opencl - 820 / 2088 0.39 71.6 / 88.1 0.81
opencl wmma - 1320 / 2088 0.63 71.1 / 88.1 0.81

Where llama.cpp stays ahead, and why:

  • GPU prompts on the F32 tile kernels (0.28 to 0.92): llama.cpp's HIP build multiplies on the matrix units; OKLM does too with OKLM_CL_WMMA=1 (0.63 to 1.71 here), but its results are not the F32 kernels' bits and greedy text can part at near ties, so it stays opt-in.
  • GPU prompts with the matrix units: Llama-3.2-1B and Qwen3-8B Q4_K_M 0.81 to 0.82, OLMoE 0.84 (the matrix products are within about 10 % of llama.cpp's; the rest is element-wise ops and the launches between them), GLM-4.7-Flash 0.63: its attention (MLA, heads of 576) has no tiled kernel (the tiles stop at 256 dimensions, and a 576-wide tile does not fit 64 KB of local memory with these block sizes), so the per-token kernel takes half of the prompt's GPU time (4.1 ms per layer).
  • GPU decoding (0.71 to 0.88, the small SmolLM2 ahead): the matrix-vector products run at 570 to 590 GB/s, near this GPU's bandwidth; the gap is the 500 to 550 launches per token (RMS norms of single heads, element wise ops, MoE routing, each 3 to 8 microseconds on the device): 3.1 ms of kernels per Granite token against llama.cpp's 2.75 ms in all, which fuses norms with their products, the MoE routing into one kernel, and replays the token as a HIP graph.
  • CPU prompts on the K-quant files: Llama-3.2-1B 0.87 to 0.92 below 16 threads, GLM-4.7-Flash 0.89 to 0.95, Qwen3-8B Q4_K_M 0.94 to 0.98: 85 % of a Llama-3.2-1B prompt is in matrix products (Q4_K 73 %) at about a third of the AVX-512 VNNI peak; the packed Q4_K kernel spends a third of its instructions outside the dot products (nibble unpacking, the scale multiplies, the mins term).
  • CPU decoding of the MoE models at one thread (0.88 to 0.94): their expert matrices have no packed copy, so round 1 does not reach them.
  • CPU decoding of SmolLM2 from four threads (0.82 to 0.88): 211 matrix products per token of 26 microseconds on average, so the team's start and end of each loop (a few microseconds) weigh; llama.cpp runs a token's graph on one thread pool with one barrier per node.

Round 4 (GPU): GPU prompts on the 8-bit matrix units, fused launches, latent-attention tiles

Three changes to the OpenCL backend (opencl.md), measured with THREADS="" sh tools/bench_vs_llama.sh (the GPU rows only, load 2.9 when it started, 0.08 hardware threads busy):

  • Prompts on the 8-bit matrix units, by default. On a gfx11 device the products of 64 columns or more with Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 or Q5_0 weights round the activations to 8 bits exactly as the CPU rounds them (q8_act) and multiply the integer levels with v_wmma_i32_16x16x16_iu8 (wq_<type>, and wqi_<type> for the grouped expert products of MoE prompts), each block's integer sum scaled and added as the CPU's dot product does it: the CPU's products bit for bit (test_opencl_q8: every output of 30 products and 12 indexed products equal), so the switch is on by default (OKLM_CL_Q8=0: the F32 kernels). Steps that paid on the way: the activations rounded by work-groups of 64 items (149 to 13 microseconds a product on Llama-3.2-1B), the weights unpacked with 16-byte vectors, the scale multiplies on 24-bit integers (Q6_K 400 to 253 microseconds a launch), and the mins term of Q4_K / Q5_K as two more 16-deep products of the 8 mins by the block sums split into 7-bit halves instead of 64 integer multiplies an element (Q4_K 539 to 280 microseconds, the F16 matrix units' 283).
  • Fused launches. The backend holds back a MoE layer's softmax router and expert choice, and Qwen3's query and key head norms (and the query rotation when Q and K have their own products), and runs each sequence as one launch when it completes (moe_route, norm_rope; OKLM_CL_FUSE=0 turns it off). The same bits: the logits after a 190-token prompt and 16 greedy tokens hash the same with and without, on the eight models.
  • Latent attention's prompts on tiles (attention_mla, heads of 576 with an F16 cache): GLM-4.7-Flash 1337 to 2174 t/s with the 8-bit products. The same 16 greedy tokens as the per-token kernel.
model pp512 before (F32 tiles) pp512 now llama.cpp ratio tg128 before / now llama.cpp ratio
SmolLM2-135M Q8_0 27060 40450 29226 1.38 427 / 420 381 1.10
SmolLM2-135M Q4_K_M 25895 34604 31196 1.11 385 / 371 383 0.97
Llama-3.2-1B Q4_K_M 6010 13432 17587 0.76 360 / 359 410 0.88
Qwen3-8B Q4_K_M 905 2564 3223 0.80 82.5 / 84.7 105 0.80
Qwen3-8B Q8_0 927 2775 1647 1.68 65.4 / 66.3 73.8 0.90
Granite-3.1-1B-A400M Q4_K_M 9978 16386 15385 1.07 247 / 285 362 0.79
OLMoE-1B-7B Q4_K_M 4154 8753 10225 0.86 260 / 271 359 0.76
GLM-4.7-Flash Q4_K_M 820 2156 2061 1.05 71.6 / 71.5 87.8 0.81

"Before" is the table above (OKLM at f353351). The OKLM_CL_WMMA=1 rows of the same run: 36890, 35186, 14351, 2623, 2782, 15102, 8584, 2068 t/s prompts. SmolLM2 Q4_K_M decoding measured 374 to 381 t/s over six more runs with and without the fused launches (none apply to it): noise. Greedy tokens after a 190-token prompt (16 tokens, the logits of each hashed by a scratch program): with the 8-bit products the GPU's tokens are the CPU's on Qwen3-8B Q4_K_M, OLMoE, Granite and Qwen3-8B Q8_0, where the F32 kernels part from the CPU at the first token on Qwen3-8B Q4_K_M (top two 0.15 apart) and OLMoE (0.04); on GLM-4.7-Flash both the CPU and the F32 kernels part from the 8-bit products at the eleventh token; on Llama-3.2-1B and SmolLM2 the 8-bit products give the F32 kernels' tokens (decoding, a column at a time, keeps the float kernels).

Measured and left out: the 8-bit tiles reading each lane's activation levels from global memory instead of local memory (a third of the local memory, so more work-groups a compute unit: Llama-3.2-1B's Q4_K products 312 to 475 microseconds a launch, Qwen3-8B Q4_K_M prompts 2564 to 1796 t/s). Not tried: the norm folded into the following one-column product (every work-group of the product would sum the squares of the whole row again: 16 MB more cache reads per Qwen3-8B product, more than the launch it saves, as the earlier fused-norm measure found), and a recorded command sequence per token (this driver has no cl_khr_command_buffer).

Where llama.cpp stays ahead on the GPU, and why:

  • Prompts of Llama-3.2-1B (0.76) and Qwen3-8B Q4_K_M (0.80): the Q4_K products run at the F16 matrix units' speed (280 microseconds a launch on Llama-3.2-1B against 283); llama.cpp's MMQ kernels keep more of a tile in registers and load the next tile while multiplying, and its attention and element-wise ops take less of the prompt (OKLM's ew ops 5.2 ms of a 38 ms Llama-3.2-1B prompt).
  • Decoding (0.76 to 0.90, SmolLM2 level or ahead): on Qwen3-8B 85 % of a token's GPU time is the matrix-vector products (the Q4_K ones near 40 microseconds a launch), so the fused launches change it within the noise; the MoE models gain most from them (Granite 247 to 285, OLMoE 260 to 271) and stay behind on their expert products (gate, up and down three indexed launches a layer) and launch count.

Round 4 (CPU): packed expert matrices on the CPU

The routed experts of the CPU layers get the packed copy the dense matrices have, one per expert (OKLM_CPU_PACK_EXPERTS, on by default when the copies and the mapped experts fit in the available memory with the plan's margin; never with an expert cache; moe.md), and the grouped indexed product runs the packed kernels: prompts always, decode steps with one or two threads (from three, the file layout measured 2 to 6 % faster: OLMoE 37.6 / 36.8 t/s at 4 threads, Granite 97.6 / 92.6). The copies take 4.0 GiB for OLMoE, 17.4 GiB for GLM-4.7-Flash, on top of the mapped file; the repacking runs on the worker team. The same bits (test_ops runs the grouped product on both layouts; greedy text identical with and without, OLMoE, Granite, GLM-4.7-Flash at 1 and 4 threads, the expert cache, --gpu-layers 12, --n-cpu-moe 8). Before / after (llama.cpp), t/s:

model threads pp512 tg128
OLMoE-1B-7B Q4_K_M 1 56.5 / 84.4 (56.1) 21.1 / 25.5 (23.1)
4 202 / 289 (206)
16 336 / 469 (358)
Granite-3.1-1B-A400M Q4_K_M 1 139 / 195 (139)
4 490 / 685 (493)
16 830 / 1136 (830)

Round 5: one team loop for the products sharing an input

oklm_matmul_n() runs a layer's Q, K and V, and its gate and up (the shared expert's too), as one loop of the worker team, the activation rows quantized once (core.md): a SmolLM2-135M token runs 121 loops of products instead of 211. The same bits (test_matmul compares it with one product per weight, packed and not, one and six threads). tg128 t/s, one product per weight / fused (llama.cpp from the table above):

model 1 thread 4 threads 8 threads 16 threads
SmolLM2-135M Q8_0 131 / 133 (124) 160 / 168 (194) 163 / 172 (188) 150 / 159 (178)
SmolLM2-135M Q4_K_M 139 / 142 (128) 204 / 219 (254) 215 / 232 (254) 193 / 209 (239)
Llama-3.2-1B Q4_K_M 24.1 / 24.2 (22.4) 34.8 / 35.2 (36.1) 34.9 / 35.2 (35.5) 32.6 / 33.0 (33.8)

What is left of SmolLM2's gap from four threads: a token is 4.6 ms, 92 % in matrix products (Q5_0 2.4 ms: its 576-column matrices cannot be K-quants, and their packed copy, 1.64 times the file, is read with one or two threads only; the Q8_0 head 1.0 ms), at about 23 GB/s against llama.cpp's 27.

Round 6: a leaner packed Q4_K kernel

With AVX-512 VNNI the packed Q4_K kernel accumulated each 32-column sub-block's products in int32, then multiplied them by the eight rows' scales (a 32-bit multiply, two micro-ops, ten cycles of latency) before adding them. It now accumulates each 16 columns apart: their sums fit in int16 (at most 16 x 15 x 127), so one multiply-add by the scale (its high half zero) scales and adds them, in two independent chains. The kernel is also specialized per activation row count and unrolled, so its accumulators stay in registers (rolled, the compiler put them on the stack: the first try ran Llama-3.2-1B prompts at 72 t/s against 100). The same values (test_matmul, packed against scalar; greedy text the same with and without the packed copies on Llama-3.2-1B, Qwen3-8B and SmolLM2). Llama-3.2-1B pp512 at one thread 99 to 119 t/s (llama.cpp 111); the table below has every row.

After rounds 4 to 6 (CPU, 2026-11-08)

The same script and llama.cpp build, CPU rows only (load average 6.3 from the earlier builds when it started, 0.09 hardware threads busy). t/s, OKLM / llama.cpp:

model threads pp512 OKLM / llama.cpp ratio tg128 OKLM / llama.cpp ratio
SmolLM2-135M Q8_0 1 532 / 461 1.15 140 / 122 1.14
4 1963 / 1619 1.21 168 / 189 0.89
8 3586 / 2925 1.23 172 / 185 0.93
16 4088 / 3007 1.36 159 / 170 0.93
SmolLM2-135M Q4_K_M 1 568 / 442 1.29 148 / 126 1.18
4 2078 / 1535 1.35 219 / 248 0.88
8 3815 / 2732 1.40 233 / 251 0.93
16 4226 / 2753 1.53 214 / 230 0.93
Llama-3.2-1B Q4_K_M 1 116 / 110 1.05 26.2 / 22.1 1.19
4 410 / 400 1.02 35.2 / 35.2 1.00
8 750 / 704 1.07 35.2 / 35.1 1.00
16 781 / 596 1.31 33.0 / 32.4 1.02
Qwen3-8B Q4_K_M 1 18.0 / 15.6 1.16 4.73 / 3.92 1.21
4 65.0 / 56.5 1.15 6.24 / 6.24 1.00
8 103 / 91.9 1.13 6.15 / 6.13 1.00
16 106 / 88.6 1.20 5.81 / 5.77 1.01
Qwen3-8B Q8_0 1 16.8 / 9.46 1.78 3.12 / 2.68 1.16
4 60.3 / 33.7 1.79 3.56 / 3.72 0.96
8 97.0 / 55.9 1.74 3.58 / 3.59 1.00
16 103 / 56.8 1.82 3.41 / 3.39 1.01
Granite-3.1-1B-A400M Q4_K_M 1 230 / 139 1.66 64.5 / 59.3 1.09
4 804 / 493 1.63 97.2 / 97.9 0.99
8 1386 / 831 1.67 98.5 / 98.3 1.00
16 1490 / 852 1.75 90.9 / 92.3 0.98
OLMoE-1B-7B Q4_K_M 1 97.3 / 56.1 1.73 26.9 / 23.1 1.16
4 334 / 199 1.68 37.6 / 37.9 0.99
8 593 / 355 1.67 37.6 / 37.8 1.00
16 546 / 368 1.48 35.6 / 35.7 1.00
GLM-4.7-Flash Q4_K_M 1 20.6 / 17.2 1.19 7.51 / 6.85 1.10
4 71.9 / 59.5 1.21 11.1 / 11.4 0.97
8 112 / 92.6 1.21 11.2 / 11.5 0.97
16 120 / 91.1 1.32 10.6 / 10.8 0.97

Against the table after the three rounds, no row got slower beyond the noise (Llama-3.2-1B tg128 at 4 threads 35.6 / 35.2 with llama.cpp 36.1 / 35.2, GLM-4.7-Flash 11.2 / 11.1 with 11.6 / 11.4). Every CPU prompt row is now ahead (1.02 to 1.82); decoding is ahead at one thread (1.09 to 1.21) and level from four (0.96 to 1.02) except SmolLM2-135M (0.88 to 0.93; round 5) and GLM-4.7-Flash (0.97): both read memory at about the speed llama.cpp does, with a few loops a token more.

Round 7 (GPU): decoding on 8-bit dot products, the CPU's products

One-column products (decoding, and a MoE decode step's indexed products) with Q4_K, Q5_K or Q6_K weights, rows of 2048 values or more and 1024 rows or more, now run mq_<type> on a gfx11 device by default (opencl.md): each work-group of 256 items rounds its activation column to Q8_K in local memory exactly as q8_act does (no launch added), then each wave takes two weight rows, forms each super-block's integer sums with v_dot4_i32_iu8 (eight items a super-block, their sums added across lanes with DPP) and adds the super-block's float term to the row's sum in order of k, as oklm_vec_dot_q4_k_q8_k() and its twins do: the CPU's products bit for bit (test_opencl_q8: decode shapes of the three types, plain and indexed, every output equal; OKLM_CL_Q8_MV=0: the float row kernels). One product at a time, pipelined, its weights out of the cache (a scratch program cycling through copies; microseconds, mv_<type> / mq_<type>):

product (rows x row length) mv mq
Q4_K 24576 x 4096 (Qwen3-8B gate and up) 97.1 92.6
Q4_K 6144 x 4096 32.0 27.5
Q4_K 4096 x 4096 21.1 20.7
Q4_K 4096 x 12288 (down) 52.8 51.6
Q6_K 4096 x 12288 (down) 95.7 75.2
Q6_K 151936 x 4096 (Qwen3-8B head) 793 703
Q4_K 16384 x 2048 (Llama-3.2-1B gate and up) 35.7 32.8
Q4_K 2048 x 2048 9.6 9.0
Q6_K 2048 x 8192 32.7 29.5

GPU rows (THREADS="" sh tools/bench_vs_llama.sh, load average 2.6 to 9.3 from earlier builds when each model started, 0.00 to 0.04 hardware threads busy), before (round 4) / after, t/s:

model pp512 llama.cpp ratio tg128 llama.cpp ratio
SmolLM2-135M Q8_0 40450 / 41106 29383 1.40 420 / 425 380 1.12
SmolLM2-135M Q4_K_M 34604 / 35428 31007 1.14 371 / 378 385 0.98
Llama-3.2-1B Q4_K_M 13432 / 13644 17908 0.76 359 / 368 413 0.89
Qwen3-8B Q4_K_M 2564 / 2503 3169 0.79 84.7 / 92.0 105.7 0.87
Qwen3-8B Q8_0 2775 / 2763 1647 1.68 66.3 / 66.3 74.1 0.89
Granite-3.1-1B-A400M Q4_K_M 16386 / 16626 14837 1.12 285 / 283 362 0.78
OLMoE-1B-7B Q4_K_M 8753 / 8854 10389 0.85 271 / 274 360 0.76
GLM-4.7-Flash Q4_K_M 2156 / 2156 2082 1.04 71.5 / 71.5 87.9 0.81

Qwen3-8B Q4_K_M decoding 84.2 to 91.6 t/s and Llama-3.2-1B 354 to 370 in alternated runs of the same build (OKLM_CL_Q8_MV=0 / 1, best of five); the prompt columns did not change (no prompt kernel changed: 2564 / 2503 is the noise of these runs). Greedy tokens after a 190-token prompt (16 tokens, the logits hashed): with the 8-bit decode the GPU's tokens are the CPU's on Qwen3-8B Q4_K_M, Granite and OLMoE, as they were with the float decode; on Llama-3.2-1B the float decode parted from the CPU at the sixth token (its top two 0.004 apart), the 8-bit decode at the fifth (the CPU's top two 0.09 apart: attention and the element-wise ops still round differently); on GLM-4.7-Flash the 8-bit decode gives the float decode's 16 tokens (both part from the CPU at the eleventh, as in round 4).

Measured and left out:

  • The activations rounded by a separate q8_act launch before each one-column product (the first version, one row a wave): Qwen3-8B Q4_K_M decoding 84.2 to 88.8 t/s but Llama-3.2-1B 349 to 297, one more launch per product (about 6 microseconds back to back, as much as a whole 2048 x 2048 product).
  • The column rounded in each work-group of 128 items, one row a wave: Llama-3.2-1B 362 to 322 t/s (the rounding paid by four rows only); four super-blocks per group a wave step instead of two: rows of 2048 values slower than the float kernels (16384 x 2048 40.5 against 35.7 microseconds: half the lanes read a block past the row's end).
  • The 8-bit kernels for Q8_0, Q4_0 and Q5_0 (four, then two items a block of 32): Qwen3-8B Q8_0 decoding 66.6 to 64.8 t/s.
  • The 8-bit kernels for shorter or fewer rows: SmolLM2-135M Q4_K_M decoding 412 to 370 t/s (its 576 x 1536 down products: 36 work-groups, each rounding the column for little work), Granite 283 to 277 and OLMoE 274 to 266 (rows of 1024 values).
  • Q4_K prompt tiles that load the next step while multiplying: the activation levels into registers (eight 16-byte vectors an item, 32 more registers on top of the 128 of the accumulators) halved the Q4_K prompts (Llama-3.2-1B 13848 to 8361 t/s, Qwen3-8B Q4_K_M 2584 to 1410; Q8_0, with fewer accumulators, unchanged); the weights only (12 registers): 13848 to 13675 and 2584 to 2491, within the noise or slower.

Not done: the MoE expert gate and up products as one indexed launch, a fused element-wise pass for prompts.

Where llama.cpp stays ahead on the GPU, and why:

  • Decoding (0.76 to 0.89, SmolLM2 level or ahead): the products of large matrices now run near the bandwidth the head's product reaches (Q6_K 151936 x 4096 at 730 GB/s, Q4_K 24576 x 4096 at 610); what is left is the launches (about 12 per layer, each a few microseconds back to back, where llama.cpp replays the token as a HIP graph) and the small products, whose fixed cost the 8-bit kernels do not lower. The MoE models (Granite, OLMoE) are the furthest behind: three indexed products of short rows per layer, still on the float kernels.
  • Prompts of Llama-3.2-1B (0.76) and Qwen3-8B Q4_K_M (0.79): unchanged from round 4 (both prefetching attempts above lost).

Round 7 (CPU): packed copies the file's size, decode products on fewer threads

Three changes to decoding on the CPU, all giving the same bits (test_matmul packed against scalar at both SIMD levels; greedy text of SmolLM2 Q4_K_M, SmolLM2 Q8_0 and Qwen3-8B Q8_0 the same packed, on the file layout and at one thread):

  • Q8_0 and Q5_0 copies as small as the file. The packed Q8_0 copy keeps the file's half-precision scales (272 bytes for eight rows of a block, 1.0 times the file instead of 1.06), the Q5_0 copy its nibbles as in the file and the fifth bits of each row in a 32-bit word laid out so one shift and one mask put them in place (176 bytes, 1.0 times instead of 1.64). A one-row product now reads the packed copy at any thread count when it is no larger than the file. Prompts expand each tile of Q5_0 rows once to a byte per value and run the byte kernel, so they do not pay for the unpacking (the kernel that unpacked on every call ran SmolLM2 Q4_K_M pp512 2.5 % slower).
  • One-row products on fewer threads. Bound by memory, they read it fastest with fewer readers than the team has (bench_matmul, a 107 MB Q8_0 matrix, packed, GB/s: 34.4 at 2 threads, 35.0 at 3, 34.5 at 4, 33.7 at 5, 32.8 at 6, 31.6 at 8, 30.4 at 16; Q4_K on the file layout 37.1 at 4, 36.0 at 8). A loop of the worker team can now take at most n participants (oklm_cpu_team_run_max(), the first to arrive; the chunks are those of the whole team): half the physical cores for a product on a packed copy (OKLM_MATVEC_THREADS), one thread per core on the file layout.
  • Decode attention without the split while the keys are few. With fewer tiles than threads (SmolLM2: three KV heads), attention split the tiles' arithmetic into three loops of the team (keys, rows, columns); below 2^18 multiply-adds per tile it now runs the tiles as they are, one loop (SmolLM2 Q8_0 tg1536 at 4 threads 142.5 to 146.1 t/s, the same at 8; tg128 181 to 185).

tg128 t/s before / after, each pair back to back (llama.cpp from the table above):

model threads before / after (llama.cpp)
SmolLM2-135M Q4_K_M 4 222 / 242 (248)
SmolLM2-135M Q8_0 4 171 / 185 (189)
Qwen3-8B Q8_0 4 3.56 / 3.73 (3.72)
8 3.58 / 3.68 (3.59)
16 3.41 / 3.66 (3.39)
Qwen3-8B Q4_K_M 16 5.81 / 6.11 (5.77)
Llama-3.2-1B Q4_K_M 16 33.5 / 34.8 (32.4)
Granite-3.1-1B-A400M Q4_K_M 16 92.0 / 93.8 (92.3)

pp512 of SmolLM2 Q4_K_M, before / after: 578 / 575 at one thread, 2131 / 2135 at four, 4295 / 4344 at sixteen.

Measured and left out:

  • the packed Q4_K copy (1.06 times the file) for one-row products from three threads: Llama-3.2-1B tg128 35.8 to 34.9 at 4 threads, 35.4 to 34.3 at 8; Q6_K's (1.31 times) as well: SmolLM2 Q4_K_M 242 to 233;
  • an eight-activation-row Q8_0 prompt kernel: pp512 539 to 522 at one thread; the compact Q5_0 layout read directly by the prompt kernels, four rows (2136 to 2086 at 4 threads) or eight (2118 to 2103);
  • the fifth bits as byte masks added with masked byte adds (AVX-512 BW): pp512 2066 against 2103 for the shifts; a three-input logic instruction for the shift-and-mask: no change;
  • one-row products on six threads instead of four on a packed copy (Qwen3-8B Q8_0 at 8 threads 3.57 against 3.68), on four for the file layout too (SmolLM2 Q4_K_M at 16 threads 214 against 220: its Q6_K products need more threads), a limit only for matrices of 4 MB or more (no better), and a limit on the indexed (expert) products (Granite, OLMoE: within the noise).

Round 8 (CPU): packed copies of latent attention's per-head matrices

A step of GLM-4.7-Flash at four threads (OKLM_CPU_PROFILE, 89 ms): products 55 % (the Q6_K output head and shared experts 13.8 ms, Q4_K attention 16.2 ms, Q8_0 11.2 ms, Q5_K shared experts 7.1 ms), expert products 41 % (36 ms), each at 26 to 32 GB/s; attention 2 %. Its Q8_0 matrices are mostly latent attention's per-head stacks (attn_k_b, attn_v_b, [K, N, 20]), which batched products read on the file layout. They now get one packed copy per head (counted in place of the mapped stack, as a matrix's copy; test_matmul checks batched products on a packed stack against the scalar kernels): pp512 at four threads 73.2 to 78.2 t/s, tg64 11.24 to 11.29 (a decode step reads them at memory speed either way). The greedy text of GLM-4.7-Flash is the same packed, on the file layout and at one thread.

After rounds 7 and 8 (CPU, 2026-11-08)

The same script and llama.cpp build, CPU rows only (load average 1.6 to 6 between the models, under 0.1 hardware threads busy at each start; Llama-3.2-1B and SmolLM2 Q4_K_M measured again with the last commit after a 16-thread prompt run of the first pass fell into the slow mode that comes and goes at 16 threads). t/s, OKLM / llama.cpp:

model threads pp512 OKLM / llama.cpp ratio tg128 OKLM / llama.cpp ratio
SmolLM2-135M Q8_0 1 538 / 466 1.15 141 / 122 1.16
4 1988 / 1684 1.18 184 / 193 0.96
8 3595 / 2949 1.22 179 / 187 0.95
16 4102 / 3026 1.36 165 / 175 0.94
SmolLM2-135M Q4_K_M 1 577 / 448 1.29 176 / 128 1.37
4 2123 / 1566 1.36 250 / 252 0.99
8 3821 / 2796 1.37 241 / 254 0.95
16 4306 / 2862 1.50 220 / 241 0.91
Llama-3.2-1B Q4_K_M 1 121 / 111 1.09 25.8 / 22.3 1.16
4 428 / 400 1.07 35.8 / 36.0 0.99
8 805 / 727 1.11 35.4 / 35.3 1.00
16 818 / 684 1.20 34.9 / 33.4 1.05
Qwen3-8B Q4_K_M 1 18.9 / 15.6 1.21 4.67 / 3.91 1.19
4 67.4 / 57.1 1.18 6.26 / 6.23 1.00
8 106 / 92.0 1.15 6.15 / 6.13 1.00
16 109 / 88.5 1.23 6.13 / 5.76 1.06
Qwen3-8B Q8_0 1 16.7 / 9.46 1.76 3.24 / 2.66 1.22
4 59.6 / 34.0 1.76 3.73 / 3.72 1.00
8 95.8 / 55.9 1.71 3.68 / 3.59 1.03
16 102 / 56.9 1.80 3.65 / 3.38 1.08
Granite-3.1-1B-A400M Q4_K_M 1 234 / 139 1.68 65.0 / 59.4 1.09
4 817 / 494 1.65 97.7 / 97.8 1.00
8 1389 / 831 1.67 98.5 / 98.3 1.00
16 1501 / 813 1.85 92.2 / 91.6 1.01
OLMoE-1B-7B Q4_K_M 1 99.6 / 56.1 1.78 27.0 / 23.1 1.17
4 351 / 201 1.75 37.6 / 37.8 0.99
8 601 / 357 1.68 37.7 / 37.8 1.00
16 626 / 322 1.94 36.0 / 36.0 1.00
GLM-4.7-Flash Q4_K_M 1 22.3 / 17.3 1.29 8.03 / 6.92 1.16
4 79.2 / 59.5 1.33 11.2 / 11.4 0.98
8 122 / 92.9 1.31 11.2 / 11.5 0.97
16 131 / 92.3 1.42 10.7 / 10.9 0.98

Against the table after rounds 4 to 6: decoding faster on every model from four threads (SmolLM2 Q4_K_M 219 / 249 at four, 214 / 220 at sixteen; Qwen3-8B Q8_0 3.56 / 3.73, 3.58 / 3.68, 3.41 / 3.65; Qwen3-8B Q4_K_M and Llama-3.2-1B at sixteen 5.81 / 6.13 and 33.0 / 34.9) and at one thread on the Q5_0 file (SmolLM2 Q4_K_M 148 / 176); prompts the same or faster (GLM-4.7-Flash 71.9 / 79.2 at four) but Qwen3-8B Q8_0's, 1 % slower (60.3 / 59.6 at four, 97.0 / 95.8 at eight; an A/B run: 60.6 / 59.4, and 17.4 / 17.2 at one thread): its packed copy now holds the scales as halves, one more conversion per block in the prompt kernel, the price of the decode rows above (a copy with float scales is 1.06 times the file, which round 1 found too large for decoding from three threads). Expanding each prompt tile to float scales measured slower (58.3). Decoding from four threads is level on every model now (0.97 to 1.08) except SmolLM2 from eight threads (0.91 to 0.95) and GLM-4.7-Flash (0.97 to 0.98). SmolLM2 decodes faster on four threads (250) than on eight (241) or sixteen (220), while llama.cpp holds 252 to 254 from four to eight; its products already run on four threads from eight (OKLM_MATVEC_THREADS), so the loss is in the rest of a step with more threads polling (a waiting policy without the short poll, and without the poll for workers left out of a loop, measured the same: 240.9 and 240.4 against 240.3).

Round 9 (GPU): decode norms folded into the 8-bit products, fewer launches

Three changes to the OpenCL backend's decoding (opencl.md):

  • A decode step's RMS norm folded into the product that reads it. A profile of a Llama-3.2-1B token (OKLM_CL_PROFILE=1) gave 202 launches: the matrix products 1.57 ms of GPU time, the 33 RMS norms 0.26 ms (7.8 microseconds each: one work-group of 256 for 2048 values), attention 0.18, element-wise ops 0.15, rope 0.09, and about 2.3 microseconds between dependent launches; the host spent 0.25 ms a token enqueueing, far less than the 2.7 ms the GPU took, so the launches' GPU time and the gaps between them were the target, not the host. The backend now holds back the norm of one row (2048 to 8192 values, a multiple of 256) and, when the next op is a one-column product of its output on an MQ kernel, runs both as one launch (mqn_<type>): each work-group, which already reads the whole column to round it, loads the norm's input and weights once, sums the squares, scales and rounds; the first work-group writes the norm's output. The norm kernel sums rows of those widths in runs of eight values an item (the MQ kernels' layout of a column), so fused and separate norms are the same bits (test_opencl_q8: norm output and product equal, the product the CPU's product of that output); other widths keep the old order, so SmolLM2's and Granite's logits are unchanged to the bit. Llama-3.2-1B: 202 to 169 launches a token, every norm folded.
  • The key and value cache writes in one launch (cpy2, the same bits): one launch fewer a layer.
  • The 8-bit decode products under OKLM_CL_WMMA=1. The 8-bit kernels are built with the matrix units too, for the one-column products alone: Qwen3-8B Q4_K_M decoding with OKLM_CL_WMMA=1 84 to 94 t/s (alternated runs), its greedy tokens now the default path's.

GPU rows (THREADS="" sh tools/bench_vs_llama.sh, load average 3.6 when it started, 0.05 hardware threads busy), before (round 7) / after, t/s:

model pp512 llama.cpp ratio tg128 llama.cpp ratio
SmolLM2-135M Q8_0 41106 / 40096 29242 1.37 425 / 445 381 1.17
SmolLM2-135M Q4_K_M 35428 / 35871 31229 1.15 378 / 400 383 1.04
Llama-3.2-1B Q4_K_M 13644 / 13710 17845 0.77 368 / 392 411 0.95
Qwen3-8B Q4_K_M 2503 / 2553 3200 0.80 92.0 / 94.7 105.3 0.90
Qwen3-8B Q8_0 2763 / 2728 1649 1.65 66.3 / 68.1 74.0 0.92
Granite-3.1-1B-A400M Q4_K_M 16626 / 16423 15492 1.06 283 / 279 364 0.77
OLMoE-1B-7B Q4_K_M 8854 / 8712 10291 0.85 274 / 291 360 0.81
GLM-4.7-Flash Q4_K_M 2156 / 2149 2070 1.04 71.5 / 73.1 88.0 0.83

The OKLM_CL_WMMA=1 rows of the same run: 36908, 35119, 14630, 2644, 2768, 15315, 8605, 2056 t/s prompts; 439, 396, 385, 95.4, 67.6, 256, 278, 73.4 decoding (Granite's 256 is that mode's speed before these changes too: 252 / 256 and 256 / 254 in alternated runs). No prompt kernel changed (prompt differences are the noise of these runs). Alternated runs of the same files, before / after: Llama-3.2-1B decoding 354 to 370 / 380 to 390 with the norms folded, 376 to 385 / 390 with the cache writes added; Granite 252 to 258 / 252 to 257 (nothing folds: rows of 1024). Greedy tokens after a 190-token prompt (16 tokens, the logits hashed): the same tokens as before on the eight models; the logits the same bits with OKLM_CL_FUSE=0 and without; bit-identical to before on SmolLM2 and Granite. With OKLM_CL_WMMA=1, Qwen3-8B Q4_K_M's tokens become the default path's (they parted from them at the fourth token), and OLMoE's part from both at the sixth (its prompt products are F16, its decode products now the CPU's; that mode is not the CPU's bits).

Measured and left out:

  • The folded norm reading its input twice (a pass for the sum of squares, then the rounding pass, in the norm kernel's old order, every bit as before): 3.4 microseconds more a fused launch than the product alone, against 2.8 with the one-pass version kept (Llama-3.2-1B's MQ time 279 against 269 ms over 256 tokens); the sum's loop unrolled by eight made no difference.
  • The norm kernel summing every width in runs of eight: SmolLM2's logits moved further from the CPU's (test_opencl_q8: 0.77 against 0.61), so only the foldable widths take the runs.
  • The work-group sum's last five steps inside one wave (DPP, the same pairs and bits, five barriers fewer): within the noise (Llama-3.2-1B 381 / 385, 386 / 381).
  • The norm's chunks loaded for a fixed four chunks a wave and rounded unconditionally (a compiler workaround tried first): the rounding of chunks past the row cost more than the fusion saved (354 to 363 t/s against 363 to 369).

Not done: the MoE expert gate and up products as one indexed launch with SwiGLU folded (Granite and OLMoE stay at 0.77 and 0.81: Granite's decode is 472 launches a token, 49 norms of 1024 values at 7.5 microseconds that no MQ kernel takes, 146 element-wise launches); SwiGLU folded into the down product (the backend cannot fold an in-place op: the model writes the gate's rows); the rotation and the cache writes as one launch; a faster standalone norm kernel for rows below 2048 values.

Where llama.cpp stays ahead on the GPU, and why:

  • Decoding of the MoE models (Granite 0.77, OLMoE 0.81) and GLM-4.7-Flash (0.83): many short launches a token (Granite 472), the norms of 1024 values and the expert products of short rows on the float kernels.
  • Decoding of the dense models (0.90 to 0.95; SmolLM2 ahead): the products of large matrices near the bandwidth, the rest launches.
  • Prompts of Llama-3.2-1B (0.77) and Qwen3-8B Q4_K_M (0.80): unchanged from round 7 (no prompt kernel changed this round).

Round 9 (CPU): decode loops on a fixed set of workers, the others asleep

A step of SmolLM2-135M Q4_K_M (OKLM_CPU_PROFILE, 512 tokens decoded, seconds in ops at 4 / 8 / 16 threads): 4.52 / 4.68 / 5.14, the products on four threads at every count (Q5_0 2.15 / 2.24 / 2.42) but attention (three tiles, 0.45 / 0.49 / 0.64) and the small ops run inline (rope 0.065 / 0.072 / 0.096) slower too. Three costs grew with the team, not with the work:

  • a one-row product was cut into tiles for the whole team (sixteen threads: 36 tiles of 16 rows for four participants), and its loop into as many chunks, each a compare-and-swap on the claim word;
  • a loop limited to four participants was joined by the first four workers to take a seat by compare-and-swap on one shared word, the others racing for it and failing, at every loop; and every waiting worker polled the claim word the participants were writing;
  • the workers no loop of a decoding step takes (at sixteen threads, the eight above the participant limits) kept polling between loops, which come every few microseconds, so they never slept and took half of every core from the threads that ran.

Now (team_cpu.c): a limited loop takes the caller and the workers of index 1 to n - 1 (never more than it has chunks), each worker knowing its index, so none contends for a seat; workers poll the generation of the last loop on a cache line of its own; a loop a worker is left out of does not restart its wait, so those no loop takes go to sleep after about a millisecond, and a loop wakes the sleepers only when it takes one of them. A one-row product cuts its tiles for the threads it runs on, and one on the file layout under 4 MiB of weights runs on as many threads as a packed copy's (SmolLM2's Q4_K and Q6_K down products: half a megabyte; Llama-3.2-1B's larger matrices stay on one thread per core: 35.4 t/s at 8 and 16 threads with or without, the threshold measured at 1, 4 and 8 MiB). The same bits (the loops' chunks change, no output element's arithmetic; test_threads adds a loop that must wake the workers left asleep by limited loops; greedy text through make test).

tg128 t/s, before (the last commit of round 8) / after, each pair back to back:

model 4 threads 8 threads 16 threads
SmolLM2-135M Q4_K_M 250.5 / 251.1 240.5 / 248.1 218.2 / 250.0
SmolLM2-135M Q8_0 185.1 / 184.9 178.7 / 184.3 166.1 / 184.0

Step by step on SmolLM2 Q4_K_M at 8 / 16 threads: tiles for the participants 242 / 225, fixed seats 244 / 230, the generation on its own line 245 / 230, left-out workers asleep 245 / 241, the small file-layout products on four threads 250 / 250.

Measured and left out:

  • Qwen3-8B Q8_0 prompts (row 2 of what was left): a packed Q8_0 group holding every block's scales first, then the values, so that the values of each block start on a 32-byte boundary (beside each block's scales, a quarter of the prompt kernel's 32-byte loads cross two cache lines): pp512 at four threads 59.2 / 57.9 before, 58.6 / 58.2 after, alternated runs: no change, so the split loads are not what the half-precision scales cost.
  • GLM-4.7-Flash decoding: a decoding expert product's tiles cut into 2, 4 or 8 per thread (larger contiguous reads) instead of 16 rows: tg64 at four threads 11.28 / 11.25 / 11.22 / 11.23; one-row products cut into 2, 8 or 16 tiles per thread instead of 4: 11.23 to 11.28 (SmolLM2 246 to 248), all within the noise.
  • The waiting workers prefetching the next product's weights: each decoding product records the weights of the product that followed it (the order is the same at every token), and when it ends the workers waiting for a loop issue a prefetch for each cache line of the successor's first 4 MiB (an indexed successor: its experts for this product's routing), filling the memory's idle time during attention, the small ops and the last chunks: SmolLM2 Q4_K_M 249 / 248 / 248 at 4 / 8 / 16 threads to 242 / 240 / 240; GLM-4.7-Flash 11.26 to 11.36 in one pair, then, prefetching only successors of 2 MiB or more, 11.30 / 11.28 against 11.06 / 11.12. The memory is not idle enough between products for the extra traffic to pay.

The rows the round touches (GPU=0 sh tools/bench_vs_llama.sh, the same llama.cpp build; the machine shared with other work, load average 2.3 to 3.6; the run was stopped after Llama-3.2-1B, whose prompt rows at 8 threads ran slow for both programs, 708 / 693 against 805 / 727 in the table after rounds 7 and 8). t/s, OKLM / llama.cpp:

model threads pp512 OKLM / llama.cpp ratio tg128 OKLM / llama.cpp ratio
SmolLM2-135M Q8_0 1 536 / 465 1.15 140 / 122 1.15
4 1984 / 1652 1.20 184 / 192 0.96
8 3589 / 2972 1.21 184 / 187 0.98
16 4099 / 3125 1.31 183 / 176 1.04
SmolLM2-135M Q4_K_M 1 577 / 448 1.29 175 / 128 1.37
4 2137 / 1586 1.35 251 / 252 1.00
8 3854 / 2754 1.40 249 / 254 0.98
16 4207 / 2774 1.52 250 / 233 1.07
Llama-3.2-1B Q4_K_M 1 120 / 111 1.08 25.8 / 22.3 1.16
4 438 / 401 1.09 35.7 / 35.6 1.00
8 708 / 693 1.02 35.2 / 35.1 1.00
16 795 / 665 1.20 35.4 / 32.5 1.09

SmolLM2 now decodes as fast on 8 or 16 threads as on 4 (Q4_K_M 249 to 251, Q8_0 183 to 184): from eight threads 0.98 to 1.07 of llama.cpp (was 0.91 to 0.95), level or ahead at sixteen, 2 % behind at eight (llama.cpp's own best is at eight). The other models' rows, not measured again, are those of the table after rounds 7 and 8: the change reaches them only through the loops' participants and the left-out workers' sleep (Llama-3.2-1B decoding the same above). Still behind: SmolLM2 Q8_0 at four threads (0.96), GLM-4.7-Flash decoding (0.97 to 0.98, unchanged), Qwen3-8B Q8_0 prompts 1 % below round 6 (still 1.71 to 1.80 times llama.cpp).

Round 10 (CPU): the activation row rounded with AVX2, packed copies on huge pages

The two decode rows left behind after round 9 (SmolLM2 at four and eight threads, GLM-4.7-Flash), measured before anything was changed (2026-11-12; perf is not available to an unprivileged user on this machine, so the worker team counted its own time for the measure: cycles between a loop's publication and its end, cycles of each chunk, the delay before a worker's first chunk):

  • The loops are not what costs. SmolLM2-135M Q8_0 at four threads, 151 loops a token: a worker starts its first chunk 0.15 microsecond after the loop is published, and the participants compute 99.8 % of the loops' time (GLM-4.7-Flash 97.5 %, the rest the wait for the last tile). Fewer loops or a cheaper hand-off had nothing to return.
  • The time outside the loops was the rounding of the activations. A quantized product first rounds its input row to bytes (oklm_quantize_row(), Q8_0 or Q8_K blocks) on the calling thread, the memory idle: 3.7 microseconds a product in scalar code (roundf per value), 121 products a token, 8 % of SmolLM2's step, and 4 % of GLM-4.7-Flash's.
  • GLM-4.7-Flash's products already read at the memory's rate. The 37 GB/s of a large product is bench_matmul's fastest call on one 107 MB matrix, 16 MB of which stay in the last-level cache. On weights not in cache (one-row Q4_K products of 2048 values a row over 1.15 GB of distinct matrices, four threads, huge pages) every size reads alike: 28.4 GB/s at 0.45 MB a matrix, 29.2 at 1.8 MB, 30.0 at 4 MB, 29.8 at 9 MB, 30.0 at 36 MB, 30.2 at 144 MB, 30.1 as one matrix of 1.15 GB (on 4 KiB pages 26.9, 28.8, 29.4, then 29.7); by threads 15.4 at one, 26 at two, 30 from three. Summing the same memory with 256-bit adds reads 28.1 GB/s on one thread and 31.2 on two or four: the products run at 96 % of what the memory gives, the short ones 3 % below the long ones.

Three changes, the same bits (test_matmul compares the rounding with the scalar one on halves, zero blocks and both signs of the largest magnitude, and packs more copies than a chunk holds; the logits of the decoded tokens and of a prompt, before and after, compared byte for byte: 32 tokens and a prompt of 96 on SmolLM2-135M Q8_0, Q4_K_M and IQ4_XS and on Granite at one and four threads, with the chunks and without, fewer tokens on GLM-4.7-Flash at one and four threads and on Llama-3.2-1B at four):

  • The activation row rounded with AVX2 (oklm_cpu_quantize_act(), at the AVX2 and AVX-512 levels): Q8_0 rows (the block's largest magnitude, the same product per value, roundf as the integer part plus one away from zero from a half) and Q8_K rows (the scale from the first value of the largest magnitude, rounding to nearest even, the sums of each sixteen levels).
  • A token's rotation computes its cosines and sines once for all its heads instead of once per head.
  • Packed copies on huge pages. Each copy was an allocation of its own, for a small matrix a mapping of 4 KiB pages; they are now cut one after the other from chunks aligned on 2 MiB and advised as huge pages (Linux, OKLM_CPU_PACK_HUGE=0 for the former allocation; decoding SmolLM2 Q8_0, whose copies take 145 MB, the process has 143 MB of anonymous memory on huge pages, 28 before).

tg128 t/s at four threads, step by step, each pair alternated in one session:

step SmolLM2-135M Q8_0 SmolLM2-135M Q4_K_M
rounding with AVX2 182.5 / 196.0 250.0 / 265.7
the rotation's cosines once 197.8 / 197.9 267.3 / 270.8
copies on huge pages 198.4 / 207.3 270.3 / 279.0

Before (the last commit of round 9's state, 6b05d98) / after / the same llama.cpp build, alternated, the best of five runs each (the machine shared with other work: each run started with under 1.2 hardware threads busy; llama.cpp's figures of the day are below those of the tables above on SmolLM2 Q8_0, 186 to 190 against 192):

model threads tg before / after llama.cpp ratio
SmolLM2-135M Q8_0 1 133.6 / 162.2 118.8 1.37
4 184.2 / 207.9 189.9 1.09
8 181.9 / 199.6 186.9 1.07
16 182.4 / 201.3 171.5 1.17
SmolLM2-135M Q4_K_M 1 169.3 / 201.2 129.2 1.56
4 249.5 / 279.9 256.5 1.09
8 249.8 / 278.8 257.4 1.08
16 228.3 / 277.2 239.8 1.16
GLM-4.7-Flash Q4_K_M (tg64) 4 11.50 / 11.99 11.78 1.02
8 11.47 / 11.86 11.59 1.02

SmolLM2 pp512 at four threads is unchanged by the chunks (Q4_K_M 2524 / 2529). GLM-4.7-Flash gains from the rounding alone (11.47 / 11.94 at four threads; with the chunks 11.91: its decode products read the file layout).

Measured and left out:

  • the mapped file collapsed into huge pages (MADV_COLLAPSE on each 2 MiB of the mapping, which the kernel did on ext4 without a copy in the program: 102 MB of SmolLM2 Q4_K_M's file and GLM-4.7-Flash's 18 GB mapped by 2 MiB entries): GLM-4.7-Flash tg48 at four threads 11.94 / 12.00, SmolLM2 Q4_K_M 247.0 / 247.5, within the noise, and collapsing 18 GB adds to the load;
  • the fusions the roadmap listed (norms and rotation into the loop of the product beside them, the expert gate and up as one loop): not written, the loops' hand-off measured at 0.15 microsecond and their threads busy 99.8 % of the time.

Round 10 (GPU): decoding in fewer launches

2026-11-12. What a decode step of a small model costs on the GPU is its launches (about 5 to 6 microseconds each on this driver, whatever the kernel does: Granite's step with the scale and add launches skipped went from 317 to 370 t/s, with its norms skipped to 362, with the keys' rotation and the cache writes skipped to 334), so three more sequences are held back by the backend and run as one launch each, with the bits of the separate ops (test_opencl; the logits of the three MoE models after 24 + 12 tokens are the same bits with OKLM_CL_FUSE=0, with the three knobs off, and with them on):

  • The end of a residual branch (res_norm, OKLM_CL_RES): the weighted sum of the expert outputs, the add of the shared experts' output, the residual scale, the add into the residual stream and the RMS norm of the new stream (the next block's first op). Granite: the attention branch's scale, add and norm three launches to one, the expert branch's sum, scale, add and norm four to one; 424 launches a token to 304, 314 to 379 t/s.
  • The rotation with the cache writes (rope_kv, OKLM_CL_ROPE_KV): the rotation of the queries and of the keys and the copies of the keys and values into their caches, three launches (two where Q and K are one product) to one. Granite 304 launches to 268, 379 to 399 t/s.
  • A sigmoid router's routing (moe_route, OKLM_CL_ROUTE_SIGMOID): GLM-4.7-Flash's sigmoid, expert choice and weights three launches to one, 1168 launches a token to 1076, 75.9 to 77.9 t/s.

bench_llama, 128 tokens decoded after a 16-token prompt, best of three, two alternated rounds each, the three knobs off / on; llama.cpp's HIP figures are those of round 9 (not measured again):

model launches a token tg128 off tg128 on llama.cpp ratio
Granite-3.1-1B-A400M Q4_K_M 424 / 268 305 to 307 395 to 396 364 1.09
OLMoE-1B-7B Q4_K_M 249 / 209 293 to 299 315 to 321 360 0.88
GLM-4.7-Flash Q4_K_M 1259 / 1076 73.7 to 74.0 77.9 to 78.0 88.0 0.89
SmolLM2-135M Q4_K_M 344 / 300 416 to 428 471 to 472 383 1.23
Llama-3.2-1B Q4_K_M 153 / 129 391 to 392 392 to 408 411 0.95 to 0.99

Measured and left out:

  • The norm kernel's work-group sum in two steps (three barriers instead of ten) for the rows summed without runs: within the noise (Granite 402 to 405 t/s before, 398 to 406 after), as the DPP steps of round 9 were; the kernel's time is not its barriers.
  • Not tried: the router's product inside moe_route (its launch skipped is 371 to 392 t/s on Granite, an upper bound; the one work-group of the routing would do 32 x 1024 products there, 64 x 2048 on OLMoE and GLM-4.7-Flash, most of what the launch costs), and OLMoE's two whole-row query and key norms with the rotation.

Round 11 (GPU): Q4_K prompt tiles with loader waves

2026-11-12. The 8-bit prompt tiles of Q4_K and Q5_K weights (opencl.md) now load a step while the previous one multiplies: a work-group has four more waves that only load, into a second buffer of local memory, and the scale arrays of every 8-bit tile are sized for its weight type. The CPU's products bit for bit, as before (test_opencl_q8). bench_llama, 512 random tokens in one batch, the best of three runs, before and after alternated over four rounds, llama.cpp's HIP llama-bench run before and after them (17 787 and 17 884; 3 227 and 3 201; the best kept), t/s:

model pp512 before pp512 now llama.cpp ratio before / now
Llama-3.2-1B Q4_K_M 13 726 to 13 869 14 809 to 15 009 17 884 0.78 / 0.84
Qwen3-8B Q4_K_M 2 550 to 2 562 2 738 to 2 781 3 227 0.79 / 0.86

Shorter prompts gain more (64 tokens: Llama-3.2-1B 5 493 to 6 307, Qwen3-8B 1 194 to 1 344); the other models' prompts are level or up to 3 % faster and decoding is unchanged; the steps, those figures and what was measured and left out (two and eight loader waves, Q6_K tiles with loader waves) are in opencl.md. The other rows of the table of round 9 were not measured again.

The opt-in F16 matrix units (OKLM_CL_WMMA=1, not the CPU's bits) gained the same day from their kernels reading the next step while they multiply (wm_<type>, w2_<type>), before / after alternated over three rounds, llama.cpp before and after them: Llama-3.2-1B Q4_K_M 16 228 to 17 553 t/s (llama.cpp 17 966: 0.90 to 0.98), Qwen3-8B Q4_K_M 2 825 to 3 172 (3 218: 0.88 to 0.99); the default kernels in those rounds 14 951 and 2 783.

The indexed tiles of MoE prompts (wqi_q4_k, wqi_q5_k) took the same loader waves later that day, the CPU's products bit for bit as before (test_opencl_q8). The best of three a run, before and after alternated over eleven rounds, llama.cpp's HIP llama-bench in the same session (the best of its samples), t/s:

model pp512 before pp512 now llama.cpp ratio before / now
OLMoE-1B-7B Q4_K_M 8 600 to 8 712 8 726 to 8 831 10 850 0.80 / 0.81
GLM-4.7-Flash Q4_K_M 2 175 to 2 191 2 201 to 2 223 2 078 1.05 / 1.07
Granite-3.1-1B-A400M Q4_K_M 17 878 to 18 186 17 903 to 18 173 15 529 1.17 / 1.17

1.3 % on the means of OLMoE and GLM-4.7-Flash, faster in every round; Granite level; the reasons are in opencl.md.

Inference

Command: oklm run -m <file> -p "<first 2000 bytes of tinyshakespeare.txt>" -n 128 --temp 0 --backend cpu (prompt token counts differ by tokenizer: 563 for SmolLM2), default threads (8, one per physical core), F16 KV cache. Same machine as below, 2026-10-30 (repacked weights, best of two runs; the machine was not idle, so expect a few % of noise). llama.cpp: llama-bench -p 512 -n 128 -t 8 on the same files (its generation starts from an empty context, OKLM's after the prompt).

file weight types prompt t/s generation t/s llama.cpp prompt / generation
SmolLM2-135M F16 F16 953 86 2471 / 100
SmolLM2-135M Q8_0 Q8_0 1585 125 2875 / 187
SmolLM2-135M Q4_K_M Q5_0, Q8_0, Q4_K, Q6_K 1561 149 2660 / 247
Qwen2.5-0.5B-Instruct Q4_K_M Q5_0, Q4_K, Q6_K, Q8_0 957 59 857 / 70
Qwen3-0.6B Q8_0 Q8_0 545 34 690 / 43
Llama-3.2-1B-Instruct Q4_K_M Q4_K, Q6_K 427 32 611 / 35
TinyLlama-1.1B-Chat Q4_K_M Q4_K, Q6_K 377 38 588 / 43
GPT-2 Q4_K_M Q4_K, Q5_K, Q6_K 2149 196 5335 / 313

With OKLM_THREADS=16 (every hardware thread) prompts are faster (Llama 3.2 1B: 208 t/s) and generation slower (28 t/s): prompts are compute-bound, generation memory-bound.

History of the same command (16 threads before 2026-10-30):

change Llama-3.2-1B Q4_K_M prompt / generation
2026-10-29, one activation row per dot product 104 / 27
four-row kernels (Q4_K, Q6_K), F16C scale conversion 181 / 28
polling worker team, 8 threads by default 169 / 31 (16 threads: 208 / 28)
K-quant kernels: vector scale decode, joint reductions, unrolling, eight-row Q4_K 301 / 31
Q4_K weights repacked eight rows wide, AVX-512 VNNI kernel 404 / 31
Q6_K, Q8_0, Q5_0 and Q4_0 repacked too; decode keeps the file layout 427 / 32
2026-10-31, CPU attention in tiles (F16 cache): query rows x key blocks converted once 563 / 32

CPU attention in tiles, 2026-10-31 (same machine, before / after, prompt t/s): Llama-3.2-1B Q4_K_M 482 / 563 with the 498-token prompt, 282 / 346 with 2060 tokens (8000 bytes); Qwen2.5-0.5B Q4_K_M 929 / 1191 and 492 / 726. One attention op at Llama-3.2-1B's shape (32 heads over 8 KV heads of 64, F16 cache), 512 queries after 1536 cached: 106 to 41 ms; one decode query over 4096 keys: 0.72 to 0.30 ms. The scores and value sums round differently (fused multiply-adds in a fixed order per row), so greedy text can change at near ties; perplexity over 60 kB of tinyshakespeare (1024-token windows, CPU) moves by about 0.05 %: Llama-3.2-1B 37.2659 to 37.2920, Qwen2.5-0.5B 42.6695 to 42.6825. That is the size of a change of rounding alone: the scalar kernels (sums in another order, the matrix products bit-identical) give 42.7273 for Qwen2.5-0.5B before the tiles.

The same tiles for every cache type and for decode, 2026-10-31: block-quant caches (-ctk / -ctv) dequantized a block of keys at a time with the query rounded through Q8_0 as before (it was an integer dot per key and query); decode and small batches, too few tiles for the threads, split the tiles' arithmetic by key blocks, rows and value columns (same bits, so a batch now gives exactly the logits of its tokens one by one); the score kernel runs eight independent chains whatever the rows. Prompt / generation t/s before / after, 128 tokens generated, best of two runs:

model cache 498-token prompt 2060-token prompt generation after 2060
Llama-3.2-1B Q4_K_M f16 521 / 574 361 / 419 29.4 / 29.8
Llama-3.2-1B Q4_K_M q8_0 335 / 482 199 / 341 23.5 / 28.0
Qwen2.5-0.5B Q4_K_M f16 1193 / 1169 713 / 736 52.7 / 52.2
Qwen2.5-0.5B Q4_K_M q8_0 726 / 1102 352 / 648 39.3 / 49.0

q4_0 caches gain as q8_0 (Llama-3.2-1B, 2060 tokens: 197 to 344). Perplexity (same 60 kB, 1024-token windows, CPU) before / after: Llama-3.2-1B f16 37.2920 / 37.2920, q8_0 37.4049 / 37.3907, q4_0 39.7913 / 39.6672; Qwen2.5-0.5B f16 42.6825 / 42.6825, q8_0 43.3546 / 43.2600. Qwen2.5-0.5B with q4_0 keys is unusable either way (318 / 312; llama.cpp's llama-perplexity -ctk q4_0 gives 303, 42.6 with f16 keys).

AVX2 only (OKLM_SIMD=avx2, same machine, 2026-10-30), file layout / repacked weights, prompt t/s, the same greedy text: Llama-3.2-1B Q4_K_M 313 / 409, Qwen2.5-0.5B Q4_K_M 437 / 782, SmolLM2-135M Q8_0 1045 / 1389. bench_matmul 4096 4096 8 <type> 64 1, 64 rows, ms per product, AVX2 file layout / AVX2 repacked / AVX-512 repacked: Q4_K 2.18 / 1.53 / 1.31, Q6_K 2.63 / 2.49 / 1.52, Q8_0 3.70 / 2.21 / 1.26, Q5_0 4.96 / 1.95 / 1.26, Q4_0 4.43 / 1.94 / 1.26, Q5_K 2.49 / 2.10 / 1.67, Q3_K 10.39 / 2.71 / 1.68, Q2_K 5.28 / 2.22 / 1.29, IQ4_NL 4.39 / 2.32 / 1.24, IQ4_XS 4.07 / 2.34 / 1.62 (2026-10-31; IQ4_XS's group sums are now added as integers, one float step per block, as ggml: 3.91 / 2.36 / 1.61). The AVX2 Q6_K kernel then adds two steps' pair sums in int16 and widens them with one multiply-add by the group scales (the offsets once per block): 2.49 to 1.72 ms; prompts on AVX2 only, Llama-3.2-1B Q4_K_M 479 to 504 t/s, Qwen2.5-0.5B Q4_K_M 905 to 934, the same text.

Then, same day, AVX2 / AVX-512 repacked, ms: Q5_K, Q3_K and Q6_K packed with each value in a byte of its own (no bits to put together in the kernels; the packed copies 1.6, 2.2 and 1.2 times those matrices' file size): Q5_K 2.09 / 1.67 to 1.71 / 1.40, Q3_K 2.71 / 1.68 to 1.94 / 1.30, Q6_K 1.72 / 1.52 to 1.29 / 1.30. With AVX-512 (32 vector registers), IQ4_XS and the grid IQ types take eight activation rows per call: IQ4_XS 1.61 to 1.33, IQ2_XXS to IQ1_M 1.97 to 1.58 (the other kernels measured slower that way and keep four). Tiled attention computes its softmax numerators eight at a time (Cephes' exponential, within 2 ulp of expf): the attention of a 498-token Llama-3.2-1B prompt 0.164 to 0.120 s. Prompt t/s, CPU, against llama-bench -p 512 -t 8 on the same files the same evening: Llama-3.2-1B Q4_K_M 588 (llama.cpp 513 to 690 from run to run), Q3_K_M 595 / 704, Q5_K_M 594 / 773, IQ4_XS 546 to 582 / 833; Qwen2.5-0.5B Q4_K_M 1275 / 853.

Then (2026-11-01): the compiler had kept one shared copy of most packed kernels, the activation row count a run-time value; each entry point now gets its own copy (loops unrolled, accumulators in registers) except Q2_K's and, with AVX-512, Q4_K's, which measured faster shared; with that, Q5_K, Q4_0 and IQ4_NL take eight activation rows per call with AVX-512 too. 64-row products, AVX2 / AVX-512, ms: Q5_K 1.70 / 1.40 to 1.53 / 1.20, Q3_K 1.94 / 1.30 to 1.76 / 1.21, Q6_K 1.29 / 1.30 to 1.30 / 1.20, IQ4_XS 2.39 / 1.33 to 2.14 / 1.10, Q4_0 1.94 / 1.26 to 1.97 / 1.19, IQ2_XXS 2.78 / 1.58 to 2.30 / 1.60, Q4_K 1.55 / 1.30 to 1.46 / 1.30. Prompt t/s, CPU: Llama-3.2-1B Q3_K_M 595 to 623 (llama.cpp 704), Q5_K_M 594 to 652 (773), IQ4_XS 582 to 646 (833), Q4_K_M 585. A layout with one 8-byte activation broadcast per two byte dot products was tried on Q8_0 / Q5_0 and measured slower (the kernels are bound by their instructions, not their loads: one thread's speed does not depend on K, and eight threads give 7.3 times one).

Then tiled attention: the scale and maximum of each row of scores eight at a time, its denominator summed in sixteen double lanes in a fixed order (it was one chain of dependent additions), and the value sums of the heads of a token (the same keys) in one pass over the value rows. One attention op at Llama-3.2-1B's shape (32 heads over 8 KV heads of 64, F16 cache, 8 threads): 512 queries from 0, 8.4 to 6.5 ms; after 1536 cached, 33.7 to 26.9 ms. Prompt t/s, CPU: Llama-3.2-1B Q3_K_M 642 (llama.cpp 704), Q5_K_M 679 (773), IQ4_XS 693 (833), Q4_K_M 597; perplexity unchanged to four decimals (37.3498). SiLU and SwiGLU then use the same exponential eight at a time (0.030 to 0.025 s per prompt, bound by memory from there; perplexity 37.3249). Tried and left out: Q4_K in Q5_K's byte layout (AVX-512 1.39 to 1.20 ms, AVX2 1.45 to 1.53, and twice the packed copy's memory for the most common type).

Llama-3.2-1B requantized from its Q4_K_M file (oklm quantize), the command above, 2026-10-31, prompt t/s with the file layout (OKLM_CPU_PACK=0) / repacked, generation t/s, the same greedy text both ways; llama.cpp prompt / generation on the same files:

file weight types prompt t/s generation t/s llama.cpp
Q2_K Q2_K, Q3_K, Q4_K, Q6_K 123 / 402 41 547 / 48
Q3_K_M Q3_K, Q4_K, Q5_K, Q6_K 116 / 371 36 691 / 41
Q5_K_M Q5_K, Q6_K 306 / 396 29 746 / 31
IQ4_XS IQ4_XS, Q5_K, Q6_K 195 / 361 33 930 / 37
IQ4_NL IQ4_NL, Q5_K, Q6_K 185 / 452 31 489 / 36

The grid IQ types (IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S) with AVX2 dot products instead of scalar ones (2026-10-31): bench_matmul 4096 4096 8 <type> 64, 64 rows, 96 to 128 ms per product before, 4.2 (IQ2_S) to 7.7 (IQ1_M) after; one row 0.18 to 0.31 ms. SmolLM2-135M's files of models/smollm2-135m-iq, the command above, prompt / generation t/s before and after, the same greedy text; llama.cpp on the same files:

file before after llama.cpp
IQ1_S 287 / 127 1313 / 178 3737 / 372
IQ1_M 254 / 119 1190 / 174 3615 / 368
IQ2_XXS 253 / 119 1413 / 174 3700 / 322
IQ2_XS 231 / 115 1385 / 176 3152 / 360
IQ2_M 238 / 118 1391 / 182 3164 / 355
IQ3_XXS 277 / 125 1391 / 178 3187 / 338
IQ3_S 266 / 124 1387 / 175 3171 / 339

The same types repacked (oklm_pack_iqg_x8: a 4-bit index per value into the at most 16 integers the dot product multiplies, a scale per 16 columns; about 4.6 bits per weight, so 1.5 to 2.3 times the file's size again in memory), 2026-10-31: bench_matmul 4096 4096 16 <type> 64 1, 64 rows, 3.9 (IQ2_S) to 6.9 ms (IQ1_M) with the AVX2 dot products, 2.67 ms packed with AVX2 and 1.97 ms with AVX-512 VNNI for every type. SmolLM2-135M's files, 2000-byte prompt, 64 tokens, prompt t/s with the file layout (OKLM_CPU_PACK=0) / repacked, the same greedy text: IQ1_S 1124 / 2151, IQ1_M 1102 / 1997, IQ2_XXS 1189 / 2130, IQ2_XS 1201 / 2148, IQ2_M 1207 / 2165, IQ3_XXS 1168 / 2203, IQ3_S 1195 / 2105; generation unchanged (about 185 t/s).

SmolLM2 Q4_K_M ran at 71 / 42 t/s before the AVX2 Q5_0 kernel (most of its matrices are Q5_0, because SmolLM2's 576-wide rows are not a multiple of 256). The BF16 Hugging Face checkpoint of SmolLM2, loaded directly (oklm run -m models/smollm2-135m-hf, short prompt, 64 tokens): 76 t/s generation with the AVX2 BF16 kernel (49 t/s before it).

The Q2_K to Q6_K, Q8_0, Q5_0, Q4_0, IQ4_NL, IQ4_XS and grid IQ weights of the layers run on the CPU are repacked when a context is created (cpu_pack.h, about their size again in memory, 1.2 to 2.2 times for Q6_K, Q5_K and Q3_K, whose values take a byte each; OKLM_CPU_PACK=0 turns it off): eight interleaved rows per block, one output row per 32-bit lane of an AVX-512 VNNI kernel. bench_matmul 4096 4096 8 Q4_K 64 1 (8 threads): 980 GFLOPS with the eight-row kernel, 1646 with the repacked weights; Q6_K 577 to 751, Q8_0 368 to 1146, Q5_0 326 to 1180, Q4_0 479 to 1679 (a loaded machine). Products of fewer than four activation rows (decode) keep the file layout: they are bound by memory, and the file layout is as small or smaller. Since 2026-11-08 one-row products read the packed copy with one thread, and with two when it is at most 10 % larger than the file layout (round 1 above).

Decoding on a busy machine (2026-11-08)

Generation t/s with oklm run -p "The capital of France is" -n 32 --temp 0 --backend cpu, OKLM_THREADS = 1 to 16, beside 0, 8 or 16 processes that only spin (a small C busy loop, standing for stress-ng --cpu N), on the i7-11700K (8 cores, 16 hardware threads). Before: the nilorea N_PARALLEL_TEAM (a static slice per thread, every thread checked in at the end of each loop, 20000 pause polls before sleeping). After: OKLM's team (team_cpu.c), chunks claimed one at a time, four per thread, a short poll then sched_yield() rounds (core.md). llama.cpp: llama-bench -ngl 0 -p 0 -n 32 -r 2 -t N (CPU only, OpenMP build), "-" when it took more than two minutes. Other jobs shared the machine during the whole run (load 15 to 35 before the spinners), so "0 spinners" is not idle; each row ran base, new and llama.cpp back to back.

model spinners threads before after llama.cpp
Granite-3.1-1B-A400M Q4_K_M 0 1 20.2 23.1 16.1
0 2 21.1 37.1 27.4
0 4 43.3 45.2 65.4
0 8 9.0 62.2 78.4
0 16 1.9 68.9 7.1
8 1 21.5 31.4 35.7
8 2 47.3 47.9 61.7
8 4 23.2 62.4 77.8
8 8 1.9 53.9 17.6
8 16 1.0 64.5 -
16 1 23.4 15.9 38.4
16 2 7.3 32.3 45.1
16 4 3.7 19.8 9.3
16 8 1.1 23.4 -
16 16 0.8 26.3 -
Llama-3.2-1B-Instruct Q4_K_M 0 1 5.6 5.8 6.0
0 2 12.5 10.0 11.3
0 4 13.8 14.3 14.4
0 8 10.1 30.5 27.7
0 16 2.6 25.5 5.3
8 1 12.3 11.9 15.1
8 2 20.2 21.2 24.2
8 4 19.2 25.7 30.5
8 8 3.1 24.4 12.4
8 16 1.8 26.8 0.6
16 1 12.1 9.8 11.4
16 2 4.0 10.7 10.4
16 4 2.8 8.5 6.7
16 8 1.8 9.7 0.8
16 16 1.3 11.2 -

One thread runs no team, so its row is noise from the other jobs. Before, a loop waited for its slowest thread; with more runnable threads than hardware threads one of them is always waiting for a core, and each of the few hundred loops of a token paid a scheduler time slice (Granite at 16 threads: 0.8 to 1.9 t/s). After, the decode speed stays within a factor of about two of its best whatever the thread count and the load. llama.cpp is ahead at 2 to 4 threads on a loaded machine, and falls apart like OKLM did once its threads outnumber the free cores (its OpenMP barrier waits for every thread). At a quieter moment of the same evening: Granite with 16 threads and no spinner went from 11.2 to 87.5 t/s (llama.cpp 91.2), 8 threads 84.8 / 71.7 / 90.6 (noise of a few runs).

The wait policy was chosen on the same models (median of 2 to 4 runs, Granite with 8 spinners): 20000 pause polls before yielding gave 31.4 (8 threads) and 26.2 (16 threads) t/s, 1000 polls then 32 yields of 256 polls 42.5 / 44.0, 300 polls then 512 yields of 32 polls (kept) 52.7 / 60.5; with no spinner the choices were within the noise. Two or eight chunks per thread instead of four were within the noise too (about 40 t/s each, four giving 21 and 42 in the same runs, with the long poll).

Decoding on a busy machine, the loops capped by the free cores (2026-11-11)

Two changes (core.md): a row loop's slices take at least 16384 values (a decode step's rotation and cache copies run on the calling thread), and a loop takes no more participants than the cores other programs leave free (their runnable threads, sampled every 100 ms), the kernels that depend on the thread count going by the threads available. Generation t/s as above (64 tokens), the machine otherwise quiet (load 13 to 31 from the spinners themselves), beside 0, 8 or 16 spinning processes; "before" the previous commit, two runs each; llama.cpp llama-bench -ngl 0 -p 0 -n 64 -r 2 -t N, "-" when it took more than two minutes:

model spinners threads before after llama.cpp
Llama-3.2-1B-Instruct Q4_K_M 0 2 / 4 / 8 / 16 32.5 / 34.9 / 34.7 / 34.9 32.6 / 35.0 / 34.8 / 34.8 32.4 / 35.0 / 34.8 / 31.0
8 2 / 4 / 8 / 16 26.6 / 31.8 / 33.3 / 33.4 28.6 / 31.7 / 33.4 / 33.3 27.9 / 33.6 / 31.9 / -
16 2 18.0, 13.1 19.6, 19.9 22.1
16 4 10.0, 11.3 18.2, 20.7 14.7
16 8 16.6, 17.3 18.1, 21.2 1.5
16 16 12.7, 13.5 19.8, 19.4 -
Granite-3.1-1B-A400M Q4_K_M 0 2 / 4 / 8 / 16 83.5 / 94.7 / 80.8 / 91.6 83.4 / 94.8 / 95.1 / 91.0 88.8 / 96.4 / 96.4 / 85.6
8 2 / 4 / 8 / 16 69.8 / 83.8 / 88.4 / 86.9 69.6 / 83.3 / 87.5 / 88.1 70.4 / 90.8 / 83.6 / -
16 2 51.3, 51.4 43.4, 46.0 50.6
16 4 37.5, 36.9 46.8, 40.6 28.0
16 8 35.7, 40.6 43.3, 38.8 2.7
16 16 47.5, 47.1 49.3, 49.1 -

The rows of several thread counts give the mean of two runs (Granite's 80.8 before at 8 threads and no spinner: runs of 64.5 and 97.0, the first one disturbed). With cores free (0 or 8 spinners on 16 hardware threads) nothing changes. With every hardware thread busy, each loop's threads were descheduled while holding chunks: Llama-3.2-1B decoded at about 20 t/s with one thread and 10 to 17 with more; the cap runs the loops on the one thread the free cores allow, at the one-thread speed whatever OKLM_THREADS says (llama.cpp falls apart from eight threads there). The cap alone was not enough at first: with eight threads configured but one running, the decode products still read the file layout, chosen for several threads, at 11 t/s; going by the threads available, they read the packed copy one thread reads fastest (20 t/s). Granite at two threads looked like the one row that lost (51 against 45 t/s, two runs each). Measured again with eight runs each (2026-11-11, beside 16 spinners): 44.3 t/s with the cap against 40.6 without (single runs from 36 to 51 either way), at four threads 45.5 against 37.0; one thread gives 44.8. The loss was the runs' spread, not the cap; Llama-3.2-1B's two threads lose to one as before (14.7 without the cap against 19.4 with it).

OpenCL

See opencl.md: the same files on an AMD Radeon gfx1100 generate 4 to 13 times and process prompts 17 to 71 times faster than the CPU backend (for example Llama-3.2-1B Q4_K_M: 268 against 21 tokens/s, and 5 189 against 93 for the prompt).

Matrix products on OpenCL, 2026-10-30, OKLM_CL_GEMM_TILES=0 (row kernels) against the tiled kernels, Q4_K weights, microseconds (TFLOPS):

N x K x columns row kernels tiled
8192 x 2048 x 512 1956 (8.8) 1228 (14.0)
2048 x 8192 x 512 1863 (9.2) 1404 (12.2)
2048 x 2048 x 512 576 (7.5) 515 (8.4)
512 x 2048 x 512 243 (4.4) 329 (3.3)
2048 x 2048 x 64 137 (3.9) 303 (1.8)

The tiled kernels need many tiles to fill the device's 84 compute units; below 256 tiles of 64 x 64 outputs they split K (up to 8 ranges of at least 4 steps of 32, the partial tiles added in split order), and the row kernels are kept only when K is too short for that. Prompt t/s without / with the split (OKLM_CL_GEMM_SPLIT=1 / default), 2026-10-31, the same greedy text: SmolLM2-135M Q8_0 16 874 / 17 953 (498 tokens) and 10 140 / 10 840 (2060 tokens), Qwen2.5-0.5B Q4_K_M 10 080 / 11 128 and 7 057 / 7 644, Qwen3-0.6B Q8_0 6 415 / 6 689 and 3 042 / 3 129; Llama-3.2-1B, whose products already have enough tiles, unchanged (5 185 / 5 201).

Attention kernels on OpenCL, 2026-10-31, before / after: F16 keys read eight halves at a time in the per-(head, token) kernel and in the tiles (they went through a load per element with a type test), block-quant caches dequantized into the tiles, the tiles kept for heads up to 128 and spans of 1024 keys or more with a float cache (any span with a block quant), split K in the products. Prompt / generation t/s, 498-token and 2060-token prompts, 64 tokens generated, the same greedy text:

file cache 498 tokens 2060 tokens
SmolLM2-135M Q8_0 f16 16 908 / 331 to 21 258 / 366 10 170 / 315 to 15 333 / 351
Qwen2.5-0.5B Q4_K_M f16 10 014 / 254 to 11 777 / 254 7 010 / 220 to 8 659 / 250
Qwen3-0.6B Q8_0 f16 6 423 / 204 to 8 004 / 222 3 038 / 151 to 4 083 / 199
Llama-3.2-1B Q4_K_M f16 5 223 / 266 to 5 135 / 281 4 137 / 187 to 4 459 / 250
Gemma-3-1B Q4_K_M f16 4 417 / 136 to 6 096 / 151 4 005 / 148 to 5 773 / 164
Gemma-2-2B Q4_K_M f16 2 131 / 113 to 2 436 / 125 1 250 / 109 to 1 948 / 121
SmolLM2-135M Q8_0 q8_0 17 124 to 18 311 10 059 to 13 310
Qwen2.5-0.5B Q4_K_M q8_0 9 709 to 10 956 6 603 to 8 049
Llama-3.2-1B Q4_K_M q8_0 4 878 to 5 128 3 628 to 4 262
Gemma-3-1B Q4_K_M q8_0 5 185 to 5 804 4 903 to 5 565

Generation with a q8_0 cache is unchanged (its values are still read element by element).

Prompt attention with few heads, 2026-10-31 (an intermediate step, before the faster loads made the per-(head, token) kernel the better one for Gemma): Gemma 3's 4 heads of 256 gave too few query blocks per 512-token chunk for the tiled kernel (one work-group per (head, token) instead: 1251 us per call for Gemma-3-270M); the keys of each block are now split and merged (1533 us with the merges for the same calls). Prompt t/s before / after, the same greedy text: Gemma-3-270M Q8_0 11 924 / 13 147 (498 tokens) and 10 035 / 10 161 (2060), Gemma-3-1B Q4_K_M 4 418 / 4 643 and 4 024 / 4 044, Gemma-2-2B Q4_K_M 2 145 / 2 125 and 1 245 / 1 293.

MoE decode, oklm run -p "<short prompt>" -n 128 --temp 0 (DeepSeek: 64 tokens), 2026-10-30, before and after routing on the device (expert choice, weights and weighted sums as kernels, indexed expert products, no read-back per layer); the greedy text is the same:

file OpenCL before / after CPU before / after
Granite-3.1-1B-A400M-instruct Q4_K_M 101 / 230 73 / 88
DeepSeek-Coder-V2-Lite-Instruct IQ4_XS 65 / 102 15.4 / 16.3

Training

See training.md: steps on the CPU, opencl and rocm, a full fine-tune against LoRA. 2026-11-04, AMD Radeon RX 7900 XT (gfx1100), ROCm 7.2.4, rocBLAS 4.4, a step of 4 x 256 tokens, LoRA rank 16 on Q, K, V, O:

Model (file) Full, opencl Full, rocm LoRA, opencl LoRA, rocm
SmolLM2-135M (F16) 163 ms 118 ms 126 ms 98 ms
Qwen2.5-0.5B (Q4_K_M) 500 ms 315 ms 322 ms 199 ms
Qwen3-0.6B (Q8_0) 635 ms 410 ms 443 ms 291 ms
Llama-3.2-1B (Q4_K_M) 9 s (beyond the device's memory) out of memory 679 ms 344 ms

rocBLAS's solution for each of the trainer's products against the fastest it lists (build/release/bin/tests/train/bench_rocblas --solutions, microseconds): within 5 % on most, faster for three (SmolLM2's MLP down forward 105 against 67 at 512 rows, its Q/K/V input gradient 57 against 42, the Nano's Q/K/V weight gradient 224 against 103), at most 3 % of a step; the default is kept (solution indices belong to one rocBLAS build).

Tokenizer

Command: build/release/bin/tests/tok/bench_tokenizer (part of make bench): data/tinyshakespeare.txt (1.12 MB) encoded in one call, one thread. Same machine as below, 2026-10-29.

tokenizer MB tokens seconds MB/s tokens/s
GPT-2 1.12 338025 0.133 8.4 2 536 982
Llama 2 (SentencePiece) 1.12 368633 0.748 1.5 492 529
Llama 3.2 1.12 301768 0.361 3.1 835 656
Qwen2.5 (with NFC) 1.12 301829 0.173 6.4 1 743 024
SmolLM2 1.12 341094 0.203 5.5 1 681 483

Llama 2 has no pre-tokenizer, so the whole text is one merge sequence (as in Hugging Face); Llama 3.2 pays for ignore_merges lookups of every piece.

Matrix product, CPU

Command: make bench (tests/core/bench_matmul.c), K = N = 4096, M = 1 (decode: one activation row) and M = 64 (prompt processing). "GB/s(w)" is the weight bytes read per second, the figure that bounds decode speed.

Machine: Intel Core i7-11700K (8 cores / 16 threads, AVX2, AVX-512 with VNNI), 128 GB RAM, Linux 7.2, GCC 16.2, -O2.

16 threads (2026-10-30, every SIMD level)

type kernel K x N M th ms / call GFLOPS GB/s(w)
F32 scalar 4096 x 4096 1 16 2.219 15.1 30.2
F32 avx2 4096 x 4096 1 16 2.177 15.4 30.8
F32 avx512 4096 x 4096 1 16 2.164 15.5 31.0
F32 scalar 4096 x 4096 64 16 17.451 123.1 3.8
F32 avx2 4096 x 4096 64 16 8.960 239.7 7.5
F32 avx512 4096 x 4096 64 16 9.068 236.8 7.4
F16 scalar 4096 x 4096 1 16 2.157 15.6 15.6
F16 avx2 4096 x 4096 1 16 0.858 39.1 39.1
F16 avx512 4096 x 4096 1 16 0.872 38.5 38.5
F16 scalar 4096 x 4096 64 16 127.620 16.8 0.3
F16 avx2 4096 x 4096 64 16 6.541 328.3 5.1
F16 avx512 4096 x 4096 64 16 6.419 334.6 5.2
BF16 scalar 4096 x 4096 1 16 1.017 33.0 33.0
BF16 avx2 4096 x 4096 1 16 0.877 38.3 38.3
BF16 avx512 4096 x 4096 1 16 0.863 38.9 38.9
BF16 scalar 4096 x 4096 64 16 47.687 45.0 0.7
BF16 avx2 4096 x 4096 64 16 6.507 330.0 5.2
BF16 avx512 4096 x 4096 64 16 6.501 330.3 5.2
Q8_0 scalar 4096 x 4096 1 16 0.310 108.4 57.6
Q8_0 avx2 4096 x 4096 1 16 0.203 165.3 87.8
Q8_0 avx512 4096 x 4096 1 16 0.214 156.5 83.2
Q8_0 scalar 4096 x 4096 64 16 16.189 132.7 1.1
Q8_0 avx2 4096 x 4096 64 16 8.029 267.5 2.2
Q8_0 avx512 4096 x 4096 64 16 8.258 260.0 2.2
Q4_0 scalar 4096 x 4096 1 16 0.321 104.4 29.4
Q4_0 avx2 4096 x 4096 1 16 0.203 164.9 46.4
Q4_0 avx512 4096 x 4096 1 16 0.207 162.1 45.6
Q4_0 scalar 4096 x 4096 64 16 18.903 113.6 0.5
Q4_0 avx2 4096 x 4096 64 16 8.867 242.2 1.1
Q4_0 avx512 4096 x 4096 64 16 8.699 246.9 1.1
Q4_1 scalar 4096 x 4096 1 16 0.396 84.8 26.5
Q4_1 avx2 4096 x 4096 1 16 0.234 143.6 44.9
Q4_1 avx512 4096 x 4096 1 16 0.229 146.4 45.8
Q4_1 scalar 4096 x 4096 64 16 23.336 92.0 0.4
Q4_1 avx2 4096 x 4096 64 16 13.346 160.9 0.8
Q4_1 avx512 4096 x 4096 64 16 13.089 164.1 0.8
Q5_0 scalar 4096 x 4096 1 16 2.212 15.2 5.2
Q5_0 avx2 4096 x 4096 1 16 0.232 144.9 49.8
Q5_0 avx512 4096 x 4096 1 16 0.231 145.3 50.0
Q5_0 scalar 4096 x 4096 64 16 131.219 16.4 0.1
Q5_0 avx2 4096 x 4096 64 16 8.976 239.2 1.3
Q5_0 avx512 4096 x 4096 64 16 8.854 242.5 1.3
Q5_1 scalar 4096 x 4096 1 16 1.674 20.1 7.5
Q5_1 avx2 4096 x 4096 1 16 0.265 126.8 47.5
Q5_1 avx512 4096 x 4096 1 16 0.264 127.2 47.7
Q5_1 scalar 4096 x 4096 64 16 101.385 21.2 0.1
Q5_1 avx2 4096 x 4096 64 16 14.868 144.4 0.8
Q5_1 avx512 4096 x 4096 64 16 14.934 143.8 0.8
Q2_K scalar 4096 x 4096 1 16 0.318 105.6 17.3
Q2_K avx2 4096 x 4096 1 16 0.143 234.3 38.4
Q2_K avx512 4096 x 4096 1 16 0.142 235.8 38.7
Q2_K scalar 4096 x 4096 64 16 18.951 113.3 0.3
Q2_K avx2 4096 x 4096 64 16 7.793 275.6 0.7
Q2_K avx512 4096 x 4096 64 16 7.764 276.6 0.7
Q3_K scalar 4096 x 4096 1 16 1.952 17.2 3.7
Q3_K avx2 4096 x 4096 1 16 0.289 115.9 24.9
Q3_K avx512 4096 x 4096 1 16 0.291 115.5 24.8
Q3_K scalar 4096 x 4096 64 16 118.873 18.1 0.1
Q3_K avx2 4096 x 4096 64 16 17.040 126.0 0.4
Q3_K avx512 4096 x 4096 64 16 16.942 126.8 0.4
Q4_K scalar 4096 x 4096 1 16 0.223 150.7 42.4
Q4_K avx2 4096 x 4096 1 16 0.123 272.1 76.5
Q4_K avx512 4096 x 4096 1 16 0.124 271.2 76.3
Q4_K scalar 4096 x 4096 64 16 12.851 167.1 0.7
Q4_K avx2 4096 x 4096 64 16 4.631 463.7 2.0
Q4_K avx512 4096 x 4096 64 16 4.630 463.8 2.0
Q5_K scalar 4096 x 4096 1 16 0.296 113.3 38.9
Q5_K avx2 4096 x 4096 1 16 0.147 228.8 78.7
Q5_K avx512 4096 x 4096 1 16 0.147 228.2 78.4
Q5_K scalar 4096 x 4096 64 16 17.631 121.8 0.7
Q5_K avx2 4096 x 4096 64 16 4.857 442.1 2.4
Q5_K avx512 4096 x 4096 64 16 4.961 432.9 2.3
Q6_K scalar 4096 x 4096 1 16 1.762 19.0 7.8
Q6_K avx2 4096 x 4096 1 16 0.136 246.3 101.0
Q6_K avx512 4096 x 4096 1 16 0.119 280.8 115.2
Q6_K scalar 4096 x 4096 64 16 108.128 19.9 0.1
Q6_K avx2 4096 x 4096 64 16 4.909 437.4 2.8
Q6_K avx512 4096 x 4096 64 16 3.563 602.8 3.9

1 thread (2026-10-29, before the four-row and AVX-512 kernels)

type kernel K x N M th ms / call GFLOPS GB/s(w)
Q8_0 scalar 4096 x 4096 1 1 4.368 7.7 4.1
Q8_0 avx2 4096 x 4096 1 1 2.419 13.9 7.4
Q8_0 scalar 4096 x 4096 64 1 125.553 17.1 0.1
Q8_0 avx2 4096 x 4096 64 1 113.786 18.9 0.2
Q4_K scalar 4096 x 4096 1 1 1.580 21.2 6.0
Q4_K avx2 4096 x 4096 1 1 0.968 34.7 9.8
Q4_K scalar 4096 x 4096 64 1 107.792 19.9 0.1
Q4_K avx2 4096 x 4096 64 1 63.003 34.1 0.1

Indexed (expert) product, 8 threads (2026-11-07)

build/release/bin/tests/core/bench_matmul_id 8 (tests/core/bench_matmul_id.c): a Qwen3-30B-A3B-like layer, 128 experts, 8 per token, n_embd 2048, expert feed-forward 768, Q4_K, a Zipf routing over the experts ("exp": experts used by the batch); "gate" shares one input row between a token's ranks (gate and up), "down" has one per rank. Milliseconds per call, the best of two runs, before / after grouping the rows by expert (the same bits), on a machine busy with other jobs:

toks exp gate before / after down before / after speed-up
1 8 0.096 / 0.069 0.096 / 0.094 1.0-1.4
16 59 2.150 / 1.748 2.308 / 1.914 1.2
128 124 23.21 / 7.65 22.99 / 8.64 2.7-3.0
512 128 92.80 / 24.44 96.15 / 28.56 3.4-3.8

The whole expert block of that layer (gate, up, SwiGLU, down) for prompt batches, both ways the model can run it on the CPU (second table of the same command): one ordinary product per expert after sorting the pairs on the host, with the gathers in and out (the model's path for every batch above n x 8 > 128 until 2026-11-07), and the indexed products (its path since then). Milliseconds per call, the best of three runs on a machine busy with other jobs; both give the same bits:

toks per expert indexed speed-up
128 28.56 19.75 1.45
512 75.26 57.57 1.31
2048 276.06 225.92 1.22

End to end, the same switch on a real MoE file: Granite-3.1-1B-A400M Q4_K_M (32 experts, 8 per token, n_embd 1024, expert feed-forward 512), CPU, 8 threads, a prompt of n random tokens from an empty context (oklm_eval(), batches of 512), F16 KV cache, best of three runs of four; OKLM_MOE_EXPERT_LOOP=1 (one product per expert) / default (indexed products), prompt t/s; the last logits are bit-identical:

prompt per expert t/s indexed t/s speed-up
128 545 750 1.38
512 522 692 1.33
2048 469 544 1.16

Reading the numbers

  • Decode (M = 1) is memory-bound: Q4_K and Q6_K reach 36-40 GB/s of weights, a large share of what dual-channel DDR4-3200 can deliver (51.2 GB/s theoretical; the module configuration of this machine was not recorded). F32 decode reads 64 MB per call and is bandwidth-limited on both kernels; the difference between them is run-to-run noise.
  • GCC auto-vectorizes some scalar kernels well (Q4_K scalar is fast); the scalar F16 and Q6_K kernels are not vectorized and show it.
  • Prompt processing (M = 64) of quantized weights is compute-bound. The four-row kernels unpack each weight block once for four activation rows and reduce the four columns together; with the F16C scale conversion they brought Q4_K from 310 to 415 GFLOPS, Q8_0 from 134 to 228, Q5_K from 237 to 400 (16 threads, 2048 x 2048), all bit-identical to the scalar kernels. AVX-512 + VNNI helps Q6_K (365 -> 500); for Q4_K it measured no faster than AVX2 and is not used.