Benchmarks
Measured performance of OKLM's kernels, with the machine, the command and the date of each run, so regressions and improvements can be tracked. Numbers are informational; nothing gates on them.
Status: maintained.
Against llama.cpp
sh tools/bench_vs_llama.sh measures OKLM and llama.cpp
on the same GGUF files the way llama-bench does:
pp512, 512 random tokens in one batch from an empty cache,
and tg128, 128 tokens decoded one at a time from an empty
cache, each the best of three runs after one warm-up. OKLM runs
tests/model/bench_llama.c
(make build/release/bin/tests/model/bench_llama, threads
from OKLM_THREADS), llama.cpp its llama-bench
(-o json, the best of the per-run samples), one process at
a time, each run waiting until at most 1.5 hardware threads are busy
(/proc/stat over two seconds). The CPU rows use the CPU
build of the pinned llama.cpp (build/llama.cpp/build,
GGML_NATIVE, AVX-512, its repacked x86 weights on); the GPU
rows OKLM's OpenCL backend (its defaults: since round 4 the 8-bit matrix
units for the quantized prompt products, the F32 tile kernels before;
since round 7 the K-quants' larger decode products on 8-bit dot
products; then OKLM_CL_WMMA=1) against llama.cpp's HIP
build with every layer on the GPU (-ngl 99), built for this
comparison:
cmake -B build-hip -DGGML_HIP=ON -DGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF \
-Dhip_DIR=/usr/lib/x86_64-linux-gnu/cmake/hip -Dhip-lang_DIR=/usr/lib/x86_64-linux-gnu/cmake/hip-lang \
-Dhipblas_DIR=/usr/lib/x86_64-linux-gnu/cmake/hipblas -Drocblas_DIR=/usr/lib/x86_64-linux-gnu/cmake/rocblas \
-DCMAKE_PREFIX_PATH=/usr -DCMAKE_HIP_FLAGS=-DHIP_ENABLE_WARP_SYNC_BUILTINS
cmake --build build-hip --target llama-bench -j16 # 215 s
(the Debian ROCm 6.4 packages; /opt/rocm there holds no
hipcc, and HIP 6.4 hides __ballot_sync behind
that define). The script's variables: THREADS (default
"1 4 8 16"), GPU, REPEATS,
PP, TG, LLAMA,
GPU_LLAMA, LOAD_MAX; arguments: GGUF files
(default: the models below that exist).
Current standing (after round 11, 2026-11-12)
The block closed after round 9 (the roadmap's 4); round 10 took the two CPU decode rows left. Ratios are OKLM / llama.cpp, t/s; the tables to read are after rounds 7 and 8 (CPU), then round 9 (CPU) for Llama-3.2-1B, round 10 (CPU) for SmolLM2 and GLM-4.7-Flash decoding, round 9 (GPU), and round 11 (GPU) for the prompts of Llama-3.2-1B and Qwen3-8B Q4_K_M.
- CPU prompts: ahead on every row (1.02 to 1.94; Qwen3-8B Q8_0 1.71 to 1.80, the MoE models 1.65 to 1.94).
- CPU decoding: level or ahead from four threads on every model (1.00 to 1.09; SmolLM2 1.07 to 1.09 and GLM-4.7-Flash 1.02 at four and eight threads since round 10, the other models' rows not measured again); one thread 1.09 to 1.56.
- GPU prompts: ahead on six of eight models (SmolLM2 1.15 to 1.37, Qwen3-8B Q8_0 1.65, GLM-4.7-Flash 1.04, Granite 1.06); behind on Llama-3.2-1B (0.84 since round 11, 0.77 before), Qwen3-8B Q4_K_M (0.86, 0.80 before) and OLMoE (0.85).
- GPU decoding: SmolLM2 ahead (1.04 to 1.17); the dense models 0.90 to 0.95; the MoE models 0.77 (Granite) to 0.83 (GLM-4.7-Flash). Since round 10 (2026-11-12): Granite 1.09, OLMoE 0.88, GLM-4.7-Flash 0.89.
- Under load: with spinning processes beside both, OKLM now degrades gently (decoding on a busy machine).
The gaps left and why they are not pursued are in the roadmap's known speed gaps.
First measure (2026-11-08)
i7-11700K (8 cores, 16 hardware threads, AVX-512 VNNI), 128 GB DDR4, Radeon gfx1100 (20 GB), nothing else running (load average 3.3 from the earlier build when it started, 0.04 hardware threads busy). OKLM at 665e6ae, llama.cpp v0.8.0. t/s, OKLM / llama.cpp; ratio above 1: OKLM faster.
| model | backend | threads | pp512 OKLM / llama.cpp | ratio | tg128 OKLM / llama.cpp | ratio |
|---|---|---|---|---|---|---|
| SmolLM2-135M Q8_0 | cpu | 1 | 485 / 468 | 1.04 | 82.4 / 124 | 0.66 |
| cpu | 4 | 1798 / 1680 | 1.07 | 164 / 190 | 0.86 | |
| cpu | 8 | 3234 / 2950 | 1.10 | 166 / 187 | 0.89 | |
| cpu | 16 | 3719 / 3043 | 1.22 | 154 / 177 | 0.87 | |
| opencl | - | 23637 / 29159 | 0.81 | 417 / 381 | 1.09 | |
| opencl wmma | - | 31296 / 29159 | 1.07 | 408 / 381 | 1.07 | |
| SmolLM2-135M Q4_K_M | cpu | 1 | 507 / 449 | 1.13 | 85.3 / 128 | 0.67 |
| cpu | 4 | 1886 / 1580 | 1.19 | 212 / 253 | 0.84 | |
| cpu | 8 | 3359 / 2753 | 1.22 | 217 / 252 | 0.86 | |
| cpu | 16 | 3774 / 2815 | 1.34 | 197 / 234 | 0.84 | |
| opencl | - | 22737 / 31372 | 0.72 | 371 / 384 | 0.97 | |
| opencl wmma | - | 29501 / 31372 | 0.94 | 369 / 384 | 0.96 | |
| Llama-3.2-1B Q4_K_M | cpu | 1 | 98.4 / 111 | 0.88 | 18.8 / 22.3 | 0.85 |
| cpu | 4 | 344 / 404 | 0.85 | 35.4 / 36.0 | 0.98 | |
| cpu | 8 | 650 / 713 | 0.91 | 35.1 / 35.1 | 1.00 | |
| cpu | 16 | 671 / 679 | 0.99 | 33.0 / 33.1 | 1.00 | |
| opencl | - | 5483 / 17839 | 0.31 | 348 / 412 | 0.84 | |
| opencl wmma | - | 9447 / 17839 | 0.53 | 356 / 412 | 0.86 | |
| Qwen3-8B Q4_K_M | cpu | 1 | 15.1 / 15.6 | 0.97 | 3.31 / 3.91 | 0.85 |
| cpu | 4 | 53.8 / 56.5 | 0.95 | 6.19 / 6.24 | 0.99 | |
| cpu | 8 | 86.6 / 91.5 | 0.95 | 6.11 / 6.11 | 1.00 | |
| cpu | 16 | 88.9 / 89.1 | 1.00 | 5.78 / 5.86 | 0.99 | |
| opencl | - | 855 / 3253 | 0.26 | 83.3 / 106 | 0.78 | |
| opencl wmma | - | 1386 / 3253 | 0.43 | 83.3 / 106 | 0.78 | |
| Qwen3-8B Q8_0 | cpu | 1 | 16.3 / 9.47 | 1.72 | 1.69 / 2.66 | 0.64 |
| cpu | 4 | 58.8 / 33.2 | 1.77 | 3.56 / 3.73 | 0.95 | |
| cpu | 8 | 94.0 / 56.0 | 1.68 | 3.57 / 3.60 | 0.99 | |
| cpu | 16 | 101 / 57.4 | 1.77 | 3.39 / 3.43 | 0.99 | |
| opencl | - | 876 / 1633 | 0.54 | 65.7 / 74.3 | 0.88 | |
| opencl wmma | - | 1504 / 1633 | 0.92 | 65.2 / 74.3 | 0.88 | |
| Granite-3.1-1B-A400M Q4_K_M | cpu | 1 | 134 / 139 | 0.96 | 50.8 / 59.6 | 0.85 |
| cpu | 4 | 503 / 498 | 1.01 | 96.4 / 97.7 | 0.99 | |
| cpu | 8 | 908 / 834 | 1.09 | 97.5 / 98.5 | 0.99 | |
| cpu | 16 | 830 / 854 | 0.97 | 89.6 / 94.2 | 0.95 | |
| opencl | - | 4872 / 15421 | 0.32 | 242 / 365 | 0.66 | |
| opencl wmma | - | 5480 / 15421 | 0.36 | 241 / 365 | 0.66 | |
| OLMoE-1B-7B Q4_K_M | cpu | 1 | 56.3 / 56.2 | 1.00 | 19.8 / 23.0 | 0.86 |
| cpu | 4 | 201 / 201 | 1.00 | 37.4 / 38.1 | 0.98 | |
| cpu | 8 | 372 / 318 | 1.17 | 37.5 / 37.9 | 0.99 | |
| cpu | 16 | 391 / 347 | 1.13 | 35.4 / 36.3 | 0.98 | |
| opencl | - | 2920 / 10299 | 0.28 | 265 / 362 | 0.73 | |
| opencl wmma | - | 3895 / 10299 | 0.38 | 259 / 362 | 0.71 | |
| GLM-4.7-Flash Q4_K_M | cpu | 1 | 15.3 / 17.3 | 0.89 | 5.85 / 6.86 | 0.85 |
| cpu | 4 | 54.9 / 59.2 | 0.93 | 11.2 / 11.5 | 0.97 | |
| cpu | 8 | 88.2 / 92.7 | 0.95 | 11.2 / 11.5 | 0.97 | |
| cpu | 16 | 94.6 / 88.7 | 1.07 | 10.6 / 10.8 | 0.98 | |
| opencl | - | 785 / 2090 | 0.38 | 67.9 / 88.2 | 0.77 | |
| opencl wmma | - | 986 / 2090 | 0.47 | 68.2 / 88.2 | 0.77 |
Reading it:
- CPU prompts: level or ahead on most rows (Q8_0 1.7 times llama.cpp, SmolLM2 up to 1.34, OLMoE 1.13 to 1.17 from 8 threads); behind on Llama-3.2-1B Q4_K_M (0.85 to 0.91 below 16 threads), GLM-4.7-Flash (0.89 to 0.95 below 16) and Qwen3-8B Q4_K_M (0.95 to 0.97), at every thread count alike: the kernels, not the thread team.
- CPU decoding: behind at one thread on every model (0.64 to 0.86), level from four threads (0.95 to 1.00) on everything but SmolLM2 (0.84 to 0.89), where both are bound by memory. The one-thread gap is the matrix-vector kernel: on the file layout, each output row is one dot product whose blocks add into one float in block order (the order that keeps every kernel's bits equal to the scalar one), so one thread waits on that chain; the packed copies the prompt kernels read (eight rows side by side, the same bits) were not used for one-row products.
- GPU: prompts are the largest gap, 0.26 to 0.54 with the F32 tile kernels and 0.36 to 0.94 with the matrix units (0.92 on Qwen3-8B Q8_0, 1.07 on SmolLM2 Q8_0); decoding 0.66 (Granite, MoE) to 0.97, the MoE models worst (0.66 to 0.77).
Round 1: decoding on the packed copy with one or two threads
One-row products read the packed copy (eight rows side by side) with
one thread, and with two when it is at most 10 % larger than the file
layout (Q8_0 1.06 times, Q4_K 1.06; Q6_K 1.31 and Q5_0 1.64 stay on the
file layout). The same bits (test_matmul checks it at one
and two threads; the logits of 100 prompt and 16 decoded tokens hash the
same). tg128 at one thread, before / after (llama.cpp): SmolLM2 Q8_0
82.4 / 133 (123), SmolLM2 Q4_K_M 85.3 / 142 (128), Llama-3.2-1B 18.8 /
24.5 (22.4), Qwen3-8B Q4_K_M 3.31 / 4.35 (3.92), Qwen3-8B Q8_0 1.69 /
3.11 (2.66), Granite 50.8 / 52.1 (59.6), OLMoE 19.8 / 21.1 (23.1),
GLM-4.7-Flash 5.85 / 6.45 (6.90); prompts unchanged. Two threads, before
/ after: Llama-3.2-1B 31.4 / 32.2, SmolLM2 Q8_0 136 / 162, SmolLM2
Q4_K_M 150 / 155. Measured and left out: every one-row product on the
packed copy at three threads and more (Llama-3.2-1B 35.6 to 31.6 at
three, 35.3 to 30.1 at eight: the larger Q6_K copy, bound by memory),
and a four-row kernel on the file layout with the four sums side by side
(Q8_0 +6 % at one thread, Q4_K 18.9 to 12.8). Four threads and more run
the code they ran before. The MoE models' expert products have no packed
copy, which is what is left of their one-thread gap.
Round 2: GPU prompts on larger matrix-unit tiles, tiled attention from 128 keys, one-column products
Three changes to the OpenCL backend (opencl.md): w2_<type>
replaces wm_<type> (OKLM_CL_WMMA) for
prompts of 64 columns and more (activations rounded to F16 once per
product, tiles of 64 rows by 128 columns, 64 values of k a step);
prompts with a float cache take the tiled attention from 128 keys
instead of 1024 (OKLM_CL_ATT_KEYS); one-column products
(decode, MoE indexed products) run mv_<type>, the
mm_<type> kernel without the reduction of seven
absent columns (the same bits). Greedy tokens after a 190-token prompt:
the same before and after on the seven models, and the same with and
without the matrix units (top logit within 0.0023 of the 1024-key
attention, Granite 0.064). GPU rows, before / after:
| model | pp512 F32 tiles | pp512 matrix units | tg128 |
|---|---|---|---|
| SmolLM2-135M Q8_0 | 23637 / 26925 | 31296 / 37474 | 417 / 430 |
| SmolLM2-135M Q4_K_M | 22737 / 25557 | 29501 / 35484 | 371 / 387 |
| Llama-3.2-1B Q4_K_M | 5483 / 5910 | 9447 / 14157 | 348 / 369 |
| Qwen3-8B Q4_K_M | 855 / 895 | 1386 / 2624 | 83.3 / 81.8 |
| Qwen3-8B Q8_0 | 876 / 914 | 1504 / 2775 | 65.7 / 65.2 |
| Granite-3.1-1B-A400M Q4_K_M | 4872 / 5162 | 5480 / 5585 | 242 / 251 |
| OLMoE-1B-7B Q4_K_M | 2920 / 2989 | 3895 / 4175 | 265 / 267 |
| GLM-4.7-Flash Q4_K_M | 785 / 774 | 986 / 1018 | 67.9 / 71.5 |
llama.cpp (HIP) in the same run: 29337, 31259, 17530, 3195, 1651,
15416, 10292, 2066 t/s prompts; 380, 385, 414, 106, 74.3, 365, 361, 88.3
decoding. With the matrix units the dense prompts are now 0.81 to 1.68
times llama.cpp's; the F32 tile kernels stay at 0.28 to 0.92 (they
cannot match a matrix-unit build). Qwen3-8B Q4_K_M decoding is within
the noise (83.3 / 81.8). Profile of a Llama-3.2-1B prompt
(OKLM_CL_PROFILE=1) before the attention change: 22 ms of
matrix products (about 45 TFLOPS), 12.7 ms of attention on the per-token
kernel, 2.7 ms of element-wise ops, for 43.5 ms of wall time against
llama.cpp's 28.7.
Round 3: MoE prompts grouped by expert on the GPU
Device batches of more rows than experts used to read the chosen
experts back and run one product per expert (four launches per expert
and layer, and a split-K reduction each). The indexed product now groups
the rows on the device (moe_group) and runs every expert's
tiles in one launch (gi_<type>,
wi_<type>; opencl.md, moe.md). A first version
sorted the rows with one item of the work-group (in a fixed order):
Granite 5585 to 7555 t/s; with the rows placed by every item (local
atomics) 15341. The same greedy tokens as the row kernels on the three
models, and the same bits from run to run. GPU rows, before / after
(llama.cpp):
| model | pp512 F32 tiles | pp512 matrix units | tg128 |
|---|---|---|---|
| Granite-3.1-1B-A400M Q4_K_M | 5162 / 9730 | 5585 / 15341 (15345) | 251 / 254 (364) |
| OLMoE-1B-7B Q4_K_M | 2989 / 4060 | 4175 / 8614 (10284) | 267 / 271 (358) |
| GLM-4.7-Flash Q4_K_M | 774 / 807 | 1018 / 1304 (2067) | 71.5 / 71.8 (88.1) |
After the three rounds (2026-11-08)
The same script, machine and llama.cpp builds, every row again (load average 0.74 when it started, 0.06 hardware threads busy). OKLM at f353351.
| model | backend | threads | pp512 OKLM / llama.cpp | ratio | tg128 OKLM / llama.cpp | ratio |
|---|---|---|---|---|---|---|
| SmolLM2-135M Q8_0 | cpu | 1 | 487 / 469 | 1.04 | 133 / 124 | 1.07 |
| cpu | 4 | 1804 / 1683 | 1.07 | 166 / 194 | 0.86 | |
| cpu | 8 | 3229 / 2974 | 1.09 | 166 / 188 | 0.88 | |
| cpu | 16 | 3719 / 3076 | 1.21 | 154 / 178 | 0.87 | |
| opencl | - | 27060 / 29500 | 0.92 | 427 / 381 | 1.12 | |
| opencl wmma | - | 37231 / 29500 | 1.26 | 433 / 381 | 1.14 | |
| SmolLM2-135M Q4_K_M | cpu | 1 | 509 / 450 | 1.13 | 143 / 128 | 1.11 |
| cpu | 4 | 1885 / 1580 | 1.19 | 213 / 254 | 0.84 | |
| cpu | 8 | 3351 / 2783 | 1.20 | 217 / 254 | 0.86 | |
| cpu | 16 | 3762 / 2854 | 1.32 | 197 / 239 | 0.82 | |
| opencl | - | 25895 / 31323 | 0.83 | 385 / 383 | 1.01 | |
| opencl wmma | - | 35153 / 31323 | 1.12 | 380 / 383 | 0.99 | |
| Llama-3.2-1B Q4_K_M | cpu | 1 | 99.2 / 111 | 0.89 | 24.4 / 22.4 | 1.09 |
| cpu | 4 | 345 / 397 | 0.87 | 35.6 / 36.1 | 0.99 | |
| cpu | 8 | 658 / 718 | 0.92 | 35.2 / 35.5 | 0.99 | |
| cpu | 16 | 671 / 689 | 0.97 | 33.1 / 33.8 | 0.98 | |
| opencl | - | 6010 / 17809 | 0.34 | 360 / 414 | 0.87 | |
| opencl wmma | - | 14510 / 17809 | 0.81 | 339 / 414 | 0.82 | |
| Qwen3-8B Q4_K_M | cpu | 1 | 15.2 / 15.6 | 0.98 | 4.32 / 3.92 | 1.10 |
| cpu | 4 | 54.1 / 56.5 | 0.96 | 6.23 / 6.29 | 0.99 | |
| cpu | 8 | 87.2 / 92.4 | 0.94 | 6.12 / 6.15 | 1.00 | |
| cpu | 16 | 89.6 / 89.3 | 1.00 | 5.78 / 5.85 | 0.99 | |
| opencl | - | 905 / 3236 | 0.28 | 82.5 / 106 | 0.78 | |
| opencl wmma | - | 2645 / 3236 | 0.82 | 82.0 / 106 | 0.77 | |
| Qwen3-8B Q8_0 | cpu | 1 | 16.4 / 9.46 | 1.74 | 3.13 / 2.66 | 1.18 |
| cpu | 4 | 58.8 / 33.5 | 1.76 | 3.55 / 3.73 | 0.95 | |
| cpu | 8 | 94.9 / 56.1 | 1.69 | 3.57 / 3.60 | 0.99 | |
| cpu | 16 | 101 / 57.2 | 1.77 | 3.39 / 3.43 | 0.99 | |
| opencl | - | 927 / 1629 | 0.57 | 65.4 / 74.4 | 0.88 | |
| opencl wmma | - | 2791 / 1629 | 1.71 | 65.6 / 74.4 | 0.88 | |
| Granite-3.1-1B-A400M Q4_K_M | cpu | 1 | 137 / 139 | 0.98 | 52.1 / 59.4 | 0.88 |
| cpu | 4 | 501 / 496 | 1.01 | 96.4 / 99.0 | 0.97 | |
| cpu | 8 | 891 / 833 | 1.07 | 97.0 / 98.0 | 0.99 | |
| cpu | 16 | 993 / 854 | 1.16 | 89.7 / 93.6 | 0.96 | |
| opencl | - | 9978 / 14356 | 0.70 | 247 / 321 | 0.77 | |
| opencl wmma | - | 15401 / 14356 | 1.07 | 250 / 321 | 0.78 | |
| OLMoE-1B-7B Q4_K_M | cpu | 1 | 56.2 / 56.2 | 1.00 | 21.1 / 23.2 | 0.91 |
| cpu | 4 | 201 / 203 | 0.99 | 37.3 / 38.1 | 0.98 | |
| cpu | 8 | 374 / 318 | 1.18 | 37.5 / 37.9 | 0.99 | |
| cpu | 16 | 392 / 325 | 1.21 | 35.5 / 36.3 | 0.98 | |
| opencl | - | 4154 / 10449 | 0.40 | 260 / 361 | 0.72 | |
| opencl wmma | - | 8740 / 10449 | 0.84 | 256 / 361 | 0.71 | |
| GLM-4.7-Flash Q4_K_M | cpu | 1 | 15.3 / 17.3 | 0.89 | 6.45 / 6.87 | 0.94 |
| cpu | 4 | 55.0 / 59.7 | 0.92 | 11.2 / 11.6 | 0.97 | |
| cpu | 8 | 88.3 / 92.9 | 0.95 | 11.2 / 11.5 | 0.97 | |
| cpu | 16 | 94.4 / 92.5 | 1.02 | 10.6 / 11.0 | 0.96 | |
| opencl | - | 820 / 2088 | 0.39 | 71.6 / 88.1 | 0.81 | |
| opencl wmma | - | 1320 / 2088 | 0.63 | 71.1 / 88.1 | 0.81 |
Where llama.cpp stays ahead, and why:
- GPU prompts on the F32 tile kernels (0.28 to 0.92): llama.cpp's HIP
build multiplies on the matrix units; OKLM does too with
OKLM_CL_WMMA=1(0.63 to 1.71 here), but its results are not the F32 kernels' bits and greedy text can part at near ties, so it stays opt-in. - GPU prompts with the matrix units: Llama-3.2-1B and Qwen3-8B Q4_K_M 0.81 to 0.82, OLMoE 0.84 (the matrix products are within about 10 % of llama.cpp's; the rest is element-wise ops and the launches between them), GLM-4.7-Flash 0.63: its attention (MLA, heads of 576) has no tiled kernel (the tiles stop at 256 dimensions, and a 576-wide tile does not fit 64 KB of local memory with these block sizes), so the per-token kernel takes half of the prompt's GPU time (4.1 ms per layer).
- GPU decoding (0.71 to 0.88, the small SmolLM2 ahead): the matrix-vector products run at 570 to 590 GB/s, near this GPU's bandwidth; the gap is the 500 to 550 launches per token (RMS norms of single heads, element wise ops, MoE routing, each 3 to 8 microseconds on the device): 3.1 ms of kernels per Granite token against llama.cpp's 2.75 ms in all, which fuses norms with their products, the MoE routing into one kernel, and replays the token as a HIP graph.
- CPU prompts on the K-quant files: Llama-3.2-1B 0.87 to 0.92 below 16 threads, GLM-4.7-Flash 0.89 to 0.95, Qwen3-8B Q4_K_M 0.94 to 0.98: 85 % of a Llama-3.2-1B prompt is in matrix products (Q4_K 73 %) at about a third of the AVX-512 VNNI peak; the packed Q4_K kernel spends a third of its instructions outside the dot products (nibble unpacking, the scale multiplies, the mins term).
- CPU decoding of the MoE models at one thread (0.88 to 0.94): their expert matrices have no packed copy, so round 1 does not reach them.
- CPU decoding of SmolLM2 from four threads (0.82 to 0.88): 211 matrix products per token of 26 microseconds on average, so the team's start and end of each loop (a few microseconds) weigh; llama.cpp runs a token's graph on one thread pool with one barrier per node.
Round 4 (GPU): GPU prompts on the 8-bit matrix units, fused launches, latent-attention tiles
Three changes to the OpenCL backend (opencl.md), measured with
THREADS="" sh tools/bench_vs_llama.sh (the GPU rows only,
load 2.9 when it started, 0.08 hardware threads busy):
- Prompts on the 8-bit matrix units, by default. On a
gfx11 device the products of 64 columns or more with Q4_K, Q5_K, Q6_K,
Q8_0, Q4_0 or Q5_0 weights round the activations to 8 bits exactly as
the CPU rounds them (
q8_act) and multiply the integer levels withv_wmma_i32_16x16x16_iu8(wq_<type>, andwqi_<type>for the grouped expert products of MoE prompts), each block's integer sum scaled and added as the CPU's dot product does it: the CPU's products bit for bit (test_opencl_q8: every output of 30 products and 12 indexed products equal), so the switch is on by default (OKLM_CL_Q8=0: the F32 kernels). Steps that paid on the way: the activations rounded by work-groups of 64 items (149 to 13 microseconds a product on Llama-3.2-1B), the weights unpacked with 16-byte vectors, the scale multiplies on 24-bit integers (Q6_K 400 to 253 microseconds a launch), and the mins term of Q4_K / Q5_K as two more 16-deep products of the 8 mins by the block sums split into 7-bit halves instead of 64 integer multiplies an element (Q4_K 539 to 280 microseconds, the F16 matrix units' 283). - Fused launches. The backend holds back a MoE
layer's softmax router and expert choice, and Qwen3's query and key head
norms (and the query rotation when Q and K have their own products), and
runs each sequence as one launch when it completes
(
moe_route,norm_rope;OKLM_CL_FUSE=0turns it off). The same bits: the logits after a 190-token prompt and 16 greedy tokens hash the same with and without, on the eight models. - Latent attention's prompts on tiles
(
attention_mla, heads of 576 with an F16 cache): GLM-4.7-Flash 1337 to 2174 t/s with the 8-bit products. The same 16 greedy tokens as the per-token kernel.
| model | pp512 before (F32 tiles) | pp512 now | llama.cpp | ratio | tg128 before / now | llama.cpp | ratio |
|---|---|---|---|---|---|---|---|
| SmolLM2-135M Q8_0 | 27060 | 40450 | 29226 | 1.38 | 427 / 420 | 381 | 1.10 |
| SmolLM2-135M Q4_K_M | 25895 | 34604 | 31196 | 1.11 | 385 / 371 | 383 | 0.97 |
| Llama-3.2-1B Q4_K_M | 6010 | 13432 | 17587 | 0.76 | 360 / 359 | 410 | 0.88 |
| Qwen3-8B Q4_K_M | 905 | 2564 | 3223 | 0.80 | 82.5 / 84.7 | 105 | 0.80 |
| Qwen3-8B Q8_0 | 927 | 2775 | 1647 | 1.68 | 65.4 / 66.3 | 73.8 | 0.90 |
| Granite-3.1-1B-A400M Q4_K_M | 9978 | 16386 | 15385 | 1.07 | 247 / 285 | 362 | 0.79 |
| OLMoE-1B-7B Q4_K_M | 4154 | 8753 | 10225 | 0.86 | 260 / 271 | 359 | 0.76 |
| GLM-4.7-Flash Q4_K_M | 820 | 2156 | 2061 | 1.05 | 71.6 / 71.5 | 87.8 | 0.81 |
"Before" is the table above (OKLM at f353351). The
OKLM_CL_WMMA=1 rows of the same run: 36890, 35186, 14351,
2623, 2782, 15102, 8584, 2068 t/s prompts. SmolLM2 Q4_K_M decoding
measured 374 to 381 t/s over six more runs with and without the fused
launches (none apply to it): noise. Greedy tokens after a 190-token
prompt (16 tokens, the logits of each hashed by a scratch program): with
the 8-bit products the GPU's tokens are the CPU's on Qwen3-8B Q4_K_M,
OLMoE, Granite and Qwen3-8B Q8_0, where the F32 kernels part from the
CPU at the first token on Qwen3-8B Q4_K_M (top two 0.15 apart) and OLMoE
(0.04); on GLM-4.7-Flash both the CPU and the F32 kernels part from the
8-bit products at the eleventh token; on Llama-3.2-1B and SmolLM2 the
8-bit products give the F32 kernels' tokens (decoding, a column at a
time, keeps the float kernels).
Measured and left out: the 8-bit tiles reading each lane's activation
levels from global memory instead of local memory (a third of the local
memory, so more work-groups a compute unit: Llama-3.2-1B's Q4_K products
312 to 475 microseconds a launch, Qwen3-8B Q4_K_M prompts 2564 to 1796
t/s). Not tried: the norm folded into the following one-column product
(every work-group of the product would sum the squares of the whole row
again: 16 MB more cache reads per Qwen3-8B product, more than the launch
it saves, as the earlier fused-norm measure found), and a recorded
command sequence per token (this driver has no
cl_khr_command_buffer).
Where llama.cpp stays ahead on the GPU, and why:
- Prompts of Llama-3.2-1B (0.76) and Qwen3-8B Q4_K_M (0.80): the Q4_K products run at the F16 matrix units' speed (280 microseconds a launch on Llama-3.2-1B against 283); llama.cpp's MMQ kernels keep more of a tile in registers and load the next tile while multiplying, and its attention and element-wise ops take less of the prompt (OKLM's ew ops 5.2 ms of a 38 ms Llama-3.2-1B prompt).
- Decoding (0.76 to 0.90, SmolLM2 level or ahead): on Qwen3-8B 85 % of a token's GPU time is the matrix-vector products (the Q4_K ones near 40 microseconds a launch), so the fused launches change it within the noise; the MoE models gain most from them (Granite 247 to 285, OLMoE 260 to 271) and stay behind on their expert products (gate, up and down three indexed launches a layer) and launch count.
Round 4 (CPU): packed expert matrices on the CPU
The routed experts of the CPU layers get the packed copy the dense
matrices have, one per expert (OKLM_CPU_PACK_EXPERTS, on by
default when the copies and the mapped experts fit in the available
memory with the plan's margin; never with an expert cache; moe.md), and the grouped
indexed product runs the packed kernels: prompts always, decode steps
with one or two threads (from three, the file layout measured 2 to 6 %
faster: OLMoE 37.6 / 36.8 t/s at 4 threads, Granite 97.6 / 92.6). The
copies take 4.0 GiB for OLMoE, 17.4 GiB for GLM-4.7-Flash, on top of the
mapped file; the repacking runs on the worker team. The same bits
(test_ops runs the grouped product on both layouts; greedy
text identical with and without, OLMoE, Granite, GLM-4.7-Flash at 1 and
4 threads, the expert cache, --gpu-layers 12,
--n-cpu-moe 8). Before / after (llama.cpp), t/s:
| model | threads | pp512 | tg128 |
|---|---|---|---|
| OLMoE-1B-7B Q4_K_M | 1 | 56.5 / 84.4 (56.1) | 21.1 / 25.5 (23.1) |
| 4 | 202 / 289 (206) | ||
| 16 | 336 / 469 (358) | ||
| Granite-3.1-1B-A400M Q4_K_M | 1 | 139 / 195 (139) | |
| 4 | 490 / 685 (493) | ||
| 16 | 830 / 1136 (830) |
Round 5: one team loop for the products sharing an input
oklm_matmul_n() runs a layer's Q, K and V, and its gate
and up (the shared expert's too), as one loop of the worker team, the
activation rows quantized once (core.md):
a SmolLM2-135M token runs 121 loops of products instead of 211. The same
bits (test_matmul compares it with one product per weight,
packed and not, one and six threads). tg128 t/s, one product per weight
/ fused (llama.cpp from the table above):
| model | 1 thread | 4 threads | 8 threads | 16 threads |
|---|---|---|---|---|
| SmolLM2-135M Q8_0 | 131 / 133 (124) | 160 / 168 (194) | 163 / 172 (188) | 150 / 159 (178) |
| SmolLM2-135M Q4_K_M | 139 / 142 (128) | 204 / 219 (254) | 215 / 232 (254) | 193 / 209 (239) |
| Llama-3.2-1B Q4_K_M | 24.1 / 24.2 (22.4) | 34.8 / 35.2 (36.1) | 34.9 / 35.2 (35.5) | 32.6 / 33.0 (33.8) |
What is left of SmolLM2's gap from four threads: a token is 4.6 ms, 92 % in matrix products (Q5_0 2.4 ms: its 576-column matrices cannot be K-quants, and their packed copy, 1.64 times the file, is read with one or two threads only; the Q8_0 head 1.0 ms), at about 23 GB/s against llama.cpp's 27.
Round 6: a leaner packed Q4_K kernel
With AVX-512 VNNI the packed Q4_K kernel accumulated each 32-column
sub-block's products in int32, then multiplied them by the eight rows'
scales (a 32-bit multiply, two micro-ops, ten cycles of latency) before
adding them. It now accumulates each 16 columns apart: their sums fit in
int16 (at most 16 x 15 x 127), so one multiply-add by the scale (its
high half zero) scales and adds them, in two independent chains. The
kernel is also specialized per activation row count and unrolled, so its
accumulators stay in registers (rolled, the compiler put them on the
stack: the first try ran Llama-3.2-1B prompts at 72 t/s against 100).
The same values (test_matmul, packed against scalar; greedy
text the same with and without the packed copies on Llama-3.2-1B,
Qwen3-8B and SmolLM2). Llama-3.2-1B pp512 at one thread 99 to 119 t/s
(llama.cpp 111); the table below has every row.
After rounds 4 to 6 (CPU, 2026-11-08)
The same script and llama.cpp build, CPU rows only (load average 6.3 from the earlier builds when it started, 0.09 hardware threads busy). t/s, OKLM / llama.cpp:
| model | threads | pp512 OKLM / llama.cpp | ratio | tg128 OKLM / llama.cpp | ratio |
|---|---|---|---|---|---|
| SmolLM2-135M Q8_0 | 1 | 532 / 461 | 1.15 | 140 / 122 | 1.14 |
| 4 | 1963 / 1619 | 1.21 | 168 / 189 | 0.89 | |
| 8 | 3586 / 2925 | 1.23 | 172 / 185 | 0.93 | |
| 16 | 4088 / 3007 | 1.36 | 159 / 170 | 0.93 | |
| SmolLM2-135M Q4_K_M | 1 | 568 / 442 | 1.29 | 148 / 126 | 1.18 |
| 4 | 2078 / 1535 | 1.35 | 219 / 248 | 0.88 | |
| 8 | 3815 / 2732 | 1.40 | 233 / 251 | 0.93 | |
| 16 | 4226 / 2753 | 1.53 | 214 / 230 | 0.93 | |
| Llama-3.2-1B Q4_K_M | 1 | 116 / 110 | 1.05 | 26.2 / 22.1 | 1.19 |
| 4 | 410 / 400 | 1.02 | 35.2 / 35.2 | 1.00 | |
| 8 | 750 / 704 | 1.07 | 35.2 / 35.1 | 1.00 | |
| 16 | 781 / 596 | 1.31 | 33.0 / 32.4 | 1.02 | |
| Qwen3-8B Q4_K_M | 1 | 18.0 / 15.6 | 1.16 | 4.73 / 3.92 | 1.21 |
| 4 | 65.0 / 56.5 | 1.15 | 6.24 / 6.24 | 1.00 | |
| 8 | 103 / 91.9 | 1.13 | 6.15 / 6.13 | 1.00 | |
| 16 | 106 / 88.6 | 1.20 | 5.81 / 5.77 | 1.01 | |
| Qwen3-8B Q8_0 | 1 | 16.8 / 9.46 | 1.78 | 3.12 / 2.68 | 1.16 |
| 4 | 60.3 / 33.7 | 1.79 | 3.56 / 3.72 | 0.96 | |
| 8 | 97.0 / 55.9 | 1.74 | 3.58 / 3.59 | 1.00 | |
| 16 | 103 / 56.8 | 1.82 | 3.41 / 3.39 | 1.01 | |
| Granite-3.1-1B-A400M Q4_K_M | 1 | 230 / 139 | 1.66 | 64.5 / 59.3 | 1.09 |
| 4 | 804 / 493 | 1.63 | 97.2 / 97.9 | 0.99 | |
| 8 | 1386 / 831 | 1.67 | 98.5 / 98.3 | 1.00 | |
| 16 | 1490 / 852 | 1.75 | 90.9 / 92.3 | 0.98 | |
| OLMoE-1B-7B Q4_K_M | 1 | 97.3 / 56.1 | 1.73 | 26.9 / 23.1 | 1.16 |
| 4 | 334 / 199 | 1.68 | 37.6 / 37.9 | 0.99 | |
| 8 | 593 / 355 | 1.67 | 37.6 / 37.8 | 1.00 | |
| 16 | 546 / 368 | 1.48 | 35.6 / 35.7 | 1.00 | |
| GLM-4.7-Flash Q4_K_M | 1 | 20.6 / 17.2 | 1.19 | 7.51 / 6.85 | 1.10 |
| 4 | 71.9 / 59.5 | 1.21 | 11.1 / 11.4 | 0.97 | |
| 8 | 112 / 92.6 | 1.21 | 11.2 / 11.5 | 0.97 | |
| 16 | 120 / 91.1 | 1.32 | 10.6 / 10.8 | 0.97 |
Against the table after the three rounds, no row got slower beyond the noise (Llama-3.2-1B tg128 at 4 threads 35.6 / 35.2 with llama.cpp 36.1 / 35.2, GLM-4.7-Flash 11.2 / 11.1 with 11.6 / 11.4). Every CPU prompt row is now ahead (1.02 to 1.82); decoding is ahead at one thread (1.09 to 1.21) and level from four (0.96 to 1.02) except SmolLM2-135M (0.88 to 0.93; round 5) and GLM-4.7-Flash (0.97): both read memory at about the speed llama.cpp does, with a few loops a token more.
Round 7 (GPU): decoding on 8-bit dot products, the CPU's products
One-column products (decoding, and a MoE decode step's indexed
products) with Q4_K, Q5_K or Q6_K weights, rows of 2048 values or more
and 1024 rows or more, now run mq_<type> on a gfx11
device by default (opencl.md): each
work-group of 256 items rounds its activation column to Q8_K in local
memory exactly as q8_act does (no launch added), then each
wave takes two weight rows, forms each super-block's integer sums with
v_dot4_i32_iu8 (eight items a super-block, their sums added
across lanes with DPP) and adds the super-block's float term to the
row's sum in order of k, as oklm_vec_dot_q4_k_q8_k() and
its twins do: the CPU's products bit for bit
(test_opencl_q8: decode shapes of the three types, plain
and indexed, every output equal; OKLM_CL_Q8_MV=0: the float
row kernels). One product at a time, pipelined, its weights out of the
cache (a scratch program cycling through copies; microseconds,
mv_<type> / mq_<type>):
| product (rows x row length) | mv |
mq |
|---|---|---|
| Q4_K 24576 x 4096 (Qwen3-8B gate and up) | 97.1 | 92.6 |
| Q4_K 6144 x 4096 | 32.0 | 27.5 |
| Q4_K 4096 x 4096 | 21.1 | 20.7 |
| Q4_K 4096 x 12288 (down) | 52.8 | 51.6 |
| Q6_K 4096 x 12288 (down) | 95.7 | 75.2 |
| Q6_K 151936 x 4096 (Qwen3-8B head) | 793 | 703 |
| Q4_K 16384 x 2048 (Llama-3.2-1B gate and up) | 35.7 | 32.8 |
| Q4_K 2048 x 2048 | 9.6 | 9.0 |
| Q6_K 2048 x 8192 | 32.7 | 29.5 |
GPU rows (THREADS="" sh tools/bench_vs_llama.sh, load
average 2.6 to 9.3 from earlier builds when each model started, 0.00 to
0.04 hardware threads busy), before (round 4) / after, t/s:
| model | pp512 | llama.cpp | ratio | tg128 | llama.cpp | ratio |
|---|---|---|---|---|---|---|
| SmolLM2-135M Q8_0 | 40450 / 41106 | 29383 | 1.40 | 420 / 425 | 380 | 1.12 |
| SmolLM2-135M Q4_K_M | 34604 / 35428 | 31007 | 1.14 | 371 / 378 | 385 | 0.98 |
| Llama-3.2-1B Q4_K_M | 13432 / 13644 | 17908 | 0.76 | 359 / 368 | 413 | 0.89 |
| Qwen3-8B Q4_K_M | 2564 / 2503 | 3169 | 0.79 | 84.7 / 92.0 | 105.7 | 0.87 |
| Qwen3-8B Q8_0 | 2775 / 2763 | 1647 | 1.68 | 66.3 / 66.3 | 74.1 | 0.89 |
| Granite-3.1-1B-A400M Q4_K_M | 16386 / 16626 | 14837 | 1.12 | 285 / 283 | 362 | 0.78 |
| OLMoE-1B-7B Q4_K_M | 8753 / 8854 | 10389 | 0.85 | 271 / 274 | 360 | 0.76 |
| GLM-4.7-Flash Q4_K_M | 2156 / 2156 | 2082 | 1.04 | 71.5 / 71.5 | 87.9 | 0.81 |
Qwen3-8B Q4_K_M decoding 84.2 to 91.6 t/s and Llama-3.2-1B 354 to 370
in alternated runs of the same build (OKLM_CL_Q8_MV=0 /
1, best of five); the prompt columns did not change (no
prompt kernel changed: 2564 / 2503 is the noise of these runs). Greedy
tokens after a 190-token prompt (16 tokens, the logits hashed): with the
8-bit decode the GPU's tokens are the CPU's on Qwen3-8B Q4_K_M, Granite
and OLMoE, as they were with the float decode; on Llama-3.2-1B the float
decode parted from the CPU at the sixth token (its top two 0.004 apart),
the 8-bit decode at the fifth (the CPU's top two 0.09 apart: attention
and the element-wise ops still round differently); on GLM-4.7-Flash the
8-bit decode gives the float decode's 16 tokens (both part from the CPU
at the eleventh, as in round 4).
Measured and left out:
- The activations rounded by a separate
q8_actlaunch before each one-column product (the first version, one row a wave): Qwen3-8B Q4_K_M decoding 84.2 to 88.8 t/s but Llama-3.2-1B 349 to 297, one more launch per product (about 6 microseconds back to back, as much as a whole 2048 x 2048 product). - The column rounded in each work-group of 128 items, one row a wave: Llama-3.2-1B 362 to 322 t/s (the rounding paid by four rows only); four super-blocks per group a wave step instead of two: rows of 2048 values slower than the float kernels (16384 x 2048 40.5 against 35.7 microseconds: half the lanes read a block past the row's end).
- The 8-bit kernels for Q8_0, Q4_0 and Q5_0 (four, then two items a block of 32): Qwen3-8B Q8_0 decoding 66.6 to 64.8 t/s.
- The 8-bit kernels for shorter or fewer rows: SmolLM2-135M Q4_K_M decoding 412 to 370 t/s (its 576 x 1536 down products: 36 work-groups, each rounding the column for little work), Granite 283 to 277 and OLMoE 274 to 266 (rows of 1024 values).
- Q4_K prompt tiles that load the next step while multiplying: the activation levels into registers (eight 16-byte vectors an item, 32 more registers on top of the 128 of the accumulators) halved the Q4_K prompts (Llama-3.2-1B 13848 to 8361 t/s, Qwen3-8B Q4_K_M 2584 to 1410; Q8_0, with fewer accumulators, unchanged); the weights only (12 registers): 13848 to 13675 and 2584 to 2491, within the noise or slower.
Not done: the MoE expert gate and up products as one indexed launch, a fused element-wise pass for prompts.
Where llama.cpp stays ahead on the GPU, and why:
- Decoding (0.76 to 0.89, SmolLM2 level or ahead): the products of large matrices now run near the bandwidth the head's product reaches (Q6_K 151936 x 4096 at 730 GB/s, Q4_K 24576 x 4096 at 610); what is left is the launches (about 12 per layer, each a few microseconds back to back, where llama.cpp replays the token as a HIP graph) and the small products, whose fixed cost the 8-bit kernels do not lower. The MoE models (Granite, OLMoE) are the furthest behind: three indexed products of short rows per layer, still on the float kernels.
- Prompts of Llama-3.2-1B (0.76) and Qwen3-8B Q4_K_M (0.79): unchanged from round 4 (both prefetching attempts above lost).
Round 7 (CPU): packed copies the file's size, decode products on fewer threads
Three changes to decoding on the CPU, all giving the same bits
(test_matmul packed against scalar at both SIMD levels;
greedy text of SmolLM2 Q4_K_M, SmolLM2 Q8_0 and Qwen3-8B Q8_0 the same
packed, on the file layout and at one thread):
- Q8_0 and Q5_0 copies as small as the file. The packed Q8_0 copy keeps the file's half-precision scales (272 bytes for eight rows of a block, 1.0 times the file instead of 1.06), the Q5_0 copy its nibbles as in the file and the fifth bits of each row in a 32-bit word laid out so one shift and one mask put them in place (176 bytes, 1.0 times instead of 1.64). A one-row product now reads the packed copy at any thread count when it is no larger than the file. Prompts expand each tile of Q5_0 rows once to a byte per value and run the byte kernel, so they do not pay for the unpacking (the kernel that unpacked on every call ran SmolLM2 Q4_K_M pp512 2.5 % slower).
- One-row products on fewer threads. Bound by memory,
they read it fastest with fewer readers than the team has
(
bench_matmul, a 107 MB Q8_0 matrix, packed, GB/s: 34.4 at 2 threads, 35.0 at 3, 34.5 at 4, 33.7 at 5, 32.8 at 6, 31.6 at 8, 30.4 at 16; Q4_K on the file layout 37.1 at 4, 36.0 at 8). A loop of the worker team can now take at most n participants (oklm_cpu_team_run_max(), the first to arrive; the chunks are those of the whole team): half the physical cores for a product on a packed copy (OKLM_MATVEC_THREADS), one thread per core on the file layout. - Decode attention without the split while the keys are few. With fewer tiles than threads (SmolLM2: three KV heads), attention split the tiles' arithmetic into three loops of the team (keys, rows, columns); below 2^18 multiply-adds per tile it now runs the tiles as they are, one loop (SmolLM2 Q8_0 tg1536 at 4 threads 142.5 to 146.1 t/s, the same at 8; tg128 181 to 185).
tg128 t/s before / after, each pair back to back (llama.cpp from the table above):
| model | threads | before / after (llama.cpp) |
|---|---|---|
| SmolLM2-135M Q4_K_M | 4 | 222 / 242 (248) |
| SmolLM2-135M Q8_0 | 4 | 171 / 185 (189) |
| Qwen3-8B Q8_0 | 4 | 3.56 / 3.73 (3.72) |
| 8 | 3.58 / 3.68 (3.59) | |
| 16 | 3.41 / 3.66 (3.39) | |
| Qwen3-8B Q4_K_M | 16 | 5.81 / 6.11 (5.77) |
| Llama-3.2-1B Q4_K_M | 16 | 33.5 / 34.8 (32.4) |
| Granite-3.1-1B-A400M Q4_K_M | 16 | 92.0 / 93.8 (92.3) |
pp512 of SmolLM2 Q4_K_M, before / after: 578 / 575 at one thread, 2131 / 2135 at four, 4295 / 4344 at sixteen.
Measured and left out:
- the packed Q4_K copy (1.06 times the file) for one-row products from three threads: Llama-3.2-1B tg128 35.8 to 34.9 at 4 threads, 35.4 to 34.3 at 8; Q6_K's (1.31 times) as well: SmolLM2 Q4_K_M 242 to 233;
- an eight-activation-row Q8_0 prompt kernel: pp512 539 to 522 at one thread; the compact Q5_0 layout read directly by the prompt kernels, four rows (2136 to 2086 at 4 threads) or eight (2118 to 2103);
- the fifth bits as byte masks added with masked byte adds (AVX-512 BW): pp512 2066 against 2103 for the shifts; a three-input logic instruction for the shift-and-mask: no change;
- one-row products on six threads instead of four on a packed copy (Qwen3-8B Q8_0 at 8 threads 3.57 against 3.68), on four for the file layout too (SmolLM2 Q4_K_M at 16 threads 214 against 220: its Q6_K products need more threads), a limit only for matrices of 4 MB or more (no better), and a limit on the indexed (expert) products (Granite, OLMoE: within the noise).
Round 8 (CPU): packed copies of latent attention's per-head matrices
A step of GLM-4.7-Flash at four threads
(OKLM_CPU_PROFILE, 89 ms): products 55 % (the Q6_K output
head and shared experts 13.8 ms, Q4_K attention 16.2 ms, Q8_0 11.2 ms,
Q5_K shared experts 7.1 ms), expert products 41 % (36 ms), each at 26 to
32 GB/s; attention 2 %. Its Q8_0 matrices are mostly latent attention's
per-head stacks (attn_k_b, attn_v_b,
[K, N, 20]), which batched products read on the file
layout. They now get one packed copy per head (counted in place of the
mapped stack, as a matrix's copy; test_matmul checks
batched products on a packed stack against the scalar kernels): pp512 at
four threads 73.2 to 78.2 t/s, tg64 11.24 to 11.29 (a decode step reads
them at memory speed either way). The greedy text of GLM-4.7-Flash is
the same packed, on the file layout and at one thread.
After rounds 7 and 8 (CPU, 2026-11-08)
The same script and llama.cpp build, CPU rows only (load average 1.6 to 6 between the models, under 0.1 hardware threads busy at each start; Llama-3.2-1B and SmolLM2 Q4_K_M measured again with the last commit after a 16-thread prompt run of the first pass fell into the slow mode that comes and goes at 16 threads). t/s, OKLM / llama.cpp:
| model | threads | pp512 OKLM / llama.cpp | ratio | tg128 OKLM / llama.cpp | ratio |
|---|---|---|---|---|---|
| SmolLM2-135M Q8_0 | 1 | 538 / 466 | 1.15 | 141 / 122 | 1.16 |
| 4 | 1988 / 1684 | 1.18 | 184 / 193 | 0.96 | |
| 8 | 3595 / 2949 | 1.22 | 179 / 187 | 0.95 | |
| 16 | 4102 / 3026 | 1.36 | 165 / 175 | 0.94 | |
| SmolLM2-135M Q4_K_M | 1 | 577 / 448 | 1.29 | 176 / 128 | 1.37 |
| 4 | 2123 / 1566 | 1.36 | 250 / 252 | 0.99 | |
| 8 | 3821 / 2796 | 1.37 | 241 / 254 | 0.95 | |
| 16 | 4306 / 2862 | 1.50 | 220 / 241 | 0.91 | |
| Llama-3.2-1B Q4_K_M | 1 | 121 / 111 | 1.09 | 25.8 / 22.3 | 1.16 |
| 4 | 428 / 400 | 1.07 | 35.8 / 36.0 | 0.99 | |
| 8 | 805 / 727 | 1.11 | 35.4 / 35.3 | 1.00 | |
| 16 | 818 / 684 | 1.20 | 34.9 / 33.4 | 1.05 | |
| Qwen3-8B Q4_K_M | 1 | 18.9 / 15.6 | 1.21 | 4.67 / 3.91 | 1.19 |
| 4 | 67.4 / 57.1 | 1.18 | 6.26 / 6.23 | 1.00 | |
| 8 | 106 / 92.0 | 1.15 | 6.15 / 6.13 | 1.00 | |
| 16 | 109 / 88.5 | 1.23 | 6.13 / 5.76 | 1.06 | |
| Qwen3-8B Q8_0 | 1 | 16.7 / 9.46 | 1.76 | 3.24 / 2.66 | 1.22 |
| 4 | 59.6 / 34.0 | 1.76 | 3.73 / 3.72 | 1.00 | |
| 8 | 95.8 / 55.9 | 1.71 | 3.68 / 3.59 | 1.03 | |
| 16 | 102 / 56.9 | 1.80 | 3.65 / 3.38 | 1.08 | |
| Granite-3.1-1B-A400M Q4_K_M | 1 | 234 / 139 | 1.68 | 65.0 / 59.4 | 1.09 |
| 4 | 817 / 494 | 1.65 | 97.7 / 97.8 | 1.00 | |
| 8 | 1389 / 831 | 1.67 | 98.5 / 98.3 | 1.00 | |
| 16 | 1501 / 813 | 1.85 | 92.2 / 91.6 | 1.01 | |
| OLMoE-1B-7B Q4_K_M | 1 | 99.6 / 56.1 | 1.78 | 27.0 / 23.1 | 1.17 |
| 4 | 351 / 201 | 1.75 | 37.6 / 37.8 | 0.99 | |
| 8 | 601 / 357 | 1.68 | 37.7 / 37.8 | 1.00 | |
| 16 | 626 / 322 | 1.94 | 36.0 / 36.0 | 1.00 | |
| GLM-4.7-Flash Q4_K_M | 1 | 22.3 / 17.3 | 1.29 | 8.03 / 6.92 | 1.16 |
| 4 | 79.2 / 59.5 | 1.33 | 11.2 / 11.4 | 0.98 | |
| 8 | 122 / 92.9 | 1.31 | 11.2 / 11.5 | 0.97 | |
| 16 | 131 / 92.3 | 1.42 | 10.7 / 10.9 | 0.98 |
Against the table after rounds 4 to 6: decoding faster on every model
from four threads (SmolLM2 Q4_K_M 219 / 249 at four, 214 / 220 at
sixteen; Qwen3-8B Q8_0 3.56 / 3.73, 3.58 / 3.68, 3.41 / 3.65; Qwen3-8B
Q4_K_M and Llama-3.2-1B at sixteen 5.81 / 6.13 and 33.0 / 34.9) and at
one thread on the Q5_0 file (SmolLM2 Q4_K_M 148 / 176); prompts the same
or faster (GLM-4.7-Flash 71.9 / 79.2 at four) but Qwen3-8B Q8_0's, 1 %
slower (60.3 / 59.6 at four, 97.0 / 95.8 at eight; an A/B run: 60.6 /
59.4, and 17.4 / 17.2 at one thread): its packed copy now holds the
scales as halves, one more conversion per block in the prompt kernel,
the price of the decode rows above (a copy with float scales is 1.06
times the file, which round 1 found too large for decoding from three
threads). Expanding each prompt tile to float scales measured slower
(58.3). Decoding from four threads is level on every model now (0.97 to
1.08) except SmolLM2 from eight threads (0.91 to 0.95) and GLM-4.7-Flash
(0.97 to 0.98). SmolLM2 decodes faster on four threads (250) than on
eight (241) or sixteen (220), while llama.cpp holds 252 to 254 from four
to eight; its products already run on four threads from eight
(OKLM_MATVEC_THREADS), so the loss is in the rest of a step
with more threads polling (a waiting policy without the short poll, and
without the poll for workers left out of a loop, measured the same:
240.9 and 240.4 against 240.3).
Round 9 (GPU): decode norms folded into the 8-bit products, fewer launches
Three changes to the OpenCL backend's decoding (opencl.md):
- A decode step's RMS norm folded into the product that reads
it. A profile of a Llama-3.2-1B token
(
OKLM_CL_PROFILE=1) gave 202 launches: the matrix products 1.57 ms of GPU time, the 33 RMS norms 0.26 ms (7.8 microseconds each: one work-group of 256 for 2048 values), attention 0.18, element-wise ops 0.15, rope 0.09, and about 2.3 microseconds between dependent launches; the host spent 0.25 ms a token enqueueing, far less than the 2.7 ms the GPU took, so the launches' GPU time and the gaps between them were the target, not the host. The backend now holds back the norm of one row (2048 to 8192 values, a multiple of 256) and, when the next op is a one-column product of its output on an MQ kernel, runs both as one launch (mqn_<type>): each work-group, which already reads the whole column to round it, loads the norm's input and weights once, sums the squares, scales and rounds; the first work-group writes the norm's output. The norm kernel sums rows of those widths in runs of eight values an item (the MQ kernels' layout of a column), so fused and separate norms are the same bits (test_opencl_q8: norm output and product equal, the product the CPU's product of that output); other widths keep the old order, so SmolLM2's and Granite's logits are unchanged to the bit. Llama-3.2-1B: 202 to 169 launches a token, every norm folded. - The key and value cache writes in one launch
(
cpy2, the same bits): one launch fewer a layer. - The 8-bit decode products under
OKLM_CL_WMMA=1. The 8-bit kernels are built with the matrix units too, for the one-column products alone: Qwen3-8B Q4_K_M decoding withOKLM_CL_WMMA=184 to 94 t/s (alternated runs), its greedy tokens now the default path's.
GPU rows (THREADS="" sh tools/bench_vs_llama.sh, load
average 3.6 when it started, 0.05 hardware threads busy), before (round
7) / after, t/s:
| model | pp512 | llama.cpp | ratio | tg128 | llama.cpp | ratio |
|---|---|---|---|---|---|---|
| SmolLM2-135M Q8_0 | 41106 / 40096 | 29242 | 1.37 | 425 / 445 | 381 | 1.17 |
| SmolLM2-135M Q4_K_M | 35428 / 35871 | 31229 | 1.15 | 378 / 400 | 383 | 1.04 |
| Llama-3.2-1B Q4_K_M | 13644 / 13710 | 17845 | 0.77 | 368 / 392 | 411 | 0.95 |
| Qwen3-8B Q4_K_M | 2503 / 2553 | 3200 | 0.80 | 92.0 / 94.7 | 105.3 | 0.90 |
| Qwen3-8B Q8_0 | 2763 / 2728 | 1649 | 1.65 | 66.3 / 68.1 | 74.0 | 0.92 |
| Granite-3.1-1B-A400M Q4_K_M | 16626 / 16423 | 15492 | 1.06 | 283 / 279 | 364 | 0.77 |
| OLMoE-1B-7B Q4_K_M | 8854 / 8712 | 10291 | 0.85 | 274 / 291 | 360 | 0.81 |
| GLM-4.7-Flash Q4_K_M | 2156 / 2149 | 2070 | 1.04 | 71.5 / 73.1 | 88.0 | 0.83 |
The OKLM_CL_WMMA=1 rows of the same run: 36908, 35119,
14630, 2644, 2768, 15315, 8605, 2056 t/s prompts; 439, 396, 385, 95.4,
67.6, 256, 278, 73.4 decoding (Granite's 256 is that mode's speed before
these changes too: 252 / 256 and 256 / 254 in alternated runs). No
prompt kernel changed (prompt differences are the noise of these runs).
Alternated runs of the same files, before / after: Llama-3.2-1B decoding
354 to 370 / 380 to 390 with the norms folded, 376 to 385 / 390 with the
cache writes added; Granite 252 to 258 / 252 to 257 (nothing folds: rows
of 1024). Greedy tokens after a 190-token prompt (16 tokens, the logits
hashed): the same tokens as before on the eight models; the logits the
same bits with OKLM_CL_FUSE=0 and without; bit-identical to
before on SmolLM2 and Granite. With OKLM_CL_WMMA=1,
Qwen3-8B Q4_K_M's tokens become the default path's (they parted from
them at the fourth token), and OLMoE's part from both at the sixth (its
prompt products are F16, its decode products now the CPU's; that mode is
not the CPU's bits).
Measured and left out:
- The folded norm reading its input twice (a pass for the sum of squares, then the rounding pass, in the norm kernel's old order, every bit as before): 3.4 microseconds more a fused launch than the product alone, against 2.8 with the one-pass version kept (Llama-3.2-1B's MQ time 279 against 269 ms over 256 tokens); the sum's loop unrolled by eight made no difference.
- The norm kernel summing every width in runs of eight: SmolLM2's
logits moved further from the CPU's (
test_opencl_q8: 0.77 against 0.61), so only the foldable widths take the runs. - The work-group sum's last five steps inside one wave (DPP, the same pairs and bits, five barriers fewer): within the noise (Llama-3.2-1B 381 / 385, 386 / 381).
- The norm's chunks loaded for a fixed four chunks a wave and rounded unconditionally (a compiler workaround tried first): the rounding of chunks past the row cost more than the fusion saved (354 to 363 t/s against 363 to 369).
Not done: the MoE expert gate and up products as one indexed launch with SwiGLU folded (Granite and OLMoE stay at 0.77 and 0.81: Granite's decode is 472 launches a token, 49 norms of 1024 values at 7.5 microseconds that no MQ kernel takes, 146 element-wise launches); SwiGLU folded into the down product (the backend cannot fold an in-place op: the model writes the gate's rows); the rotation and the cache writes as one launch; a faster standalone norm kernel for rows below 2048 values.
Where llama.cpp stays ahead on the GPU, and why:
- Decoding of the MoE models (Granite 0.77, OLMoE 0.81) and GLM-4.7-Flash (0.83): many short launches a token (Granite 472), the norms of 1024 values and the expert products of short rows on the float kernels.
- Decoding of the dense models (0.90 to 0.95; SmolLM2 ahead): the products of large matrices near the bandwidth, the rest launches.
- Prompts of Llama-3.2-1B (0.77) and Qwen3-8B Q4_K_M (0.80): unchanged from round 7 (no prompt kernel changed this round).
Round 9 (CPU): decode loops on a fixed set of workers, the others asleep
A step of SmolLM2-135M Q4_K_M (OKLM_CPU_PROFILE, 512
tokens decoded, seconds in ops at 4 / 8 / 16 threads): 4.52 / 4.68 /
5.14, the products on four threads at every count (Q5_0 2.15 / 2.24 /
2.42) but attention (three tiles, 0.45 / 0.49 / 0.64) and the small ops
run inline (rope 0.065 / 0.072 / 0.096) slower too. Three costs grew
with the team, not with the work:
- a one-row product was cut into tiles for the whole team (sixteen threads: 36 tiles of 16 rows for four participants), and its loop into as many chunks, each a compare-and-swap on the claim word;
- a loop limited to four participants was joined by the first four workers to take a seat by compare-and-swap on one shared word, the others racing for it and failing, at every loop; and every waiting worker polled the claim word the participants were writing;
- the workers no loop of a decoding step takes (at sixteen threads, the eight above the participant limits) kept polling between loops, which come every few microseconds, so they never slept and took half of every core from the threads that ran.
Now (team_cpu.c): a limited loop takes the caller and
the workers of index 1 to n - 1 (never more than it has chunks), each
worker knowing its index, so none contends for a seat; workers poll the
generation of the last loop on a cache line of its own; a loop a worker
is left out of does not restart its wait, so those no loop takes go to
sleep after about a millisecond, and a loop wakes the sleepers only when
it takes one of them. A one-row product cuts its tiles for the threads
it runs on, and one on the file layout under 4 MiB of weights runs on as
many threads as a packed copy's (SmolLM2's Q4_K and Q6_K down products:
half a megabyte; Llama-3.2-1B's larger matrices stay on one thread per
core: 35.4 t/s at 8 and 16 threads with or without, the threshold
measured at 1, 4 and 8 MiB). The same bits (the loops' chunks change, no
output element's arithmetic; test_threads adds a loop that
must wake the workers left asleep by limited loops; greedy text through
make test).
tg128 t/s, before (the last commit of round 8) / after, each pair back to back:
| model | 4 threads | 8 threads | 16 threads |
|---|---|---|---|
| SmolLM2-135M Q4_K_M | 250.5 / 251.1 | 240.5 / 248.1 | 218.2 / 250.0 |
| SmolLM2-135M Q8_0 | 185.1 / 184.9 | 178.7 / 184.3 | 166.1 / 184.0 |
Step by step on SmolLM2 Q4_K_M at 8 / 16 threads: tiles for the participants 242 / 225, fixed seats 244 / 230, the generation on its own line 245 / 230, left-out workers asleep 245 / 241, the small file-layout products on four threads 250 / 250.
Measured and left out:
- Qwen3-8B Q8_0 prompts (row 2 of what was left): a packed Q8_0 group holding every block's scales first, then the values, so that the values of each block start on a 32-byte boundary (beside each block's scales, a quarter of the prompt kernel's 32-byte loads cross two cache lines): pp512 at four threads 59.2 / 57.9 before, 58.6 / 58.2 after, alternated runs: no change, so the split loads are not what the half-precision scales cost.
- GLM-4.7-Flash decoding: a decoding expert product's tiles cut into 2, 4 or 8 per thread (larger contiguous reads) instead of 16 rows: tg64 at four threads 11.28 / 11.25 / 11.22 / 11.23; one-row products cut into 2, 8 or 16 tiles per thread instead of 4: 11.23 to 11.28 (SmolLM2 246 to 248), all within the noise.
- The waiting workers prefetching the next product's weights: each decoding product records the weights of the product that followed it (the order is the same at every token), and when it ends the workers waiting for a loop issue a prefetch for each cache line of the successor's first 4 MiB (an indexed successor: its experts for this product's routing), filling the memory's idle time during attention, the small ops and the last chunks: SmolLM2 Q4_K_M 249 / 248 / 248 at 4 / 8 / 16 threads to 242 / 240 / 240; GLM-4.7-Flash 11.26 to 11.36 in one pair, then, prefetching only successors of 2 MiB or more, 11.30 / 11.28 against 11.06 / 11.12. The memory is not idle enough between products for the extra traffic to pay.
The rows the round touches
(GPU=0 sh tools/bench_vs_llama.sh, the same llama.cpp
build; the machine shared with other work, load average 2.3 to 3.6; the
run was stopped after Llama-3.2-1B, whose prompt rows at 8 threads ran
slow for both programs, 708 / 693 against 805 / 727 in the table after
rounds 7 and 8). t/s, OKLM / llama.cpp:
| model | threads | pp512 OKLM / llama.cpp | ratio | tg128 OKLM / llama.cpp | ratio |
|---|---|---|---|---|---|
| SmolLM2-135M Q8_0 | 1 | 536 / 465 | 1.15 | 140 / 122 | 1.15 |
| 4 | 1984 / 1652 | 1.20 | 184 / 192 | 0.96 | |
| 8 | 3589 / 2972 | 1.21 | 184 / 187 | 0.98 | |
| 16 | 4099 / 3125 | 1.31 | 183 / 176 | 1.04 | |
| SmolLM2-135M Q4_K_M | 1 | 577 / 448 | 1.29 | 175 / 128 | 1.37 |
| 4 | 2137 / 1586 | 1.35 | 251 / 252 | 1.00 | |
| 8 | 3854 / 2754 | 1.40 | 249 / 254 | 0.98 | |
| 16 | 4207 / 2774 | 1.52 | 250 / 233 | 1.07 | |
| Llama-3.2-1B Q4_K_M | 1 | 120 / 111 | 1.08 | 25.8 / 22.3 | 1.16 |
| 4 | 438 / 401 | 1.09 | 35.7 / 35.6 | 1.00 | |
| 8 | 708 / 693 | 1.02 | 35.2 / 35.1 | 1.00 | |
| 16 | 795 / 665 | 1.20 | 35.4 / 32.5 | 1.09 |
SmolLM2 now decodes as fast on 8 or 16 threads as on 4 (Q4_K_M 249 to 251, Q8_0 183 to 184): from eight threads 0.98 to 1.07 of llama.cpp (was 0.91 to 0.95), level or ahead at sixteen, 2 % behind at eight (llama.cpp's own best is at eight). The other models' rows, not measured again, are those of the table after rounds 7 and 8: the change reaches them only through the loops' participants and the left-out workers' sleep (Llama-3.2-1B decoding the same above). Still behind: SmolLM2 Q8_0 at four threads (0.96), GLM-4.7-Flash decoding (0.97 to 0.98, unchanged), Qwen3-8B Q8_0 prompts 1 % below round 6 (still 1.71 to 1.80 times llama.cpp).
Round 10 (CPU): the activation row rounded with AVX2, packed copies on huge pages
The two decode rows left behind after round 9 (SmolLM2 at four and
eight threads, GLM-4.7-Flash), measured before anything was changed
(2026-11-12; perf is not available to an unprivileged user
on this machine, so the worker team counted its own time for the
measure: cycles between a loop's publication and its end, cycles of each
chunk, the delay before a worker's first chunk):
- The loops are not what costs. SmolLM2-135M Q8_0 at four threads, 151 loops a token: a worker starts its first chunk 0.15 microsecond after the loop is published, and the participants compute 99.8 % of the loops' time (GLM-4.7-Flash 97.5 %, the rest the wait for the last tile). Fewer loops or a cheaper hand-off had nothing to return.
- The time outside the loops was the rounding of the
activations. A quantized product first rounds its input row to
bytes (
oklm_quantize_row(), Q8_0 or Q8_K blocks) on the calling thread, the memory idle: 3.7 microseconds a product in scalar code (roundfper value), 121 products a token, 8 % of SmolLM2's step, and 4 % of GLM-4.7-Flash's. - GLM-4.7-Flash's products already read at the memory's
rate. The 37 GB/s of a large product is
bench_matmul's fastest call on one 107 MB matrix, 16 MB of which stay in the last-level cache. On weights not in cache (one-row Q4_K products of 2048 values a row over 1.15 GB of distinct matrices, four threads, huge pages) every size reads alike: 28.4 GB/s at 0.45 MB a matrix, 29.2 at 1.8 MB, 30.0 at 4 MB, 29.8 at 9 MB, 30.0 at 36 MB, 30.2 at 144 MB, 30.1 as one matrix of 1.15 GB (on 4 KiB pages 26.9, 28.8, 29.4, then 29.7); by threads 15.4 at one, 26 at two, 30 from three. Summing the same memory with 256-bit adds reads 28.1 GB/s on one thread and 31.2 on two or four: the products run at 96 % of what the memory gives, the short ones 3 % below the long ones.
Three changes, the same bits (test_matmul compares the
rounding with the scalar one on halves, zero blocks and both signs of
the largest magnitude, and packs more copies than a chunk holds; the
logits of the decoded tokens and of a prompt, before and after, compared
byte for byte: 32 tokens and a prompt of 96 on SmolLM2-135M Q8_0, Q4_K_M
and IQ4_XS and on Granite at one and four threads, with the chunks and
without, fewer tokens on GLM-4.7-Flash at one and four threads and on
Llama-3.2-1B at four):
- The activation row rounded with AVX2
(
oklm_cpu_quantize_act(), at the AVX2 and AVX-512 levels): Q8_0 rows (the block's largest magnitude, the same product per value,roundfas the integer part plus one away from zero from a half) and Q8_K rows (the scale from the first value of the largest magnitude, rounding to nearest even, the sums of each sixteen levels). - A token's rotation computes its cosines and sines once for all its heads instead of once per head.
- Packed copies on huge pages. Each copy was an
allocation of its own, for a small matrix a mapping of 4 KiB pages; they
are now cut one after the other from chunks aligned on 2 MiB and advised
as huge pages (Linux,
OKLM_CPU_PACK_HUGE=0for the former allocation; decoding SmolLM2 Q8_0, whose copies take 145 MB, the process has 143 MB of anonymous memory on huge pages, 28 before).
tg128 t/s at four threads, step by step, each pair alternated in one session:
| step | SmolLM2-135M Q8_0 | SmolLM2-135M Q4_K_M |
|---|---|---|
| rounding with AVX2 | 182.5 / 196.0 | 250.0 / 265.7 |
| the rotation's cosines once | 197.8 / 197.9 | 267.3 / 270.8 |
| copies on huge pages | 198.4 / 207.3 | 270.3 / 279.0 |
Before (the last commit of round 9's state, 6b05d98) /
after / the same llama.cpp build, alternated, the best of five runs each
(the machine shared with other work: each run started with under 1.2
hardware threads busy; llama.cpp's figures of the day are below those of
the tables above on SmolLM2 Q8_0, 186 to 190 against 192):
| model | threads | tg before / after | llama.cpp | ratio |
|---|---|---|---|---|
| SmolLM2-135M Q8_0 | 1 | 133.6 / 162.2 | 118.8 | 1.37 |
| 4 | 184.2 / 207.9 | 189.9 | 1.09 | |
| 8 | 181.9 / 199.6 | 186.9 | 1.07 | |
| 16 | 182.4 / 201.3 | 171.5 | 1.17 | |
| SmolLM2-135M Q4_K_M | 1 | 169.3 / 201.2 | 129.2 | 1.56 |
| 4 | 249.5 / 279.9 | 256.5 | 1.09 | |
| 8 | 249.8 / 278.8 | 257.4 | 1.08 | |
| 16 | 228.3 / 277.2 | 239.8 | 1.16 | |
| GLM-4.7-Flash Q4_K_M (tg64) | 4 | 11.50 / 11.99 | 11.78 | 1.02 |
| 8 | 11.47 / 11.86 | 11.59 | 1.02 |
SmolLM2 pp512 at four threads is unchanged by the chunks (Q4_K_M 2524 / 2529). GLM-4.7-Flash gains from the rounding alone (11.47 / 11.94 at four threads; with the chunks 11.91: its decode products read the file layout).
Measured and left out:
- the mapped file collapsed into huge pages
(
MADV_COLLAPSEon each 2 MiB of the mapping, which the kernel did on ext4 without a copy in the program: 102 MB of SmolLM2 Q4_K_M's file and GLM-4.7-Flash's 18 GB mapped by 2 MiB entries): GLM-4.7-Flash tg48 at four threads 11.94 / 12.00, SmolLM2 Q4_K_M 247.0 / 247.5, within the noise, and collapsing 18 GB adds to the load; - the fusions the roadmap listed (norms and rotation into the loop of the product beside them, the expert gate and up as one loop): not written, the loops' hand-off measured at 0.15 microsecond and their threads busy 99.8 % of the time.
Round 10 (GPU): decoding in fewer launches
2026-11-12. What a decode step of a small model costs on the GPU is
its launches (about 5 to 6 microseconds each on this driver, whatever
the kernel does: Granite's step with the scale and add launches skipped
went from 317 to 370 t/s, with its norms skipped to 362, with the keys'
rotation and the cache writes skipped to 334), so three more sequences
are held back by the backend and run as one launch each, with the bits
of the separate ops (test_opencl; the logits of the three
MoE models after 24 + 12 tokens are the same bits with
OKLM_CL_FUSE=0, with the three knobs off, and with them
on):
- The end of a residual branch
(
res_norm,OKLM_CL_RES): the weighted sum of the expert outputs, the add of the shared experts' output, the residual scale, the add into the residual stream and the RMS norm of the new stream (the next block's first op). Granite: the attention branch's scale, add and norm three launches to one, the expert branch's sum, scale, add and norm four to one; 424 launches a token to 304, 314 to 379 t/s. - The rotation with the cache writes
(
rope_kv,OKLM_CL_ROPE_KV): the rotation of the queries and of the keys and the copies of the keys and values into their caches, three launches (two where Q and K are one product) to one. Granite 304 launches to 268, 379 to 399 t/s. - A sigmoid router's routing (
moe_route,OKLM_CL_ROUTE_SIGMOID): GLM-4.7-Flash's sigmoid, expert choice and weights three launches to one, 1168 launches a token to 1076, 75.9 to 77.9 t/s.
bench_llama, 128 tokens decoded after a 16-token prompt,
best of three, two alternated rounds each, the three knobs off / on;
llama.cpp's HIP figures are those of round 9 (not measured again):
| model | launches a token | tg128 off | tg128 on | llama.cpp | ratio |
|---|---|---|---|---|---|
| Granite-3.1-1B-A400M Q4_K_M | 424 / 268 | 305 to 307 | 395 to 396 | 364 | 1.09 |
| OLMoE-1B-7B Q4_K_M | 249 / 209 | 293 to 299 | 315 to 321 | 360 | 0.88 |
| GLM-4.7-Flash Q4_K_M | 1259 / 1076 | 73.7 to 74.0 | 77.9 to 78.0 | 88.0 | 0.89 |
| SmolLM2-135M Q4_K_M | 344 / 300 | 416 to 428 | 471 to 472 | 383 | 1.23 |
| Llama-3.2-1B Q4_K_M | 153 / 129 | 391 to 392 | 392 to 408 | 411 | 0.95 to 0.99 |
Measured and left out:
- The norm kernel's work-group sum in two steps (three barriers instead of ten) for the rows summed without runs: within the noise (Granite 402 to 405 t/s before, 398 to 406 after), as the DPP steps of round 9 were; the kernel's time is not its barriers.
- Not tried: the router's product inside
moe_route(its launch skipped is 371 to 392 t/s on Granite, an upper bound; the one work-group of the routing would do 32 x 1024 products there, 64 x 2048 on OLMoE and GLM-4.7-Flash, most of what the launch costs), and OLMoE's two whole-row query and key norms with the rotation.
Round 11 (GPU): Q4_K prompt tiles with loader waves
2026-11-12. The 8-bit prompt tiles of Q4_K and Q5_K weights (opencl.md) now load a step while the
previous one multiplies: a work-group has four more waves that only
load, into a second buffer of local memory, and the scale arrays of
every 8-bit tile are sized for its weight type. The CPU's products bit
for bit, as before (test_opencl_q8).
bench_llama, 512 random tokens in one batch, the best of
three runs, before and after alternated over four rounds, llama.cpp's
HIP llama-bench run before and after them (17 787 and 17
884; 3 227 and 3 201; the best kept), t/s:
| model | pp512 before | pp512 now | llama.cpp | ratio before / now |
|---|---|---|---|---|
| Llama-3.2-1B Q4_K_M | 13 726 to 13 869 | 14 809 to 15 009 | 17 884 | 0.78 / 0.84 |
| Qwen3-8B Q4_K_M | 2 550 to 2 562 | 2 738 to 2 781 | 3 227 | 0.79 / 0.86 |
Shorter prompts gain more (64 tokens: Llama-3.2-1B 5 493 to 6 307, Qwen3-8B 1 194 to 1 344); the other models' prompts are level or up to 3 % faster and decoding is unchanged; the steps, those figures and what was measured and left out (two and eight loader waves, Q6_K tiles with loader waves) are in opencl.md. The other rows of the table of round 9 were not measured again.
The opt-in F16 matrix units (OKLM_CL_WMMA=1, not the
CPU's bits) gained the same day from their kernels reading the next step
while they multiply (wm_<type>,
w2_<type>), before / after alternated over three
rounds, llama.cpp before and after them: Llama-3.2-1B Q4_K_M 16 228 to
17 553 t/s (llama.cpp 17 966: 0.90 to 0.98), Qwen3-8B Q4_K_M 2 825 to 3
172 (3 218: 0.88 to 0.99); the default kernels in those rounds 14 951
and 2 783.
The indexed tiles of MoE prompts (wqi_q4_k,
wqi_q5_k) took the same loader waves later that day, the
CPU's products bit for bit as before (test_opencl_q8). The
best of three a run, before and after alternated over eleven rounds,
llama.cpp's HIP llama-bench in the same session (the best
of its samples), t/s:
| model | pp512 before | pp512 now | llama.cpp | ratio before / now |
|---|---|---|---|---|
| OLMoE-1B-7B Q4_K_M | 8 600 to 8 712 | 8 726 to 8 831 | 10 850 | 0.80 / 0.81 |
| GLM-4.7-Flash Q4_K_M | 2 175 to 2 191 | 2 201 to 2 223 | 2 078 | 1.05 / 1.07 |
| Granite-3.1-1B-A400M Q4_K_M | 17 878 to 18 186 | 17 903 to 18 173 | 15 529 | 1.17 / 1.17 |
1.3 % on the means of OLMoE and GLM-4.7-Flash, faster in every round; Granite level; the reasons are in opencl.md.
Inference
Command:
oklm run -m <file> -p "<first 2000 bytes of tinyshakespeare.txt>" -n 128 --temp 0 --backend cpu
(prompt token counts differ by tokenizer: 563 for SmolLM2), default
threads (8, one per physical core), F16 KV cache. Same machine as below,
2026-10-30 (repacked weights, best of two runs; the machine was not
idle, so expect a few % of noise). llama.cpp:
llama-bench -p 512 -n 128 -t 8 on the same files (its
generation starts from an empty context, OKLM's after the prompt).
| file | weight types | prompt t/s | generation t/s | llama.cpp prompt / generation |
|---|---|---|---|---|
| SmolLM2-135M F16 | F16 | 953 | 86 | 2471 / 100 |
| SmolLM2-135M Q8_0 | Q8_0 | 1585 | 125 | 2875 / 187 |
| SmolLM2-135M Q4_K_M | Q5_0, Q8_0, Q4_K, Q6_K | 1561 | 149 | 2660 / 247 |
| Qwen2.5-0.5B-Instruct Q4_K_M | Q5_0, Q4_K, Q6_K, Q8_0 | 957 | 59 | 857 / 70 |
| Qwen3-0.6B Q8_0 | Q8_0 | 545 | 34 | 690 / 43 |
| Llama-3.2-1B-Instruct Q4_K_M | Q4_K, Q6_K | 427 | 32 | 611 / 35 |
| TinyLlama-1.1B-Chat Q4_K_M | Q4_K, Q6_K | 377 | 38 | 588 / 43 |
| GPT-2 Q4_K_M | Q4_K, Q5_K, Q6_K | 2149 | 196 | 5335 / 313 |
With OKLM_THREADS=16 (every hardware thread) prompts are
faster (Llama 3.2 1B: 208 t/s) and generation slower (28 t/s): prompts
are compute-bound, generation memory-bound.
History of the same command (16 threads before 2026-10-30):
| change | Llama-3.2-1B Q4_K_M prompt / generation |
|---|---|
| 2026-10-29, one activation row per dot product | 104 / 27 |
| four-row kernels (Q4_K, Q6_K), F16C scale conversion | 181 / 28 |
| polling worker team, 8 threads by default | 169 / 31 (16 threads: 208 / 28) |
| K-quant kernels: vector scale decode, joint reductions, unrolling, eight-row Q4_K | 301 / 31 |
| Q4_K weights repacked eight rows wide, AVX-512 VNNI kernel | 404 / 31 |
| Q6_K, Q8_0, Q5_0 and Q4_0 repacked too; decode keeps the file layout | 427 / 32 |
| 2026-10-31, CPU attention in tiles (F16 cache): query rows x key blocks converted once | 563 / 32 |
CPU attention in tiles, 2026-10-31 (same machine, before / after, prompt t/s): Llama-3.2-1B Q4_K_M 482 / 563 with the 498-token prompt, 282 / 346 with 2060 tokens (8000 bytes); Qwen2.5-0.5B Q4_K_M 929 / 1191 and 492 / 726. One attention op at Llama-3.2-1B's shape (32 heads over 8 KV heads of 64, F16 cache), 512 queries after 1536 cached: 106 to 41 ms; one decode query over 4096 keys: 0.72 to 0.30 ms. The scores and value sums round differently (fused multiply-adds in a fixed order per row), so greedy text can change at near ties; perplexity over 60 kB of tinyshakespeare (1024-token windows, CPU) moves by about 0.05 %: Llama-3.2-1B 37.2659 to 37.2920, Qwen2.5-0.5B 42.6695 to 42.6825. That is the size of a change of rounding alone: the scalar kernels (sums in another order, the matrix products bit-identical) give 42.7273 for Qwen2.5-0.5B before the tiles.
The same tiles for every cache type and for decode, 2026-10-31:
block-quant caches (-ctk / -ctv) dequantized a
block of keys at a time with the query rounded through Q8_0 as before
(it was an integer dot per key and query); decode and small batches, too
few tiles for the threads, split the tiles' arithmetic by key blocks,
rows and value columns (same bits, so a batch now gives exactly the
logits of its tokens one by one); the score kernel runs eight
independent chains whatever the rows. Prompt / generation t/s before /
after, 128 tokens generated, best of two runs:
| model | cache | 498-token prompt | 2060-token prompt | generation after 2060 |
|---|---|---|---|---|
| Llama-3.2-1B Q4_K_M | f16 | 521 / 574 | 361 / 419 | 29.4 / 29.8 |
| Llama-3.2-1B Q4_K_M | q8_0 | 335 / 482 | 199 / 341 | 23.5 / 28.0 |
| Qwen2.5-0.5B Q4_K_M | f16 | 1193 / 1169 | 713 / 736 | 52.7 / 52.2 |
| Qwen2.5-0.5B Q4_K_M | q8_0 | 726 / 1102 | 352 / 648 | 39.3 / 49.0 |
q4_0 caches gain as q8_0 (Llama-3.2-1B, 2060 tokens: 197 to 344).
Perplexity (same 60 kB, 1024-token windows, CPU) before / after:
Llama-3.2-1B f16 37.2920 / 37.2920, q8_0 37.4049 / 37.3907, q4_0 39.7913
/ 39.6672; Qwen2.5-0.5B f16 42.6825 / 42.6825, q8_0 43.3546 / 43.2600.
Qwen2.5-0.5B with q4_0 keys is unusable either way (318 / 312;
llama.cpp's llama-perplexity -ctk q4_0 gives 303, 42.6 with
f16 keys).
AVX2 only (OKLM_SIMD=avx2, same machine, 2026-10-30),
file layout / repacked weights, prompt t/s, the same greedy text:
Llama-3.2-1B Q4_K_M 313 / 409, Qwen2.5-0.5B Q4_K_M 437 / 782,
SmolLM2-135M Q8_0 1045 / 1389.
bench_matmul 4096 4096 8 <type> 64 1, 64 rows, ms per
product, AVX2 file layout / AVX2 repacked / AVX-512 repacked: Q4_K 2.18
/ 1.53 / 1.31, Q6_K 2.63 / 2.49 / 1.52, Q8_0 3.70 / 2.21 / 1.26, Q5_0
4.96 / 1.95 / 1.26, Q4_0 4.43 / 1.94 / 1.26, Q5_K 2.49 / 2.10 / 1.67,
Q3_K 10.39 / 2.71 / 1.68, Q2_K 5.28 / 2.22 / 1.29, IQ4_NL 4.39 / 2.32 /
1.24, IQ4_XS 4.07 / 2.34 / 1.62 (2026-10-31; IQ4_XS's group sums are now
added as integers, one float step per block, as ggml: 3.91 / 2.36 /
1.61). The AVX2 Q6_K kernel then adds two steps' pair sums in int16 and
widens them with one multiply-add by the group scales (the offsets once
per block): 2.49 to 1.72 ms; prompts on AVX2 only, Llama-3.2-1B Q4_K_M
479 to 504 t/s, Qwen2.5-0.5B Q4_K_M 905 to 934, the same text.
Then, same day, AVX2 / AVX-512 repacked, ms: Q5_K, Q3_K and Q6_K
packed with each value in a byte of its own (no bits to put together in
the kernels; the packed copies 1.6, 2.2 and 1.2 times those matrices'
file size): Q5_K 2.09 / 1.67 to 1.71 / 1.40, Q3_K 2.71 / 1.68 to 1.94 /
1.30, Q6_K 1.72 / 1.52 to 1.29 / 1.30. With AVX-512 (32 vector
registers), IQ4_XS and the grid IQ types take eight activation rows per
call: IQ4_XS 1.61 to 1.33, IQ2_XXS to IQ1_M 1.97 to 1.58 (the other
kernels measured slower that way and keep four). Tiled attention
computes its softmax numerators eight at a time (Cephes' exponential,
within 2 ulp of expf): the attention of a 498-token Llama-3.2-1B prompt
0.164 to 0.120 s. Prompt t/s, CPU, against
llama-bench -p 512 -t 8 on the same files the same evening:
Llama-3.2-1B Q4_K_M 588 (llama.cpp 513 to 690 from run to run), Q3_K_M
595 / 704, Q5_K_M 594 / 773, IQ4_XS 546 to 582 / 833; Qwen2.5-0.5B
Q4_K_M 1275 / 853.
Then (2026-11-01): the compiler had kept one shared copy of most packed kernels, the activation row count a run-time value; each entry point now gets its own copy (loops unrolled, accumulators in registers) except Q2_K's and, with AVX-512, Q4_K's, which measured faster shared; with that, Q5_K, Q4_0 and IQ4_NL take eight activation rows per call with AVX-512 too. 64-row products, AVX2 / AVX-512, ms: Q5_K 1.70 / 1.40 to 1.53 / 1.20, Q3_K 1.94 / 1.30 to 1.76 / 1.21, Q6_K 1.29 / 1.30 to 1.30 / 1.20, IQ4_XS 2.39 / 1.33 to 2.14 / 1.10, Q4_0 1.94 / 1.26 to 1.97 / 1.19, IQ2_XXS 2.78 / 1.58 to 2.30 / 1.60, Q4_K 1.55 / 1.30 to 1.46 / 1.30. Prompt t/s, CPU: Llama-3.2-1B Q3_K_M 595 to 623 (llama.cpp 704), Q5_K_M 594 to 652 (773), IQ4_XS 582 to 646 (833), Q4_K_M 585. A layout with one 8-byte activation broadcast per two byte dot products was tried on Q8_0 / Q5_0 and measured slower (the kernels are bound by their instructions, not their loads: one thread's speed does not depend on K, and eight threads give 7.3 times one).
Then tiled attention: the scale and maximum of each row of scores eight at a time, its denominator summed in sixteen double lanes in a fixed order (it was one chain of dependent additions), and the value sums of the heads of a token (the same keys) in one pass over the value rows. One attention op at Llama-3.2-1B's shape (32 heads over 8 KV heads of 64, F16 cache, 8 threads): 512 queries from 0, 8.4 to 6.5 ms; after 1536 cached, 33.7 to 26.9 ms. Prompt t/s, CPU: Llama-3.2-1B Q3_K_M 642 (llama.cpp 704), Q5_K_M 679 (773), IQ4_XS 693 (833), Q4_K_M 597; perplexity unchanged to four decimals (37.3498). SiLU and SwiGLU then use the same exponential eight at a time (0.030 to 0.025 s per prompt, bound by memory from there; perplexity 37.3249). Tried and left out: Q4_K in Q5_K's byte layout (AVX-512 1.39 to 1.20 ms, AVX2 1.45 to 1.53, and twice the packed copy's memory for the most common type).
Llama-3.2-1B requantized from its Q4_K_M file
(oklm quantize), the command above, 2026-10-31, prompt t/s
with the file layout (OKLM_CPU_PACK=0) / repacked,
generation t/s, the same greedy text both ways; llama.cpp prompt /
generation on the same files:
| file | weight types | prompt t/s | generation t/s | llama.cpp |
|---|---|---|---|---|
| Q2_K | Q2_K, Q3_K, Q4_K, Q6_K | 123 / 402 | 41 | 547 / 48 |
| Q3_K_M | Q3_K, Q4_K, Q5_K, Q6_K | 116 / 371 | 36 | 691 / 41 |
| Q5_K_M | Q5_K, Q6_K | 306 / 396 | 29 | 746 / 31 |
| IQ4_XS | IQ4_XS, Q5_K, Q6_K | 195 / 361 | 33 | 930 / 37 |
| IQ4_NL | IQ4_NL, Q5_K, Q6_K | 185 / 452 | 31 | 489 / 36 |
The grid IQ types (IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS,
IQ3_S) with AVX2 dot products instead of scalar ones (2026-10-31):
bench_matmul 4096 4096 8 <type> 64, 64 rows, 96 to
128 ms per product before, 4.2 (IQ2_S) to 7.7 (IQ1_M) after; one row
0.18 to 0.31 ms. SmolLM2-135M's files of
models/smollm2-135m-iq, the command above, prompt /
generation t/s before and after, the same greedy text; llama.cpp on the
same files:
| file | before | after | llama.cpp |
|---|---|---|---|
| IQ1_S | 287 / 127 | 1313 / 178 | 3737 / 372 |
| IQ1_M | 254 / 119 | 1190 / 174 | 3615 / 368 |
| IQ2_XXS | 253 / 119 | 1413 / 174 | 3700 / 322 |
| IQ2_XS | 231 / 115 | 1385 / 176 | 3152 / 360 |
| IQ2_M | 238 / 118 | 1391 / 182 | 3164 / 355 |
| IQ3_XXS | 277 / 125 | 1391 / 178 | 3187 / 338 |
| IQ3_S | 266 / 124 | 1387 / 175 | 3171 / 339 |
The same types repacked (oklm_pack_iqg_x8: a 4-bit index
per value into the at most 16 integers the dot product multiplies, a
scale per 16 columns; about 4.6 bits per weight, so 1.5 to 2.3 times the
file's size again in memory), 2026-10-31:
bench_matmul 4096 4096 16 <type> 64 1, 64 rows, 3.9
(IQ2_S) to 6.9 ms (IQ1_M) with the AVX2 dot products, 2.67 ms packed
with AVX2 and 1.97 ms with AVX-512 VNNI for every type. SmolLM2-135M's
files, 2000-byte prompt, 64 tokens, prompt t/s with the file layout
(OKLM_CPU_PACK=0) / repacked, the same greedy text: IQ1_S
1124 / 2151, IQ1_M 1102 / 1997, IQ2_XXS 1189 / 2130, IQ2_XS 1201 / 2148,
IQ2_M 1207 / 2165, IQ3_XXS 1168 / 2203, IQ3_S 1195 / 2105; generation
unchanged (about 185 t/s).
SmolLM2 Q4_K_M ran at 71 / 42 t/s before the AVX2 Q5_0 kernel (most
of its matrices are Q5_0, because SmolLM2's 576-wide rows are not a
multiple of 256). The BF16 Hugging Face checkpoint of SmolLM2, loaded
directly (oklm run -m models/smollm2-135m-hf, short prompt,
64 tokens): 76 t/s generation with the AVX2 BF16 kernel (49 t/s before
it).
The Q2_K to Q6_K, Q8_0, Q5_0, Q4_0, IQ4_NL, IQ4_XS and grid IQ
weights of the layers run on the CPU are repacked when a context is
created (cpu_pack.h, about their size again in memory, 1.2
to 2.2 times for Q6_K, Q5_K and Q3_K, whose values take a byte each;
OKLM_CPU_PACK=0 turns it off): eight interleaved rows per
block, one output row per 32-bit lane of an AVX-512 VNNI kernel.
bench_matmul 4096 4096 8 Q4_K 64 1 (8 threads): 980 GFLOPS
with the eight-row kernel, 1646 with the repacked weights; Q6_K 577 to
751, Q8_0 368 to 1146, Q5_0 326 to 1180, Q4_0 479 to 1679 (a loaded
machine). Products of fewer than four activation rows (decode) keep the
file layout: they are bound by memory, and the file layout is as small
or smaller. Since 2026-11-08 one-row products read the packed copy with
one thread, and with two when it is at most 10 % larger than the file
layout (round 1 above).
Decoding on a busy machine (2026-11-08)
Generation t/s with
oklm run -p "The capital of France is" -n 32 --temp 0 --backend cpu,
OKLM_THREADS = 1 to 16, beside 0, 8 or 16 processes that
only spin (a small C busy loop, standing for
stress-ng --cpu N), on the i7-11700K (8 cores, 16 hardware
threads). Before: the nilorea N_PARALLEL_TEAM (a static
slice per thread, every thread checked in at the end of each loop, 20000
pause polls before sleeping). After: OKLM's team
(team_cpu.c), chunks claimed one at a time, four per
thread, a short poll then sched_yield() rounds (core.md). llama.cpp:
llama-bench -ngl 0 -p 0 -n 32 -r 2 -t N (CPU only, OpenMP
build), "-" when it took more than two minutes. Other jobs shared the
machine during the whole run (load 15 to 35 before the spinners), so "0
spinners" is not idle; each row ran base, new and llama.cpp back to
back.
| model | spinners | threads | before | after | llama.cpp |
|---|---|---|---|---|---|
| Granite-3.1-1B-A400M Q4_K_M | 0 | 1 | 20.2 | 23.1 | 16.1 |
| 0 | 2 | 21.1 | 37.1 | 27.4 | |
| 0 | 4 | 43.3 | 45.2 | 65.4 | |
| 0 | 8 | 9.0 | 62.2 | 78.4 | |
| 0 | 16 | 1.9 | 68.9 | 7.1 | |
| 8 | 1 | 21.5 | 31.4 | 35.7 | |
| 8 | 2 | 47.3 | 47.9 | 61.7 | |
| 8 | 4 | 23.2 | 62.4 | 77.8 | |
| 8 | 8 | 1.9 | 53.9 | 17.6 | |
| 8 | 16 | 1.0 | 64.5 | - | |
| 16 | 1 | 23.4 | 15.9 | 38.4 | |
| 16 | 2 | 7.3 | 32.3 | 45.1 | |
| 16 | 4 | 3.7 | 19.8 | 9.3 | |
| 16 | 8 | 1.1 | 23.4 | - | |
| 16 | 16 | 0.8 | 26.3 | - | |
| Llama-3.2-1B-Instruct Q4_K_M | 0 | 1 | 5.6 | 5.8 | 6.0 |
| 0 | 2 | 12.5 | 10.0 | 11.3 | |
| 0 | 4 | 13.8 | 14.3 | 14.4 | |
| 0 | 8 | 10.1 | 30.5 | 27.7 | |
| 0 | 16 | 2.6 | 25.5 | 5.3 | |
| 8 | 1 | 12.3 | 11.9 | 15.1 | |
| 8 | 2 | 20.2 | 21.2 | 24.2 | |
| 8 | 4 | 19.2 | 25.7 | 30.5 | |
| 8 | 8 | 3.1 | 24.4 | 12.4 | |
| 8 | 16 | 1.8 | 26.8 | 0.6 | |
| 16 | 1 | 12.1 | 9.8 | 11.4 | |
| 16 | 2 | 4.0 | 10.7 | 10.4 | |
| 16 | 4 | 2.8 | 8.5 | 6.7 | |
| 16 | 8 | 1.8 | 9.7 | 0.8 | |
| 16 | 16 | 1.3 | 11.2 | - |
One thread runs no team, so its row is noise from the other jobs. Before, a loop waited for its slowest thread; with more runnable threads than hardware threads one of them is always waiting for a core, and each of the few hundred loops of a token paid a scheduler time slice (Granite at 16 threads: 0.8 to 1.9 t/s). After, the decode speed stays within a factor of about two of its best whatever the thread count and the load. llama.cpp is ahead at 2 to 4 threads on a loaded machine, and falls apart like OKLM did once its threads outnumber the free cores (its OpenMP barrier waits for every thread). At a quieter moment of the same evening: Granite with 16 threads and no spinner went from 11.2 to 87.5 t/s (llama.cpp 91.2), 8 threads 84.8 / 71.7 / 90.6 (noise of a few runs).
The wait policy was chosen on the same models (median of 2 to 4 runs, Granite with 8 spinners): 20000 pause polls before yielding gave 31.4 (8 threads) and 26.2 (16 threads) t/s, 1000 polls then 32 yields of 256 polls 42.5 / 44.0, 300 polls then 512 yields of 32 polls (kept) 52.7 / 60.5; with no spinner the choices were within the noise. Two or eight chunks per thread instead of four were within the noise too (about 40 t/s each, four giving 21 and 42 in the same runs, with the long poll).
Decoding on a busy machine, the loops capped by the free cores (2026-11-11)
Two changes (core.md): a row
loop's slices take at least 16384 values (a decode step's rotation and
cache copies run on the calling thread), and a loop takes no more
participants than the cores other programs leave free (their runnable
threads, sampled every 100 ms), the kernels that depend on the thread
count going by the threads available. Generation t/s as above (64
tokens), the machine otherwise quiet (load 13 to 31 from the spinners
themselves), beside 0, 8 or 16 spinning processes; "before" the previous
commit, two runs each; llama.cpp
llama-bench -ngl 0 -p 0 -n 64 -r 2 -t N, "-" when it took
more than two minutes:
| model | spinners | threads | before | after | llama.cpp |
|---|---|---|---|---|---|
| Llama-3.2-1B-Instruct Q4_K_M | 0 | 2 / 4 / 8 / 16 | 32.5 / 34.9 / 34.7 / 34.9 | 32.6 / 35.0 / 34.8 / 34.8 | 32.4 / 35.0 / 34.8 / 31.0 |
| 8 | 2 / 4 / 8 / 16 | 26.6 / 31.8 / 33.3 / 33.4 | 28.6 / 31.7 / 33.4 / 33.3 | 27.9 / 33.6 / 31.9 / - | |
| 16 | 2 | 18.0, 13.1 | 19.6, 19.9 | 22.1 | |
| 16 | 4 | 10.0, 11.3 | 18.2, 20.7 | 14.7 | |
| 16 | 8 | 16.6, 17.3 | 18.1, 21.2 | 1.5 | |
| 16 | 16 | 12.7, 13.5 | 19.8, 19.4 | - | |
| Granite-3.1-1B-A400M Q4_K_M | 0 | 2 / 4 / 8 / 16 | 83.5 / 94.7 / 80.8 / 91.6 | 83.4 / 94.8 / 95.1 / 91.0 | 88.8 / 96.4 / 96.4 / 85.6 |
| 8 | 2 / 4 / 8 / 16 | 69.8 / 83.8 / 88.4 / 86.9 | 69.6 / 83.3 / 87.5 / 88.1 | 70.4 / 90.8 / 83.6 / - | |
| 16 | 2 | 51.3, 51.4 | 43.4, 46.0 | 50.6 | |
| 16 | 4 | 37.5, 36.9 | 46.8, 40.6 | 28.0 | |
| 16 | 8 | 35.7, 40.6 | 43.3, 38.8 | 2.7 | |
| 16 | 16 | 47.5, 47.1 | 49.3, 49.1 | - |
The rows of several thread counts give the mean of two runs
(Granite's 80.8 before at 8 threads and no spinner: runs of 64.5 and
97.0, the first one disturbed). With cores free (0 or 8 spinners on 16
hardware threads) nothing changes. With every hardware thread busy, each
loop's threads were descheduled while holding chunks: Llama-3.2-1B
decoded at about 20 t/s with one thread and 10 to 17 with more; the cap
runs the loops on the one thread the free cores allow, at the one-thread
speed whatever OKLM_THREADS says (llama.cpp falls apart
from eight threads there). The cap alone was not enough at first: with
eight threads configured but one running, the decode products still read
the file layout, chosen for several threads, at 11 t/s; going by the
threads available, they read the packed copy one thread reads fastest
(20 t/s). Granite at two threads looked like the one row that lost (51
against 45 t/s, two runs each). Measured again with eight runs each
(2026-11-11, beside 16 spinners): 44.3 t/s with the cap against 40.6
without (single runs from 36 to 51 either way), at four threads 45.5
against 37.0; one thread gives 44.8. The loss was the runs' spread, not
the cap; Llama-3.2-1B's two threads lose to one as before (14.7 without
the cap against 19.4 with it).
OpenCL
See opencl.md: the same files on an AMD Radeon gfx1100 generate 4 to 13 times and process prompts 17 to 71 times faster than the CPU backend (for example Llama-3.2-1B Q4_K_M: 268 against 21 tokens/s, and 5 189 against 93 for the prompt).
Matrix products on OpenCL, 2026-10-30,
OKLM_CL_GEMM_TILES=0 (row kernels) against the tiled
kernels, Q4_K weights, microseconds (TFLOPS):
| N x K x columns | row kernels | tiled |
|---|---|---|
| 8192 x 2048 x 512 | 1956 (8.8) | 1228 (14.0) |
| 2048 x 8192 x 512 | 1863 (9.2) | 1404 (12.2) |
| 2048 x 2048 x 512 | 576 (7.5) | 515 (8.4) |
| 512 x 2048 x 512 | 243 (4.4) | 329 (3.3) |
| 2048 x 2048 x 64 | 137 (3.9) | 303 (1.8) |
The tiled kernels need many tiles to fill the device's 84 compute
units; below 256 tiles of 64 x 64 outputs they split K (up to 8 ranges
of at least 4 steps of 32, the partial tiles added in split order), and
the row kernels are kept only when K is too short for that. Prompt t/s
without / with the split (OKLM_CL_GEMM_SPLIT=1 / default),
2026-10-31, the same greedy text: SmolLM2-135M Q8_0 16 874 / 17 953 (498
tokens) and 10 140 / 10 840 (2060 tokens), Qwen2.5-0.5B Q4_K_M 10 080 /
11 128 and 7 057 / 7 644, Qwen3-0.6B Q8_0 6 415 / 6 689 and 3 042 / 3
129; Llama-3.2-1B, whose products already have enough tiles, unchanged
(5 185 / 5 201).
Attention kernels on OpenCL, 2026-10-31, before / after: F16 keys read eight halves at a time in the per-(head, token) kernel and in the tiles (they went through a load per element with a type test), block-quant caches dequantized into the tiles, the tiles kept for heads up to 128 and spans of 1024 keys or more with a float cache (any span with a block quant), split K in the products. Prompt / generation t/s, 498-token and 2060-token prompts, 64 tokens generated, the same greedy text:
| file | cache | 498 tokens | 2060 tokens |
|---|---|---|---|
| SmolLM2-135M Q8_0 | f16 | 16 908 / 331 to 21 258 / 366 | 10 170 / 315 to 15 333 / 351 |
| Qwen2.5-0.5B Q4_K_M | f16 | 10 014 / 254 to 11 777 / 254 | 7 010 / 220 to 8 659 / 250 |
| Qwen3-0.6B Q8_0 | f16 | 6 423 / 204 to 8 004 / 222 | 3 038 / 151 to 4 083 / 199 |
| Llama-3.2-1B Q4_K_M | f16 | 5 223 / 266 to 5 135 / 281 | 4 137 / 187 to 4 459 / 250 |
| Gemma-3-1B Q4_K_M | f16 | 4 417 / 136 to 6 096 / 151 | 4 005 / 148 to 5 773 / 164 |
| Gemma-2-2B Q4_K_M | f16 | 2 131 / 113 to 2 436 / 125 | 1 250 / 109 to 1 948 / 121 |
| SmolLM2-135M Q8_0 | q8_0 | 17 124 to 18 311 | 10 059 to 13 310 |
| Qwen2.5-0.5B Q4_K_M | q8_0 | 9 709 to 10 956 | 6 603 to 8 049 |
| Llama-3.2-1B Q4_K_M | q8_0 | 4 878 to 5 128 | 3 628 to 4 262 |
| Gemma-3-1B Q4_K_M | q8_0 | 5 185 to 5 804 | 4 903 to 5 565 |
Generation with a q8_0 cache is unchanged (its values are still read element by element).
Prompt attention with few heads, 2026-10-31 (an intermediate step, before the faster loads made the per-(head, token) kernel the better one for Gemma): Gemma 3's 4 heads of 256 gave too few query blocks per 512-token chunk for the tiled kernel (one work-group per (head, token) instead: 1251 us per call for Gemma-3-270M); the keys of each block are now split and merged (1533 us with the merges for the same calls). Prompt t/s before / after, the same greedy text: Gemma-3-270M Q8_0 11 924 / 13 147 (498 tokens) and 10 035 / 10 161 (2060), Gemma-3-1B Q4_K_M 4 418 / 4 643 and 4 024 / 4 044, Gemma-2-2B Q4_K_M 2 145 / 2 125 and 1 245 / 1 293.
MoE decode,
oklm run -p "<short prompt>" -n 128 --temp 0
(DeepSeek: 64 tokens), 2026-10-30, before and after routing on the
device (expert choice, weights and weighted sums as kernels, indexed
expert products, no read-back per layer); the greedy text is the
same:
| file | OpenCL before / after | CPU before / after |
|---|---|---|
| Granite-3.1-1B-A400M-instruct Q4_K_M | 101 / 230 | 73 / 88 |
| DeepSeek-Coder-V2-Lite-Instruct IQ4_XS | 65 / 102 | 15.4 / 16.3 |
Training
See training.md: steps on the CPU,
opencl and rocm, a full fine-tune against
LoRA. 2026-11-04, AMD Radeon RX 7900 XT (gfx1100), ROCm 7.2.4, rocBLAS
4.4, a step of 4 x 256 tokens, LoRA rank 16 on Q, K, V, O:
| Model (file) | Full, opencl |
Full, rocm |
LoRA, opencl |
LoRA, rocm |
|---|---|---|---|---|
| SmolLM2-135M (F16) | 163 ms | 118 ms | 126 ms | 98 ms |
| Qwen2.5-0.5B (Q4_K_M) | 500 ms | 315 ms | 322 ms | 199 ms |
| Qwen3-0.6B (Q8_0) | 635 ms | 410 ms | 443 ms | 291 ms |
| Llama-3.2-1B (Q4_K_M) | 9 s (beyond the device's memory) | out of memory | 679 ms | 344 ms |
rocBLAS's solution for each of the trainer's products against the
fastest it lists
(build/release/bin/tests/train/bench_rocblas --solutions,
microseconds): within 5 % on most, faster for three (SmolLM2's MLP down
forward 105 against 67 at 512 rows, its Q/K/V input gradient 57 against
42, the Nano's Q/K/V weight gradient 224 against 103), at most 3 % of a
step; the default is kept (solution indices belong to one rocBLAS
build).
Tokenizer
Command: build/release/bin/tests/tok/bench_tokenizer
(part of make bench): data/tinyshakespeare.txt (1.12 MB)
encoded in one call, one thread. Same machine as below, 2026-10-29.
| tokenizer | MB | tokens | seconds | MB/s | tokens/s |
|---|---|---|---|---|---|
| GPT-2 | 1.12 | 338025 | 0.133 | 8.4 | 2 536 982 |
| Llama 2 (SentencePiece) | 1.12 | 368633 | 0.748 | 1.5 | 492 529 |
| Llama 3.2 | 1.12 | 301768 | 0.361 | 3.1 | 835 656 |
| Qwen2.5 (with NFC) | 1.12 | 301829 | 0.173 | 6.4 | 1 743 024 |
| SmolLM2 | 1.12 | 341094 | 0.203 | 5.5 | 1 681 483 |
Llama 2 has no pre-tokenizer, so the whole text is one merge sequence
(as in Hugging Face); Llama 3.2 pays for ignore_merges
lookups of every piece.
Matrix product, CPU
Command: make bench
(tests/core/bench_matmul.c), K = N = 4096, M = 1 (decode:
one activation row) and M = 64 (prompt processing). "GB/s(w)" is the
weight bytes read per second, the figure that bounds decode speed.
Machine: Intel Core i7-11700K (8 cores / 16 threads, AVX2, AVX-512
with VNNI), 128 GB RAM, Linux 7.2, GCC 16.2, -O2.
16 threads (2026-10-30, every SIMD level)
| type | kernel | K x N | M | th | ms / call | GFLOPS | GB/s(w) |
|---|---|---|---|---|---|---|---|
| F32 | scalar | 4096 x 4096 | 1 | 16 | 2.219 | 15.1 | 30.2 |
| F32 | avx2 | 4096 x 4096 | 1 | 16 | 2.177 | 15.4 | 30.8 |
| F32 | avx512 | 4096 x 4096 | 1 | 16 | 2.164 | 15.5 | 31.0 |
| F32 | scalar | 4096 x 4096 | 64 | 16 | 17.451 | 123.1 | 3.8 |
| F32 | avx2 | 4096 x 4096 | 64 | 16 | 8.960 | 239.7 | 7.5 |
| F32 | avx512 | 4096 x 4096 | 64 | 16 | 9.068 | 236.8 | 7.4 |
| F16 | scalar | 4096 x 4096 | 1 | 16 | 2.157 | 15.6 | 15.6 |
| F16 | avx2 | 4096 x 4096 | 1 | 16 | 0.858 | 39.1 | 39.1 |
| F16 | avx512 | 4096 x 4096 | 1 | 16 | 0.872 | 38.5 | 38.5 |
| F16 | scalar | 4096 x 4096 | 64 | 16 | 127.620 | 16.8 | 0.3 |
| F16 | avx2 | 4096 x 4096 | 64 | 16 | 6.541 | 328.3 | 5.1 |
| F16 | avx512 | 4096 x 4096 | 64 | 16 | 6.419 | 334.6 | 5.2 |
| BF16 | scalar | 4096 x 4096 | 1 | 16 | 1.017 | 33.0 | 33.0 |
| BF16 | avx2 | 4096 x 4096 | 1 | 16 | 0.877 | 38.3 | 38.3 |
| BF16 | avx512 | 4096 x 4096 | 1 | 16 | 0.863 | 38.9 | 38.9 |
| BF16 | scalar | 4096 x 4096 | 64 | 16 | 47.687 | 45.0 | 0.7 |
| BF16 | avx2 | 4096 x 4096 | 64 | 16 | 6.507 | 330.0 | 5.2 |
| BF16 | avx512 | 4096 x 4096 | 64 | 16 | 6.501 | 330.3 | 5.2 |
| Q8_0 | scalar | 4096 x 4096 | 1 | 16 | 0.310 | 108.4 | 57.6 |
| Q8_0 | avx2 | 4096 x 4096 | 1 | 16 | 0.203 | 165.3 | 87.8 |
| Q8_0 | avx512 | 4096 x 4096 | 1 | 16 | 0.214 | 156.5 | 83.2 |
| Q8_0 | scalar | 4096 x 4096 | 64 | 16 | 16.189 | 132.7 | 1.1 |
| Q8_0 | avx2 | 4096 x 4096 | 64 | 16 | 8.029 | 267.5 | 2.2 |
| Q8_0 | avx512 | 4096 x 4096 | 64 | 16 | 8.258 | 260.0 | 2.2 |
| Q4_0 | scalar | 4096 x 4096 | 1 | 16 | 0.321 | 104.4 | 29.4 |
| Q4_0 | avx2 | 4096 x 4096 | 1 | 16 | 0.203 | 164.9 | 46.4 |
| Q4_0 | avx512 | 4096 x 4096 | 1 | 16 | 0.207 | 162.1 | 45.6 |
| Q4_0 | scalar | 4096 x 4096 | 64 | 16 | 18.903 | 113.6 | 0.5 |
| Q4_0 | avx2 | 4096 x 4096 | 64 | 16 | 8.867 | 242.2 | 1.1 |
| Q4_0 | avx512 | 4096 x 4096 | 64 | 16 | 8.699 | 246.9 | 1.1 |
| Q4_1 | scalar | 4096 x 4096 | 1 | 16 | 0.396 | 84.8 | 26.5 |
| Q4_1 | avx2 | 4096 x 4096 | 1 | 16 | 0.234 | 143.6 | 44.9 |
| Q4_1 | avx512 | 4096 x 4096 | 1 | 16 | 0.229 | 146.4 | 45.8 |
| Q4_1 | scalar | 4096 x 4096 | 64 | 16 | 23.336 | 92.0 | 0.4 |
| Q4_1 | avx2 | 4096 x 4096 | 64 | 16 | 13.346 | 160.9 | 0.8 |
| Q4_1 | avx512 | 4096 x 4096 | 64 | 16 | 13.089 | 164.1 | 0.8 |
| Q5_0 | scalar | 4096 x 4096 | 1 | 16 | 2.212 | 15.2 | 5.2 |
| Q5_0 | avx2 | 4096 x 4096 | 1 | 16 | 0.232 | 144.9 | 49.8 |
| Q5_0 | avx512 | 4096 x 4096 | 1 | 16 | 0.231 | 145.3 | 50.0 |
| Q5_0 | scalar | 4096 x 4096 | 64 | 16 | 131.219 | 16.4 | 0.1 |
| Q5_0 | avx2 | 4096 x 4096 | 64 | 16 | 8.976 | 239.2 | 1.3 |
| Q5_0 | avx512 | 4096 x 4096 | 64 | 16 | 8.854 | 242.5 | 1.3 |
| Q5_1 | scalar | 4096 x 4096 | 1 | 16 | 1.674 | 20.1 | 7.5 |
| Q5_1 | avx2 | 4096 x 4096 | 1 | 16 | 0.265 | 126.8 | 47.5 |
| Q5_1 | avx512 | 4096 x 4096 | 1 | 16 | 0.264 | 127.2 | 47.7 |
| Q5_1 | scalar | 4096 x 4096 | 64 | 16 | 101.385 | 21.2 | 0.1 |
| Q5_1 | avx2 | 4096 x 4096 | 64 | 16 | 14.868 | 144.4 | 0.8 |
| Q5_1 | avx512 | 4096 x 4096 | 64 | 16 | 14.934 | 143.8 | 0.8 |
| Q2_K | scalar | 4096 x 4096 | 1 | 16 | 0.318 | 105.6 | 17.3 |
| Q2_K | avx2 | 4096 x 4096 | 1 | 16 | 0.143 | 234.3 | 38.4 |
| Q2_K | avx512 | 4096 x 4096 | 1 | 16 | 0.142 | 235.8 | 38.7 |
| Q2_K | scalar | 4096 x 4096 | 64 | 16 | 18.951 | 113.3 | 0.3 |
| Q2_K | avx2 | 4096 x 4096 | 64 | 16 | 7.793 | 275.6 | 0.7 |
| Q2_K | avx512 | 4096 x 4096 | 64 | 16 | 7.764 | 276.6 | 0.7 |
| Q3_K | scalar | 4096 x 4096 | 1 | 16 | 1.952 | 17.2 | 3.7 |
| Q3_K | avx2 | 4096 x 4096 | 1 | 16 | 0.289 | 115.9 | 24.9 |
| Q3_K | avx512 | 4096 x 4096 | 1 | 16 | 0.291 | 115.5 | 24.8 |
| Q3_K | scalar | 4096 x 4096 | 64 | 16 | 118.873 | 18.1 | 0.1 |
| Q3_K | avx2 | 4096 x 4096 | 64 | 16 | 17.040 | 126.0 | 0.4 |
| Q3_K | avx512 | 4096 x 4096 | 64 | 16 | 16.942 | 126.8 | 0.4 |
| Q4_K | scalar | 4096 x 4096 | 1 | 16 | 0.223 | 150.7 | 42.4 |
| Q4_K | avx2 | 4096 x 4096 | 1 | 16 | 0.123 | 272.1 | 76.5 |
| Q4_K | avx512 | 4096 x 4096 | 1 | 16 | 0.124 | 271.2 | 76.3 |
| Q4_K | scalar | 4096 x 4096 | 64 | 16 | 12.851 | 167.1 | 0.7 |
| Q4_K | avx2 | 4096 x 4096 | 64 | 16 | 4.631 | 463.7 | 2.0 |
| Q4_K | avx512 | 4096 x 4096 | 64 | 16 | 4.630 | 463.8 | 2.0 |
| Q5_K | scalar | 4096 x 4096 | 1 | 16 | 0.296 | 113.3 | 38.9 |
| Q5_K | avx2 | 4096 x 4096 | 1 | 16 | 0.147 | 228.8 | 78.7 |
| Q5_K | avx512 | 4096 x 4096 | 1 | 16 | 0.147 | 228.2 | 78.4 |
| Q5_K | scalar | 4096 x 4096 | 64 | 16 | 17.631 | 121.8 | 0.7 |
| Q5_K | avx2 | 4096 x 4096 | 64 | 16 | 4.857 | 442.1 | 2.4 |
| Q5_K | avx512 | 4096 x 4096 | 64 | 16 | 4.961 | 432.9 | 2.3 |
| Q6_K | scalar | 4096 x 4096 | 1 | 16 | 1.762 | 19.0 | 7.8 |
| Q6_K | avx2 | 4096 x 4096 | 1 | 16 | 0.136 | 246.3 | 101.0 |
| Q6_K | avx512 | 4096 x 4096 | 1 | 16 | 0.119 | 280.8 | 115.2 |
| Q6_K | scalar | 4096 x 4096 | 64 | 16 | 108.128 | 19.9 | 0.1 |
| Q6_K | avx2 | 4096 x 4096 | 64 | 16 | 4.909 | 437.4 | 2.8 |
| Q6_K | avx512 | 4096 x 4096 | 64 | 16 | 3.563 | 602.8 | 3.9 |
1 thread (2026-10-29, before the four-row and AVX-512 kernels)
| type | kernel | K x N | M | th | ms / call | GFLOPS | GB/s(w) |
|---|---|---|---|---|---|---|---|
| Q8_0 | scalar | 4096 x 4096 | 1 | 1 | 4.368 | 7.7 | 4.1 |
| Q8_0 | avx2 | 4096 x 4096 | 1 | 1 | 2.419 | 13.9 | 7.4 |
| Q8_0 | scalar | 4096 x 4096 | 64 | 1 | 125.553 | 17.1 | 0.1 |
| Q8_0 | avx2 | 4096 x 4096 | 64 | 1 | 113.786 | 18.9 | 0.2 |
| Q4_K | scalar | 4096 x 4096 | 1 | 1 | 1.580 | 21.2 | 6.0 |
| Q4_K | avx2 | 4096 x 4096 | 1 | 1 | 0.968 | 34.7 | 9.8 |
| Q4_K | scalar | 4096 x 4096 | 64 | 1 | 107.792 | 19.9 | 0.1 |
| Q4_K | avx2 | 4096 x 4096 | 64 | 1 | 63.003 | 34.1 | 0.1 |
Indexed (expert) product, 8 threads (2026-11-07)
build/release/bin/tests/core/bench_matmul_id 8
(tests/core/bench_matmul_id.c): a Qwen3-30B-A3B-like layer,
128 experts, 8 per token, n_embd 2048, expert feed-forward 768, Q4_K, a
Zipf routing over the experts ("exp": experts used by the batch); "gate"
shares one input row between a token's ranks (gate and up), "down" has
one per rank. Milliseconds per call, the best of two runs, before /
after grouping the rows by expert (the same bits), on a machine busy
with other jobs:
| toks | exp | gate before / after | down before / after | speed-up |
|---|---|---|---|---|
| 1 | 8 | 0.096 / 0.069 | 0.096 / 0.094 | 1.0-1.4 |
| 16 | 59 | 2.150 / 1.748 | 2.308 / 1.914 | 1.2 |
| 128 | 124 | 23.21 / 7.65 | 22.99 / 8.64 | 2.7-3.0 |
| 512 | 128 | 92.80 / 24.44 | 96.15 / 28.56 | 3.4-3.8 |
The whole expert block of that layer (gate, up, SwiGLU, down) for prompt batches, both ways the model can run it on the CPU (second table of the same command): one ordinary product per expert after sorting the pairs on the host, with the gathers in and out (the model's path for every batch above n x 8 > 128 until 2026-11-07), and the indexed products (its path since then). Milliseconds per call, the best of three runs on a machine busy with other jobs; both give the same bits:
| toks | per expert | indexed | speed-up |
|---|---|---|---|
| 128 | 28.56 | 19.75 | 1.45 |
| 512 | 75.26 | 57.57 | 1.31 |
| 2048 | 276.06 | 225.92 | 1.22 |
End to end, the same switch on a real MoE file: Granite-3.1-1B-A400M
Q4_K_M (32 experts, 8 per token, n_embd 1024, expert feed-forward 512),
CPU, 8 threads, a prompt of n random tokens from an empty context
(oklm_eval(), batches of 512), F16 KV cache, best of three
runs of four; OKLM_MOE_EXPERT_LOOP=1 (one product per
expert) / default (indexed products), prompt t/s; the last logits are
bit-identical:
| prompt | per expert t/s | indexed t/s | speed-up |
|---|---|---|---|
| 128 | 545 | 750 | 1.38 |
| 512 | 522 | 692 | 1.33 |
| 2048 | 469 | 544 | 1.16 |
Reading the numbers
- Decode (M = 1) is memory-bound: Q4_K and Q6_K reach 36-40 GB/s of weights, a large share of what dual-channel DDR4-3200 can deliver (51.2 GB/s theoretical; the module configuration of this machine was not recorded). F32 decode reads 64 MB per call and is bandwidth-limited on both kernels; the difference between them is run-to-run noise.
- GCC auto-vectorizes some scalar kernels well (Q4_K scalar is fast); the scalar F16 and Q6_K kernels are not vectorized and show it.
- Prompt processing (M = 64) of quantized weights is compute-bound. The four-row kernels unpack each weight block once for four activation rows and reduce the four columns together; with the F16C scale conversion they brought Q4_K from 310 to 415 GFLOPS, Q8_0 from 134 to 228, Q5_K from 237 to 400 (16 threads, 2048 x 2048), all bit-identical to the scalar kernels. AVX-512 + VNNI helps Q6_K (365 -> 500); for Q4_K it measured no faster than AVX2 and is not used.