Kernel tuning: codebase map
The survey that kernel-autotune-plan.md asks for before any tuning code: how OKLM is built, which kernels run on the forward path, where they are chosen, where a model's shapes become known, and how deterministic the results are. Line numbers are those of commit b1c1266 (2026-10-31).
Status: maintained (the survey before the tuner; see 02-parameterization.md for what it searches).
Build, run, test
- C17, GNU make, no other build system.
make -j16builds the library,build/release/bin/oklmand the tests;make testsbuilds the tests only (not the CLI). - Run:
build/release/bin/oklm run -m <model.gguf> -p "<prompt>" -n 128 --temp 0 --backend cpu|opencl. - Tests:
make test(24 test binaries),make check,make doc(Doxygen, 0 warnings),make docs-lint; sanitizers once a day (make asan-test,make tsan-test,make asan-opencl),make wine-test. - Benchmarks exist already:
tests/core/bench_matmul.c(one matrix product, any type, SIMD level and thread count,[K] [N] [threads] [type] [rows] [pack]),tests/tok/bench_tokenizer.c, and the end-to-end timingsoklm runprints (prompt and generation t/s).
Backends
- CPU (
src/core/cpu/): scalar, AVX2 and AVX-512 VNNI kernels, chosen at run time (OKLM_SIMD,oklm_cpu_set_simd()); one polling worker team (OKLM_THREADS, default one thread per physical core). - OpenCL (
src/core/opencl/): one backend instance per device, kernels inkernels.clbuilt at first use (OpenCL C 1.2). - Selection:
oklm_backend_select()(src/core/backend.c), fromoklm_context_params.backendorOKLM_BACKEND; a context puts its last layers and the head on the device, the rest on the CPU.
Dispatch
Every op goes through oklm_op_run()
(src/core/ops.c:55), which hands the descriptor to the
backend owning the destination: cpu_compute_op()
(src/core/cpu/backend_cpu.c:110) or
cl_compute()
(src/core/opencl/backend_cl.c:2508), each a switch over the
op. Inside, the kernel variant is chosen per call from the shapes, the
types and a few thresholds; there is no dispatch table and no per-model
state, so a tuner can only steer these choices through new state the
backends read.
Kernels of the forward path
CPU
| Kernel | Where | Prompt / decode | Knobs today |
|---|---|---|---|
| matrix product, file layout | oklm_cpu_matmul()
src/core/cpu/matmul_cpu.c:559, dot products per type in
simd_avx2.c, simd_avx512.c,
simd_avx2_iq.c |
decode (1 row), and prompts for unpacked types | oklm_cpu_tuning.matmul_rows 16 output rows per task
(cpu.h:111), one even share per thread below
FEW_ROWS rows |
| matrix product, repacked weights | gemm_pack.inc (e.g. gemm_q4k_x8 :51,
gemm_iqg_x8 :861), packed by
oklm_cpu_pack_weight() pack_cpu.c:440 |
prompts (2 rows or more) | rows in groups of 8; 4 activation rows per call;
OKLM_CPU_PACK=0 turns packing off |
| attention | oklm_cpu_attention() attention_cpu.c:689:
tiles (:397), split path (:581), pairs (:97) |
both | ATTN_TILE_ROWS 32 (attention_cpu.c:214),
oklm_cpu_tuning.attn_keys 64 (cpu.h:114); runs
of heads chosen from the group size |
| RMSNorm, RoPE, SwiGLU, copies, get_rows | ops_cpu.c |
both | rows per task |
| MoE routing | moe_cpu.c |
both | none |
Results must stay bit-identical to the scalar kernels (packed and SIMD kernels) and independent of the thread count (attention): any tuned CPU variant has to keep the same order of every sum, so the knobs that are safe are the work split (rows per task, tile sizes that do not change a row's order) rather than the arithmetic.
OpenCL
| Kernel | Where | Prompt / decode | Knobs today |
|---|---|---|---|
mm_<type> row kernels |
kernels.cl:1887 (MM_KERNEL), launched by
run_matmul() backend_cl.c:1646 |
decode, small prompts | WG_MM 64, MM_ROWS 4, MM_COLS
8 (backend_cl.c:137) |
gm_<type> tiled kernels,
gm_reduce |
kernels.cl:2158 (GM_KERNEL) |
prompts | GM_TILES 256 (OKLM_CL_GEMM_TILES), split
of K up to GM_SPLIT 8 (OKLM_CL_GEMM_SPLIT),
tile 64 x 64, K step 32 |
attention, attention_merge |
kernels.cl:769, run_attention()
backend_cl.c:1089 |
decode, short prompts, Gemma | WG_ATT 256, key split for few pairs (4 work-groups per
compute unit, ranges of 128 keys or more) |
attention_tile64/128/256 |
kernels.cl:940 (AT_KERNEL) |
prompts | OKLM_CL_ATT_BLOCKS, OKLM_CL_ATT_KEYS
(1024), 32 / 16 queries per block, key split for few blocks (aimed at 4
x the blocks that fill the device) |
norm, rope, ew,
quantize_rows, hadamard, MoE kernels |
kernels.cl |
both | WG_NORM 256 |
The OpenCL thresholds are already environment knobs, set by hand from measurements on one gfx1100 (42 compute units): they are what a tuner would search first, per device.
Hot path
Llama-3.2-1B Q4_K_M, a 510-token prompt and 128 generated tokens,
greedy, OKLM_CPU_PROFILE=1 / OKLM_CL_PROFILE=1
(OpenCL profiling waits after every launch, so its decode figures are
dominated by launch overhead: 1.1 s of host time for 0.41 s of kernel
time, against about 4 ms per token unprofiled).
| Prompt (510 tokens) | CPU (1.00 s of ops) | OpenCL (95 ms of kernels) |
|---|---|---|
| matrix products | 78.7 % (Q4_K 65 %, Q6_K 14 %) | 82.3 % (gm_q4_k 65.3 %, gm_q6_k 13.5 %,
row kernels 3.3 %) |
| attention | 16.4 % | 13.0 % |
SwiGLU / ew |
3.0 % | 3.3 % |
| RoPE, norms, copies | 1.9 % | 1.5 % |
| Prompt + 128 tokens | CPU (4.95 s of ops) | OpenCL (0.41 s of kernels) |
|---|---|---|
| matrix products | 91.0 % (Q4_K 57.6 %, Q6_K 33.5 %) | 72.5 % (row kernels 54.3 %, tiled 18.2 %) |
| attention | 6.1 % | 12.6 % |
norms, RoPE, ew |
2.8 % | 14.9 % |
Matrix products dominate everywhere; attention matters for prompts and long contexts; the rest matters on OpenCL decode only as launch count.
Model load path
oklm_model_load()(src/model/model.c:1139): GGUF or Hugging Face directory, hyper-parameters, every tensor bound with its type and shape (bind_llama(),bind_gpt2()). Architecture, quantization types and shapes are all known when it returns.oklm_context_new()(model.c:2035): the backend and device placement (model_to_backends():1556), repacking for the CPU (model_pack_cpu():1097), KV cache, activation buffers. The device is known only here: the earliest point where a (device, model shapes) key exists is the start of the first context of a model.
Persistent state
None. OKLM writes no cache or configuration file; every setting is an
argument, a context parameter or an environment variable. A tuning cache
would be the first: $XDG_CACHE_HOME/oklm/tuning/
(~/.cache/oklm/ when unset) is the usual place on
Linux.
Determinism
- CPU: bit-identical from run to run and across thread counts; the
packed and SIMD kernels give the scalar kernels' bits
(
test_matmul), a batch gives the logits of its tokens one by one and attention the same bits at any thread count (test_attention,test_model). - OpenCL: every sum in a fixed order, so runs repeat exactly; contexts
spread over two instances give one device's logits exactly
(
test_multi_device). Different kernels (row vs tiled, split vs not) round differently. - So the correctness gate can be bitwise for CPU work-split knobs and for an OpenCL knob that keeps the kernel and its order (work-group size of an elementwise kernel), and needs a tolerance for knobs that change the kernel or the reduction order (tile thresholds, split factors).
Where the plan does not match the codebase
- The plan's file layout (
src/tuning/,bench/) and generic "KernelConfig" types fit; the languages and build steps it lists do not apply (C17 and make only, no templates or JIT besides OpenCL's own compile). - A harness exists in part:
bench_matmultimes one product per type and SIMD level, and both backends report per-op profiles. - CPU kernels are constrained to the scalar kernels' bits: their search space is the work split, not tile shapes that reorder sums.
- OpenCL kernels are compiled once from one source; variants by
work-group size or tile would be compile-time macros (
-Doptions at build) or more kernels in the source. - The plan's commit message form (
tune: phase N - ...) conflicts with the project's (<module>: <summary>, no phase names); its "ask before changing the public API" applies to the context parameters a tuning switch would need (oklm_context_params).