OKLMdocumentation v1.0.3
On this page

Kernel tuning: codebase map

The survey that kernel-autotune-plan.md asks for before any tuning code: how OKLM is built, which kernels run on the forward path, where they are chosen, where a model's shapes become known, and how deterministic the results are. Line numbers are those of commit b1c1266 (2026-10-31).

Status: maintained (the survey before the tuner; see 02-parameterization.md for what it searches).

Build, run, test

  • C17, GNU make, no other build system. make -j16 builds the library, build/release/bin/oklm and the tests; make tests builds the tests only (not the CLI).
  • Run: build/release/bin/oklm run -m <model.gguf> -p "<prompt>" -n 128 --temp 0 --backend cpu|opencl.
  • Tests: make test (24 test binaries), make check, make doc (Doxygen, 0 warnings), make docs-lint; sanitizers once a day (make asan-test, make tsan-test, make asan-opencl), make wine-test.
  • Benchmarks exist already: tests/core/bench_matmul.c (one matrix product, any type, SIMD level and thread count, [K] [N] [threads] [type] [rows] [pack]), tests/tok/bench_tokenizer.c, and the end-to-end timings oklm run prints (prompt and generation t/s).

Backends

  • CPU (src/core/cpu/): scalar, AVX2 and AVX-512 VNNI kernels, chosen at run time (OKLM_SIMD, oklm_cpu_set_simd()); one polling worker team (OKLM_THREADS, default one thread per physical core).
  • OpenCL (src/core/opencl/): one backend instance per device, kernels in kernels.cl built at first use (OpenCL C 1.2).
  • Selection: oklm_backend_select() (src/core/backend.c), from oklm_context_params.backend or OKLM_BACKEND; a context puts its last layers and the head on the device, the rest on the CPU.

Dispatch

Every op goes through oklm_op_run() (src/core/ops.c:55), which hands the descriptor to the backend owning the destination: cpu_compute_op() (src/core/cpu/backend_cpu.c:110) or cl_compute() (src/core/opencl/backend_cl.c:2508), each a switch over the op. Inside, the kernel variant is chosen per call from the shapes, the types and a few thresholds; there is no dispatch table and no per-model state, so a tuner can only steer these choices through new state the backends read.

Kernels of the forward path

CPU

Kernel Where Prompt / decode Knobs today
matrix product, file layout oklm_cpu_matmul() src/core/cpu/matmul_cpu.c:559, dot products per type in simd_avx2.c, simd_avx512.c, simd_avx2_iq.c decode (1 row), and prompts for unpacked types oklm_cpu_tuning.matmul_rows 16 output rows per task (cpu.h:111), one even share per thread below FEW_ROWS rows
matrix product, repacked weights gemm_pack.inc (e.g. gemm_q4k_x8 :51, gemm_iqg_x8 :861), packed by oklm_cpu_pack_weight() pack_cpu.c:440 prompts (2 rows or more) rows in groups of 8; 4 activation rows per call; OKLM_CPU_PACK=0 turns packing off
attention oklm_cpu_attention() attention_cpu.c:689: tiles (:397), split path (:581), pairs (:97) both ATTN_TILE_ROWS 32 (attention_cpu.c:214), oklm_cpu_tuning.attn_keys 64 (cpu.h:114); runs of heads chosen from the group size
RMSNorm, RoPE, SwiGLU, copies, get_rows ops_cpu.c both rows per task
MoE routing moe_cpu.c both none

Results must stay bit-identical to the scalar kernels (packed and SIMD kernels) and independent of the thread count (attention): any tuned CPU variant has to keep the same order of every sum, so the knobs that are safe are the work split (rows per task, tile sizes that do not change a row's order) rather than the arithmetic.

OpenCL

Kernel Where Prompt / decode Knobs today
mm_<type> row kernels kernels.cl:1887 (MM_KERNEL), launched by run_matmul() backend_cl.c:1646 decode, small prompts WG_MM 64, MM_ROWS 4, MM_COLS 8 (backend_cl.c:137)
gm_<type> tiled kernels, gm_reduce kernels.cl:2158 (GM_KERNEL) prompts GM_TILES 256 (OKLM_CL_GEMM_TILES), split of K up to GM_SPLIT 8 (OKLM_CL_GEMM_SPLIT), tile 64 x 64, K step 32
attention, attention_merge kernels.cl:769, run_attention() backend_cl.c:1089 decode, short prompts, Gemma WG_ATT 256, key split for few pairs (4 work-groups per compute unit, ranges of 128 keys or more)
attention_tile64/128/256 kernels.cl:940 (AT_KERNEL) prompts OKLM_CL_ATT_BLOCKS, OKLM_CL_ATT_KEYS (1024), 32 / 16 queries per block, key split for few blocks (aimed at 4 x the blocks that fill the device)
norm, rope, ew, quantize_rows, hadamard, MoE kernels kernels.cl both WG_NORM 256

The OpenCL thresholds are already environment knobs, set by hand from measurements on one gfx1100 (42 compute units): they are what a tuner would search first, per device.

Hot path

Llama-3.2-1B Q4_K_M, a 510-token prompt and 128 generated tokens, greedy, OKLM_CPU_PROFILE=1 / OKLM_CL_PROFILE=1 (OpenCL profiling waits after every launch, so its decode figures are dominated by launch overhead: 1.1 s of host time for 0.41 s of kernel time, against about 4 ms per token unprofiled).

Prompt (510 tokens) CPU (1.00 s of ops) OpenCL (95 ms of kernels)
matrix products 78.7 % (Q4_K 65 %, Q6_K 14 %) 82.3 % (gm_q4_k 65.3 %, gm_q6_k 13.5 %, row kernels 3.3 %)
attention 16.4 % 13.0 %
SwiGLU / ew 3.0 % 3.3 %
RoPE, norms, copies 1.9 % 1.5 %
Prompt + 128 tokens CPU (4.95 s of ops) OpenCL (0.41 s of kernels)
matrix products 91.0 % (Q4_K 57.6 %, Q6_K 33.5 %) 72.5 % (row kernels 54.3 %, tiled 18.2 %)
attention 6.1 % 12.6 %
norms, RoPE, ew 2.8 % 14.9 %

Matrix products dominate everywhere; attention matters for prompts and long contexts; the rest matters on OpenCL decode only as launch count.

Model load path

  • oklm_model_load() (src/model/model.c:1139): GGUF or Hugging Face directory, hyper-parameters, every tensor bound with its type and shape (bind_llama(), bind_gpt2()). Architecture, quantization types and shapes are all known when it returns.
  • oklm_context_new() (model.c:2035): the backend and device placement (model_to_backends() :1556), repacking for the CPU (model_pack_cpu() :1097), KV cache, activation buffers. The device is known only here: the earliest point where a (device, model shapes) key exists is the start of the first context of a model.

Persistent state

None. OKLM writes no cache or configuration file; every setting is an argument, a context parameter or an environment variable. A tuning cache would be the first: $XDG_CACHE_HOME/oklm/tuning/ (~/.cache/oklm/ when unset) is the usual place on Linux.

Determinism

  • CPU: bit-identical from run to run and across thread counts; the packed and SIMD kernels give the scalar kernels' bits (test_matmul), a batch gives the logits of its tokens one by one and attention the same bits at any thread count (test_attention, test_model).
  • OpenCL: every sum in a fixed order, so runs repeat exactly; contexts spread over two instances give one device's logits exactly (test_multi_device). Different kernels (row vs tiled, split vs not) round differently.
  • So the correctness gate can be bitwise for CPU work-split knobs and for an OpenCL knob that keeps the kernel and its order (work-group size of an elementwise kernel), and needs a tolerance for knobs that change the kernel or the reduction order (tile thresholds, split factors).

Where the plan does not match the codebase

  • The plan's file layout (src/tuning/, bench/) and generic "KernelConfig" types fit; the languages and build steps it lists do not apply (C17 and make only, no templates or JIT besides OpenCL's own compile).
  • A harness exists in part: bench_matmul times one product per type and SIMD level, and both backends report per-op profiles.
  • CPU kernels are constrained to the scalar kernels' bits: their search space is the work split, not tile shapes that reorder sums.
  • OpenCL kernels are compiled once from one source; variants by work-group size or tile would be compile-time macros (-D options at build) or more kernels in the source.
  • The plan's commit message form (tune: phase N - ...) conflicts with the project's (<module>: <summary>, no phase names); its "ask before changing the public API" applies to the context parameters a tuning switch would need (oklm_context_params).