OKLMdocumentation v1.0.3
On this page

ROCm design

The audit (R0) of rocm_spec.md: what OKLM has for a ROCm/HIP training path and LoRA, what each later phase reuses, extends or creates, and the five decisions the specification leaves to R0. The inventory with its evidence is the "ROCm" section of gap-analysis.md.

Status: audit approved (2026-11-04) with its recommendations; R1 done with rocBLAS in hipBLASLt's place (approved 2026-11-04); R2 done (--backend rocm); LoRA next.

Repository layout

Directory Role
apps/ the oklm command (oklm.c) and the GUI
include/oklm/ public headers, one directory per module (core, io, tok, model, mm, harness, server, client, train)
src/core/ tensors, arenas, dtypes and quantization, the op dispatcher (ops.c, backend.c), the CPU backend (cpu/), the OpenCL backend (opencl/: run-time loader, backend, inference kernels)
src/io/ GGUF and safetensors reading and writing, conversion, quantization
src/tok/ tokenizers (BPE, Unigram, WordPiece) and their trainer
src/model/ inference: architectures, contexts, the paged cache, sampling, grammars, speculative decoding
src/mm/ images, sounds, video and documents
src/train/ training: CPU kernels (layers_train.c, gemm_train.c), the OpenCL device (train_cl.c, train_kernels.cl), the trainer (train.c), checkpoints, reinforcement learning
src/harness/, src/server/, src/client/ the agent, the HTTP server, the client
tests/ one program per module (oklm_test.h), tolerances in tests/tolerances.h, fixtures; benches
tools/ reference generators (llama.cpp, llm.c, Hugging Face), benches
docs/ documentation, ADRs, specifications, design notes

Existing abstractions

Name File Backend-specific? Extendable? Notes
oklm_backend_iface include/oklm/core/backend.h:51-84 no yes buffers, upload / download, supports / compute of an op descriptor, synchronize, streams, memory and free memory; used by inference (src/core/ops.c), not by the trainer
The trainer's device seam src/train/train_internal.h:560-759 OpenCL by name (tr_cl_*) yes one launcher per CPU kernel tr_*, same arguments; train.c picks one or the other per call (k_matmul_forward, train.c:1191-1196)
Device memory as fake addresses src/train/train_cl.c:95-112 (tr_cl_buf), tr_cl_alloc OpenCL yes pointers never dereferenced on the host, so the trainer's arithmetic is the same on both sides; a launcher maps a pointer to (buffer, offset)
The device product src/train/train_cl.c:903 (gemm), train_kernels.cl:167 (tr_gemm) OpenCL yes strides on both operands (sai, sak, sbk, sbj), accumulate flag, batched (gemm_batch, train_cl.c:623-630), the attention's batches mapping query heads to KV heads (pairs, train_cl.c:1105)
The CPU product src/train/gemm_train.c:233 (tr_gemm) no reference F32, multiply then add, scalar / AVX2 / AVX-512 giving the same bits
Run-time library loading src/core/opencl/cl_loader.c:31-35, 68 (oklm_cl_load), cl_loader.h OpenCL the model to copy dlopen / LoadLibrary, its own declarations of the API's types and functions (no system header)
Shared OpenCL handles src/core/opencl/backend_cl.h:19-43 (oklm_cl_handles, oklm_opencl_handles) OpenCL as is the trainer borrows the inference backend's device, context and queue
Errors and logging OKLM_ERR_* (include/oklm/oklm.h), n_log (nilorea) no as is the device seam keeps its first error and returns it from tr_cl_finish()
Test harness tests/oklm_test.h, tests/tolerances.h:87 (TOL_TRAIN_DEVICE 1e-4) no as is finite-difference gradients, device against CPU, llm.c as a test-only oracle

OpenCL path breakdown

  • Generic: the trainer's forward and backward (train.c), its parameter layout and fake-address arithmetic, the CPU kernels, the checkpoints, the test harness.
  • OpenCL-specific, isolated: src/core/opencl/ (inference) and src/train/train_cl.c with train_kernels.cl (training). No OpenCL type or call appears anywhere else (a search for cl_mem, cl_kernel, cl_int, clEnqueue, cl_context, cl_command_queue outside those two places finds nothing). The trainer chooses its device through one pointer, tr->cl (train.c:654-662).
  • Ownership: the trainer owns its device side (tr_cl_new / tr_cl_free), which owns its buffers and kernels; the OpenCL context and queue belong to the inference backend instance and are borrowed.
  • The products: every product of a training step, the attention's included, goes through gemm() in train_cl.c: one function to route elsewhere.

Existing math primitives

  • Products: tr_gemm (CPU, gemm_train.c:233; device, train_kernels.cl:167).
  • Norms (LayerNorm, RMSNorm, per-head RMSNorm), softmax with cross-entropy, attention (causal, grouped KV), GELU, SwiGLU, RoPE (with frequency factors), dropout, AdamW, gradient norm: CPU in layers_train.c, device in train_kernels.cl, declared side by side in train_internal.h.
  • Inference ops (include/oklm/core/ops.h) with CPU and OpenCL kernels, quantized matrix kernels for every GGUF type on both.
  • Layout: row-major throughout (include/oklm/core/tensor.h); a weight is OC rows of C inputs (train_internal.h:16-17). hipBLASLt is column-major: a row-major C = A B is its C^T = B^T A^T, the operands swapped.

Existing training code

  • The trainer: GPT-2 and Llama-family blocks (Qwen2 biases, Qwen3 head norms, grouped KV), losses, AdamW, gradient clipping and accumulation, learning-rate schedules, dropout, evaluation (src/train/train.c, layers_train.c; training.md).
  • Checkpoints with optimizer state and resume (src/train/checkpoint.c, ADR-0008); fine-tuning from Hugging Face checkpoints; reinforcement learning (src/train/rl.c).
  • On OpenCL: the whole step on the device, deterministic (no atomics), within 1e-4 relative of the CPU (tests/train/test_train_cl.c).
  • LoRA or adapters: none. No A / B naming to match. (Added since by R3 to R5: oklm train --lora, training.md.)
  • Activation recompute: none: every activation of the step is kept (train.c:538-574 sizes them). (Added since, on 2026-11-04: oklm train --recompute, training.md.)

Build system

  • No USE_* switches. OpenCL is loaded at run time by cl_loader.c (Makefile:239), so one binary runs with or without it; the backend is chosen at run time (--backend, the trainer's backend option, include/oklm/train/train.h:71).
  • Optional dependencies found by pkg-config (Allegro, OpenSSL, Ogg); -ldl linked (Makefile:159); kernels embedded as generated includes (Makefile:525-530).
  • MinGW builds the same sources for Windows; ROCm has no part there.

Baseline (task 8)

OKLM_CL_PROFILE=1 (each launch waited for and timed), gfx1100, 2026-11-04:

Run Step Products (tr_gemm) Their share Products' rate
Llama Nano, 4.95M, 16 x 256 47 ms (profiled) 3052 launches in 20 steps, 38.6 ms a step 69 % about 3.6 TFLOP/s
SmolLM2-135M fine-tune, 4 x 4 x 128 528 ms 18808 launches in 6 steps, 429 ms a step 69.5 % about 3.8 TFLOP/s (6 x 2048 x 134.5M FLOP a step)

About 150 to 800 products a micro-batch, many small (the attention's per-head batches). R1 measures hipBLASLt on the same shapes; a product must then win more than what handing it over costs (below, decision 3).

Reuse / extend / create decision table

Component Decision Reason
Backend abstraction reuse the trainer's device seam, not oklm_backend_iface the trainer never uses the backend interface; its seam already pairs every CPU kernel with a device launcher, and the products have one entry, gemm()
Device memory API reuse the fake-address allocator; extend it to shared virtual memory only if R1 shows HIP can read it the trainer's arithmetic depends on it; a second allocator would split parameters and activations between two runtimes
GEMM wrapper extend gemm() in train_cl.c: hipBLASLt for the products it wins, the OpenCL kernel for the rest (the attention's head-mapped batches, small products) one function, strides already explicit; hipBLASLt's strided batches cannot express the query-to-KV head mapping
Numerical test harness reuse oklm_test.h, TOL_TRAIN_DEVICE, the device-against-CPU pattern of test_train_cl.c it already holds devices to the CPU
LoRA (R3, R4, R7) create, in the trainer first (CPU reference, then the OpenCL device), independent of HIP OKLM has none; the plan's GEMM composition is plain products that both existing devices run

The five decisions of "Before starting (OKLM)":

  1. How HIP is linked: loaded at run time. hipblaslt.h cannot be compiled as C: it includes <memory>, <regex> and <vector> unconditionally (/opt/rocm/include/hipblaslt/hipblaslt.h:54-56), which the plan's constraint 7 says to report. Its functions are exported as C symbols (nm -D libhipblaslt.so: hipblasLtCreate, hipblasLtMatmul, hipblasLtMatmulAlgoGetHeuristic, ... unmangled), so a loader like cl_loader.c with OKLM's own declarations of the few functions, opaque handles and enum values needed reaches them without the header. hip_runtime_api.h compiles as C with __HIP_PLATFORM_AMD__, but the loader declares its handful of HIP functions too, so no ROCm header is needed to build and one binary still runs without ROCm (and MinGW is untouched). The enum values are checked numerically by R1's test. rocblas.h does compile as C, a fallback needing your approval (the plan's R1 pitfall).
  2. Where it sits: inside the trainer's OpenCL device, as the engine of its products, not a new oklm_backend_iface instance. A HIP backend for inference would need every inference op, which HIP cannot provide in C (no device kernels without HIP C++, no HIPRTC).
  3. The other kernels stay on OpenCL, and only products move. That holds only if the two runtimes share memory: ROCm's OpenCL and HIP sit on the same HSA runtime, so a buffer allocated as OpenCL shared virtual memory (clSVMAlloc) may be a pointer hipBLASLt can read on the same device; R1 tests it first. If it is not, each product would copy its operands both ways, and the plan stops at R1 (LoRA through R7 on the existing devices). Each hand-over also needs the OpenCL queue drained before the HIP stream starts and the reverse (no shared events): only products large enough to pay for two waits move.
  4. LoRA's conventions: the paper's naming (A down to the rank, B up, B zero at start, scaling = alpha / rank). Targets chosen by name (q, k, v, o, gate, up, down; default q, k, v, o). Files in llama.cpp's GGUF adapter format, so llama.cpp reads them too: general.type = adapter, adapter.type = lora, adapter.lora.alpha, tensors <base name>.lora_a and .lora_b (build/llama.cpp/src/llama-adapter.cpp:205-207, 276-284, 374 for the checks it makes); the optimizer state for resuming in OKLM's checkpoint sidecar.
  5. The 27B target is out of this plan's reach. The trainer keeps every weight, gradient, moment and activation in F32: 27B is 108 GB of weights alone. A frozen base would have to stay quantized (Q4_K about 16 GB of the 20), its products dequantized by tiles, the backward pass through it a transposed quantized product OKLM does not have (dX = dY W with W quantized), activations recomputed (none today) and possibly layers kept in host memory. R7 should stop at a frozen F32 or F16 base (SmolLM2-135M, Qwen3-0.6B, up to about 4B); the 27B path is a milestone of its own.

Risks

  1. Hand-over cost. Products are many and small (150 to 800 a micro-batch); waiting on the OpenCL queue and the HIP stream for each (tens of microseconds each) can cost more than hipBLASLt saves on small shapes. Mitigation: route only large products, measured in R1.
  2. Memory sharing between the runtimes is not documented by AMD; if clSVMAlloc memory is not a valid HIP device pointer, decision 3 fails and R2 to R6 do not follow.
  3. Our own declarations of hipBLASLt's enums drift with ROCm releases: the loader checks hipblasLtGetVersion against the version it was written for and refuses others; R1's test catches a wrong value as wrong numbers.
  4. Determinism. The trainer gives the same result on every run; hipBLASLt's algorithms may split K with atomic adds. The heuristic's choice must be restricted (or measured) to keep runs reproducible.
  5. FP32 algorithms on gfx1100: the heuristic may return none for some shapes or transposes (the plan's R1 pitfall); the OpenCL kernel stays the fallback for those.
  6. Column-major operand swaps and leading dimensions: silent garbage when wrong; R1's test compares every transpose combination the trainer uses against tr_gemm.
  7. Numerics: hipBLASLt fuses multiply and add where OKLM's kernels do not; results differ from the CPU in the last bits, within TOL_TRAIN_DEVICE.
  1. R1 first tests memory sharing, then times hipBLASLt against the OpenCL kernel on the baseline's shapes, the hand-over cost included. Only if both pass (sharing works, products clearly faster after the waits) do R2 to R6 follow, as the routing inside gemm() above rather than a new backend.
  2. LoRA does not wait for HIP. R3 and R4's GEMM compositions are products both existing devices run; R7 (the CPU reference, then the OpenCL device, files, oklm train --lora, oklm run --lora) can start right after R1 whatever R1 decides, and R3 to R5's checks become the trainer's own gradient tests.
  3. The 27B path is split out of R7 into a milestone of its own (frozen quantized base, transposed quantized products, activation recompute, host offload), after LoRA works on small models.
  4. No HIP header in the build (decision 1): a run-time loader, so the C-only rule holds without a C++ header and one binary serves machines with and without ROCm.

R1: measurements

Run outside the build (throwaway programs, 2026-11-04), as R1's gate before any code: hipBLASLt reached through OKLM's own declarations (the header being C++), the trainer's OpenCL launchers called through its internal header, rocBLAS for comparison only.

  • Declarations and numerics: hipBLASLt 1.2.2 (hipblasLtGetVersion 100202) through our own C declarations gives every transpose the trainer uses within 7e-6 of a CPU product (77 x 53 x 129).

  • Memory sharing works: a buffer from clSVMAlloc is a device pointer to HIP (hipPointerGetAttributes: type device, device 0), and a hipBLASLt product on such pointers is correct (1e-5); an idle hand-over (clFinish then hipStreamSynchronize) costs 2.2 microseconds.

  • hipBLASLt is not faster than the OpenCL kernel. Its heuristic offers 4 algorithms for FP32 on gfx1100; the best of them reaches 4.4 to 7.4 TFLOP/s on the trainer's shapes, about the OpenCL kernel's speed and sometimes below it (the Nano's output projection, 4096 x 2048 x 256: 790 microseconds against 419).

  • rocBLAS is 1.5 to 3.8 times faster (rocblas_sgemm, its header compiles as C), the trainer's products as it issues them, microseconds:

    Layer (rows x inputs x outputs) Forward, OpenCL / rocBLAS Backward (input and weight gradients), OpenCL / rocBLAS
    SmolLM2 Q/K/V, 512 x 576 x 960 179 / 95 343 / 212
    SmolLM2 MLP up, 512 x 576 x 3072 393 / 103 745 / 223
    SmolLM2 MLP down, 512 x 1536 x 576 178 / 117 264 / 120
    SmolLM2 output, 512 x 576 x 49152 2971 / 1142 10393 / 6215
    Nano Q/K/V, 4096 x 256 x 512 123 / 58 647 / 283
    Nano MLP up, 4096 x 256 x 1408 310 / 130 890 / 379
    Nano output, 4096 x 256 x 2048 419 / 180 1189 / 473

    For SmolLM2-135M this is about 305 ms of products a step against 135 ms (the attention's per-head products staying on OpenCL), so a step of 528 ms near 360 (about 1.45 times), before the hand-overs.

By R1's rule hipBLASLt stopped there; rocBLAS, the plan's fallback, was approved in its place (2026-11-04) and takes it everywhere in the design above: run-time loading with our own declarations, products routed from the trainer's OpenCL device on shared memory, the rest unchanged.

In the tree (R1's deliverables):

  • src/train/hip_loader.c: the HIP runtime and rocBLAS opened at run time (OKLM_HIP_LIB, OKLM_ROCBLAS_LIB to name them, empty to disable; nothing on Windows, nor in an ASan/TSan build unless named), the entry points and values declared there.
  • src/train/gemm_rocblas.c: tr_rb_gemm() takes tr_gemm()'s operands (row-major, either operand read transposed through its strides) and calls rocblas_sgemm with the operands swapped; HIP device OKLM_HIP_DEVICE (0).
  • tests/train/test_rocblas.c: every operand layout, accumulation, padded rows and sizes of 1 against tr_gemm() within 7e-6 (PASSED); skipped without ROCm.
  • tests/train/bench_rocblas.c: the table above through the loader (1.7 to 3.5 times: SmolLM2's Q/K/V 1.7, MLP up 3.5, MLP down 1.8, output 1.7; the Nano's 2.2 to 2.5).

R2: the trainer's products through rocBLAS

--backend rocm (the trainer's backend option): the first OpenCL device with its large single products through rocBLAS. The files R2 changed, as approved: src/core/opencl/cl_loader.c (clSVMAlloc, clSVMFree resolved when present; the inference backend untouched), src/train/train_cl.c, src/train/train.c, include/oklm/train/train.h, apps/oklm.c, tests/train/test_train_cl.c.

  • Memory: each trainer buffer is shared virtual memory wrapped in a buffer (CL_MEM_USE_HOST_PTR), so every OpenCL launcher is unchanged and gemm() hands rocBLAS the shared address of the same bytes (checked first outside the tree: OpenCL writes, rocBLAS computes, OpenCL reads the product back within 2e-6). Coarse-grained SVM is device memory committed when allocated, where the opencl backend's plain buffers may overflow into host memory; once the device is full (or past OKLM_ROCM_DEVICE_MB), a buffer is allocated as fine-grained SVM, which this driver places in host memory the device and rocBLAS read over the bus (2026-11-08: Qwen3-8B LoRA at 6144 tokens, the activations 17.1 GB then the 5.0 GB packed base, which went to the host; see training.md).
  • Routing: a single product of at least 2^22 multiply-adds (OKLM_ROCBLAS_MIN) goes to rocBLAS; the attention's batched products, whose batches map query heads to KV heads, stay on the OpenCL kernel.
  • Hand-overs: switching from OpenCL to rocBLAS drains the OpenCL queue, and back the HIP stream, only when the side left has work queued; consecutive products do not wait for each other.
  • Checks: every test of tests/train/test_train_cl.c runs again on rocm with every single product through rocBLAS: the forward and backward pass against the CPU (logits and gradients within 1e-6), five AdamW steps (within 2.3e-5), the same bits on two runs, dropout, weighted rows, resuming on the CPU.
  • Speed: the Llama Nano 38 to 26.5 ms a step, SmolLM2-135M 390 to 310 ms (training.md); the threshold makes no difference on these models (every single product is large).

16-bit product inputs (2026-11-08)

--matmul-precision bf16|f16 (matmul_type): tr_to16 rounds both operands of a product rocBLAS computes (at least 128 in every size) into one scratch buffer, kept in their layouts so the transpose flags and leading dimensions are those of the F32 call (the kernel walks a transposed operand along its contiguous dimension); rocblas_gemm_ex then writes a 16-bit product with F32 accumulation, which tr_from16 widens into, or adds to, the F32 output. rocBLAS 5 on gfx1100 has tuned kernels only for 16-bit outputs: bf16_r inputs with an f32_r output measured 1.5 to 2 times slower than sgemm and 12 to 20 times with the second operand transposed, a bf16_r output 2.5 to 4 times faster than sgemm (shapes of SmolLM2-135M's layers and head and 4096 x 4096). A BF16 base under --quantized-base is read in place: tr_cl_dequantize() records the BF16 rows it unpacked as they lie, and an operand within them points rocBLAS at the rows themselves. Measured in training.md: BF16 follows F32's validation curve to the third decimal, F16 alone does not, Llama-3.2-1B's LoRA step 350 to 227 ms. With --loss-scale dynamic (loss_scale: the gradients of the loss times a scale, halved on an overflow with the update skipped, doubled after 200 finite updates) F16 follows F32 too (2026-11-08).

The weights' 16-bit copy (2026-11-08): tr_cl_set_weights16() names the parameter buffer, whose rounding train_cl.c keeps in a second buffer of 2 bytes a parameter, element for element, with a flag per 65 536 floats. An operand of gemm16() within the parameters points rocBLAS at the copy, after tr_to16 has rounded the stale chunks it spans; tr_adamw and tr_adamw16 write the rounding of each parameter they update (w16_store, the rounding tr_to16 does) and mark the chunks they cover whole current; an upload marks its chunks stale. Each weight is thus rounded once per update instead of at each of its products (forward, input gradient, recomputed forward, each accumulated batch, evaluations), and a LoRA run's frozen base once for the run. The copy is allocated on the first 16-bit product (after the activations), in coarse-grained memory only and when 512 MiB stay free after it (TR_CL_W16_RESERVE), else not at all; OKLM_ROCM_W16_CACHE=0 turns it off. Under --quantized-base the frozen matrices are not in the parameter buffer: each is unpacked and rounded before its product as before (a BF16 base still read in place), and no copy is made. The scratch also remembers the left operand it holds rounded (address and view): the next product reading the same elements, as a weight's gradient reads the output gradient its input gradient's product has just rounded, skips the rounding; any other kernel, an upload, an F32 product or a 16-bit output over those elements forgets it. The bits are those of rounding every time (the same rounding of the same values), checked by test_train_cl and by the checkpoints of 20-step runs (training.md).

R3 to R5 and R7: LoRA

LoRA in the trainer, on the CPU reference, the OpenCL device and rocm alike, as recommended (deviation 2); the decisions in ADR-0020, the use in training.md.

  • R3, forward: h = s (x A^T), then y += h B^T after the Q/K/V (or packed QKV), output, gate/up and down products, one adapter per weight of the model file on its columns of the packed product (src/train/train.c: lora_forward).
  • R4, backward: dB += dy^T h, dh = s (dy B), dA += dh^T x, dx += dh A (lora_backward); the model's weight gradients skipped (the products' dw, the norms' and the head norms' parameter passes, the embeddings' backward), the input gradients kept.
  • R5, validation: every adapter of the five tiny architectures against finite differences, every target on; the model's parameters and gradients untouched after steps; a fresh adapter giving the model's loss to the bit (tests/train/test_train.c, test_lora); one pass and five steps on opencl and on rocm (every single product through rocBLAS) against the CPU (tests/train/test_train_cl.c); the device's split products against a double-precision reference (test_skinny).
  • R7, end to end: adapter files in llama.cpp's format with the checkpoint's sidecar (src/train/checkpoint.c, oklm_trainer_load_lora(); resuming bit-identical, another rank refused); oklm train --lora R --lora-alpha A --lora-targets LIST, -m <model> --resume <adapter>; oklm_lora_merge() (src/io/lora.c), oklm lora-merge, --lora on run, chat, ppl and serve (test_lora_merge: the merged model's loss is the adapted trainer's). llama.cpp reads the adapters: its perplexity with --lora matches it on OKLM's merged model (27.3985 and 27.3989, SmolLM2-135M).
  • Device products: LoRA's rank-wide products have one to sixteen output tiles and an inner size of every row of the batch; split along the inner size into a batch of parts (about 64 work groups in all) summed in order, and the scale taken as the product's factor, a step for every projection of SmolLM2-135M went from 197 to 140 ms on opencl and 125 to 102 ms on rocm (a full fine-tune: 156 and 110 ms). Adjacent tensors of the same weight decay are updated in one AdamW launch.
  • Not done: the 27B path (a quantized frozen model, activations recomputed) stays split out, as decided at R0. Since then (2026-11-04): activation recompute, the head in chunks, and the blocks' matrices kept in their file's types (--quantized-base, unpacked by tiles of one matrix before each product rather than a transposed quantized kernel; training.md).

R6: performance baseline

Measured 2026-11-04 (training.md, benchmarks.md).

  • Solutions (task 1): hipBLASLt was replaced by rocBLAS at R1, so rocBLAS's choice was checked instead: bench_rocblas --solutions times its default against every solution rocblas_gemm_ex_get_solutions lists for each product (both loaded at run time, optional). The default is within 5 % on most products and gives up at most 3 % of a step on the others (SmolLM2's MLP down forward, the Nano's Q/K/V weight gradient); nothing is hardcoded, as solution indices belong to one rocBLAS build. rocBLAS 4.4 aborts on some listed solutions of a 49 152-column output head, which the bench skips.
  • Steps (tasks 2, 3): full fine-tune against LoRA (rank 16, Q, K, V, O), 4 x 256 tokens, on opencl and rocm: SmolLM2-135M, Qwen2.5-0.5B, Qwen3-0.6B and Llama-3.2-1B. LoRA is 1.3 to 1.6 times faster than a full step, rocm 1.3 to 2 times faster than opencl; Llama-3.2-1B only trains with LoRA (344 ms a step on rocm): a full fine-tune needs 16 bytes a parameter.
  • Memory: the gradients and AdamW moments now hold only the trainable floats (train_off): with LoRA the model has neither, its gradient pointers NULL. Before, every LoRA run allocated three model-sized buffers it never used.
  • Profile: the Llama-3.2-1B LoRA step on rocm is 79 % rocBLAS products, 10 % norms, nothing else above 2 %: no hotspot for a custom kernel, so the plan's decision point (a .hip unit) is not reached.