OKLMdocumentation v1.0.3
On this page

OpenCL backend

How OKLM runs inference on a GPU through OpenCL: how the runtime is found, how a device is chosen, where tensors live, the kernels, the fallback to the CPU, how results compare with the CPU and llama.cpp, and the measured speed.

Status: maintained.

Using it

oklm devices                                  # lists the OpenCL device when one is found
oklm run -m model.gguf -p "Hello"             # auto: OpenCL when available, else the CPU
oklm run -m model.gguf -p "Hello" --backend cpu
OKLM_BACKEND=opencl oklm chat -m model.gguf

From C, a context runs on the backend named in its parameters:

oklm_context_params cp = oklm_context_default_params();   /* backend "cpu" */
cp.backend = "opencl";                                    /* or "auto", or NULL for $OKLM_BACKEND */
oklm_context_new(model, &cp, &ctx);

The library's default stays the CPU, so programs and tests that do not ask get the same numbers on every machine; the oklm commands ask for auto.

OKLM_BACKEND picks the backend when a program does not, and OKLM_CL_DEVICES / OKLM_CL_DEVICE the device(s). Without them every discrete GPU is used; an integrated GPU (one sharing the host's memory, as CL_DEVICE_HOST_UNIFIED_MEMORY says, such as a Core i7's UHD 750 beside a Radeon) only when there is no discrete one, since splitting the layers onto it slows the discrete GPU down; the log marks it integrated. On a gfx11 device the prompt products of the common quantized types run on its 8-bit matrix units by default, giving the CPU's products bit for bit (wq_<type> below; OKLM_CL_Q8=0 for the F32 kernels), and so do the decode products of the K-quants (mq_<type>, OKLM_CL_Q8_MV). The prompt products of the quantized types that have no 8-bit kernel (the grid IQ types, Q2_K, Q3_K, Q4_1, Q5_1, IQ4_NL, IQ4_XS, MXFP4, NVFP4, the ternary and low-bit types) run on its float matrix units by default, their activations rounded to F16 as Vulkan's cooperative matrices take them (w2_<type>, wm_<type>, wi_<type> below; OKLM_CL_WMMA=0 for the F32 kernels; Types without an 8-bit kernel). OKLM_CL_WMMA (1 or f16, bf16) runs every prompt product on the F16 / BF16 matrix units instead (opt-in: not the CPU's bits for the common types); its decode products still take mq_<type> (Qwen3-8B Q4_K_M decoding with OKLM_CL_WMMA=1 84 to 94 t/s; its greedy tokens are the default path's up to a near tie: the ninth of 16 after a 193-token prompt, the seventh of 64 after a 497-token one, 2026-11-12). The kernels' thresholds (OKLM_CL_GEMM_*, OKLM_CL_ATT_*) and OKLM_CL_PROFILE are described with every other variable in build.md.

Runtime loading

Nothing of OpenCL is needed to build OKLM: src/core/opencl/cl_loader.h declares the part of the OpenCL 1.2 API the backend uses, and cl_loader.c loads the ICD loader with dlopen (LoadLibrary on Windows) and resolves 27 entry points. The backend is probed the first time opencl or auto is asked for (runtime loaded, devices listed, context and queue created, kernels compiled), so programs that never ask pay nothing, and a binary runs unchanged on a machine without OpenCL, where auto then selects the CPU.

Where tensors live

  • Device memory is handed out as ranges of fake addresses (never dereferenced on the host). Tensor views keep their pointer arithmetic, and the backend translates data to a buffer and an offset when it launches a kernel. Every kernel takes tensors as buffer, offset, sizes and byte strides, like oklm_tensor, so strided and permuted views need no copy.
  • Weights: copied once per model into one device buffer when the first OpenCL context of the model is created. Token and position embeddings stay in host memory: they are only looked up, one row per token.
  • Each context allocates its KV cache and activation buffers on the device. Per step, only the positions go up and the logits come back; token ids stay on the host for the embedding lookup.
  • --gpu-layers N (oklm_context_params.n_gpu_layers, alias -ngl): only the last N layers and the output head go to the device, as with llama.cpp's -ngl; the first layers run on the CPU with their KV cache in host memory, and the residual stream is uploaded once per step between the two parts. The head, often the largest matrix of a small model, always gets the device. -1 (default) puts there as many of the last layers as fit in the device memory (all of them when the model fits): the head, the layers' weights and KV cache, the activation buffers and, with --gpu-stream, the two streaming slots and the cache mirrors are counted against the memory minus 8 % (at least 256 MiB) for the driver and other programs, and weights and caches must each fit the largest single buffer; a notice says how many fit. OKLM_GPU_MEM_MB replaces the device's size (to test a smaller device). One placement per model: a second context must use the same N.
  • --n-cpu-moe N|all (oklm_context_params.n_cpu_moe, as llama.cpp's flag): the routed experts (ffn_gate_exps, ffn_up_exps, ffn_down_exps) of the first N layers stay in host memory and run on the CPU, while the rest of those layers (attention, norms, router, shared experts) goes to the device like any device layer. Experts are most of a MoE model's weights but each token reads a few of them, so they are what belongs in host memory when the model is larger than the device; with --gpu-layers left out the fit counts the memory they leave free. Such a layer crosses devices once each way: after the router (on the device) the chosen experts, their weights and the normalized rows are read back, the shared expert is queued and started on the device, the CPU computes the routed experts (grouped indexed products, through --expert-cache when it is on) and their weighted sum, which is written back and added to the shared expert's output. See moe.md.
  • --expert-gpu GB|auto (oklm_context_params.expert_gpu): a device expert tier for those layers (every layer's experts in host memory when given without --n-cpu-moe). Each layer keeps device stacks of expert slots (an equal share of GB; auto: the free device memory once the context is placed, less the fit's margin) holding its most routed experts by the usage file's counts and the routing seen, refilled every 32 steps. After the router the chosen experts are mapped to slots (a slot of zeros for the others) and uploaded; the device's indexed products over the slots (one product per slot for batches with more pairs than experts) and their weighted sum run while the CPU computes the other pairs, whose sum is written up and added. A layer holding all its experts runs as a device layer. See moe.md.
  • --gpu-stream B (oklm_context_params.stream_batch): with --gpu-layers below the layer count, batches of at least B tokens (prompts, perplexity windows) run the CPU layers on the device too. Their weights go through two device slots sized for the largest of them: layer l + 1 is copied on a second, transfer queue while layer l computes from the other slot (events order the copies after the last use of a slot and the kernels after their copies; OpenCL 1.1 drivers copy on the compute queue). Those layers' caches are mirrored on the device: rows written by a streamed batch are copied back to the host, rows written on the CPU are copied up before the next streamed batch. Shorter batches (decode) keep those layers on the CPU, where they are not limited by the bus. Perplexity of Llama-3.2-1B Q4_K_M over 60 kB, 512-token windows: 14.9 s with every layer on the device, 129 s with 8 of 16 layers there and 16.4 s with the other 8 streamed, 211 s / 16.5 s with 1 layer there; the logits are those of the all-device run.

Kernels

All in src/core/opencl/kernels.cl (OpenCL C 1.2, built with -cl-std=CL1.2), embedded in the library at build time.

Kernel Ops Notes
ew add, mul, scale, silu, sigmoid, gelu, gelu_erf, swiglu, geglu, sigglu, softcap, cpy any 4-D strides, broadcast of the second operand; F32 / F16 / BF16 elements, conversions round to nearest even like the CPU
norm rms_norm, layer_norm one work-group of 256 per row, optional weight and bias; rows of 2048 values or more, a multiple of 256, summed in runs of eight values an item (the MQ kernels' layout, so a norm folded into them gives the same bits), the others one value an item in turn
rope rope normal and NeoX layouts, partial rotation, frequency factors (with one position a token or three: oklm_rope_parts_factors()), linear and YaRN scaling; angles built by the same repeated products as the CPU; one position a token, or three with a band of pairs each (oklm_rope_parts(), M-RoPE; the fused norm_rope and rope_kv launches take one)
attention_tile64, attention_tile128, attention_tile256 attention prompts (16 rows or more, a row a query of one head) with heads of at most 256 dimensions: a work-group of 128 items per block of 32 rows (16 for heads over 128), the rows the queries of the query heads that read one KV head (a divisor of the group: Qwen3-4B's 4 heads take 8 queries a block, so a 4-token piece fills half a block and each key is read once for its 4 heads), walks the keys the block can see in tiles of 32 (16), each key tile (transposed) and value tile loaded once into local memory for the whole block; the items of a query each score a few keys of the tile, then update the query's online softmax and their share of its output dimensions; causal window and softcap as below. F16 tiles loaded eight halves at a time, block-quant caches dequantized 32 elements at a time. Used by default for heads up to 128, with a block-quant cache or from 128 keys (OKLM_CL_ATT_KEYS) with a float cache: the kernel below measured faster for heads of 256 (Gemma's), and for spans under 128 (the threshold was 1024 until 2026-11-08, when 512-token prompts measured 1.1 to 1.25 times faster on the tiles: Llama-3.2-1B 11 786 to 14 787 t/s, Qwen3-8B Q4_K_M 2 262 to 2 641, with the matrix units). With fewer blocks than keep the device busy (2 per compute unit, 4 for heads over 128), the keys of each block are split into up to 16 ranges of at least 4 tiles, aimed at four times that many work-groups, each writing its queries' running max, sum and unscaled output for attention_merge. Block-quant caches measured no faster on these kernels and keep the one below. Queries that see each other (mutual: an image's rows in Gemma 3) take these kernels too, and attention_tile64h / 128h / attention_wm64h / 128h below: a last key is given (last, -1 for causal rows), every block walks the keys up to it and no key before it is masked for coming after a query, the window still starting from each query's own position; under the same thresholds, so Gemma 3 27B's heads of 128 are on the tiles and the 4B's and 12B's heads of 256 are not: measured on 2026-11-14 (RX 7900 XT, OKLM_CL_PROFILE=1, Gemma 3 4B with llama.cpp's test image: 256 rows over 261 keys, 8 heads of 256 on 4, in 34 layers, and the 10 rows of text after them) 20.0 ms on the kernel below, 25.6 to 35.5 ms on attention_tile256 (OKLM_CL_ATT_BLOCKS 168, 64 and 1). The output accumulators are indexed by compile-time bounds, so they stay in registers
attention_tile64h, attention_tile128h attention the same for F16 caches (heads a multiple of 8, up to 128): the key and value tiles kept as halves in local memory (their bits, read through vload_half: no fp16 extension needed), about two thirds of the float kernels' local memory, so more work-groups share a compute unit; the key tile written by consecutive items at consecutive keys (no bank conflicts), the next tile loaded into registers while one is used. Qwen3-4B at 16k keys: 512 prompt tokens in 1584 ms against 3380, a 16-token piece in 62 ms against 215
attention_wm64h, attention_wm128h attention with OKLM_CL_WMMA (unless OKLM_CL_WMMA_ATT=0), in place of attention_tile64h / attention_tile128h (heads a multiple of 16, up to 128; the same arguments, rows, key tiles of 32, masks, split of the keys and attention_merge): both products on the matrix units as 16 x 16 x 16 blocks of v_wmma_f32_16x16x16_f16 with F32 sums, F16 inputs whatever OKLM_CL_WMMA's type (the caches are F16). The 32 queries of a block are rounded to F16 once into local memory; per key tile, the four waves each take one 16 x 16 block of the scores (the scale and soft cap applied to the F32 sums, then the masks), items 0 to 31 each run one row's softmax step (the probabilities rounded to F16, their sum added as rounded), and P V runs as two row blocks by the head's 16-wide column blocks, half of them a wave, the accumulators rescaled row by row; the value tile is kept transposed in local memory for that product. Results within the rounding of the queries and probabilities to F16 (2e-4 at worst against the CPU on values and keys in [-1, 1], test_opencl_wmma)
attention_bi64, attention_bi80, attention_bi128 attention bidirectional attention (an encoder's rows: the vision encoders' blocks, Whisper's encoder, an embedding model's): from 16 rows and OKLM_CL_ATT_KEYS keys (with OKLM_CL_ATT_BLOCKS set: from 16 rows; 0: never), F32 or F16 keys and values, heads a multiple of 4 up to 64, 80 (SigLIP's 72, Qwen2-VL's 80) and 128, the variants differing by the local memory their bound takes. The work-groups of the prompt kernels: 128 items per block of 32 rows (a row a query of one head, the heads that read one KV head sharing a block) by tiles of 32 keys; no position, window, mask or block table. One buffer of local memory holds a tile's keys (transposed), then, once every score is taken, its values: with the probabilities and the rows' maxima a work-group takes 16 KB for heads up to 80 (14 KB up to 64, 23 KB up to 128) where a tile of keys, one of values and the scores took 30 (26, 43), and local memory bounds how many work-groups a compute unit runs at once. An item reads its share of a tile into registers, four elements a load, while the items still compute: the values during the scores, the next tile's keys while the values are added. The four items of a row hold its query in registers, score 8 keys each and keep the scores in registers, share their maximum, turn their own scores into probabilities once the row's four maxima give the tile's (an online softmax, one exponential a score), and add the tile's values to their quarter of the output; the sums are the ones of the layout with two tiles, in the same order. A head that is not a multiple of 16 is laid out as the next one (72 as 80), its last dimensions zeros that are not written. With fewer blocks than fill the device the keys of each block are split into ranges of 4 tiles or more, merged by attention_merge (the split of the prompt kernels). With a span of keys a query (oklm_attention_spans(): the windowed blocks of Qwen2.5-VL's encoder, a window's rows seeing each other) the kernels read a table of first key and one past the last a query: a block walks the tiles from the first key any of its rows sees to the last, each row keeping the scores of its own span, so the windows of a block are one launch whatever their sizes and wherever they fall in the blocks of 32 rows; the keys are then one range, from 16 rows whatever the keys' count. No other kernel reads the table: BF16 or block-quant keys, a head over 128 or not a multiple of 4, fewer than 16 rows or OKLM_CL_ATT_BLOCKS=0 return OKLM_ERR_UNSUPPORTED, and the caller attends span by span (one launch a window, as before). The sums are the attention kernel's in another order: results within the tests' tolerance of the CPU's (2e-6 + 2e-4 relative). Measured on 2026-11-14 (RX 7900 XT, OKLM_CL_PROFILE=1): the 27 blocks of Gemma 3's encoder (4 096 rows, 16 heads of 72) 2.64 s of attention on the attention kernel, 0.50 s on these with two tiles and 0.40 s with one buffer (0.46 s on the variant of 128, whose work-group takes 23 KB); Whisper large-v3-turbo's 32 blocks (1 500 rows, 20 heads of 64) 0.44 s, 0.095 s and 0.085 s; the 32 blocks of Qwen2-VL 2B on a picture of 2 000 x 2 000 pixels (4 900 rows, 16 heads of 80) 0.91 s -> 0.72 s; a decode row unchanged (Llama 3.2 1B Q4_K_M, bench_llama: pp512 14 802 to 14 989 before, 14 814 to 14 847 after; tg128 398 to 403 and 406 to 409)
attention, attention_merge attention grouped-query, causal, with queries that see each other (mutual, an image's rows in Gemma 3: the last key they all see is a parameter, the window's start stays each query's own) or bidirectional (an encoder's rows: the cache's last key and no window, more queries than keys allowed); the first take the prompt tiles where causal rows do (heads up to 128 by default) and this kernel otherwise, the second has its own (attention_bi64 and the others, above) and comes here with what they do not take (BF16 or block-quant keys or values, heads over 128 or not a multiple of 4, fewer than 16 rows or than OKLM_CL_ATT_KEYS keys, few blocks over fewer than 8 tiles of keys). What this kernel takes of the models on disk was listed on 2026-11-14 (RX 7900 XT, OKLM_CL_PROFILE=1, each launch waited for): the decode steps of every model (one row); prompts under 16 rows or under OKLM_CL_ATT_KEYS keys; Gemma 3's image rows on heads of 256 (34 launches, 17.5 to 18.3 ms of a prompt's 0.95 s of kernel time, faster here than on attention_tile256); the cross-attention of Whisper's decoder, one row over the encoder's 1 500 F32 keys (38 us a launch: 4.4 ms of jfk.wav with large-v3-turbo, 39 ms of 0.62 s with large-v3's 32 decoder blocks); a text encoder's single sequence over fewer than 128 keys (8.7 to 10.6 us a launch, where attention_bi64 took 10.9 to 20.1 us with OKLM_CL_ATT_BLOCKS=1); a batch of several sequences left this kernel on 2026-11-14: it is one launch of attention_bi64 a block, each row given its sequence's rows as its span of keys (oklm_attention_spans()), where it was a launch a sequence here (364 lines of a page in 9 batches: bge-small 4 308 launches of 12.4 us and 60 on the tiles, 55.8 ms of 0.162 s of kernel time -> 108 launches of 102 us, 11.0 ms of 0.118 s; nomic-embed-text 81.2 ms of 0.380 s -> 21.7 ms of 0.331 s; the vectors 2.7e-7 and 3e-7 from the CPU's, 2e-7 and 4e-7 before). The cross-attention of Whisper's decoder stays: a launch reads the 1 500 keys and values of 20 heads of 64, 15.4 MB in F32, in 38 us, the memory's rate (about 400 GB/s), and a block's row needs the block before it, so neither the work-groups' shape nor a launch for several blocks has anything to win (large-v3 on jfk.wav, 2026-11-14: 2 112 launches here for the decoder's self- and cross-attention, 47.7 ms, and 1 056 of attention_merge, 5.4 ms, of 0.49 s of kernel time). No caller gives bidirectional attention BF16 or block-quant keys (an encoder's keys are its activations, F32) and no encoder on disk has heads over 128 or not a multiple of 4 (32, 64, 72 and 80): those operands have no tiles; sliding window, softcap, F32, F16, BF16 or block-quant cache (F16 keys read eight at a time, F32 keys four, block-quant keys dequantized 32 at a time, values element by element). Keys are split across work-groups when there are few (head, token) pairs (decode), each split runs an online softmax over chunks of 512 keys, and a merge kernel combines the partial results; with attention sinks (gpt-oss: one score per query head, a device tensor) the merge kernel adds each head's sink to its sum as one more term, so an op with sinks always goes through it: the per-token kernel then splits the keys in two at least, and the tiled prompt kernels write their partials even for one range; every item of a work-group takes part in the value sum; heads up to 1024 wide (latent attention's shared 576-wide head, the 512-wide heads of Gemma 4's blocks without window: each item takes several dimensions over all keys)
get_rows, softmax_rows get_rows, softmax gathers from float tables on the device (MoE routing; host tables such as token embeddings and quantized tables fall back), unmasked softmax of short rows (router probabilities), one work-group of 64 per row
moe_select, moe_weights, moe_sum moe_select, moe_weights, moe_sum MoE routing without a read-back: expert choice with selection bias and groups (one work-group of 64 per token, scores in local memory, up to 1024 experts; up to 256 each competing expert counts those ranked before it and the k-th writes slot k, one barrier in place of a reduction per pick; the same order as the CPU, so the same experts), weights of the chosen experts, and the weighted sum of the K expert outputs in rank order
ssm_conv, gated_delta, gated_delta_rows ssm_conv, gated_delta a Gated DeltaNet block's short causal convolution (one item per channel over the tokens in order, the ring updated) and its recurrence. gated_delta: one work-group of 128 per value head, a state row an item, read and written in device memory at each token (a decode step). gated_delta_rows, from 8 tokens with key heads up to 128: the state in registers over the tokens, read before the first and written after the last; a work-group of 256 items per 32 state rows of a head, an item holding 4 rows by 4 key dimensions, the rows' sums reduced through local memory. Its sums are not in gated_delta's order: the CPU's values within the op test's tolerance (1e-5 absolute, 1e-4 relative), not the step kernel's bits. A prompt of 512 tokens of Ternary Bonsai 2 4B: 2.14 to 1.46 ms a launch (2026-11-14)
quantize_rows, hadamard KV cache cache writes into Q8_0, Q4_0, Q4_1, Q5_0, Q5_1 with the CPU's algorithms (one item per block); Walsh-Hadamard rotation, one work-group per group, butterflies in local memory
hadamard_fold hadamard with signs, reordered heads or groups above 256 (oklm_hadamard_fold()) the input of a product with a weight stored in a rotated basis (Ternary Bonsai 2: H (D x), blocks of 512 and 4096) and a rotated embedding row turned back (D (H z)): a work-group of 64 items per group with 16 KiB of local memory (groups up to 4096), each element times its sign before or after the butterflies, the value heads of a row read in the order they were folded in; one launch where the CPU makes three passes. Three stages a pass: the 8 elements of a pass's group are a stride apart and take each other's values alone, so an item computes their twelve butterflies in registers, the first pass reading the input and the last one writing the output (a block of 512: three passes and two exchanges through local memory where nine stages made nine; the butterflies are the same, so the same bits; one or two stages left over run a pair an item)
norm_fold rms_norm + hadamard (oklm_hadamard_fold()) a decode step's RMS norm and the rotation of its output in one launch (OKLM_CL_FUSE): the backend holds the norm of one row (weighted, a multiple of 256 values from 2048, the widths the norm kernel sums in runs) and runs it with the rotation that reads its output when that comes next, else on its own; one work-group of 256 items per group of the rotation, each summing the row's squares as norm does, writing its part of the norm's output and rotating it in hadamard_fold's passes. The same bits as the two ops (test_opencl, on every device)
moe_group, gi_<type>, wi_<type> matmul_id indexed products of more rows than experts (prompts of MoE layers): moe_group (one work-group) counts each expert's rows, lays out a table of tiles of up to 64 rows per expert and places the rows (atomically: their order within an expert varies, the products do not), then one launch of gi_<type> (the gm_<type> steps, 64 weight rows by one tile entry, rows read and written through the table; wi_<type> on the matrix units of a gfx11 device for the types wm_<type> takes there) multiplies every tile; chunks of 8192 rows. Replaced reading the experts back and one product per expert (OLMoE-1B-7B prompts 4175 to 8614 t/s with the matrix units, 2026-11-08)
q8_act, wq_<type>, wqi_<type> matmul, matmul_id on a gfx11 device by default (OKLM_CL_Q8), in place of the tiled kernels for products of 64 columns or more (OKLM_CL_Q8_COLS) with Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 or Q5_0 weights: the CPU's products, bit for bit. q8_act rounds the activations to 8 bits exactly as the CPU does (Q8_K blocks of 256 for the K-quants, Q8_0 blocks of 32 for the others: the same scale, levels and block sums; its divisions correctly rounded through double precision), then wq_<type> (64 weight rows by 128 columns, four waves, 128 values of k a step) puts the weights' integer levels in local memory unscaled and multiplies them with v_wmma_i32_16x16x16_iu8 (exact 32-bit sums): each block's integer sum is scaled and added as the CPU's dot product does it (oklm_vec_dot_q4_k_q8_k() and the others: the sub-block scales applied to integers, the mins term of Q4_K / Q5_K as two more 16-deep products of the 8 mins by the block sums split into 7-bit halves, then d isum - dmin summs per super-block, in order of k, no fused multiply-adds, K never split). The Q4_K and Q5_K tiles have four more waves that only load (a work-group of 256 items): while the four multiplying waves work on a step, the loader waves put the next step's weight levels, scales and activation levels in a second buffer of local memory, so a step's loads hide behind the previous step's products and the multiplying waves keep their registers for the accumulators (the same sums in the same order, the same bits). wqi_<type> does the same for the grouped indexed products of MoE prompts (128 weight rows by a tile entry of up to 64 rows; four waves for Q6_K, Q8_0, Q4_0 and Q5_0, eight with the same loader waves for Q4_K and Q5_K, a loader item taking both halves of a weight row and half a column: 64 000 bytes of local memory). Decoding: mq_<type> below
mvg_<type>, mqg_<type> matmul_id + matmul_id + swiglu a MoE layer's gate and up products and their SwiGLU in one decode launch (OKLM_CL_FUSE and OKLM_CL_MVG): the backend holds the gate product, then the up product on the same activations and experts, and runs the three when the SwiGLU of their outputs comes; mvg_<type> on the MV kernels (every type), mqg_<type> on the 8-bit ones (Q4_K, Q5_K, Q6_K: the column rounded once); each product summed in its own kernel's order, the up output written too, so the same bits as the three ops
cpy2 two cpy two copies of F32 tensors into float-like ones in one launch (OKLM_CL_FUSE: a held copy runs with the next one): a decode step's keys and values into their caches
res_norm moe_sum + add + scale + add + rms_norm the end of a residual branch in one launch (OKLM_CL_FUSE and OKLM_CL_RES), one work-group of 256 items per token: the backend holds the weighted sum of the expert outputs, the add of another tensor into it (the shared experts' output), its scale (a residual multiplier), all in place on the branch's output, and the add of that output into the residual stream, and runs them with the RMS norm of the new stream when it comes next (the next block's input norm), else on their own as soon as anything else comes; each stage optional but the add into the stream, a lone op run as it is. Each stage repeats its kernel's operations (no fused multiply-adds, the norm's sum of squares over the same elements per item and the same reduction; the branch's output and the stream written as the separate launches write them), so the same bits as the two to five ops. An item reads four of its elements before it writes any of them (a row of 2880 values is eleven or twelve elements an item, and a load, an operation and a store an element waited for the memory once an element): 16.1 to 10.7 us a launch on gpt-oss-20b, 49 launches a step (OKLM_CL_PROFILE=1, 2026-11-14)
rope_kv rope (+ rope) + two cpy a step's rotation of its queries and keys and the writes of its keys and values into their caches in one launch (OKLM_CL_FUSE and OKLM_CL_ROPE_KV): the backend holds the rotation in place of the queries (or of the queries and keys as rows of one tensor), the rotation of the keys with the same parameters, and the copy of the rotated keys into their cache (F32, F16 or BF16), and runs them with the copy of the values that follows; an item per dimension pair rotates as rope does and, on a key row, stores the pair into the cache too (converted as ew converts it), the items after those copy the values. The same bits as the three or four ops; a quantized cache keeps its own kernel
moe_route, norm_rope softmax + moe_select + moe_weights; two rms_norm + rope launches fused by the backend (OKLM_CL_FUSE): a held-back op is run with the ones that complete it, or on its own as soon as anything else comes; the fused kernels repeat the separate kernels' operations, so the results are the same bits (moe_route: a MoE layer's softmax router, expert choice and weights in one launch per layer, and a sigmoid router's likewise, OKLM_CL_ROUTE_SIGMOID; norm_rope: Qwen3's per-head query and key norms and their rotation, one launch instead of three or four)
attention_mla attention prompts of latent attention's shared head (heads of 257 to 576 dimensions, F16 cache, the values the keys' own tensor: DeepSeek V2 / V3, GLM-4.7-Flash; OKLM_CL_ATT_MLA): blocks of 16 rows (query, head) pairs, the heads of a token sharing each tile of 16 keys, kept as halves in local memory for both the scores and the values; each of the 16 items of a row holds a slice of its query and output (the scores' partial sums added by the item of each key), the keys split into ranges as the other tiles when there are few blocks. GLM-4.7-Flash prompts 1 337 to 2 174 t/s (2026-11-08)
mm_<type> matmul every weight type OKLM computes with: F32, F16, BF16, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, IQ4_NL, IQ4_XS, MXFP4, NVFP4, TQ1_0, TQ2_0, Q1_0, Q2_0, the grid IQ types, and PQ2_0 and PTQ1_0 (the two types of PrismML's llama.cpp fork, decoded like Q2_0 and TQ1_0 with a scale per 128). A work-group of 64 items computes 4 weight rows for a block of 8 activation columns; 16 items per row take groups of 32 weights in turn (adjacent items read adjacent blocks), dequantize each group once and use it for every column; fixed reduction order; optional accumulation into the destination (oklm_matmul_acc); with an index buffer, each batch index multiplies the expert matrix it names (oklm_matmul_id, one column per (token, rank) pair)
mq_<type> matmul, matmul_id on a gfx11 device by default (OKLM_CL_Q8_MV, with OKLM_CL_Q8), in place of mv_<type> for one-column products (decode, and the indexed products of MoE decode steps) with Q4_K, Q5_K or Q6_K weights, rows of 2048 values or more and 1024 rows or more: the CPU's products, bit for bit. Each work-group of 256 items first rounds its activation column to Q8_K in local memory exactly as q8_act does (each wave 256 values at a time, the largest magnitude found with cross-lane DPP steps, the divisions through double precision), so no launch is added; then each wave takes two weight rows: eight items a super-block, each forming 32 levels' products with the 4-way 8-bit dot products (v_dot4_i32_iu8), the scales applied to the integer sums, the sums of the eight items added (DPP), and each super-block's d isum - dmin summs added to the row's sum in order of k (the wave's four groups read with readlane); the activations are read once for both rows, every load of a wave step (two super-blocks a group) issued before any term is added. Measured slower and left to mv_<type>: Q8_0, Q4_0 and Q5_0 (their 8-bit twins 3 to 7 % slower: Qwen3-8B Q8_0 decoding 66.6 to 64.8 t/s), and shorter or fewer rows, where each work-group pays the column's rounding for little work (SmolLM2-135M's 576 x 1536 down products: its decoding 412 to 370 t/s; Granite's and OLMoE's rows of 1024 values 2 to 3 % slower). mqn_<type>: the same on the input of a held decode-step RMS norm (Fewer launches)
mv_<type> matmul, matmul_id in place of mm_<type> for one activation column (decode) and for the indexed products (one column per expert row): the same rows, lanes, group order and reduction tree, so the same bits, without reducing the seven columns that are not there (Llama-3.2-1B decode 349 to 362 t/s, 2026-11-08)
gm_<type> matmul the same types, for prompts (32 columns or more, at least 256 tiles of 64 x 64 outputs, K a multiple of 32): a work-group of 8 x 8 items per tile; for each step of 32 values of k, each item dequantizes one group of 32 weights and loads 32 activations into local memory (k-major), then multiplies its 8 x 8 outputs, two float8 local reads per k for 64 products. The next step's weights and activations are loaded into registers while the current step multiplies; sums run in a fixed order. Products with fewer tiles split K into up to 8 ranges of at least 4 steps (one work-group per tile and range, partial tiles in a scratch buffer, added in range order by gm_reduce, so the sums keep a fixed order); with K too short for that, the row kernels, which have more work-groups to spread over the device. Each output is one running sum over the K terms of its range, where the CPU and the row kernels keep several partial sums: further from the CPU's where the terms cancel, most through Whisper's encoder (Ultravox's embeddings on jfk.wav 3.0e-4 relative from the CPU's, 3.1e-5 with OKLM_CL_GEMM_TILES=0; the vision encoders 1.4e-5 to 5.3e-5). Measured on 2026-11-14 and not kept: a sum a step of 32 terms added to the total gives 4.5e-5 and one every four steps 1.1e-5, for 7 % more time in the kernel (Gemma 3's encoder 228 -> 245 ms of gm_f16, Whisper's 140 -> 149 ms: the second set of 64 sums takes as many registers again); the partial sums added into the output buffer instead, 313 ms. These are the F32 products: 14 to 15 Tflop/s on the encoders (gm_f16: 0.14 s of Whisper large-v3-turbo's encoder, 0.22 s of Gemma 3's), against 0.035 s and 0.050 s on the matrix units with OKLM_CL_WMMA=1, which round the activations to F16 as Vulkan's do
wm_<type> matmul on a gfx11 device, in place of gm_<type> (same arguments, tiles, split K and gm_reduce), by default for quantized weights that have no 8-bit kernel and whose product was not asked with the activations at F32's precision (oklm_matmul_wide()), for every product with OKLM_CL_WMMA (1, f16, bf16), for none with OKLM_CL_WMMA=0: the RDNA3 matrix units through clang's __builtin_amdgcn_wmma_f32_16x16x16_{f16,bf16}_w32, which the ROCm OpenCL compiler accepts in OpenCL C (built with -DOKLM_WMMA, so other devices never compile them). Per step of 32 values of k, each of the 64 items dequantizes one weight row's group and loads one activation column, both rounded to F16 (or BF16) into local memory; each of the two waves multiplies 32 weight rows by the 64 columns as 2 x 4 blocks of 16 x 16 with F32 accumulators, two 16-deep products per step. The next step's weight group and activation column are read into registers while the current step multiplies, as gm_<type> does. A zero weight is given to the units as +0 whatever its sign (their sums depend on it in their last bits). Results within the inputs' rounding of the F32 kernels
w2_<type>, w2_cvt matmul in place of wm_<type> (the same products) for prompts of 64 columns or more: w2_cvt rounds the activations to F16 (BF16) once per product, then a work-group of four waves takes 64 weight rows by 128 columns, 64 values of k a step: each item dequantizes one row's group of 32 and copies one column's 64 halves as 16-byte vectors, each wave multiplies 32 rows by 64 columns (the same 2 x 4 blocks as wm_<type>); each dequantized weight serves 128 columns instead of 64 and no work-group rounds activations; the next step's weight group and 64 halves are read into registers while the current step multiplies. The split of K is gm_<type>'s. Results within the inputs' rounding, and the bits wm_<type> gives for the same product (the same halves summed in the same order; Qwen3-8B Q4_K_M prompts 1 386 to 2 262 t/s, Llama-3.2-1B 9 323 to 11 909, 2026-11-08)

The float kernels multiply dequantized weights by F32 activations. The CPU (like llama.cpp) rounds the activations to 8 bits first for quantized weights, so their results are closer to the exact product of the quantized weights than the CPU's. On a gfx11 device the prompt products of Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 and Q5_0 weights take the 8-bit kernels instead (wq_<type>, by default): the activations rounded as the CPU rounds them and the CPU's products bit for bit, so a prompt's greedy tokens follow the CPU's where the float kernels part from it at a near tie (Qwen3-8B Q4_K_M, OLMoE-1B-7B: the 16 greedy tokens after a 190-token prompt are the CPU's, the float kernels' differ from the first or second; 2026-11-08). Decoding (one column) takes the 8-bit kernels too for Q4_K, Q5_K and Q6_K weights, rows of 2048 values and more, 1024 rows and more (mq_<type>, the CPU's products bit for bit; OKLM_CL_Q8_MV); the other one-column products keep the float row kernels.

Fewer launches

Generation of small models is bound by the number of kernels per token (each costs a few microseconds of dispatch even when its work is tiny), so the Llama-family forward pass merges what it can without adding work:

  • in the device copy of the weights, Q, K and V matrices of the same type are stored back to back and also described as one matrix (likewise the Q/K/V biases and the gate/up matrices): one product instead of three, and one instead of two;
  • K heads follow the Q heads in the fused output, so one RoPE launch rotates both;
  • the output and down projections add into the residual stream (oklm_matmul_acc) instead of writing a temporary and adding it.

Per token: SmolLM2-135M from 514 launches to 322, Llama-3.2-1B from 274 to 194. The backend also holds back the ops of two short sequences and runs each as one launch when it completes (OKLM_CL_FUSE; the same bits, anything else first runs what is held): a MoE layer's routing (softmax, expert choice, weights: moe_route, OLMoE-1B-7B decoding 264 to 278 t/s), a MoE layer's gate and up products with their SwiGLU (mvg_<type>, mqg_<type>; 2026-11-11, decoding 128 tokens, three runs each: Granite-3.1-1B-A400M 281 to 302 t/s, OLMoE-1B-7B 276 to 289, GLM-4.7-Flash 70.9 to 72.4; Granite's 472 launches a token to 425) and Qwen3's per-head query and key norms with their rotation (norm_rope; Qwen3-8B decoding within the noise: its products, not its launches, take the time). A decode step's RMS norm (one row of 2048 to 8192 values, a multiple of 256) is held back too and folded into the following one-column product when that product takes an MQ kernel (mqn_<type>): each work-group of that kernel already reads the whole column to round it, so it loads the norm's input and weights once, sums the squares in the norm kernel's order, scales and rounds; its first work-group writes the norm's output for the ops that read it later. The norm kernel sums rows of those widths in the same order (runs of eight values an item, the MQ kernels' layout of a column), so the fused and the separate norm give the same bits (test_opencl_q8). Every norm of Llama-3.2-1B's decode steps goes (202 to 169 launches a token, decoding about 367 to 385 t/s), and so do Qwen3-8B Q4_K_M's. Two copies that follow each other (a step's new keys and values into a converting cache) run as one launch (cpy2, each element converted as the ew kernel converts it: the same bits), one launch fewer a layer (Llama-3.2-1B decoding about 1 to 2 % faster). The end of a residual branch is one launch too (res_norm, OKLM_CL_RES): the weighted sum of the expert outputs, the add of the shared experts' output, the residual scale, the add into the residual stream and the RMS norm of the new stream (the next block's first op), which were up to five launches of a few microseconds of work each; and so is a step's rotation with its cache writes (rope_kv, OKLM_CL_ROPE_KV: the rotation of the queries and of the keys, the copies of the keys and the values into their caches, two to four launches before). Both give the bits of the separate ops (test_opencl: every stage combination and layout against the ops kept apart, and the launch counts). A sigmoid router's routing (GLM-4.7-Flash) is one launch as a softmax router's is (moe_route, OKLM_CL_ROUTE_SIGMOID). Decoding 128 tokens after 16, alternated runs, 2026-11-12, without / with the three: Granite-3.1-1B-A400M 306 to 395 t/s (424 launches a token to 268), OLMoE-1B-7B 296 to 318 (249 to 209), GLM-4.7-Flash 73.8 to 77.9 (1259 to 1076), SmolLM2-135M Q4_K_M 422 to 471 (344 to 300), Llama-3.2-1B 391 to 400 (153 to 129) (benchmarks.md). A step's norm of a model whose weights are stored in a rotated basis (Ternary Bonsai 2) is followed by the rotation of its output, not by a product: the two are one launch (norm_fold), 65 launches fewer in a step of Ternary Bonsai 2 4B. With the rotation's three stages a pass (8.5 to 5.5 us a launch, OKLM_CL_PROFILE=1) and one transformed input for the two products of a Gated DeltaNet block that read the same one (24 launches fewer, on every backend), that file's step went from 612 launches and 6.04 ms of kernel time to 523 and 4.87, and its decoding from 138.6 to 162.5 tokens a second (Vulkan 169.3 to 177.3 with the transformed input shared; bench_llama, 128 tokens, the best of four or five runs, 2026-11-14). Two other fusions were measured and left out: RMSNorm inside the float matrix product (every work-group recomputes the norm of its columns: prompt processing halved, Llama-3.2 decode at 500 tokens of context fell from 165 to 103 tokens/s) and SwiGLU inside the down projection (TinyLlama: 8 % fewer launches for 44 % more GPU time). Files whose Q/K/V or gate/up matrices have different types (Qwen2.5 Q4_K_M) keep separate products.

Fallback

An op whose operands or types no kernel takes (an operand in host memory, a weight type without a mm_ kernel, softmax, cross-entropy, argmax...) runs on the CPU backend over host copies of the byte ranges it touches, and the destination range is written back. It is logged once per op at level INFO. During inference only the embedding lookup falls back, by design.

Verification

Test What Result
tests/core/test_opencl.c every kernel against the CPU on random shapes (the float kernels of every type: OKLM_CL_Q8=0, OKLM_CL_WMMA=0) (sizes not multiple of the work-group sizes, broadcasts, a permuted view, 1 300 keys across attention chunks and splits, bidirectional attention on its tiles (rows and keys that are not multiples of a tile's 32, heads of 36, 64, 72, 80, 100 and 128, one head and several, two heads a KV head, a soft-cap, few blocks whose keys are split into 2 and 8 ranges) and off them (BF16 keys, a head of 66), the same tiles with a span of keys a query (test_attention_spans: windows of unequal sizes, of 1 to 100 rows, smaller than a tile and over several, blocks of rows that straddle windows, one window over everything, 2 to 16 heads, two a KV head, a soft-cap, spans that are not windows; BF16 keys, a head of 66 and 9 rows refused), all 21 weight types at decode, prompt and batched shapes, accumulation; the products a convolution is made of on a device (test_conv_views: one product a position of the kernel, with the activations at F32's precision, over a view of every stride-th row from that position on, the first written and the others accumulated: kernels of 3 and 7, strides of 1 and 2, 96 to 1 280 channels, against the convolution in doubles, within 6.8e-8 of sum w x
tests/core/test_opencl_q8.c on a gfx11 device: the 8-bit matrix-unit products (wq_<type>, wqi_<type>) of Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 and Q5_0 weights against the CPU's (partial tiles, batches, K not a multiple of the step, one to eight super-blocks of the K-quants and an odd count over several tiles, accumulation, rows and columns of zeros, ties of magnitude, grouped indexed products shared or not, of 300 and 800 rows over several tiles an expert and three tiles of weight rows, partial ones, with one, two and five super-blocks); the one-column products of the K-quants (mq_<type>: rows of 2048 to 4864 values, batches, accumulation, indexed products of one and three tokens); the prompt products of the types without an 8-bit kernel as the device runs them by default (200 rows by 150 columns in a batch of two and 200 by 40, K split, for each of the 21 types; 1030 by 1000 over whole and partial tiles for IQ2_XXS, IQ2_S, IQ3_XXS, IQ3_S, Q3_K, MXFP4 and PQ2_0; indexed products of 300 rows over 8 experts for IQ2_XXS and MXFP4) against a double reference, and the same products asked at F32's precision; a 100-token SmolLM2 Q8_0 / Q4_K_M prompt against the CPU every product bit-identical to the CPU's; the default products of the other types (each of the 21) within 2e-3 of the sum of magnitudes (1.1e-4 at worst measured), those asked at F32's precision within 1e-5 (3.1e-7) and not the first ones' bits; logits within 0.61 to 0.63 of the CPU's (0.80 to 0.92 with the float kernels), the same top token
tests/core/test_opencl_wmma.c on a gfx11 device: a probe kernel calling clang's __builtin_amdgcn_wmma_f32_16x16x16_f16_w32 and ..._bf16_w32 from OpenCL C on one 16 x 16 x 16 tile, and on one whose zeros are +0 then -0; the wm_<type> and w2_<type> kernels (OKLM_CL_WMMA) on F32, F16, BF16, Q8_0, Q4_0, Q4_K and Q6_K weights, partial tiles, batches and split K (a range of K ending on half a W2 step), and on F32 weights whose zeros are +0 then -0; the prompt attention of F16 caches on the matrix units (attention_wm64h, attention_wm128h: grouped heads, keys split and merged, heads of 64 and 80, a window, a soft cap) against the CPU's the probe within 4e-8 of the CPU relative to the sum of magnitudes (182 of 256 values exact: the units do not round their F32 sums as IEEE additions; with 30 zeros of A given as -0 instead of +0, 36 of 256 sums change in their last bits, both within 8.4e-8); products within 2e-3 (F16) / 1.6e-2 (BF16) of the sum of magnitudes, 1.5e-4 at worst measured with F16, and the same bits whatever the sign of the zero weights; attention within 1e-3 (2e-4 at worst) and not the CPU's bits. Skipped elsewhere
tests/model/test_model.c tiny llama / qwen3 / gpt2 against the float64 reference, every layer on the device and split between CPU and device same as the CPU: 9.5e-7, 1.5e-5, 2.6e-5 (split: 9.5e-7, 1.4e-5, 3.7e-5); batched and token-by-token logits bit-identical
tests/model/test_infer_public.c SmolLM2 F16 / Q8_0 / Q4_K_M and GPT-2 Q4_K_M against the float64 reference (the float kernels: OKLM_CL_Q8=0) 4.6e-5, 3.0e-5, 5.6e-5 and 3.2e-4 (logits near 100); every greedy choice equal
same the corpus against llama.cpp F16 and BF16: identical continuations. Quantized: logit noise of 1 to 2.4 (llama.cpp rounds activations), every point where the continuations part a near tie for llama.cpp (margin at most 0.44)

Kernel results are deterministic: the same inputs give the same bits on every run.

Speed

oklm run with the first 2000 bytes of Tiny Shakespeare as prompt (about 500 tokens) and 128 generated tokens, AMD Radeon gfx1100 (42 compute units, 20 GiB), tokens per second. CPU: the CPU backend on an Intel Core i7-11700K, 16 threads (2026-10-30, quiet machine). OpenCL: best of three runs, 2026-10-30, measured while other heavy jobs were running on the machine, so a quiet machine can only do better; "first" is the first version of the backend, "now" the current one.

file CPU prompt OpenCL prompt, first / now CPU generation OpenCL generation, first / now
SmolLM2-135M F16 888 8 977 / 14 807 47 224 / 317
SmolLM2-135M Q8_0 372 8 501 / 14 614 60 249 / 340
SmolLM2-135M Q4_K_M 397 7 819 / 14 286 81 242 / 310
SmolLM2-135M BF16 checkpoint 746 8 528 / 14 660 72 243 / 322
Qwen2.5-0.5B Q4_K_M 141 4 117 / 10 013 33 192 / 251
Qwen3-0.6B Q8_0 95 2 597 / 6 446 21 122 / 203
Llama-3.2-1B Q4_K_M 93 1 816 / 5 189 21 165 / 268
TinyLlama-1.1B Q4_K_M 100 1 812 / 4 563 24 184 / 265
GPT-2 Q4_K_M 409 9 112 / 22 718 101 421 / 552

From first to now: the fused launches above, a Q6_K dequantizer without a per-element branch (Llama-3.2's Q6_K products went from 348 to 49 microseconds per launch), 4 rows per work-group in the matrix products (half the GPU time of one row per work-group on prompts), and the tiled matrix products for prompts (10 to 28 % on prompts: an 8192 x 2048 Q4_K product of 512 columns takes 1.23 ms instead of 1.96, 14 TFLOPS against 9), then the tiled prompt attention (Llama-3.2-1B's attention 42 to 12 ms per 500-token prompt; Qwen3-0.6B prompts 3 549 to 6 446 t/s, Phi-3.5-mini 849 to 1 343).

The matrix units (OKLM_CL_WMMA, 2026-11-08, quiet GPU, oklm run with the first 2000 bytes of Tiny Shakespeare and 64 greedy tokens, best of three): Qwen3-8B Q4_K_M prompts 831 t/s with the F32 kernels, 1 320 with F16 inputs (1.59 x; its Q4_K products 2.93 to 1.54 ms a launch on average), 1 137 with BF16 (1.37 x); Llama-3.2-1B Q4_K_M 4 938, 8 713 (1.76 x) and 6 887. Decoding (a column at a time, the row kernels) is unchanged. The 64 greedy tokens are the F32 kernels' with F16 inputs on both models, and with BF16 on Qwen3-8B; Llama-3.2-1B with BF16 parts at one token. Perplexity of Qwen3-8B (first 30 KB of Tiny Shakespeare, 512-token windows): 19.9952 (F32 kernels), 19.9972 (F16), 20.0010 (BF16); an F32 KV cache instead of the default F16 moves it as much (19.9962). Off by default: the greedy answers can part at near ties.

The prompt attention on the matrix units too (attention_wm<hd>h, with OKLM_CL_WMMA, 2026-11-11, quiet GPU, best of three; OKLM_CL_WMMA_ATT=0 for the half tiles): Qwen3-8B Q4_K_M prompts of 497 tokens 2 478 to 2 619 t/s, of 2 059 tokens 1 992 to 2 476 (its attention 291 to 97 ms of the GPU time at 2 000 tokens, 23.4 to 9.4 ms at 500); Llama-3.2-1B Q4_K_M 14 175 to 15 370 and 10 840 to 14 277. Perplexity (the same 30 KB, 512-token windows) 19.9965 to 19.9962 on Qwen3-8B, 41.4164 to 41.4163 on Llama-3.2-1B; the 64 greedy tokens after the first 2 000 bytes part at the third or fourth token on both, as those of the matrix-unit products part from the default kernels' (Qwen3-8B: at the fifth).

Measured and left out (2026-11-11): prompt products through rocBLAS's gemm_ex, as llama.cpp's hipBLAS route does (each weight dequantized to F16 rows in shared virtual memory, the activations rounded to F16, F32 accumulation into an F16 product, widened back). rocBLAS alone runs Qwen3-8B's shapes at 36 to 71 TFLOPS with an F16 product (12 TFLOPS with an F32 one) at 512 and 2048 columns; but the 8-bit kernels the products take by default already do a 500-token prompt's 180 products in 150 ms, while rocBLAS took 161 ms for the products alone (about 125 at the rates it reaches by itself) before the dequantization (114 ms with a first kernel, at least about 20 at the device's bandwidth) and the conversions (about 30 ms with the OpenCL queue drained before each product). Qwen3-8B Q4_K_M prompts: 2385 t/s with the default kernels, 1595 through rocBLAS (500 tokens); 1994 against 1665 at 2000 tokens. The code was not kept.

The matrix-unit products reading the next step while they multiply (wm_<type>, w2_<type>, with OKLM_CL_WMMA=1; 2026-11-12, quiet GPU, bench_llama, the best of three, without / with alternated over three rounds, t/s). An item keeps the next step's dequantized weight group and activation column in registers (32 floats and 32 floats in wm_<type>, 32 floats and 64 halves in w2_<type>) beside its 64 float accumulators, where the same thing on the 8-bit Q4_K tiles, with 64 integer accumulators more, had halved the prompts:

prompt, OKLM_CL_WMMA=1 without with llama.cpp
Llama-3.2-1B Q4_K_M, 512 tokens 16 228 17 553 17 966
Qwen3-8B Q4_K_M, 512 tokens 2 825 3 172 3 218
Llama-3.2-1B Q4_K_M, 48 tokens (wm_<type>) 5 759 6 863
Qwen3-8B Q4_K_M, 48 tokens (wm_<type>) 1 216 1 523
SmolLM2-135M Q8_0, 512 tokens 46 027 50 440
SmolLM2-135M Q4_K_M, 512 tokens 44 317 44 966
Qwen3-8B Q8_0, 512 tokens 2 996 3 276
Granite-3.1-1B-A400M Q4_K_M, 512 tokens 17 792 18 595
OLMoE-1B-7B Q4_K_M, 512 tokens 8 835 8 916

Llama-3.2-1B decoding (other kernels) 407 / 409. With it the opt-in matrix-unit mode runs the prompts of Llama-3.2-1B and Qwen3-8B Q4_K_M close to llama.cpp's rate (0.98 and 0.99; the default 8-bit kernels, which give the CPU's bits, 0.84 and 0.86). The outputs stay within the inputs' rounding (test_opencl_wmma, the same worst differences) but are not always the bits of the kernels that did not read ahead: with F16 inputs, of the test's 25 products the four of w2_<type> with Q4_0 or Q6_K weights changed (10 to 47 % of their outputs, by at most 9.3e-8 of the sum of magnitudes; both versions within 4.8e-7 of the exact product of the rounded inputs); nothing changed with BF16 inputs. The cause is the sign of zero weights, not the steps read: a level of zero times a negative scale (Q4_0's and Q6_K's scales are signed) is -0, and the matrix units' sums depend, in their last bits, on the sign of a zero input (the test's probe: 36 of 256 sums of a tile change when 30 zeros of A become -0). wm_<type> always rounded such a weight to the half -0; w2_<type> as compiled before stored +0 and now stores -0 as well. With every zero forced to +0 in both versions, the 25 products are the same bits before and after, the range of K ending on half a step included.

Since then the kernels give the units +0 for every zero weight (wm_cvt_w: + 0.0f on the dequantized weight before it is rounded, in wm_<type>, w2_<type> and wi_<type>), so the bits no longer depend on the sign a dequantization, or the compiler, leaves on a zero. The four w2_<type> products are again the bits of the kernels that did not read ahead; three of the test's wm_<type> products with Q6_K or Q4_0 weights, which always had -0, changed in their turn, and with BF16 inputs the same seven products changed (all within the same worst differences: the test's figures did not move). test_opencl_wmma runs a product of F32 weights with one zero in eight as +0 then as -0 on wm_f32 and on w2_f32 and asks for the same bits (both failed before). Two forms were measured (2026-11-12, bench_llama, 512 tokens, OKLM_CL_WMMA=1, the builds alternated over three and four rounds, the best run of each, t/s):

prompt, OKLM_CL_WMMA=1 zeros as dequantized + 0.0f on the weight (kept) the half -0 replaced in wm_cvt
Llama-3.2-1B Q4_K_M, 512 tokens 17 260 17 440 16 618
Qwen3-8B Q4_K_M, 512 tokens 3 166 3 179 2 982

The two forms give the same bits on the test's 25 products, F16 and BF16. Comparing the rounded half with 0x8000 in wm_cvt (weights and activations) cost 4 to 6 % in every round and was not kept; the float addition costs nothing measurable. What it leaves: with F16 inputs a negative weight of at most half the least subnormal half (2^-25) still rounds to -0 (a property of the value, not of the compiler), and the activations keep their sign. The perplexities of the table below with zero weights as +0: 41.4163 and 19.9962 with OKLM_CL_WMMA=1, the figures of the kernels that did not read ahead.

The same sign was the whole difference between w2_<type> and wm_<type>, which W2_KERNEL's comment had put down to the compiler scheduling the dequantization differently. Checked with a build that could send every product to wm_<type>, on the test's 25 products: before the kernels read ahead, w2_<type> and wm_<type> differed on four, exactly the products of 64 columns or more with Q4_0 or Q6_K weights (where w2_<type> stored +0 and wm_<type> -0); once both stored -0 they gave the same bits on all 25, and they still do with every zero as +0, F16 and BF16. The two kernels sum the same halves in the same order with the same split of K, so their results are the same bits; why the earlier w2_<type> lost the sign in compilation was not looked into.

Quality of the mode after that change (2026-11-12, oklm ppl on the first 30 KB of Tiny Shakespeare, 3 825 tokens scored in 512-token windows; oklm run with its first 2 000 bytes, 497 tokens, and 64 greedy tokens), the kernels of before rebuilt and run in the same session:

perplexity Llama-3.2-1B Q4_K_M Qwen3-8B Q4_K_M
default (8-bit kernels, the CPU's products) 41.5610 20.0465
F32 kernels (OKLM_CL_Q8=0) 41.4166 19.9948
OKLM_CL_WMMA=1, before / now 41.4163 / 41.4168 19.9962 / 19.9965
OKLM_CL_WMMA=1 OKLM_CL_WMMA_ATT=0, before / now 41.4164 / 41.4159 19.9965 / 19.9987
OKLM_CL_WMMA=bf16, before / now 41.4211 / 41.4211 20.0056 / 20.0056

The F16 figures moved by at most 0.0022 (1.1e-4 of the value), less than the distance between the F32 kernels and the mode and far less than the distance from either to the default kernels, whose activations are rounded to 8 bits as the CPU's; BF16 did not move (the same bits). The 64 greedy tokens: on Llama-3.2-1B the F16 mode gives the F32 kernels' 64 tokens before and now, and parts from the default kernels' at the fifth; on Qwen3-8B it gave other tokens than before from the fifth (as it parts from the F32 kernels' at the fifth and from the default kernels' at the seventh), the fifth token being a near tie that every pair of kernel families splits on. With BF16 the tokens are those of before on both models (Llama-3.2-1B parts from the F32 kernels' at the fifth, Qwen3-8B at the nineteenth). The matrix-unit attention against the half tiles (OKLM_CL_WMMA_ATT=0): the same 64 tokens on Qwen3-8B, parting at the fourteenth on Llama-3.2-1B (at the fifth on both before).

The mode against the CPU backend (2026-11-12, the same text, windows and Q4_K_M files, oklm ppl -c 512 --backend cpu then --backend opencl; the matrix-unit kernels with every zero weight as +0):

perplexity Llama-3.2-1B Q4_K_M Qwen3-8B Q4_K_M
CPU backend 41.4967 20.1612
OpenCL, default (8-bit kernels) 41.5610 20.0465
OpenCL, F32 kernels (OKLM_CL_Q8=0) 41.4166 19.9948
OpenCL, OKLM_CL_WMMA=1 41.4163 19.9962

The mode is 0.19 % under the CPU on Llama-3.2-1B and 0.82 % under it on Qwen3-8B; the default OpenCL path is 0.15 % over and 0.57 % under. The mode is thus 0.04 and 0.25 % of the value further from the CPU than the default path is, on 3 825 tokens, where neither the sign nor the size of such gaps says which backend is the better one. The default path's prompt products are the CPU's bits, yet its perplexity is not the CPU's: which of its other operations account for the 0.15 and 0.57 % was not looked into.

Types without an 8-bit kernel

The 8-bit kernels take Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 and Q5_0. A prompt product of any other quantized type (the grid IQ types, Q2_K, Q3_K, Q4_1, Q5_1, IQ4_NL, IQ4_XS, MXFP4, NVFP4, the ternary and low-bit types) ran on gm_<type>, the F32 tiles, and a file made of such types paid it on every matrix: Muse Glimmer 30B UD-IQ2_XXS (IQ2_XXS, IQ2_S, IQ3_XXS and IQ3_S matrices) read a prompt at a third of Vulkan's rate. Since 2026-11-14 a gfx11 device gives these products its float matrix units by default (w2_<type> from 64 columns, wm_<type> under, wi_<type> for the indexed products of a MoE prompt), the activations rounded to F16 as Vulkan's cooperative matrices take them; the 8-bit kernels keep their types, float weights and a product asked with the activations at F32's precision (oklm_matmul_wide(): the encoders) keep the F32 tiles, and OKLM_CL_WMMA=0 gives every one of these products the F32 tiles again. Other devices have the F32 tiles alone.

Where a 512-token prompt of Muse Glimmer went (RX 7900 XT, OKLM_CL_PROFILE=1, each launch waited for; 52 launches a kernel a prompt, 104 for IQ2_XXS), time a launch on the F32 tiles and on the units:

Weights F32 tiles Share of the prompt's kernel time Matrix units
IQ2_S 18.6 ms 49.0 % 4.62 ms
IQ3_XXS 10.6 ms 28.0 % 2.42 ms
IQ2_XXS 2.22 ms 11.7 % 0.57 ms
IQ3_S 2.05 ms 5.4 % 0.53 ms
the prompt, kernel time 1.98 s 0.56 s

Prompt of 512 tokens / decode of 128, tokens a second (bench_llama, the best of three runs, 2026-11-14; the decode products are the row kernels' either way):

File F32 tiles (OKLM_CL_WMMA=0) Matrix units (default) Vulkan
Muse Glimmer 30B UD-IQ2_XXS 253.3 / 29.4 865 to 868 / 29.5 834 to 839 / 37.1 to 37.3
Ternary Bonsai 2 4B PQ2_0 1276 / 136.7 2544 / 137.3 3411 / 172.4
gpt-oss-20b Q4_K_M (MXFP4 experts, indexed) 1680 / 126.1 3313 / 126.4 3738 / 163.9
Llama-3.2-1B Q4_K_M (no such type) 14 568 to 14 644 / 391 to 406 14 545 to 14 643 / 404 to 407

The decode rates of this table and Ternary Bonsai's prompts were taken up after it: Decoding on these files.

Perplexity (oklm ppl -c 512, 1275 tokens of this repository's documentation: the first 14 000 bytes of harness.md), the CPU, the F32 tiles, the units, Vulkan:

File CPU F32 tiles Matrix units (default) Vulkan
Muse Glimmer 30B UD-IQ2_XXS 30.7146 30.7098 30.7160 30.7671
Ternary Bonsai 2 4B PQ2_0 47.9792 47.9281 47.9281 (28.6707 against 28.6734 after the first window) 47.9271
gpt-oss-20b Q4_K_M 64.3276 61.8951 61.6072 63.2014

The units move a perplexity by 2e-4 of its value on Muse Glimmer and by 5e-3 on gpt-oss-20b, on which the backends already stand 2 to 4 % apart (its router's near ties on each backend's rounding of the activations, llama.cpp's 62.70 among them: inference.md). On the files' llama.cpp recordings (test_archs_public, whose prompts are short) Muse Glimmer kept its figures (156 of 160 choices, logits within 3.66, 0.171 on average; 157 of 160 and 4.36, 0.163 from the CPU), gpt-oss-20b its choices (158 of 160, 155 from the CPU) with its largest logit differences at 4.05 and 2.88 where they were 4.03 and 2.50, and the Bonsai files theirs (test_bonsai_public).

Decoding on these files

Decoding on the three files was below Vulkan's rate (Ternary Bonsai 2 4B 138.6 against 169.3 tokens a second, gpt-oss-20b 127 to 128 against 164, Muse Glimmer 28.7 against 36.7). Where a step goes (RX 7900 XT, 2026-11-14, OKLM_CL_PROFILE=1, each launch waited for; then the step timed with each kernel's launches left out in turn):

  • No decoder reads a table a weight. The cure of Vulkan's MXFP4 products (vulkan.md) does not apply: with the 16-entry codebooks of MXFP4, NVFP4, IQ4_NL and IQ4_XS computed in registers, gpt-oss-20b's indexed product went from 31.9 to 35.9 us and its prompt tiles from 1186 to 1220 us, so the constant arrays stay. PQ2_0's codes and IQ2_S's indexes and signs read as 32-bit words, and the activations read as vectors, gave nothing or cost (IQ2_S 236 to 274 us).
  • The products are at the memory's rate. Ternary Bonsai's head, 159 MB of PQ2_0, takes 299 us (530 GB/s) and its 169 products 2.8 ms of a step; gpt-oss-20b's indexed product of four experts is 17.6 MB in 32 us (Vulkan's takes 34) and its Q8_0 head 579 MB in 854 us; Muse Glimmer's IQ2_S product is 85 MB in 236 us (360 GB/s), its IQ3_XXS 51 MB in 113 us (450 GB/s).
  • The rest is launches. A launch whose work is nothing costs 2.2 to 3.5 us of kernel time, and a step loses about as much again for each one (a Ternary Bonsai step without its 88 elementwise launches is 0.59 ms shorter, 6.7 us a launch, where they are 3.5 us of kernel time). The host is not the bound: 1.7 us a launch in the backend's calls, 0.5 of them setting arguments. A step of Ternary Bonsai 2 4B was 612 launches (19 a block) and 6.04 ms of kernel time, of gpt-oss-20b 600 (25 a block) and 6.4 ms, of Muse Glimmer 902 (17 a block) and 31.4 ms.

What changed:

  • Ternary Bonsai 2: the rotation of a folded weight's input three stages a pass (hadamard_fold, 8.5 to 5.5 us), a step's norm and the rotation of its output in one launch (norm_fold), one rotation for the two products of a Gated DeltaNet block that read the same input (the model's graph, every backend): 523 launches and 4.87 ms of kernel time a step, the products 2.8 ms of them, the rotations 0.72, the recurrence 0.33, the elementwise ops 0.32.
  • gpt-oss-20b and Muse Glimmer: the end of a residual branch (res_norm) reads four elements an item before it writes them, 16.1 to 10.7 us on gpt-oss-20b's rows of 2880 values (49 launches a step) and 14.6 to 13.1 us on Muse Glimmer's of 6656 with the norm before it (236 a step).
  • Ternary Bonsai's prompts: the Gated DeltaNet recurrence was 44 % of a prompt's kernel time (2.14 ms a launch for 512 tokens, the state read and written in device memory at every token), the products 32 %; gated_delta_rows keeps the state in registers over the tokens (1.46 ms a launch, 36 % of the kernel time, the products 38 %). Its sums are in another order than the step kernel's: the file's perplexity on OpenCL over 254 tokens moved from 32.8443 to 32.8405 (the CPU's is 32.9058 before and after), its choices and logit distances on llama.cpp's recording did not (test_bonsai_public).

Prompt of 512 tokens / decode of 128, tokens a second (bench_llama, the best of three to five runs, before and after in the same session):

File OpenCL before OpenCL Vulkan Vulkan / OpenCL, decode
Ternary Bonsai 2 4B PQ2_0 2522 to 2582 / 137.7 to 138.6 3041 to 3050 / 163.1 3420 / 180.9 (169.3 before the shared rotation) 1.11
gpt-oss-20b Q4_K_M 3304 / 127.3 to 128.1 3323 to 3346 / 129.9 to 130.6 3656 / 163.6 1.25
Muse Glimmer 30B UD-IQ2_XXS 858 / 28.7 875 / 29.2 831 / 36.7 1.26
Llama-3.2-1B Q4_K_M (none of these kernels in a step) 14 771 to 15 338 / 398.3 to 408.9 15 097 to 15 151 / 396.1 to 411.1

What keeps gpt-oss-20b and Muse Glimmer below Vulkan is the count of their launches, not a kernel: of gpt-oss-20b's 25 launches a block, seven add a bias after a product (three of them after a gather of the experts' rows) and four are the routing (the bias, the choice, the weights, the softmax, in an order moe_route does not take), 3 us of kernel time each; Muse Glimmer's products are 26 of its 31 ms, and a norm followed by the end of a residual branch are two launches of 13 us at each of its 104 branch ends. A row kernel that adds a bias and a routing kernel for gpt-oss's order were not written.

The unit kernels at start

The float unit kernels are a second program, built at the first product that takes them: kernels.cl compiled with OKLM_UNITS_ONLY, which leaves every other kernel out (the decoders and the constants they share stay), so the 91 kernels (wm_, w2_, wi_ of the 30 weight types and w2_cvt) cost their 0.3 s once, in the first prompt of 32 columns of a model that holds such a type, and nothing for a model without one. Wall time of oklm run -n 1 on SmolLM2-135M Q4_K_M (every matrix on an 8-bit kernel; the best of six runs, 2026-11-14): 0.87 s with the unit kernels in the first program as they were, 0.55 s then with OKLM_CL_WMMA=0, 0.56 s now by default (1.12, 0.74 and 0.76 s in a later round with the CPU busy). Ternary Bonsai 2 4B with a prompt of a hundred words, which builds them: 1.78 s before, 1.76 s now. With OKLM_CL_WMMA asking for every product (1, f16, bf16) the unit kernels are in the first program, built at start, with the matrix-unit attention. When the second program does not build, the F32 tiles take these products from then on, with a notice.

The Q4_K and Q5_K prompt tiles with loader waves (2026-11-12, quiet GPU, bench_llama with 512 random tokens, the best of three, the builds alternated over three or four rounds, llama.cpp's HIP build in the same session). Two steps, each the CPU's products bit for bit (test_opencl_q8):

  • The scale arrays of a tile sized for its weight type (a Q4_K tile used those of the largest type: 36 864 bytes of local memory to 32 256) and a column's block sums as one 16-byte read instead of two: Llama-3.2-1B Q4_K_M 13 858 to 14 343 t/s, Qwen3-8B Q4_K_M 2 544 to 2 641.
  • Four loader waves beside the four that multiply (wq_q4_k, wq_q5_k: a work-group of 256 items, two steps of levels and two super-blocks of scales in local memory): 14 343 to 14 904 and 2 641 to 2 781; Llama-3.2-1B's Q4_K products 284 to 258 microseconds a launch, Qwen3-8B's 806 to 743 (OKLM_CL_PROFILE=1).
prompt before now llama.cpp ratio before / now
Llama-3.2-1B Q4_K_M, 512 tokens 13 869 15 009 17 884 0.78 / 0.84
Qwen3-8B Q4_K_M, 512 tokens 2 562 2 781 3 227 0.79 / 0.86
Llama-3.2-1B Q4_K_M, 200 / 100 / 64 tokens 10 750 / 7 930 / 5 493 11 825 / 9 070 / 6 307
Qwen3-8B Q4_K_M, 200 / 100 / 64 tokens 2 027 / 1 730 / 1 194 2 253 / 1 947 / 1 344

The other models' prompts of 512 tokens, before / now (the smaller arrays alone reach their kernels): SmolLM2-135M Q8_0 40 535 / 40 772, Q4_K_M 36 499 / 37 030, Qwen3-8B Q8_0 2 748 / 2 735, Granite-3.1-1B-A400M 17 670 / 18 095, OLMoE-1B-7B 8 581 / 8 804, GLM-4.7-Flash 2 133 / 2 185; Llama-3.2-1B decoding (no kernel of its own changed) 404 / 405.

Measured and left out: two loader waves instead of four (each item two half rows and two columns: 13 382 and 2 408 t/s, slower than no loader wave at all); eight loader waves (four for the weights, four for the activations: 15 127 against 14 992 and 2 798 against 2 769, under 1 %); the Q6_K tiles with loader waves (a Q4_K_M file's Q6_K products are 13 to 14 % of a prompt's GPU time: 15 029 against 14 905 and 2 797 against 2 780, under 1 %, not worth a third kernel body).

The indexed tiles of MoE prompts with the same loader waves (wqi_q4_k, wqi_q5_k; 2026-11-12, quiet GPU, bench_llama with 512 random tokens, the best of three a run, the builds alternated over eleven rounds, six for the last two rows; llama.cpp's HIP llama-bench in the same session, the best of its samples, its first one being 10 to 50 % slower than the others in every run that day). The plain and the indexed tiles are now one body of two shapes, and a row's 8 mins are kept as 8 bytes, widened in registers. The CPU's products bit for bit (test_opencl_q8):

prompt, 512 tokens before, mean / best now, mean / best rounds faster llama.cpp ratio before / now
OLMoE-1B-7B Q4_K_M 8 652 / 8 712 8 763 / 8 831 11 of 11 10 850 0.80 / 0.81
GLM-4.7-Flash Q4_K_M 2 184 / 2 191 2 213 / 2 223 11 of 11 2 078 1.05 / 1.07
Granite-3.1-1B-A400M Q4_K_M 18 022 / 18 186 18 038 / 18 173 5 of 11 15 529 1.17 / 1.17
Llama-3.2-1B Q4_K_M (wq_q4_k) 14 838 / 14 891 14 935 / 14 997 6 of 6
Qwen3-8B Q4_K_M (wq_q4_k) 2 768 / 2 783 2 768 / 2 773 3 of 6

A small gain, 1.3 % on OLMoE and on GLM-4.7-Flash, none on Granite: the Q4_K products of four prompts take 122.5 to 120.1 ms on OLMoE, 380 to 366 on GLM-4.7-Flash and 104 either way on Granite (OKLM_CL_PROFILE=1, both kinds of tile counted together), where the plain tiles had gained 9 %. An indexed tile is 128 weight rows by at most 64 rows of activations: for the same count of products it unpacks twice the weight levels of a plain tile (with their scales) and copies half the activation levels, and an expert's last tile is seldom full (OLMoE's 4 096 rows of a 512-token prompt are 64 an expert on average), so its loads weigh more against its products than a plain tile's; the likely reason the loader waves, four at most beside the four that multiply in a work-group of 256 items, hide less of them. Not measured further.

Several devices

Each chosen device is its own backend instance (context, queue, kernels, buffers, its own range of fake addresses). A context spreads its device layers over every instance of the selected backend's kind (oklm_context_params.n_devices, --devices N; 0 = all): consecutive layer ranges in instance order, the output head with the last range on the selected instance, in proportion to how many layers fit in each device's memory (or to their memory sizes with --gpu-layers N). Each device holds its layers' weights, KV cache, activation buffers and its own copy of the rope factor tensors; the residual stream is handed from one device to the next through the host after its last layer. The first layers can still run on the CPU (--gpu-layers below the layer count), and weight streaming (--gpu-stream) runs them on the head's device for large batches (its slots and cache mirrors counted in that device's share), the residual then handed to the first other device. test_multi_device opens the first device twice and checks that contexts spread over both instances give the logits of a one-device context exactly, for dense, GPT-2, MoE and DeepSeek fixtures, and for SmolLM2-135M with 10 layers streamed and 20 over the two devices. A forward pass of a spread context is one pass on every device it uses (oklm_backend_pass_begin_all(), the devices' locks taken in the order of their registration): the test also runs a spread context, a context on the head's device and an image encoder on the other device in three threads at once, each giving the floats it gives alone (server.md). The Vulkan instances spread a context the same way; a context never mixes OpenCL and Vulkan instances (vulkan.md).

Training

The trainer has its own kernels (src/train/train_kernels.cl), built on an instance's device, context and queue (oklm_opencl_handles()), since training needs backward passes the inference ops do not have; see training.md. OKLM_CL_PROFILE=1 also times them.

Limits

  • A model larger than the devices' memory is split with --gpu-layers, prompts stream the rest with --gpu-stream, decode runs it on the CPU; a MoE model can instead keep its routed experts in host memory (--n-cpu-moe) and every other weight on the device, the hottest experts on the device too (--expert-gpu). Layers are split across devices, not tensors (no row split, see the roadmap's closed items).
  • Every weight type OKLM reads has a matrix-product kernel; the grid IQ codebooks are constant arrays prepended to the kernel source.
  • A context that records its activations (--observe) copies the rows it wants into staging memory on the device as the pass goes (clEnqueueCopyBuffer, two rows of the stream a block) and reads it back once when the pass ends: one blocking clEnqueueReadBuffer, the keys an attention row needs queued before it without waiting (observe.md, with what it costs).
  • A build with ASan or TSan leaves the OpenCL backend out on its own (GPU drivers leak and start threads the sanitizers cannot follow), so make asan-test, tsan-test and the CI's sanitizer stages never load a driver; OKLM_OPENCL_LIB naming the ICD loader loads it anyway, which is how make asan-opencl runs the OpenCL tests under ASan / UBSan for memory errors, without leak reports.