OpenCL backend
How OKLM runs inference on a GPU through OpenCL: how the runtime is found, how a device is chosen, where tensors live, the kernels, the fallback to the CPU, how results compare with the CPU and llama.cpp, and the measured speed.
Status: maintained.
Using it
oklm devices # lists the OpenCL device when one is found
oklm run -m model.gguf -p "Hello" # auto: OpenCL when available, else the CPU
oklm run -m model.gguf -p "Hello" --backend cpu
OKLM_BACKEND=opencl oklm chat -m model.ggufFrom C, a context runs on the backend named in its parameters:
oklm_context_params cp = oklm_context_default_params(); /* backend "cpu" */
cp.backend = "opencl"; /* or "auto", or NULL for $OKLM_BACKEND */
oklm_context_new(model, &cp, &ctx);The library's default stays the CPU, so programs and tests that do
not ask get the same numbers on every machine; the oklm
commands ask for auto.
OKLM_BACKEND picks the backend when a program does not,
and OKLM_CL_DEVICES / OKLM_CL_DEVICE the
device(s). Without them every discrete GPU is used; an integrated GPU
(one sharing the host's memory, as
CL_DEVICE_HOST_UNIFIED_MEMORY says, such as a Core i7's UHD
750 beside a Radeon) only when there is no discrete one, since splitting
the layers onto it slows the discrete GPU down; the log marks it
integrated. On a gfx11 device the prompt products of the
common quantized types run on its 8-bit matrix units by default, giving
the CPU's products bit for bit (wq_<type> below;
OKLM_CL_Q8=0 for the F32 kernels), and so do the decode
products of the K-quants (mq_<type>,
OKLM_CL_Q8_MV). The prompt products of the quantized types
that have no 8-bit kernel (the grid IQ types, Q2_K, Q3_K, Q4_1, Q5_1,
IQ4_NL, IQ4_XS, MXFP4, NVFP4, the ternary and low-bit types) run on its
float matrix units by default, their activations rounded to F16 as
Vulkan's cooperative matrices take them (w2_<type>,
wm_<type>, wi_<type> below;
OKLM_CL_WMMA=0 for the F32 kernels; Types without an 8-bit
kernel). OKLM_CL_WMMA (1 or
f16, bf16) runs every prompt product on the
F16 / BF16 matrix units instead (opt-in: not the CPU's bits for the
common types); its decode products still take
mq_<type> (Qwen3-8B Q4_K_M decoding with
OKLM_CL_WMMA=1 84 to 94 t/s; its greedy tokens are the
default path's up to a near tie: the ninth of 16 after a 193-token
prompt, the seventh of 64 after a 497-token one, 2026-11-12). The
kernels' thresholds (OKLM_CL_GEMM_*,
OKLM_CL_ATT_*) and OKLM_CL_PROFILE are
described with every other variable in build.md.
Runtime loading
Nothing of OpenCL is needed to build OKLM:
src/core/opencl/cl_loader.h declares the part of the OpenCL
1.2 API the backend uses, and cl_loader.c loads the ICD
loader with dlopen (LoadLibrary on Windows)
and resolves 27 entry points. The backend is probed the first time
opencl or auto is asked for (runtime loaded,
devices listed, context and queue created, kernels compiled), so
programs that never ask pay nothing, and a binary runs unchanged on a
machine without OpenCL, where auto then selects the
CPU.
Where tensors live
- Device memory is handed out as ranges of fake addresses (never
dereferenced on the host). Tensor views keep their pointer arithmetic,
and the backend translates
datato a buffer and an offset when it launches a kernel. Every kernel takes tensors as buffer, offset, sizes and byte strides, likeoklm_tensor, so strided and permuted views need no copy. - Weights: copied once per model into one device buffer when the first OpenCL context of the model is created. Token and position embeddings stay in host memory: they are only looked up, one row per token.
- Each context allocates its KV cache and activation buffers on the device. Per step, only the positions go up and the logits come back; token ids stay on the host for the embedding lookup.
--gpu-layers N(oklm_context_params.n_gpu_layers, alias-ngl): only the last N layers and the output head go to the device, as with llama.cpp's-ngl; the first layers run on the CPU with their KV cache in host memory, and the residual stream is uploaded once per step between the two parts. The head, often the largest matrix of a small model, always gets the device.-1(default) puts there as many of the last layers as fit in the device memory (all of them when the model fits): the head, the layers' weights and KV cache, the activation buffers and, with--gpu-stream, the two streaming slots and the cache mirrors are counted against the memory minus 8 % (at least 256 MiB) for the driver and other programs, and weights and caches must each fit the largest single buffer; a notice says how many fit.OKLM_GPU_MEM_MBreplaces the device's size (to test a smaller device). One placement per model: a second context must use the same N.--n-cpu-moe N|all(oklm_context_params.n_cpu_moe, as llama.cpp's flag): the routed experts (ffn_gate_exps,ffn_up_exps,ffn_down_exps) of the first N layers stay in host memory and run on the CPU, while the rest of those layers (attention, norms, router, shared experts) goes to the device like any device layer. Experts are most of a MoE model's weights but each token reads a few of them, so they are what belongs in host memory when the model is larger than the device; with--gpu-layersleft out the fit counts the memory they leave free. Such a layer crosses devices once each way: after the router (on the device) the chosen experts, their weights and the normalized rows are read back, the shared expert is queued and started on the device, the CPU computes the routed experts (grouped indexed products, through--expert-cachewhen it is on) and their weighted sum, which is written back and added to the shared expert's output. See moe.md.--expert-gpu GB|auto(oklm_context_params.expert_gpu): a device expert tier for those layers (every layer's experts in host memory when given without--n-cpu-moe). Each layer keeps device stacks of expert slots (an equal share of GB;auto: the free device memory once the context is placed, less the fit's margin) holding its most routed experts by the usage file's counts and the routing seen, refilled every 32 steps. After the router the chosen experts are mapped to slots (a slot of zeros for the others) and uploaded; the device's indexed products over the slots (one product per slot for batches with more pairs than experts) and their weighted sum run while the CPU computes the other pairs, whose sum is written up and added. A layer holding all its experts runs as a device layer. See moe.md.--gpu-stream B(oklm_context_params.stream_batch): with--gpu-layersbelow the layer count, batches of at least B tokens (prompts, perplexity windows) run the CPU layers on the device too. Their weights go through two device slots sized for the largest of them: layer l + 1 is copied on a second, transfer queue while layer l computes from the other slot (events order the copies after the last use of a slot and the kernels after their copies; OpenCL 1.1 drivers copy on the compute queue). Those layers' caches are mirrored on the device: rows written by a streamed batch are copied back to the host, rows written on the CPU are copied up before the next streamed batch. Shorter batches (decode) keep those layers on the CPU, where they are not limited by the bus. Perplexity of Llama-3.2-1B Q4_K_M over 60 kB, 512-token windows: 14.9 s with every layer on the device, 129 s with 8 of 16 layers there and 16.4 s with the other 8 streamed, 211 s / 16.5 s with 1 layer there; the logits are those of the all-device run.
Kernels
All in src/core/opencl/kernels.cl (OpenCL C 1.2, built
with -cl-std=CL1.2), embedded in the library at build
time.
| Kernel | Ops | Notes |
|---|---|---|
ew |
add, mul, scale, silu, sigmoid, gelu, gelu_erf, swiglu, geglu, sigglu, softcap, cpy | any 4-D strides, broadcast of the second operand; F32 / F16 / BF16 elements, conversions round to nearest even like the CPU |
norm |
rms_norm, layer_norm | one work-group of 256 per row, optional weight and bias; rows of 2048 values or more, a multiple of 256, summed in runs of eight values an item (the MQ kernels' layout, so a norm folded into them gives the same bits), the others one value an item in turn |
rope |
rope | normal and NeoX layouts, partial rotation, frequency factors (with
one position a token or three: oklm_rope_parts_factors()),
linear and YaRN scaling; angles built by the same repeated products as
the CPU; one position a token, or three with a band of pairs each
(oklm_rope_parts(), M-RoPE; the fused
norm_rope and rope_kv launches take one) |
attention_tile64, attention_tile128,
attention_tile256 |
attention | prompts (16 rows or more, a row a query of one head) with heads of
at most 256 dimensions: a work-group of 128 items per block of 32 rows
(16 for heads over 128), the rows the queries of the query heads that
read one KV head (a divisor of the group: Qwen3-4B's 4 heads take 8
queries a block, so a 4-token piece fills half a block and each key is
read once for its 4 heads), walks the keys the block can see in tiles of
32 (16), each key tile (transposed) and value tile loaded once into
local memory for the whole block; the items of a query each score a few
keys of the tile, then update the query's online softmax and their share
of its output dimensions; causal window and softcap as below. F16 tiles
loaded eight halves at a time, block-quant caches dequantized 32
elements at a time. Used by default for heads up to 128, with a
block-quant cache or from 128 keys (OKLM_CL_ATT_KEYS) with
a float cache: the kernel below measured faster for heads of 256
(Gemma's), and for spans under 128 (the threshold was 1024 until
2026-11-08, when 512-token prompts measured 1.1 to 1.25 times faster on
the tiles: Llama-3.2-1B 11 786 to 14 787 t/s, Qwen3-8B Q4_K_M 2 262 to 2
641, with the matrix units). With fewer blocks than keep the device busy
(2 per compute unit, 4 for heads over 128), the keys of each block are
split into up to 16 ranges of at least 4 tiles, aimed at four times that
many work-groups, each writing its queries' running max, sum and
unscaled output for attention_merge. Block-quant caches
measured no faster on these kernels and keep the one below. Queries that
see each other (mutual: an image's rows in Gemma 3) take
these kernels too, and attention_tile64h /
128h / attention_wm64h / 128h
below: a last key is given (last, -1 for causal rows),
every block walks the keys up to it and no key before it is masked for
coming after a query, the window still starting from each query's own
position; under the same thresholds, so Gemma 3 27B's heads of 128 are
on the tiles and the 4B's and 12B's heads of 256 are not: measured on
2026-11-14 (RX 7900 XT, OKLM_CL_PROFILE=1, Gemma 3 4B with
llama.cpp's test image: 256 rows over 261 keys, 8 heads of 256 on 4, in
34 layers, and the 10 rows of text after them) 20.0 ms on the kernel
below, 25.6 to 35.5 ms on attention_tile256
(OKLM_CL_ATT_BLOCKS 168, 64 and 1). The output accumulators
are indexed by compile-time bounds, so they stay in registers |
attention_tile64h, attention_tile128h |
attention | the same for F16 caches (heads a multiple of 8, up to 128): the key
and value tiles kept as halves in local memory (their bits, read through
vload_half: no fp16 extension needed), about two thirds of
the float kernels' local memory, so more work-groups share a compute
unit; the key tile written by consecutive items at consecutive keys (no
bank conflicts), the next tile loaded into registers while one is used.
Qwen3-4B at 16k keys: 512 prompt tokens in 1584 ms against 3380, a
16-token piece in 62 ms against 215 |
attention_wm64h, attention_wm128h |
attention | with OKLM_CL_WMMA (unless
OKLM_CL_WMMA_ATT=0), in place of
attention_tile64h / attention_tile128h (heads
a multiple of 16, up to 128; the same arguments, rows, key tiles of 32,
masks, split of the keys and attention_merge): both
products on the matrix units as 16 x 16 x 16 blocks of
v_wmma_f32_16x16x16_f16 with F32 sums, F16 inputs whatever
OKLM_CL_WMMA's type (the caches are F16). The 32 queries of
a block are rounded to F16 once into local memory; per key tile, the
four waves each take one 16 x 16 block of the scores (the scale and soft
cap applied to the F32 sums, then the masks), items 0 to 31 each run one
row's softmax step (the probabilities rounded to F16, their sum added as
rounded), and P V runs as two row blocks by the head's 16-wide column
blocks, half of them a wave, the accumulators rescaled row by row; the
value tile is kept transposed in local memory for that product. Results
within the rounding of the queries and probabilities to F16 (2e-4 at
worst against the CPU on values and keys in [-1, 1],
test_opencl_wmma) |
attention_bi64, attention_bi80,
attention_bi128 |
attention | bidirectional attention (an encoder's rows: the vision encoders'
blocks, Whisper's encoder, an embedding model's): from 16 rows and
OKLM_CL_ATT_KEYS keys (with OKLM_CL_ATT_BLOCKS
set: from 16 rows; 0: never), F32 or F16 keys and values,
heads a multiple of 4 up to 64, 80 (SigLIP's 72, Qwen2-VL's 80) and 128,
the variants differing by the local memory their bound takes. The
work-groups of the prompt kernels: 128 items per block of 32 rows (a row
a query of one head, the heads that read one KV head sharing a block) by
tiles of 32 keys; no position, window, mask or block table. One buffer
of local memory holds a tile's keys (transposed), then, once every score
is taken, its values: with the probabilities and the rows' maxima a
work-group takes 16 KB for heads up to 80 (14 KB up to 64, 23 KB up to
128) where a tile of keys, one of values and the scores took 30 (26,
43), and local memory bounds how many work-groups a compute unit runs at
once. An item reads its share of a tile into registers, four elements a
load, while the items still compute: the values during the scores, the
next tile's keys while the values are added. The four items of a row
hold its query in registers, score 8 keys each and keep the scores in
registers, share their maximum, turn their own scores into probabilities
once the row's four maxima give the tile's (an online softmax, one
exponential a score), and add the tile's values to their quarter of the
output; the sums are the ones of the layout with two tiles, in the same
order. A head that is not a multiple of 16 is laid out as the next one
(72 as 80), its last dimensions zeros that are not written. With fewer
blocks than fill the device the keys of each block are split into ranges
of 4 tiles or more, merged by attention_merge (the split of
the prompt kernels). With a span of keys a query
(oklm_attention_spans(): the windowed blocks of
Qwen2.5-VL's encoder, a window's rows seeing each other) the kernels
read a table of first key and one past the last a query: a block walks
the tiles from the first key any of its rows sees to the last, each row
keeping the scores of its own span, so the windows of a block are one
launch whatever their sizes and wherever they fall in the blocks of 32
rows; the keys are then one range, from 16 rows whatever the keys'
count. No other kernel reads the table: BF16 or block-quant keys, a head
over 128 or not a multiple of 4, fewer than 16 rows or
OKLM_CL_ATT_BLOCKS=0 return
OKLM_ERR_UNSUPPORTED, and the caller attends span by span
(one launch a window, as before). The sums are the
attention kernel's in another order: results within the
tests' tolerance of the CPU's (2e-6 + 2e-4 relative). Measured on
2026-11-14 (RX 7900 XT, OKLM_CL_PROFILE=1): the 27 blocks
of Gemma 3's encoder (4 096 rows, 16 heads of 72) 2.64 s of attention on
the attention kernel, 0.50 s on these with two tiles and
0.40 s with one buffer (0.46 s on the variant of 128, whose work-group
takes 23 KB); Whisper large-v3-turbo's 32 blocks (1 500 rows, 20 heads
of 64) 0.44 s, 0.095 s and 0.085 s; the 32 blocks of Qwen2-VL 2B on a
picture of 2 000 x 2 000 pixels (4 900 rows, 16 heads of 80) 0.91 s
-> 0.72 s; a decode row unchanged (Llama 3.2 1B Q4_K_M,
bench_llama: pp512 14 802 to 14 989 before, 14 814 to 14
847 after; tg128 398 to 403 and 406 to 409) |
attention, attention_merge |
attention | grouped-query, causal, with queries that see each other
(mutual, an image's rows in Gemma 3: the last key they all
see is a parameter, the window's start stays each query's own) or
bidirectional (an encoder's rows: the cache's last key and no window,
more queries than keys allowed); the first take the prompt tiles where
causal rows do (heads up to 128 by default) and this kernel otherwise,
the second has its own (attention_bi64 and the others,
above) and comes here with what they do not take (BF16 or block-quant
keys or values, heads over 128 or not a multiple of 4, fewer than 16
rows or than OKLM_CL_ATT_KEYS keys, few blocks over fewer
than 8 tiles of keys). What this kernel takes of the models on disk was
listed on 2026-11-14 (RX 7900 XT, OKLM_CL_PROFILE=1, each
launch waited for): the decode steps of every model (one row); prompts
under 16 rows or under OKLM_CL_ATT_KEYS keys; Gemma 3's
image rows on heads of 256 (34 launches, 17.5 to 18.3 ms of a prompt's
0.95 s of kernel time, faster here than on
attention_tile256); the cross-attention of Whisper's
decoder, one row over the encoder's 1 500 F32 keys (38 us a launch: 4.4
ms of jfk.wav with large-v3-turbo, 39 ms of 0.62 s with large-v3's 32
decoder blocks); a text encoder's single sequence over fewer than 128
keys (8.7 to 10.6 us a launch, where attention_bi64 took
10.9 to 20.1 us with OKLM_CL_ATT_BLOCKS=1); a batch of
several sequences left this kernel on 2026-11-14: it is one launch of
attention_bi64 a block, each row given its sequence's rows
as its span of keys (oklm_attention_spans()), where it was
a launch a sequence here (364 lines of a page in 9 batches: bge-small 4
308 launches of 12.4 us and 60 on the tiles, 55.8 ms of 0.162 s of
kernel time -> 108 launches of 102 us, 11.0 ms of 0.118 s;
nomic-embed-text 81.2 ms of 0.380 s -> 21.7 ms of 0.331 s; the
vectors 2.7e-7 and 3e-7 from the CPU's, 2e-7 and 4e-7 before). The
cross-attention of Whisper's decoder stays: a launch reads the 1 500
keys and values of 20 heads of 64, 15.4 MB in F32, in 38 us, the
memory's rate (about 400 GB/s), and a block's row needs the block before
it, so neither the work-groups' shape nor a launch for several blocks
has anything to win (large-v3 on jfk.wav, 2026-11-14: 2 112 launches
here for the decoder's self- and cross-attention, 47.7 ms, and 1 056 of
attention_merge, 5.4 ms, of 0.49 s of kernel time). No
caller gives bidirectional attention BF16 or block-quant keys (an
encoder's keys are its activations, F32) and no encoder on disk has
heads over 128 or not a multiple of 4 (32, 64, 72 and 80): those
operands have no tiles; sliding window, softcap, F32, F16, BF16 or
block-quant cache (F16 keys read eight at a time, F32 keys four,
block-quant keys dequantized 32 at a time, values element by element).
Keys are split across work-groups when there are few (head, token) pairs
(decode), each split runs an online softmax over chunks of 512 keys, and
a merge kernel combines the partial results; with attention sinks
(gpt-oss: one score per query head, a device tensor) the merge kernel
adds each head's sink to its sum as one more term, so an op with sinks
always goes through it: the per-token kernel then splits the keys in two
at least, and the tiled prompt kernels write their partials even for one
range; every item of a work-group takes part in the value sum; heads up
to 1024 wide (latent attention's shared 576-wide head, the 512-wide
heads of Gemma 4's blocks without window: each item takes several
dimensions over all keys) |
get_rows, softmax_rows |
get_rows, softmax | gathers from float tables on the device (MoE routing; host tables such as token embeddings and quantized tables fall back), unmasked softmax of short rows (router probabilities), one work-group of 64 per row |
moe_select, moe_weights,
moe_sum |
moe_select, moe_weights, moe_sum | MoE routing without a read-back: expert choice with selection bias and groups (one work-group of 64 per token, scores in local memory, up to 1024 experts; up to 256 each competing expert counts those ranked before it and the k-th writes slot k, one barrier in place of a reduction per pick; the same order as the CPU, so the same experts), weights of the chosen experts, and the weighted sum of the K expert outputs in rank order |
ssm_conv, gated_delta,
gated_delta_rows |
ssm_conv, gated_delta | a Gated DeltaNet block's short causal convolution (one item per
channel over the tokens in order, the ring updated) and its recurrence.
gated_delta: one work-group of 128 per value head, a state
row an item, read and written in device memory at each token (a decode
step). gated_delta_rows, from 8 tokens with key heads up to
128: the state in registers over the tokens, read before the first and
written after the last; a work-group of 256 items per 32 state rows of a
head, an item holding 4 rows by 4 key dimensions, the rows' sums reduced
through local memory. Its sums are not in gated_delta's
order: the CPU's values within the op test's tolerance (1e-5 absolute,
1e-4 relative), not the step kernel's bits. A prompt of 512 tokens of
Ternary Bonsai 2 4B: 2.14 to 1.46 ms a launch (2026-11-14) |
quantize_rows, hadamard |
KV cache | cache writes into Q8_0, Q4_0, Q4_1, Q5_0, Q5_1 with the CPU's algorithms (one item per block); Walsh-Hadamard rotation, one work-group per group, butterflies in local memory |
hadamard_fold |
hadamard with signs, reordered heads or groups above 256
(oklm_hadamard_fold()) |
the input of a product with a weight stored in a rotated basis
(Ternary Bonsai 2: H (D x), blocks of 512 and 4096) and a
rotated embedding row turned back (D (H z)): a work-group
of 64 items per group with 16 KiB of local memory (groups up to 4096),
each element times its sign before or after the butterflies, the value
heads of a row read in the order they were folded in; one launch where
the CPU makes three passes. Three stages a pass: the 8 elements of a
pass's group are a stride apart and take each other's values alone, so
an item computes their twelve butterflies in registers, the first pass
reading the input and the last one writing the output (a block of 512:
three passes and two exchanges through local memory where nine stages
made nine; the butterflies are the same, so the same bits; one or two
stages left over run a pair an item) |
norm_fold |
rms_norm + hadamard (oklm_hadamard_fold()) |
a decode step's RMS norm and the rotation of its output in one
launch (OKLM_CL_FUSE): the backend holds the norm of one
row (weighted, a multiple of 256 values from 2048, the widths the norm
kernel sums in runs) and runs it with the rotation that reads its output
when that comes next, else on its own; one work-group of 256 items per
group of the rotation, each summing the row's squares as
norm does, writing its part of the norm's output and
rotating it in hadamard_fold's passes. The same bits as the
two ops (test_opencl, on every device) |
moe_group, gi_<type>,
wi_<type> |
matmul_id | indexed products of more rows than experts (prompts of MoE layers):
moe_group (one work-group) counts each expert's rows, lays
out a table of tiles of up to 64 rows per expert and places the rows
(atomically: their order within an expert varies, the products do not),
then one launch of gi_<type> (the
gm_<type> steps, 64 weight rows by one tile entry,
rows read and written through the table; wi_<type> on
the matrix units of a gfx11 device for the types
wm_<type> takes there) multiplies every tile; chunks
of 8192 rows. Replaced reading the experts back and one product per
expert (OLMoE-1B-7B prompts 4175 to 8614 t/s with the matrix units,
2026-11-08) |
q8_act, wq_<type>,
wqi_<type> |
matmul, matmul_id | on a gfx11 device by default (OKLM_CL_Q8), in place of
the tiled kernels for products of 64 columns or more
(OKLM_CL_Q8_COLS) with Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 or Q5_0
weights: the CPU's products, bit for bit. q8_act rounds the
activations to 8 bits exactly as the CPU does (Q8_K blocks of 256 for
the K-quants, Q8_0 blocks of 32 for the others: the same scale, levels
and block sums; its divisions correctly rounded through double
precision), then wq_<type> (64 weight rows by 128
columns, four waves, 128 values of k a step) puts the weights' integer
levels in local memory unscaled and multiplies them with
v_wmma_i32_16x16x16_iu8 (exact 32-bit sums): each block's
integer sum is scaled and added as the CPU's dot product does it
(oklm_vec_dot_q4_k_q8_k() and the others: the sub-block
scales applied to integers, the mins term of Q4_K / Q5_K as two more
16-deep products of the 8 mins by the block sums split into 7-bit
halves, then d isum - dmin summs per super-block, in order
of k, no fused multiply-adds, K never split). The Q4_K and Q5_K tiles
have four more waves that only load (a work-group of 256 items): while
the four multiplying waves work on a step, the loader waves put the next
step's weight levels, scales and activation levels in a second buffer of
local memory, so a step's loads hide behind the previous step's products
and the multiplying waves keep their registers for the accumulators (the
same sums in the same order, the same bits).
wqi_<type> does the same for the grouped indexed
products of MoE prompts (128 weight rows by a tile entry of up to 64
rows; four waves for Q6_K, Q8_0, Q4_0 and Q5_0, eight with the same
loader waves for Q4_K and Q5_K, a loader item taking both halves of a
weight row and half a column: 64 000 bytes of local memory). Decoding:
mq_<type> below |
mvg_<type>, mqg_<type> |
matmul_id + matmul_id + swiglu | a MoE layer's gate and up products and their SwiGLU in one decode
launch (OKLM_CL_FUSE and OKLM_CL_MVG): the
backend holds the gate product, then the up product on the same
activations and experts, and runs the three when the SwiGLU of their
outputs comes; mvg_<type> on the MV kernels (every
type), mqg_<type> on the 8-bit ones (Q4_K, Q5_K,
Q6_K: the column rounded once); each product summed in its own kernel's
order, the up output written too, so the same bits as the three ops |
cpy2 |
two cpy | two copies of F32 tensors into float-like ones in one launch
(OKLM_CL_FUSE: a held copy runs with the next one): a
decode step's keys and values into their caches |
res_norm |
moe_sum + add + scale + add + rms_norm | the end of a residual branch in one launch
(OKLM_CL_FUSE and OKLM_CL_RES), one work-group
of 256 items per token: the backend holds the weighted sum of the expert
outputs, the add of another tensor into it (the shared experts' output),
its scale (a residual multiplier), all in place on the branch's output,
and the add of that output into the residual stream, and runs them with
the RMS norm of the new stream when it comes next (the next block's
input norm), else on their own as soon as anything else comes; each
stage optional but the add into the stream, a lone op run as it is. Each
stage repeats its kernel's operations (no fused multiply-adds, the
norm's sum of squares over the same elements per item and the same
reduction; the branch's output and the stream written as the separate
launches write them), so the same bits as the two to five ops. An item
reads four of its elements before it writes any of them (a row of 2880
values is eleven or twelve elements an item, and a load, an operation
and a store an element waited for the memory once an element): 16.1 to
10.7 us a launch on gpt-oss-20b, 49 launches a step
(OKLM_CL_PROFILE=1, 2026-11-14) |
rope_kv |
rope (+ rope) + two cpy | a step's rotation of its queries and keys and the writes of its keys
and values into their caches in one launch (OKLM_CL_FUSE
and OKLM_CL_ROPE_KV): the backend holds the rotation in
place of the queries (or of the queries and keys as rows of one tensor),
the rotation of the keys with the same parameters, and the copy of the
rotated keys into their cache (F32, F16 or BF16), and runs them with the
copy of the values that follows; an item per dimension pair rotates as
rope does and, on a key row, stores the pair into the cache
too (converted as ew converts it), the items after those
copy the values. The same bits as the three or four ops; a quantized
cache keeps its own kernel |
moe_route, norm_rope |
softmax + moe_select + moe_weights; two rms_norm + rope | launches fused by the backend (OKLM_CL_FUSE): a
held-back op is run with the ones that complete it, or on its own as
soon as anything else comes; the fused kernels repeat the separate
kernels' operations, so the results are the same bits
(moe_route: a MoE layer's softmax router, expert choice and
weights in one launch per layer, and a sigmoid router's likewise,
OKLM_CL_ROUTE_SIGMOID; norm_rope: Qwen3's
per-head query and key norms and their rotation, one launch instead of
three or four) |
attention_mla |
attention | prompts of latent attention's shared head (heads of 257 to 576
dimensions, F16 cache, the values the keys' own tensor: DeepSeek V2 /
V3, GLM-4.7-Flash; OKLM_CL_ATT_MLA): blocks of 16 rows
(query, head) pairs, the heads of a token sharing each tile of 16 keys,
kept as halves in local memory for both the scores and the values; each
of the 16 items of a row holds a slice of its query and output (the
scores' partial sums added by the item of each key), the keys split into
ranges as the other tiles when there are few blocks. GLM-4.7-Flash
prompts 1 337 to 2 174 t/s (2026-11-08) |
mm_<type> |
matmul | every weight type OKLM computes with: F32, F16, BF16, Q4_0, Q4_1,
Q5_0, Q5_1, Q8_0, Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, IQ4_NL, IQ4_XS, MXFP4,
NVFP4, TQ1_0, TQ2_0, Q1_0, Q2_0, the grid IQ types, and PQ2_0 and PTQ1_0
(the two types of PrismML's llama.cpp fork, decoded like Q2_0 and TQ1_0
with a scale per 128). A work-group of 64 items computes 4 weight rows
for a block of 8 activation columns; 16 items per row take groups of 32
weights in turn (adjacent items read adjacent blocks), dequantize each
group once and use it for every column; fixed reduction order; optional
accumulation into the destination (oklm_matmul_acc); with
an index buffer, each batch index multiplies the expert matrix it names
(oklm_matmul_id, one column per (token, rank) pair) |
mq_<type> |
matmul, matmul_id | on a gfx11 device by default (OKLM_CL_Q8_MV, with
OKLM_CL_Q8), in place of mv_<type> for
one-column products (decode, and the indexed products of MoE decode
steps) with Q4_K, Q5_K or Q6_K weights, rows of 2048 values or more and
1024 rows or more: the CPU's products, bit for bit. Each work-group of
256 items first rounds its activation column to Q8_K in local memory
exactly as q8_act does (each wave 256 values at a time, the
largest magnitude found with cross-lane DPP steps, the divisions through
double precision), so no launch is added; then each wave takes two
weight rows: eight items a super-block, each forming 32 levels' products
with the 4-way 8-bit dot products (v_dot4_i32_iu8), the
scales applied to the integer sums, the sums of the eight items added
(DPP), and each super-block's d isum - dmin summs added to
the row's sum in order of k (the wave's four groups read with
readlane); the activations are read once for both rows,
every load of a wave step (two super-blocks a group) issued before any
term is added. Measured slower and left to mv_<type>:
Q8_0, Q4_0 and Q5_0 (their 8-bit twins 3 to 7 % slower: Qwen3-8B Q8_0
decoding 66.6 to 64.8 t/s), and shorter or fewer rows, where each
work-group pays the column's rounding for little work (SmolLM2-135M's
576 x 1536 down products: its decoding 412 to 370 t/s; Granite's and
OLMoE's rows of 1024 values 2 to 3 % slower).
mqn_<type>: the same on the input of a held
decode-step RMS norm (Fewer launches) |
mv_<type> |
matmul, matmul_id | in place of mm_<type> for one activation column
(decode) and for the indexed products (one column per expert row): the
same rows, lanes, group order and reduction tree, so the same bits,
without reducing the seven columns that are not there (Llama-3.2-1B
decode 349 to 362 t/s, 2026-11-08) |
gm_<type> |
matmul | the same types, for prompts (32 columns or more, at least 256 tiles
of 64 x 64 outputs, K a multiple of 32): a work-group of 8 x 8 items per
tile; for each step of 32 values of k, each item dequantizes one group
of 32 weights and loads 32 activations into local memory (k-major), then
multiplies its 8 x 8 outputs, two float8 local reads per k
for 64 products. The next step's weights and activations are loaded into
registers while the current step multiplies; sums run in a fixed order.
Products with fewer tiles split K into up to 8 ranges of at least 4
steps (one work-group per tile and range, partial tiles in a scratch
buffer, added in range order by gm_reduce, so the sums keep
a fixed order); with K too short for that, the row kernels, which have
more work-groups to spread over the device. Each output is one running
sum over the K terms of its range, where the CPU and the row kernels
keep several partial sums: further from the CPU's where the terms
cancel, most through Whisper's encoder (Ultravox's embeddings on jfk.wav
3.0e-4 relative from the CPU's, 3.1e-5 with
OKLM_CL_GEMM_TILES=0; the vision encoders 1.4e-5 to
5.3e-5). Measured on 2026-11-14 and not kept: a sum a step of 32 terms
added to the total gives 4.5e-5 and one every four steps 1.1e-5, for 7 %
more time in the kernel (Gemma 3's encoder 228 -> 245 ms of
gm_f16, Whisper's 140 -> 149 ms: the second set of 64
sums takes as many registers again); the partial sums added into the
output buffer instead, 313 ms. These are the F32 products: 14 to 15
Tflop/s on the encoders (gm_f16: 0.14 s of Whisper
large-v3-turbo's encoder, 0.22 s of Gemma 3's), against 0.035 s and
0.050 s on the matrix units with OKLM_CL_WMMA=1, which
round the activations to F16 as Vulkan's do |
wm_<type> |
matmul | on a gfx11 device, in place of gm_<type> (same
arguments, tiles, split K and gm_reduce), by default for
quantized weights that have no 8-bit kernel and whose product was not
asked with the activations at F32's precision
(oklm_matmul_wide()), for every product with
OKLM_CL_WMMA (1, f16,
bf16), for none with OKLM_CL_WMMA=0: the RDNA3
matrix units through clang's
__builtin_amdgcn_wmma_f32_16x16x16_{f16,bf16}_w32, which
the ROCm OpenCL compiler accepts in OpenCL C (built with
-DOKLM_WMMA, so other devices never compile them). Per step
of 32 values of k, each of the 64 items dequantizes one weight row's
group and loads one activation column, both rounded to F16 (or BF16)
into local memory; each of the two waves multiplies 32 weight rows by
the 64 columns as 2 x 4 blocks of 16 x 16 with F32 accumulators, two
16-deep products per step. The next step's weight group and activation
column are read into registers while the current step multiplies, as
gm_<type> does. A zero weight is given to the units
as +0 whatever its sign (their sums depend on it in their last bits).
Results within the inputs' rounding of the F32 kernels |
w2_<type>, w2_cvt |
matmul | in place of wm_<type> (the same products) for
prompts of 64 columns or more: w2_cvt rounds the
activations to F16 (BF16) once per product, then a work-group of four
waves takes 64 weight rows by 128 columns, 64 values of k a step: each
item dequantizes one row's group of 32 and copies one column's 64 halves
as 16-byte vectors, each wave multiplies 32 rows by 64 columns (the same
2 x 4 blocks as wm_<type>); each dequantized weight
serves 128 columns instead of 64 and no work-group rounds activations;
the next step's weight group and 64 halves are read into registers while
the current step multiplies. The split of K is
gm_<type>'s. Results within the inputs' rounding, and
the bits wm_<type> gives for the same product (the
same halves summed in the same order; Qwen3-8B Q4_K_M prompts 1 386 to 2
262 t/s, Llama-3.2-1B 9 323 to 11 909, 2026-11-08) |
The float kernels multiply dequantized weights by F32 activations.
The CPU (like llama.cpp) rounds the activations to 8 bits first for
quantized weights, so their results are closer to the exact product of
the quantized weights than the CPU's. On a gfx11 device the prompt
products of Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 and Q5_0 weights take the 8-bit
kernels instead (wq_<type>, by default): the
activations rounded as the CPU rounds them and the CPU's products bit
for bit, so a prompt's greedy tokens follow the CPU's where the float
kernels part from it at a near tie (Qwen3-8B Q4_K_M, OLMoE-1B-7B: the 16
greedy tokens after a 190-token prompt are the CPU's, the float kernels'
differ from the first or second; 2026-11-08). Decoding (one column)
takes the 8-bit kernels too for Q4_K, Q5_K and Q6_K weights, rows of
2048 values and more, 1024 rows and more (mq_<type>,
the CPU's products bit for bit; OKLM_CL_Q8_MV); the other
one-column products keep the float row kernels.
Fewer launches
Generation of small models is bound by the number of kernels per token (each costs a few microseconds of dispatch even when its work is tiny), so the Llama-family forward pass merges what it can without adding work:
- in the device copy of the weights, Q, K and V matrices of the same type are stored back to back and also described as one matrix (likewise the Q/K/V biases and the gate/up matrices): one product instead of three, and one instead of two;
- K heads follow the Q heads in the fused output, so one RoPE launch rotates both;
- the output and down projections add into the residual stream
(
oklm_matmul_acc) instead of writing a temporary and adding it.
Per token: SmolLM2-135M from 514 launches to 322, Llama-3.2-1B from
274 to 194. The backend also holds back the ops of two short sequences
and runs each as one launch when it completes
(OKLM_CL_FUSE; the same bits, anything else first runs what
is held): a MoE layer's routing (softmax, expert choice, weights:
moe_route, OLMoE-1B-7B decoding 264 to 278 t/s), a MoE
layer's gate and up products with their SwiGLU
(mvg_<type>, mqg_<type>;
2026-11-11, decoding 128 tokens, three runs each: Granite-3.1-1B-A400M
281 to 302 t/s, OLMoE-1B-7B 276 to 289, GLM-4.7-Flash 70.9 to 72.4;
Granite's 472 launches a token to 425) and Qwen3's per-head query and
key norms with their rotation (norm_rope; Qwen3-8B decoding
within the noise: its products, not its launches, take the time). A
decode step's RMS norm (one row of 2048 to 8192 values, a multiple of
256) is held back too and folded into the following one-column product
when that product takes an MQ kernel (mqn_<type>):
each work-group of that kernel already reads the whole column to round
it, so it loads the norm's input and weights once, sums the squares in
the norm kernel's order, scales and rounds; its first work-group writes
the norm's output for the ops that read it later. The norm kernel sums
rows of those widths in the same order (runs of eight values an item,
the MQ kernels' layout of a column), so the fused and the separate norm
give the same bits (test_opencl_q8). Every norm of
Llama-3.2-1B's decode steps goes (202 to 169 launches a token, decoding
about 367 to 385 t/s), and so do Qwen3-8B Q4_K_M's. Two copies that
follow each other (a step's new keys and values into a converting cache)
run as one launch (cpy2, each element converted as the
ew kernel converts it: the same bits), one launch fewer a
layer (Llama-3.2-1B decoding about 1 to 2 % faster). The end of a
residual branch is one launch too (res_norm,
OKLM_CL_RES): the weighted sum of the expert outputs, the
add of the shared experts' output, the residual scale, the add into the
residual stream and the RMS norm of the new stream (the next block's
first op), which were up to five launches of a few microseconds of work
each; and so is a step's rotation with its cache writes
(rope_kv, OKLM_CL_ROPE_KV: the rotation of the
queries and of the keys, the copies of the keys and the values into
their caches, two to four launches before). Both give the bits of the
separate ops (test_opencl: every stage combination and
layout against the ops kept apart, and the launch counts). A sigmoid
router's routing (GLM-4.7-Flash) is one launch as a softmax router's is
(moe_route, OKLM_CL_ROUTE_SIGMOID). Decoding
128 tokens after 16, alternated runs, 2026-11-12, without / with the
three: Granite-3.1-1B-A400M 306 to 395 t/s (424 launches a token to
268), OLMoE-1B-7B 296 to 318 (249 to 209), GLM-4.7-Flash 73.8 to 77.9
(1259 to 1076), SmolLM2-135M Q4_K_M 422 to 471 (344 to 300),
Llama-3.2-1B 391 to 400 (153 to 129) (benchmarks.md).
A step's norm of a model whose weights are stored in a rotated basis
(Ternary Bonsai 2) is followed by the rotation of its output, not by a
product: the two are one launch (norm_fold), 65 launches
fewer in a step of Ternary Bonsai 2 4B. With the rotation's three stages
a pass (8.5 to 5.5 us a launch, OKLM_CL_PROFILE=1) and one
transformed input for the two products of a Gated DeltaNet block that
read the same one (24 launches fewer, on every backend), that file's
step went from 612 launches and 6.04 ms of kernel time to 523 and 4.87,
and its decoding from 138.6 to 162.5 tokens a second (Vulkan 169.3 to
177.3 with the transformed input shared; bench_llama, 128
tokens, the best of four or five runs, 2026-11-14). Two other fusions
were measured and left out: RMSNorm inside the float matrix product
(every work-group recomputes the norm of its columns: prompt processing
halved, Llama-3.2 decode at 500 tokens of context fell from 165 to 103
tokens/s) and SwiGLU inside the down projection (TinyLlama: 8 % fewer
launches for 44 % more GPU time). Files whose Q/K/V or gate/up matrices
have different types (Qwen2.5 Q4_K_M) keep separate products.
Fallback
An op whose operands or types no kernel takes (an operand in host
memory, a weight type without a mm_ kernel, softmax,
cross-entropy, argmax...) runs on the CPU backend over host copies of
the byte ranges it touches, and the destination range is written back.
It is logged once per op at level INFO. During inference only the
embedding lookup falls back, by design.
Verification
| Test | What | Result |
|---|---|---|
tests/core/test_opencl.c |
every kernel against the CPU on random shapes (the float kernels of
every type: OKLM_CL_Q8=0, OKLM_CL_WMMA=0)
(sizes not multiple of the work-group sizes, broadcasts, a permuted
view, 1 300 keys across attention chunks and splits, bidirectional
attention on its tiles (rows and keys that are not multiples of a tile's
32, heads of 36, 64, 72, 80, 100 and 128, one head and several, two
heads a KV head, a soft-cap, few blocks whose keys are split into 2 and
8 ranges) and off them (BF16 keys, a head of 66), the same tiles with a
span of keys a query (test_attention_spans: windows of
unequal sizes, of 1 to 100 rows, smaller than a tile and over several,
blocks of rows that straddle windows, one window over everything, 2 to
16 heads, two a KV head, a soft-cap, spans that are not windows; BF16
keys, a head of 66 and 9 rows refused), all 21 weight types at decode,
prompt and batched shapes, accumulation; the products a convolution is
made of on a device (test_conv_views: one product a
position of the kernel, with the activations at F32's precision, over a
view of every stride-th row from that position on, the first written and
the others accumulated: kernels of 3 and 7, strides of 1 and 2, 96 to 1
280 channels, against the convolution in doubles, within 6.8e-8 of
sum |
w x |
tests/core/test_opencl_q8.c |
on a gfx11 device: the 8-bit matrix-unit products
(wq_<type>, wqi_<type>) of Q4_K,
Q5_K, Q6_K, Q8_0, Q4_0 and Q5_0 weights against the CPU's (partial
tiles, batches, K not a multiple of the step, one to eight super-blocks
of the K-quants and an odd count over several tiles, accumulation, rows
and columns of zeros, ties of magnitude, grouped indexed products shared
or not, of 300 and 800 rows over several tiles an expert and three tiles
of weight rows, partial ones, with one, two and five super-blocks); the
one-column products of the K-quants (mq_<type>: rows
of 2048 to 4864 values, batches, accumulation, indexed products of one
and three tokens); the prompt products of the types without an 8-bit
kernel as the device runs them by default (200 rows by 150 columns in a
batch of two and 200 by 40, K split, for each of the 21 types; 1030 by
1000 over whole and partial tiles for IQ2_XXS, IQ2_S, IQ3_XXS, IQ3_S,
Q3_K, MXFP4 and PQ2_0; indexed products of 300 rows over 8 experts for
IQ2_XXS and MXFP4) against a double reference, and the same products
asked at F32's precision; a 100-token SmolLM2 Q8_0 / Q4_K_M prompt
against the CPU |
every product bit-identical to the CPU's; the default products of the other types (each of the 21) within 2e-3 of the sum of magnitudes (1.1e-4 at worst measured), those asked at F32's precision within 1e-5 (3.1e-7) and not the first ones' bits; logits within 0.61 to 0.63 of the CPU's (0.80 to 0.92 with the float kernels), the same top token |
tests/core/test_opencl_wmma.c |
on a gfx11 device: a probe kernel calling clang's
__builtin_amdgcn_wmma_f32_16x16x16_f16_w32 and
..._bf16_w32 from OpenCL C on one 16 x 16 x 16 tile, and on
one whose zeros are +0 then -0; the wm_<type> and
w2_<type> kernels (OKLM_CL_WMMA) on F32,
F16, BF16, Q8_0, Q4_0, Q4_K and Q6_K weights, partial tiles, batches and
split K (a range of K ending on half a W2 step), and on F32 weights
whose zeros are +0 then -0; the prompt attention of F16 caches on the
matrix units (attention_wm64h,
attention_wm128h: grouped heads, keys split and merged,
heads of 64 and 80, a window, a soft cap) against the CPU's |
the probe within 4e-8 of the CPU relative to the sum of magnitudes (182 of 256 values exact: the units do not round their F32 sums as IEEE additions; with 30 zeros of A given as -0 instead of +0, 36 of 256 sums change in their last bits, both within 8.4e-8); products within 2e-3 (F16) / 1.6e-2 (BF16) of the sum of magnitudes, 1.5e-4 at worst measured with F16, and the same bits whatever the sign of the zero weights; attention within 1e-3 (2e-4 at worst) and not the CPU's bits. Skipped elsewhere |
tests/model/test_model.c |
tiny llama / qwen3 / gpt2 against the float64 reference, every layer on the device and split between CPU and device | same as the CPU: 9.5e-7, 1.5e-5, 2.6e-5 (split: 9.5e-7, 1.4e-5, 3.7e-5); batched and token-by-token logits bit-identical |
tests/model/test_infer_public.c |
SmolLM2 F16 / Q8_0 / Q4_K_M and GPT-2 Q4_K_M against the float64
reference (the float kernels: OKLM_CL_Q8=0) |
4.6e-5, 3.0e-5, 5.6e-5 and 3.2e-4 (logits near 100); every greedy choice equal |
| same | the corpus against llama.cpp | F16 and BF16: identical continuations. Quantized: logit noise of 1 to 2.4 (llama.cpp rounds activations), every point where the continuations part a near tie for llama.cpp (margin at most 0.44) |
Kernel results are deterministic: the same inputs give the same bits on every run.
Speed
oklm run with the first 2000 bytes of Tiny Shakespeare
as prompt (about 500 tokens) and 128 generated tokens, AMD Radeon
gfx1100 (42 compute units, 20 GiB), tokens per second. CPU: the CPU
backend on an Intel Core i7-11700K, 16 threads (2026-10-30, quiet
machine). OpenCL: best of three runs, 2026-10-30, measured while other
heavy jobs were running on the machine, so a quiet machine can only do
better; "first" is the first version of the backend, "now" the current
one.
| file | CPU prompt | OpenCL prompt, first / now | CPU generation | OpenCL generation, first / now |
|---|---|---|---|---|
| SmolLM2-135M F16 | 888 | 8 977 / 14 807 | 47 | 224 / 317 |
| SmolLM2-135M Q8_0 | 372 | 8 501 / 14 614 | 60 | 249 / 340 |
| SmolLM2-135M Q4_K_M | 397 | 7 819 / 14 286 | 81 | 242 / 310 |
| SmolLM2-135M BF16 checkpoint | 746 | 8 528 / 14 660 | 72 | 243 / 322 |
| Qwen2.5-0.5B Q4_K_M | 141 | 4 117 / 10 013 | 33 | 192 / 251 |
| Qwen3-0.6B Q8_0 | 95 | 2 597 / 6 446 | 21 | 122 / 203 |
| Llama-3.2-1B Q4_K_M | 93 | 1 816 / 5 189 | 21 | 165 / 268 |
| TinyLlama-1.1B Q4_K_M | 100 | 1 812 / 4 563 | 24 | 184 / 265 |
| GPT-2 Q4_K_M | 409 | 9 112 / 22 718 | 101 | 421 / 552 |
From first to now: the fused launches above, a Q6_K dequantizer without a per-element branch (Llama-3.2's Q6_K products went from 348 to 49 microseconds per launch), 4 rows per work-group in the matrix products (half the GPU time of one row per work-group on prompts), and the tiled matrix products for prompts (10 to 28 % on prompts: an 8192 x 2048 Q4_K product of 512 columns takes 1.23 ms instead of 1.96, 14 TFLOPS against 9), then the tiled prompt attention (Llama-3.2-1B's attention 42 to 12 ms per 500-token prompt; Qwen3-0.6B prompts 3 549 to 6 446 t/s, Phi-3.5-mini 849 to 1 343).
The matrix units (OKLM_CL_WMMA, 2026-11-08, quiet GPU,
oklm run with the first 2000 bytes of Tiny Shakespeare and
64 greedy tokens, best of three): Qwen3-8B Q4_K_M prompts 831 t/s with
the F32 kernels, 1 320 with F16 inputs (1.59 x; its Q4_K products 2.93
to 1.54 ms a launch on average), 1 137 with BF16 (1.37 x); Llama-3.2-1B
Q4_K_M 4 938, 8 713 (1.76 x) and 6 887. Decoding (a column at a time,
the row kernels) is unchanged. The 64 greedy tokens are the F32 kernels'
with F16 inputs on both models, and with BF16 on Qwen3-8B; Llama-3.2-1B
with BF16 parts at one token. Perplexity of Qwen3-8B (first 30 KB of
Tiny Shakespeare, 512-token windows): 19.9952 (F32 kernels), 19.9972
(F16), 20.0010 (BF16); an F32 KV cache instead of the default F16 moves
it as much (19.9962). Off by default: the greedy answers can part at
near ties.
The prompt attention on the matrix units too
(attention_wm<hd>h, with OKLM_CL_WMMA,
2026-11-11, quiet GPU, best of three; OKLM_CL_WMMA_ATT=0
for the half tiles): Qwen3-8B Q4_K_M prompts of 497 tokens 2 478 to 2
619 t/s, of 2 059 tokens 1 992 to 2 476 (its attention 291 to 97 ms of
the GPU time at 2 000 tokens, 23.4 to 9.4 ms at 500); Llama-3.2-1B
Q4_K_M 14 175 to 15 370 and 10 840 to 14 277. Perplexity (the same 30
KB, 512-token windows) 19.9965 to 19.9962 on Qwen3-8B, 41.4164 to
41.4163 on Llama-3.2-1B; the 64 greedy tokens after the first 2 000
bytes part at the third or fourth token on both, as those of the
matrix-unit products part from the default kernels' (Qwen3-8B: at the
fifth).
Measured and left out (2026-11-11): prompt products through rocBLAS's
gemm_ex, as llama.cpp's hipBLAS route does (each weight
dequantized to F16 rows in shared virtual memory, the activations
rounded to F16, F32 accumulation into an F16 product, widened back).
rocBLAS alone runs Qwen3-8B's shapes at 36 to 71 TFLOPS with an F16
product (12 TFLOPS with an F32 one) at 512 and 2048 columns; but the
8-bit kernels the products take by default already do a 500-token
prompt's 180 products in 150 ms, while rocBLAS took 161 ms for the
products alone (about 125 at the rates it reaches by itself) before the
dequantization (114 ms with a first kernel, at least about 20 at the
device's bandwidth) and the conversions (about 30 ms with the OpenCL
queue drained before each product). Qwen3-8B Q4_K_M prompts: 2385 t/s
with the default kernels, 1595 through rocBLAS (500 tokens); 1994
against 1665 at 2000 tokens. The code was not kept.
The matrix-unit products reading the next step while they multiply
(wm_<type>, w2_<type>, with
OKLM_CL_WMMA=1; 2026-11-12, quiet GPU,
bench_llama, the best of three, without / with alternated
over three rounds, t/s). An item keeps the next step's dequantized
weight group and activation column in registers (32 floats and 32 floats
in wm_<type>, 32 floats and 64 halves in
w2_<type>) beside its 64 float accumulators, where
the same thing on the 8-bit Q4_K tiles, with 64 integer accumulators
more, had halved the prompts:
prompt, OKLM_CL_WMMA=1 |
without | with | llama.cpp |
|---|---|---|---|
| Llama-3.2-1B Q4_K_M, 512 tokens | 16 228 | 17 553 | 17 966 |
| Qwen3-8B Q4_K_M, 512 tokens | 2 825 | 3 172 | 3 218 |
Llama-3.2-1B Q4_K_M, 48 tokens (wm_<type>) |
5 759 | 6 863 | |
Qwen3-8B Q4_K_M, 48 tokens (wm_<type>) |
1 216 | 1 523 | |
| SmolLM2-135M Q8_0, 512 tokens | 46 027 | 50 440 | |
| SmolLM2-135M Q4_K_M, 512 tokens | 44 317 | 44 966 | |
| Qwen3-8B Q8_0, 512 tokens | 2 996 | 3 276 | |
| Granite-3.1-1B-A400M Q4_K_M, 512 tokens | 17 792 | 18 595 | |
| OLMoE-1B-7B Q4_K_M, 512 tokens | 8 835 | 8 916 |
Llama-3.2-1B decoding (other kernels) 407 / 409. With it the opt-in
matrix-unit mode runs the prompts of Llama-3.2-1B and Qwen3-8B Q4_K_M
close to llama.cpp's rate (0.98 and 0.99; the default 8-bit kernels,
which give the CPU's bits, 0.84 and 0.86). The outputs stay within the
inputs' rounding (test_opencl_wmma, the same worst
differences) but are not always the bits of the kernels that did not
read ahead: with F16 inputs, of the test's 25 products the four of
w2_<type> with Q4_0 or Q6_K weights changed (10 to 47
% of their outputs, by at most 9.3e-8 of the sum of magnitudes; both
versions within 4.8e-7 of the exact product of the rounded inputs);
nothing changed with BF16 inputs. The cause is the sign of zero weights,
not the steps read: a level of zero times a negative scale (Q4_0's and
Q6_K's scales are signed) is -0, and the matrix units' sums depend, in
their last bits, on the sign of a zero input (the test's probe: 36 of
256 sums of a tile change when 30 zeros of A become -0).
wm_<type> always rounded such a weight to the half
-0; w2_<type> as compiled before stored +0 and now
stores -0 as well. With every zero forced to +0 in both versions, the 25
products are the same bits before and after, the range of K ending on
half a step included.
Since then the kernels give the units +0 for every zero weight
(wm_cvt_w: + 0.0f on the dequantized weight
before it is rounded, in wm_<type>,
w2_<type> and wi_<type>), so the
bits no longer depend on the sign a dequantization, or the compiler,
leaves on a zero. The four w2_<type> products are
again the bits of the kernels that did not read ahead; three of the
test's wm_<type> products with Q6_K or Q4_0 weights,
which always had -0, changed in their turn, and with BF16 inputs the
same seven products changed (all within the same worst differences: the
test's figures did not move). test_opencl_wmma runs a
product of F32 weights with one zero in eight as +0 then as -0 on
wm_f32 and on w2_f32 and asks for the same
bits (both failed before). Two forms were measured (2026-11-12,
bench_llama, 512 tokens, OKLM_CL_WMMA=1, the
builds alternated over three and four rounds, the best run of each,
t/s):
prompt, OKLM_CL_WMMA=1 |
zeros as dequantized | + 0.0f on the weight (kept) |
the half -0 replaced in wm_cvt |
|---|---|---|---|
| Llama-3.2-1B Q4_K_M, 512 tokens | 17 260 | 17 440 | 16 618 |
| Qwen3-8B Q4_K_M, 512 tokens | 3 166 | 3 179 | 2 982 |
The two forms give the same bits on the test's 25 products, F16 and
BF16. Comparing the rounded half with 0x8000 in wm_cvt
(weights and activations) cost 4 to 6 % in every round and was not kept;
the float addition costs nothing measurable. What it leaves: with F16
inputs a negative weight of at most half the least subnormal half
(2^-25) still rounds to -0 (a property of the value, not of the
compiler), and the activations keep their sign. The perplexities of the
table below with zero weights as +0: 41.4163 and 19.9962 with
OKLM_CL_WMMA=1, the figures of the kernels that did not
read ahead.
The same sign was the whole difference between
w2_<type> and wm_<type>, which
W2_KERNEL's comment had put down to the compiler scheduling
the dequantization differently. Checked with a build that could send
every product to wm_<type>, on the test's 25
products: before the kernels read ahead, w2_<type>
and wm_<type> differed on four, exactly the products
of 64 columns or more with Q4_0 or Q6_K weights (where
w2_<type> stored +0 and wm_<type>
-0); once both stored -0 they gave the same bits on all 25, and they
still do with every zero as +0, F16 and BF16. The two kernels sum the
same halves in the same order with the same split of K, so their results
are the same bits; why the earlier w2_<type> lost the
sign in compilation was not looked into.
Quality of the mode after that change (2026-11-12,
oklm ppl on the first 30 KB of Tiny Shakespeare, 3 825
tokens scored in 512-token windows; oklm run with its first
2 000 bytes, 497 tokens, and 64 greedy tokens), the kernels of before
rebuilt and run in the same session:
| perplexity | Llama-3.2-1B Q4_K_M | Qwen3-8B Q4_K_M |
|---|---|---|
| default (8-bit kernels, the CPU's products) | 41.5610 | 20.0465 |
F32 kernels (OKLM_CL_Q8=0) |
41.4166 | 19.9948 |
OKLM_CL_WMMA=1, before / now |
41.4163 / 41.4168 | 19.9962 / 19.9965 |
OKLM_CL_WMMA=1 OKLM_CL_WMMA_ATT=0, before / now |
41.4164 / 41.4159 | 19.9965 / 19.9987 |
OKLM_CL_WMMA=bf16, before / now |
41.4211 / 41.4211 | 20.0056 / 20.0056 |
The F16 figures moved by at most 0.0022 (1.1e-4 of the value), less
than the distance between the F32 kernels and the mode and far less than
the distance from either to the default kernels, whose activations are
rounded to 8 bits as the CPU's; BF16 did not move (the same bits). The
64 greedy tokens: on Llama-3.2-1B the F16 mode gives the F32 kernels' 64
tokens before and now, and parts from the default kernels' at the fifth;
on Qwen3-8B it gave other tokens than before from the fifth (as it parts
from the F32 kernels' at the fifth and from the default kernels' at the
seventh), the fifth token being a near tie that every pair of kernel
families splits on. With BF16 the tokens are those of before on both
models (Llama-3.2-1B parts from the F32 kernels' at the fifth, Qwen3-8B
at the nineteenth). The matrix-unit attention against the half tiles
(OKLM_CL_WMMA_ATT=0): the same 64 tokens on Qwen3-8B,
parting at the fourteenth on Llama-3.2-1B (at the fifth on both
before).
The mode against the CPU backend (2026-11-12, the same text, windows
and Q4_K_M files, oklm ppl -c 512 --backend cpu then
--backend opencl; the matrix-unit kernels with every zero
weight as +0):
| perplexity | Llama-3.2-1B Q4_K_M | Qwen3-8B Q4_K_M |
|---|---|---|
| CPU backend | 41.4967 | 20.1612 |
| OpenCL, default (8-bit kernels) | 41.5610 | 20.0465 |
OpenCL, F32 kernels (OKLM_CL_Q8=0) |
41.4166 | 19.9948 |
OpenCL, OKLM_CL_WMMA=1 |
41.4163 | 19.9962 |
The mode is 0.19 % under the CPU on Llama-3.2-1B and 0.82 % under it on Qwen3-8B; the default OpenCL path is 0.15 % over and 0.57 % under. The mode is thus 0.04 and 0.25 % of the value further from the CPU than the default path is, on 3 825 tokens, where neither the sign nor the size of such gaps says which backend is the better one. The default path's prompt products are the CPU's bits, yet its perplexity is not the CPU's: which of its other operations account for the 0.15 and 0.57 % was not looked into.
Types without an 8-bit kernel
The 8-bit kernels take Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 and Q5_0. A
prompt product of any other quantized type (the grid IQ types, Q2_K,
Q3_K, Q4_1, Q5_1, IQ4_NL, IQ4_XS, MXFP4, NVFP4, the ternary and low-bit
types) ran on gm_<type>, the F32 tiles, and a file
made of such types paid it on every matrix: Muse Glimmer 30B UD-IQ2_XXS
(IQ2_XXS, IQ2_S, IQ3_XXS and IQ3_S matrices) read a prompt at a third of
Vulkan's rate. Since 2026-11-14 a gfx11 device gives these products its
float matrix units by default (w2_<type> from 64
columns, wm_<type> under,
wi_<type> for the indexed products of a MoE prompt),
the activations rounded to F16 as Vulkan's cooperative matrices take
them; the 8-bit kernels keep their types, float weights and a product
asked with the activations at F32's precision
(oklm_matmul_wide(): the encoders) keep the F32 tiles, and
OKLM_CL_WMMA=0 gives every one of these products the F32
tiles again. Other devices have the F32 tiles alone.
Where a 512-token prompt of Muse Glimmer went (RX 7900 XT,
OKLM_CL_PROFILE=1, each launch waited for; 52 launches a
kernel a prompt, 104 for IQ2_XXS), time a launch on the F32 tiles and on
the units:
| Weights | F32 tiles | Share of the prompt's kernel time | Matrix units |
|---|---|---|---|
| IQ2_S | 18.6 ms | 49.0 % | 4.62 ms |
| IQ3_XXS | 10.6 ms | 28.0 % | 2.42 ms |
| IQ2_XXS | 2.22 ms | 11.7 % | 0.57 ms |
| IQ3_S | 2.05 ms | 5.4 % | 0.53 ms |
| the prompt, kernel time | 1.98 s | 0.56 s |
Prompt of 512 tokens / decode of 128, tokens a second
(bench_llama, the best of three runs, 2026-11-14; the
decode products are the row kernels' either way):
| File | F32 tiles (OKLM_CL_WMMA=0) |
Matrix units (default) | Vulkan |
|---|---|---|---|
| Muse Glimmer 30B UD-IQ2_XXS | 253.3 / 29.4 | 865 to 868 / 29.5 | 834 to 839 / 37.1 to 37.3 |
| Ternary Bonsai 2 4B PQ2_0 | 1276 / 136.7 | 2544 / 137.3 | 3411 / 172.4 |
| gpt-oss-20b Q4_K_M (MXFP4 experts, indexed) | 1680 / 126.1 | 3313 / 126.4 | 3738 / 163.9 |
| Llama-3.2-1B Q4_K_M (no such type) | 14 568 to 14 644 / 391 to 406 | 14 545 to 14 643 / 404 to 407 |
The decode rates of this table and Ternary Bonsai's prompts were taken up after it: Decoding on these files.
Perplexity (oklm ppl -c 512, 1275 tokens of this
repository's documentation: the first 14 000 bytes of
harness.md), the CPU, the F32 tiles, the units, Vulkan:
| File | CPU | F32 tiles | Matrix units (default) | Vulkan |
|---|---|---|---|---|
| Muse Glimmer 30B UD-IQ2_XXS | 30.7146 | 30.7098 | 30.7160 | 30.7671 |
| Ternary Bonsai 2 4B PQ2_0 | 47.9792 | 47.9281 | 47.9281 (28.6707 against 28.6734 after the first window) | 47.9271 |
| gpt-oss-20b Q4_K_M | 64.3276 | 61.8951 | 61.6072 | 63.2014 |
The units move a perplexity by 2e-4 of its value on Muse Glimmer and
by 5e-3 on gpt-oss-20b, on which the backends already stand 2 to 4 %
apart (its router's near ties on each backend's rounding of the
activations, llama.cpp's 62.70 among them: inference.md).
On the files' llama.cpp recordings (test_archs_public,
whose prompts are short) Muse Glimmer kept its figures (156 of 160
choices, logits within 3.66, 0.171 on average; 157 of 160 and 4.36,
0.163 from the CPU), gpt-oss-20b its choices (158 of 160, 155 from the
CPU) with its largest logit differences at 4.05 and 2.88 where they were
4.03 and 2.50, and the Bonsai files theirs
(test_bonsai_public).
Decoding on these files
Decoding on the three files was below Vulkan's rate (Ternary Bonsai 2
4B 138.6 against 169.3 tokens a second, gpt-oss-20b 127 to 128 against
164, Muse Glimmer 28.7 against 36.7). Where a step goes (RX 7900 XT,
2026-11-14, OKLM_CL_PROFILE=1, each launch waited for; then
the step timed with each kernel's launches left out in turn):
- No decoder reads a table a weight. The cure of Vulkan's MXFP4 products (vulkan.md) does not apply: with the 16-entry codebooks of MXFP4, NVFP4, IQ4_NL and IQ4_XS computed in registers, gpt-oss-20b's indexed product went from 31.9 to 35.9 us and its prompt tiles from 1186 to 1220 us, so the constant arrays stay. PQ2_0's codes and IQ2_S's indexes and signs read as 32-bit words, and the activations read as vectors, gave nothing or cost (IQ2_S 236 to 274 us).
- The products are at the memory's rate. Ternary Bonsai's head, 159 MB of PQ2_0, takes 299 us (530 GB/s) and its 169 products 2.8 ms of a step; gpt-oss-20b's indexed product of four experts is 17.6 MB in 32 us (Vulkan's takes 34) and its Q8_0 head 579 MB in 854 us; Muse Glimmer's IQ2_S product is 85 MB in 236 us (360 GB/s), its IQ3_XXS 51 MB in 113 us (450 GB/s).
- The rest is launches. A launch whose work is nothing costs 2.2 to 3.5 us of kernel time, and a step loses about as much again for each one (a Ternary Bonsai step without its 88 elementwise launches is 0.59 ms shorter, 6.7 us a launch, where they are 3.5 us of kernel time). The host is not the bound: 1.7 us a launch in the backend's calls, 0.5 of them setting arguments. A step of Ternary Bonsai 2 4B was 612 launches (19 a block) and 6.04 ms of kernel time, of gpt-oss-20b 600 (25 a block) and 6.4 ms, of Muse Glimmer 902 (17 a block) and 31.4 ms.
What changed:
- Ternary Bonsai 2: the rotation of a folded weight's input three
stages a pass (
hadamard_fold, 8.5 to 5.5 us), a step's norm and the rotation of its output in one launch (norm_fold), one rotation for the two products of a Gated DeltaNet block that read the same input (the model's graph, every backend): 523 launches and 4.87 ms of kernel time a step, the products 2.8 ms of them, the rotations 0.72, the recurrence 0.33, the elementwise ops 0.32. - gpt-oss-20b and Muse Glimmer: the end of a residual branch
(
res_norm) reads four elements an item before it writes them, 16.1 to 10.7 us on gpt-oss-20b's rows of 2880 values (49 launches a step) and 14.6 to 13.1 us on Muse Glimmer's of 6656 with the norm before it (236 a step). - Ternary Bonsai's prompts: the Gated DeltaNet recurrence was 44 % of
a prompt's kernel time (2.14 ms a launch for 512 tokens, the state read
and written in device memory at every token), the products 32 %;
gated_delta_rowskeeps the state in registers over the tokens (1.46 ms a launch, 36 % of the kernel time, the products 38 %). Its sums are in another order than the step kernel's: the file's perplexity on OpenCL over 254 tokens moved from 32.8443 to 32.8405 (the CPU's is 32.9058 before and after), its choices and logit distances on llama.cpp's recording did not (test_bonsai_public).
Prompt of 512 tokens / decode of 128, tokens a second
(bench_llama, the best of three to five runs, before and
after in the same session):
| File | OpenCL before | OpenCL | Vulkan | Vulkan / OpenCL, decode |
|---|---|---|---|---|
| Ternary Bonsai 2 4B PQ2_0 | 2522 to 2582 / 137.7 to 138.6 | 3041 to 3050 / 163.1 | 3420 / 180.9 (169.3 before the shared rotation) | 1.11 |
| gpt-oss-20b Q4_K_M | 3304 / 127.3 to 128.1 | 3323 to 3346 / 129.9 to 130.6 | 3656 / 163.6 | 1.25 |
| Muse Glimmer 30B UD-IQ2_XXS | 858 / 28.7 | 875 / 29.2 | 831 / 36.7 | 1.26 |
| Llama-3.2-1B Q4_K_M (none of these kernels in a step) | 14 771 to 15 338 / 398.3 to 408.9 | 15 097 to 15 151 / 396.1 to 411.1 |
What keeps gpt-oss-20b and Muse Glimmer below Vulkan is the count of
their launches, not a kernel: of gpt-oss-20b's 25 launches a block,
seven add a bias after a product (three of them after a gather of the
experts' rows) and four are the routing (the bias, the choice, the
weights, the softmax, in an order moe_route does not take),
3 us of kernel time each; Muse Glimmer's products are 26 of its 31 ms,
and a norm followed by the end of a residual branch are two launches of
13 us at each of its 104 branch ends. A row kernel that adds a bias and
a routing kernel for gpt-oss's order were not written.
The unit kernels at start
The float unit kernels are a second program, built at the first
product that takes them: kernels.cl compiled with
OKLM_UNITS_ONLY, which leaves every other kernel out (the
decoders and the constants they share stay), so the 91 kernels
(wm_, w2_, wi_ of the 30 weight
types and w2_cvt) cost their 0.3 s once, in the first
prompt of 32 columns of a model that holds such a type, and nothing for
a model without one. Wall time of oklm run -n 1 on
SmolLM2-135M Q4_K_M (every matrix on an 8-bit kernel; the best of six
runs, 2026-11-14): 0.87 s with the unit kernels in the first program as
they were, 0.55 s then with OKLM_CL_WMMA=0, 0.56 s now by
default (1.12, 0.74 and 0.76 s in a later round with the CPU busy).
Ternary Bonsai 2 4B with a prompt of a hundred words, which builds them:
1.78 s before, 1.76 s now. With OKLM_CL_WMMA asking for
every product (1, f16, bf16) the
unit kernels are in the first program, built at start, with the
matrix-unit attention. When the second program does not build, the F32
tiles take these products from then on, with a notice.
The Q4_K and Q5_K prompt tiles with loader waves (2026-11-12, quiet
GPU, bench_llama with 512 random tokens, the best of three,
the builds alternated over three or four rounds, llama.cpp's HIP build
in the same session). Two steps, each the CPU's products bit for bit
(test_opencl_q8):
- The scale arrays of a tile sized for its weight type (a Q4_K tile used those of the largest type: 36 864 bytes of local memory to 32 256) and a column's block sums as one 16-byte read instead of two: Llama-3.2-1B Q4_K_M 13 858 to 14 343 t/s, Qwen3-8B Q4_K_M 2 544 to 2 641.
- Four loader waves beside the four that multiply
(
wq_q4_k,wq_q5_k: a work-group of 256 items, two steps of levels and two super-blocks of scales in local memory): 14 343 to 14 904 and 2 641 to 2 781; Llama-3.2-1B's Q4_K products 284 to 258 microseconds a launch, Qwen3-8B's 806 to 743 (OKLM_CL_PROFILE=1).
| prompt | before | now | llama.cpp | ratio before / now |
|---|---|---|---|---|
| Llama-3.2-1B Q4_K_M, 512 tokens | 13 869 | 15 009 | 17 884 | 0.78 / 0.84 |
| Qwen3-8B Q4_K_M, 512 tokens | 2 562 | 2 781 | 3 227 | 0.79 / 0.86 |
| Llama-3.2-1B Q4_K_M, 200 / 100 / 64 tokens | 10 750 / 7 930 / 5 493 | 11 825 / 9 070 / 6 307 | ||
| Qwen3-8B Q4_K_M, 200 / 100 / 64 tokens | 2 027 / 1 730 / 1 194 | 2 253 / 1 947 / 1 344 |
The other models' prompts of 512 tokens, before / now (the smaller arrays alone reach their kernels): SmolLM2-135M Q8_0 40 535 / 40 772, Q4_K_M 36 499 / 37 030, Qwen3-8B Q8_0 2 748 / 2 735, Granite-3.1-1B-A400M 17 670 / 18 095, OLMoE-1B-7B 8 581 / 8 804, GLM-4.7-Flash 2 133 / 2 185; Llama-3.2-1B decoding (no kernel of its own changed) 404 / 405.
Measured and left out: two loader waves instead of four (each item two half rows and two columns: 13 382 and 2 408 t/s, slower than no loader wave at all); eight loader waves (four for the weights, four for the activations: 15 127 against 14 992 and 2 798 against 2 769, under 1 %); the Q6_K tiles with loader waves (a Q4_K_M file's Q6_K products are 13 to 14 % of a prompt's GPU time: 15 029 against 14 905 and 2 797 against 2 780, under 1 %, not worth a third kernel body).
The indexed tiles of MoE prompts with the same loader waves
(wqi_q4_k, wqi_q5_k; 2026-11-12, quiet GPU,
bench_llama with 512 random tokens, the best of three a
run, the builds alternated over eleven rounds, six for the last two
rows; llama.cpp's HIP llama-bench in the same session, the
best of its samples, its first one being 10 to 50 % slower than the
others in every run that day). The plain and the indexed tiles are now
one body of two shapes, and a row's 8 mins are kept as 8 bytes, widened
in registers. The CPU's products bit for bit
(test_opencl_q8):
| prompt, 512 tokens | before, mean / best | now, mean / best | rounds faster | llama.cpp | ratio before / now |
|---|---|---|---|---|---|
| OLMoE-1B-7B Q4_K_M | 8 652 / 8 712 | 8 763 / 8 831 | 11 of 11 | 10 850 | 0.80 / 0.81 |
| GLM-4.7-Flash Q4_K_M | 2 184 / 2 191 | 2 213 / 2 223 | 11 of 11 | 2 078 | 1.05 / 1.07 |
| Granite-3.1-1B-A400M Q4_K_M | 18 022 / 18 186 | 18 038 / 18 173 | 5 of 11 | 15 529 | 1.17 / 1.17 |
Llama-3.2-1B Q4_K_M (wq_q4_k) |
14 838 / 14 891 | 14 935 / 14 997 | 6 of 6 | ||
Qwen3-8B Q4_K_M (wq_q4_k) |
2 768 / 2 783 | 2 768 / 2 773 | 3 of 6 |
A small gain, 1.3 % on OLMoE and on GLM-4.7-Flash, none on Granite:
the Q4_K products of four prompts take 122.5 to 120.1 ms on OLMoE, 380
to 366 on GLM-4.7-Flash and 104 either way on Granite
(OKLM_CL_PROFILE=1, both kinds of tile counted together),
where the plain tiles had gained 9 %. An indexed tile is 128 weight rows
by at most 64 rows of activations: for the same count of products it
unpacks twice the weight levels of a plain tile (with their scales) and
copies half the activation levels, and an expert's last tile is seldom
full (OLMoE's 4 096 rows of a 512-token prompt are 64 an expert on
average), so its loads weigh more against its products than a plain
tile's; the likely reason the loader waves, four at most beside the four
that multiply in a work-group of 256 items, hide less of them. Not
measured further.
Several devices
Each chosen device is its own backend instance (context, queue,
kernels, buffers, its own range of fake addresses). A context spreads
its device layers over every instance of the selected backend's kind
(oklm_context_params.n_devices, --devices N; 0
= all): consecutive layer ranges in instance order, the output head with
the last range on the selected instance, in proportion to how many
layers fit in each device's memory (or to their memory sizes with
--gpu-layers N). Each device holds its layers' weights, KV
cache, activation buffers and its own copy of the rope factor tensors;
the residual stream is handed from one device to the next through the
host after its last layer. The first layers can still run on the CPU
(--gpu-layers below the layer count), and weight streaming
(--gpu-stream) runs them on the head's device for large
batches (its slots and cache mirrors counted in that device's share),
the residual then handed to the first other device.
test_multi_device opens the first device twice and checks
that contexts spread over both instances give the logits of a one-device
context exactly, for dense, GPT-2, MoE and DeepSeek fixtures, and for
SmolLM2-135M with 10 layers streamed and 20 over the two devices. A
forward pass of a spread context is one pass on every device it uses
(oklm_backend_pass_begin_all(), the devices' locks taken in
the order of their registration): the test also runs a spread context, a
context on the head's device and an image encoder on the other device in
three threads at once, each giving the floats it gives alone (server.md). The Vulkan
instances spread a context the same way; a context never mixes OpenCL
and Vulkan instances (vulkan.md).
Training
The trainer has its own kernels
(src/train/train_kernels.cl), built on an instance's
device, context and queue (oklm_opencl_handles()), since
training needs backward passes the inference ops do not have; see training.md.
OKLM_CL_PROFILE=1 also times them.
Limits
- A model larger than the devices' memory is split with
--gpu-layers, prompts stream the rest with--gpu-stream, decode runs it on the CPU; a MoE model can instead keep its routed experts in host memory (--n-cpu-moe) and every other weight on the device, the hottest experts on the device too (--expert-gpu). Layers are split across devices, not tensors (no row split, see the roadmap's closed items). - Every weight type OKLM reads has a matrix-product kernel; the grid IQ codebooks are constant arrays prepended to the kernel source.
- A context that records its activations (
--observe) copies the rows it wants into staging memory on the device as the pass goes (clEnqueueCopyBuffer, two rows of the stream a block) and reads it back once when the pass ends: one blockingclEnqueueReadBuffer, the keys an attention row needs queued before it without waiting (observe.md, with what it costs). - A build with ASan or TSan leaves the OpenCL backend out on its own
(GPU drivers leak and start threads the sanitizers cannot follow), so
make asan-test,tsan-testand the CI's sanitizer stages never load a driver;OKLM_OPENCL_LIBnaming the ICD loader loads it anyway, which is howmake asan-openclruns the OpenCL tests under ASan / UBSan for memory errors, without leak reports.