Vulkan backend
How OKLM runs on a GPU through Vulkan (ADR-0023): how the API is loaded, how devices are chosen, how tensors are addressed, how shaders are built and dispatched, which file holds which ops, the fallback to the CPU, and how it is tested.
Status: working on AMD (RADV) and Intel (ANV); NVIDIA untested. Steps
V1 to V3 of the roadmap's GPU
backend section are done: Vulkan is the main GPU backend and OpenCL
the fallback. Which of the two auto takes on a device is in
auto.
Using it
oklm run -m model.gguf -p "Hello" --backend vulkan
OKLM_BACKEND=vulkan oklm chat -m model.gguf
OKLM_VK_DEVICE=Intel oklm run -m model.gguf -p "Hello" --backend vulkanFrom C, oklm_backend_get("vulkan") probes the backend on
first use (as opencl does) and returns NULL without a
usable device; oklm_backend_select("vulkan") then gives the
CPU with a warning. oklm_vulkan_device_count() and
oklm_vulkan_device_info() describe the instances. The
backend registers below the CPU's priority, so auto keeps
choosing OpenCL or the CPU until Vulkan reaches parity (auto);
OKLM_BACKEND_ORDER=vulkan,opencl,cpu makes
auto take it before.
Building
The shaders are GLSL compute shaders compiled to SPIR-V at build time
by glslc (Debian's glslc package, or shaderc),
targeting Vulkan 1.2, and embedded in the library. Without
glslc the library is built without the Vulkan backend
(OKLM_HAVE_VULKAN undefined,
src/core/vulkan/vk_none.c in its place), as is a build with
FORCE_NO_VULKAN=1; GLSLC=path names another
compiler. make show-config prints
HAVE_VULKAN.
Nothing of Vulkan is needed otherwise: no SDK, no headers, no library
to link. The MinGW build (make MINGW=1) embeds the same
SPIR-V (the compiler runs on the build host) and loads
vulkan-1.dll at run time; it builds, but has not been run
on Windows.
Runtime loading
src/core/vulkan/vk_loader.h declares the part of the
Vulkan 1.3 API the backend uses (types, constants, structures with the
layouts of vulkan_core.h), and vk_loader.c
opens the loader with dlopen (libvulkan.so.1;
vulkan-1.dll with LoadLibrary on Windows),
takes vkGetInstanceProcAddr from it and every other entry
point through vkGetInstanceProcAddr and
vkGetDeviceProcAddr. OKLM_VULKAN_LIB names
another loader (empty: Vulkan disabled). A sanitizer build loads no
driver unless OKLM_VULKAN_LIB names one, as with
OpenCL.
Devices
The probe creates one Vulkan instance (API 1.3), lists the physical devices and logs each one: name, driver and its version, Vulkan version, memory, kind (discrete, integrated, CPU implementation), subgroup sizes, and whether it has cooperative matrices, integer dot products (and whether they are accelerated), buffer device addresses, 16-bit floats, 8-bit integers:
Vulkan device 0: AMD Radeon RX 7900 XT (RADV NAVI31) (radv Mesa 26.1.6-1, Vulkan 1.4.354), 20.0 GiB, discrete; subgroup 64 (32-64), cooperative matrix, integer dot product (accelerated), buffer device address, fp16, int8
Vulkan device 1: Intel(R) Graphics (RKL GT1) (Intel open-source Mesa driver Mesa 26.1.6-1, Vulkan 1.4.354), 93.5 GiB, integrated (shares the host's memory); subgroup 32 (8-32), integer dot product (accelerated), buffer device address, fp16, int8
Vulkan device 2: llvmpipe (LLVM 21.1.8, 256 bits) (llvmpipe Mesa 26.1.6-1 (LLVM 21.1.8), Vulkan 1.4.354), 124.6 GiB, CPU implementation; subgroup 8 (8-8), cooperative matrix, integer dot product, buffer device address, fp16, int8
Each chosen device becomes a backend instance: vulkan,
vulkan1, ...
OKLM_VK_DEVICES: a comma-separated list of indexes or name substrings (of the device or the driver name:radv,Intel), one instance each (a device listed twice gives two independent instances), orall: every GPU, discrete ones first.OKLM_VK_DEVICE: one device, whenOKLM_VK_DEVICESis unset.- Neither: every discrete GPU; integrated (or virtual) GPUs only when there is no discrete one, the rule of the OpenCL backend.
- A CPU implementation (llvmpipe) is taken only when named.
A device needs Vulkan 1.2, buffer device addresses, 64-bit integers
and 16-bit storage in shaders, and a compute queue; one without is
logged and left out. The logical device enables, where the device has
them, 16-bit floats, 8-bit and 16-bit integers, 8-bit storage, subgroup
size control and full subgroups, integer dot products, cooperative
matrices (VK_KHR_cooperative_matrix) and
VK_EXT_memory_budget (the free memory the backend reports),
so the groups of ops can use them without touching device creation. The
queue is a compute-only family when there is one.
Addressing: buffer device addresses
Every buffer is created with a buffer device address (Vulkan 1.2),
and a tensor's data is that address, plus a per-instance
bias (k << 52 for instance k) that keeps two
instances' ranges apart. A shader gets the addresses of its tensors and
their byte strides in its push constants and reads and writes through
GL_EXT_buffer_reference; there are no descriptor sets.
The other scheme, descriptors with fake addresses mapped back to (buffer, offset) at dispatch as the OpenCL backend does, was left aside:
- a view's pointer arithmetic carries straight into the shader, with no translation at dispatch (the translation is kept only for copies);
- every pipeline shares one pipeline layout (one push-constant range), with no descriptor set layouts, pools or updates, and an op may take any number of tensors;
- a recorded dispatch holds its addresses in its push constants, which suits the decode step recorded once and replayed (V2d).
Buffer device addresses are core in Vulkan 1.2 and required by 1.3: RADV, ANV and NVIDIA's driver have them. The cost is 64-bit integers in shaders (present on all three) and that an element must sit at an address aligned to its type, which tensors do. Both local GPUs run every test this way.
Memory and copies
Each allocation is a device-local buffer with its own memory (on RADV
one allocation is at most 4 GiB, reported as the largest buffer). An
upload of up to 64 KiB (positions, cache cells, a token's activations)
is copied into a host-visible buffer of the command buffer being
recorded and recorded as a copy from there: it is not waited for, and
the host's memory can be reused at once. Larger uploads, and downloads,
go through a host-visible staging buffer (grown to 64 MiB, larger copies
in pieces) and wait for their copy; download_many queues
its copies together and waits once.
Commands are recorded in call order into a ring of four command
buffers. A memory barrier goes before a command only when it is needed:
an op's first command gets one when the op reads or writes a byte range
written since the last barrier, or writes one read since then (its
tensors' spans, and the scratch buffers it asked for); every further
command of the same op gets one (a group chains its dispatches through
scratch freely). The Q, K and V products of a layer, or its gate and up
products, thus run without a barrier between them. The ops that update a
source in place (SSM_CONV, GATED_DELTA) and commands outside an op get
one as before. A command buffer is submitted when a copy needs the host
to wait, on synchronize or flush, or after 64
commands (between ops), and recording continues in the next one. A freed
buffer is destroyed once the device is idle (the next wait), not when it
is freed; when 64 of them wait, or 256 MiB of them (a model's weights, a
context's cache), the free itself waits for the device and destroys
them, so that what is allocated next finds the memory. Before 2026-11-14
only the count did: a model of 18.5 GiB freed and placed again otherwise
in the same process (Qwen2.5-VL 32B, 61 of 64 layers then 32 of 64 with
a paged cache, test_mm_public) was allocated beside its own
freed weights, past the card's 20 GiB: the context failed once, and once
ran its prompt at 11 rows/s where it takes 42.
Shaders and pipelines
src/core/vulkan/
vk_loader.h, vk_loader.c the API's declarations and its loading
vk_device.c instance, device listing and choice, device creation
vk_runtime.c buffers, copies, command buffers, pipelines, dispatch, push constants
backend_vk.c the backend interface, the routing of each op, the CPU fallback, the probe
vk_internal.h the internal API the op files use
vk_ops_<group>.c one host file per group of ops
shaders/common.glsl what every shader includes
shaders/<group>/<name>.comp the shaders of each group
The build compiles every
shaders/<group>/<name>.comp it finds
(glslc --target-env=vulkan1.2 -O -Werror, with
shaders/ on the include path) to a C initializer in
$(BUILD)/gen/vk/, and writes a table of them by name
($(BUILD)/gen/vk_shaders.inc) that
vk_runtime.c includes; adding a shader needs no Makefile
edit. <name> is the pipeline's name and must be
unique across the groups (the build stops on a duplicate).
common.glsl declares the buffer references
(RefF32, RefU32, RefU16),
load_f() and store_f() for F32, F16 and BF16
elements, the conversions done on the integer bits (F32 to F16 and BF16
with round to nearest even, as the CPU does, whatever the device's float
modes), and GLOBAL_INDEX_1D().
oklm_vk_pipeline_get(s, name, spec) builds a pipeline on
first use and caches it by name and variant: up to 8 specialization
constants (32-bit, ids 0 to 7; by convention id 0 is the work-group
size, layout(local_size_x_id = 0) in;), a required subgroup
size, full subgroups. A shader's parameters are one push-constant block
of scalar members (uint64_t, uint,
int, float) filled in the same order with
oklm_vk_push_*() (oklm_vk_push_addr() for a
tensor, which marks the block failed when the tensor is not on the
instance; oklm_vk_push_u32_checked() for a size that must
fit 32 bits). The block is at most 256 bytes, or the device's limit.
oklm_vk_dispatch() records a dispatch,
oklm_vk_dispatch_1d() spreads n invocations over the work
group grid (the groups beyond the device's x limit, 65535 on the UHD
750, in y). oklm_vk_copy(), oklm_vk_fill() and
oklm_vk_scratch() (a scratch buffer per group) complete the
set; vk_internal.h documents each.
Groups of ops
backend_vk.c's switch routes every op to its group's
entry point, which returns 1 for what it does not take; the next group
in the route, else the CPU fallback, then runs it. A group adds shaders
in its own host file and directory and never edits the switch.
| Group | Host file, shaders, test | Ops | Roadmap |
|---|---|---|---|
| element-wise | vk_ops_ew.c, shaders/ew/,
test_vulkan.c |
ADD, MUL, SCALE, SILU, GELU, GELU_ERF, SWIGLU, GEGLU, SIGGLU, SOFTCAP, SIGMOID, CPY between float types | V1 (done) |
| norm | vk_ops_norm.c, shaders/norm/,
test_vulkan_norm.c |
RMS_NORM, LAYER_NORM, ROPE, SOFTMAX, GET_ROWS, SET_ROWS, CPY into quantized types and byte copies (after the element-wise group), ARGMAX, TOPK, CROSS_ENTROPY | V2a |
| matmul | vk_ops_matmul.c, shaders/matmul/,
test_vulkan_matmul.c |
MATMUL with one activation column, every weight type | V2b |
| attention | vk_ops_attention.c, shaders/attention/,
test_vulkan_attention.c |
ATTENTION | V2c |
| gemm | vk_ops_gemm.c, shaders/gemm/,
test_vulkan_gemm.c |
MATMUL with several columns (after matmul) | V3a |
| q8 | vk_ops_q8.c, shaders/q8/,
test_vulkan_q8.c |
MATMUL and MATMUL_ID on integer dot products (tried first) | V3b |
| moe | vk_ops_moe.c, shaders/moe/,
test_vulkan_moe.c |
MATMUL_ID (after q8), MOE_SELECT, MOE_WEIGHTS, MOE_SUM; the fused
routing and gate / up / SwiGLU (held by backend_vk.c) |
V3c |
| ssm | vk_ops_ssm.c, shaders/ssm/,
test_vulkan_ssm.c |
SSM_CONV, GATED_DELTA, HADAMARD, latent attention (ATTENTION after the attention group) | V3d |
Adding an op: a shader in the group's directory; in the group's host file, a function that checks the operands (returning 1 for what the shader does not handle), fills the push block, gets the pipeline and dispatches; a check against the CPU in the group's test file.
Element-wise ops
vk_ops_ew.c runs every element-wise op on one shader
(shaders/ew/ew.comp), one invocation per element of the
destination, 4-D with any strides, the second source broadcast. Indexing
an element in general takes 64-bit divisions of the invocation's linear
index by the sizes, which GPUs emulate: they were most of a large op's
time (the SwiGLU of a 512-token prompt of Llama-3.2-1B, 4.2 million
elements: 834 us on the RX 7900 XT). Two variants of the shader (its
specialization constant 1) index without them, when the destination and
the sources have one shape:
- contiguous tensors: element i at i times the element size in each;
- rows at a constant stride (a view of half of wider rows: the gate and up outputs of a fused gate / up product), rows of 128 elements or more: the dispatch has x over a row's elements and y over the rows.
The same arithmetic per element, so the same bits. The SwiGLU above
then takes 58 us, and a 512-token prompt on the RX 7900 XT goes from 11
020 to 15 150 t/s on Llama-3.2-1B and from 1 970 to 2 340 t/s on
Qwen3-8B (Q4_K_M, 2026-11-12); a broadcast, a permuted view or shorter
strided rows keep the general indexing, and
OKLM_VK_EW_FLAT=0 gives it to every op.
test_vulkan.c checks the row-strided variant on the halves
of rows of 600 over two batch dimensions (a sum and an F16 copy bit for
bit, the gated unit in place within the activations' tolerance, the
other half untouched).
One-column products
vk_ops_matmul.c takes MATMUL with up to 16 activation
columns (decode, and the small batches of speculative decoding or
several sequences) for every weight type OKLM computes: F32, F16, BF16,
Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q2_K to Q6_K, IQ4_NL, IQ4_XS, MXFP4,
NVFP4, TQ1_0, TQ2_0, Q1_0, Q2_0, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S,
IQ1_S, IQ1_M, and PQ2_0 and PTQ1_0 (types 142 and 143 of PrismML's
llama.cpp fork: Q2_0's and TQ1_0's codes with a scale per 128, the
prompt products and the indexed ones too). Activations and destination
are F32 (columns 16-byte aligned), iparams[0] = 1 adds the
product to the destination (oklm_matmul_acc()), and the
weights broadcast over dimensions 2 and 3 as on the CPU. A product of
more than 16 columns returns 1, so the routing tries the prompt products
(vk_ops_gemm.c) next: past 16 columns the accumulators no
longer fit the registers and a tile that reuses each dequantized weight
for many columns is the faster shape.
The shader (shaders/matmul/mv.glsl, built once per
weight type as mv_<type>,
shaders/matmul/mv_<type>.comp defining the type)
gives each work-group of 64 invocations two weight rows (one column) or
one row (2 to 16 columns: 2, 4, 8 or 16 per pass, the columns past the
product's skipped). The invocations take the row's groups of 32 weights
in turn, adjacent invocations reading adjacent blocks; each group is
dequantized and used at once for every column (only one group of weights
is live in registers: holding several rows' groups made ANV's compiler
spill and take minutes), so a weight block is read once per product. The
partial sums are reduced by a shared-memory tree in a fixed order, so
accumulating a product twice doubles it exactly. The columns per pass
and the rows per work-group are specialization constants: one pipeline
per type and combination, built on first use. The type is fixed at
compile time rather than by a specialization constant: a module holding
all 28 decoders is 370 KB of SPIR-V, and the drivers took seconds per
pipeline over it (minutes for the test's variants on ANV); one type's
module is 17 to 66 KB.
shaders/dequant.glsl is the one place the block layout
and the dequantization of every weight type live; every shader that
reads weights (these products, the prompt products, the 8-bit products,
the mixture-of-experts products) includes it after
common.glsl:
#include "common.glsl"
#include "dequant.glsl"
const uint TYPE = 12u; // an oklm_dtype code (here Q4_K), from the .comp's #define
...
float v[32];
dq_group(TYPE, row_address, j, v); // weights 32 j .. 32 j + 31 of the rowdq_group() computes what the OpenCL dequantizers
(dq_<type> in kernels.cl) and
oklm_dequantize_row() compute. Give it a compile-time
constant as the type (one .comp per type defining it, as
shaders/matmul/mv_<type>.comp do): the compiler then
keeps only that type's code and tables. A specialization constant works
too, but leaves every decoder in the module for the driver to compile
through. The row address must be a multiple of
DQ_ALIGN(type) (16 for the float types, Q4_K and Q5_K,
which are read with 16-byte loads; 4 for Q2_K, IQ4_XS, NVFP4, IQ1_M; 1
for MXFP4; 2 for the others), which the host checks on the tensor's
address and strides before choosing a shader (dq_align() in
oklm_vk_dq_align(),
src/core/vulkan/vk_dequant.h); the file's header lists
every type's block size, bytes and alignment. The type codes are
T_* (OKLM's). The data is handled with 32-bit integers only
(64-bit integers serve the addresses): Intel's Gen12 emulates 64-bit
integers.
A code is never the index of a constant array. The two 16-entry
codebooks (IQ4_NL's and IQ4_XS's grid, the E2M1 values of MXFP4 and
NVFP4) are computed from constants in registers: dq_iq4nl()
holds the grid as signed bytes in four words and sign-extends the code's
byte, dq_fp4() reads the magnitude from the 4-bit fields of
one word and takes bit 3 as the sign, both to the integer the table
held, so the weights are the same bits. An array indexed by the code is
a memory load per weight on RADV, about twice the time of a product:
with the arrays gpt-oss-20b (MXFP4 experts) decoded at 96.7 t/s and
SmolLM2-135M IQ4_XS at 391, without them at 163.9 and at 650 to 707 (RX
7900 XT, 2026-11-14; Mixture of
experts). The shaders of an IQ4_NL cache take the grid the same way
(iq4nl_grid() in common.glsl, for
norm_store_rows, norm_get_rows and the
attention's cache rows, which each held a constant array): a row's
quantization bisects the grid for each of a block's 32 values at sixteen
scales, some three thousand reads a block, one invocation a block.
Llama-3.2-1B Q4_K_M with an IQ4_NL cache decoded at 93.5 t/s with the
arrays, the same at 64, 256 and 1024 tokens of context (the writes, not
the attention's reads), and at 154.5 without them; its prompt of 512
tokens went from 11 900 to 12 190 t/s to 12 610 (a Q4_0 cache: 447 to
456 and 15 230 to 15 370; F16: 470 to 483 and 15 680 to 16 110; 1024
tokens decoded, the best of three runs, 2026-11-14). The bytes written
and the values read are unchanged (test_vulkan_norm,
test_vulkan_attention, on the RX 7900 XT and the Intel UHD
750).
The grid IQ types (IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ1_S,
IQ1_M) read their codebooks (the tables of
src/core/iq_grids.inc, 33 KB) from a device buffer each
instance uploads once, oklm_vk_iq_grids(s): the shader sets
the global dq_grids to that address (passed in its push
constants) before decoding, the tables at the DQ_GRID_*
word offsets. Constant arrays in the shader were copied into every
invocation's memory by the drivers. vk_dequant.h declares
oklm_vk_dq_align(), oklm_vk_dq_needs_grids()
and oklm_vk_iq_grids() for the other groups.
On Intel GPUs IQ2_S, IQ1_S and IQ1_M are left to the CPU fallback: on the UHD 750 (ANV, Mesa 26.1) their pipelines take over 30 seconds each to build and then lose the device or give wrong results, while the same shaders pass on RADV and llvmpipe. Every other type runs on both GPUs.
tests/core/test_vulkan_matmul.c checks every type on
every GPU against a double sum over the dequantized weights (within
1e-5 sum |w x| + 1e-6, test_opencl.c's matmul
tolerance): one and 7 columns (2, 3 and 16 too for F32, Q8_0, Q4_K,
Q6_K, IQ2_XS), row counts that are not multiples of the work-group's
rows, rows of one block and of several hundred groups; then the
accumulated product, which must double every value exactly (IQ2_S, IQ1_S
and IQ1_M skipped on Intel, IQ3_S's pipelines there limited to the
decode one, ANV building each in about 10 seconds before its cache keeps
them); weights broadcast over each batch dimension, strided activation
columns, F16 denormal weights, and products of 17 and 40 F32 columns
(past the group). With OKLM_VK_BENCH=1 it also measures, on
the discrete GPUs, the decode product's weight bandwidth for Q4_K, Q6_K,
Q8_0 and F16 at Llama-3.2-1B and Qwen3-8B sizes, each product on another
copy of the weights (320 MB in all, past the 7900 XT's 80 MB Infinity
Cache, which otherwise serves the weights of repeated products at up to
1.5 TB/s); OKLM_VKT_CL=1 adds the OpenCL backend's on the
same shapes, OKLM_VKT_BENCH_ONLY=1 skips the checks:
OKLM_VK_BENCH=1 OKLM_VKT_CL=1 OKLM_VKT_BENCH_ONLY=1 OKLM_VK_DEVICE=radv build/release/bin/tests/core/test_vulkan_matmulOn the RX 7900 XT (RADV), weight bytes read per second, Vulkan and the OpenCL backend's decode kernels on the same shapes:
| Type | K x N | MB | Vulkan | OpenCL |
|---|---|---|---|---|
| Q4_K | 2048 x 8192 | 9.4 | 494 GB/s (19.1 us) | 455 GB/s |
| Q4_K | 8192 x 2048 | 9.4 | 417 GB/s | 340 GB/s |
| Q4_K | 4096 x 12288 | 28.3 | 558 GB/s (50.8 us) | 514 GB/s |
| Q4_K | 12288 x 4096 | 28.3 | 514 GB/s | 512 GB/s |
| Q6_K | 2048 x 8192 | 13.8 | 533 GB/s | 484 GB/s |
| Q6_K | 8192 x 2048 | 13.8 | 488 GB/s | 421 GB/s |
| Q6_K | 4096 x 12288 | 41.3 | 603 GB/s | 583 GB/s |
| Q6_K | 12288 x 4096 | 41.3 | 534 GB/s | 475 GB/s |
| Q8_0 | 2048 x 8192 | 17.8 | 592 GB/s | 520 GB/s |
| Q8_0 | 8192 x 2048 | 17.8 | 587 GB/s | 521 GB/s |
| Q8_0 | 4096 x 12288 | 53.5 | 659 GB/s | 584 GB/s |
| Q8_0 | 12288 x 4096 | 53.5 | 611 GB/s | 562 GB/s |
| F16 | 2048 x 8192 | 33.6 | 449 to 480 GB/s | 537 GB/s |
| F16 | 8192 x 2048 | 33.6 | 578 to 600 GB/s | 623 GB/s |
| F16 | 4096 x 12288 | 100.7 | 526 to 590 GB/s | 651 GB/s |
| F16 | 12288 x 4096 | 100.7 | 558 to 640 GB/s | 619 GB/s |
Every product pays about 5.7 us of dispatch and barrier besides its reads (the intercept of the two sizes); past it the Q8_0 product streams at about 730 GB/s, Q4_K and F16 near 680 GB/s, Q6_K near 600 GB/s. F32 and BF16 weights, read with plain 16-byte loads, reach 655 to 700 GB/s on these shapes: the shader's ceiling, near the 7900 XT's practical bandwidth. F16 varies between runs over the range shown and stays a little behind the OpenCL kernel; the quantized types are ahead of it.
8-bit products
vk_ops_q8.c is tried before the float products. It takes
MATMUL of up to 16 activation columns and the decode steps of MATMUL_ID
(up to 64 (slot, token) pairs) with Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 or Q5_0
weights and gives the CPU's products bit for bit: the activations are
rounded to 8 bits as the CPU rounds them, each block's integer sums are
formed with the 4-way 8-bit dot products of
VK_KHR_shader_integer_dot_product
(dotPacked4x8EXT, exact), and the float terms are formed
and added as the CPU's dot products form and add them. A greedy decode
on the 8-bit path therefore follows the CPU's tokens, which the float
products only approach. It needs integer dot products and pipelines of
32-wide subgroups with arithmetic, clustered, ballot and shuffle
operations (both local GPUs have them, the 4x8-bit packed product
accelerated on both); elsewhere it returns 1 and the float groups
run.
The shader (shaders/q8/q8mv.glsl, built per weight type
as q8mv_<type>) gives each work-group of 256
invocations (eight subgroups of 32) 16 weight rows, two per subgroup.
The work-group first rounds its activation columns (one to four of them,
as shared memory allows) into shared memory, so no launch is added:
- Q4_K, Q5_K, Q6_K: blocks of 256 as
oklm_quantize_row_q8_k()(a subgroup per block, eight values a lane): the largest magnitude and, on ties, the first one gives the scale's sign; levelsmin(roundEven(iscale x), 127)withiscale = -127 / max; the block's scale1 / iscale; the sums of each 32 levels for the mins term; - Q8_0, Q4_0, Q5_0: blocks of 32 as
oklm_quantize_row_q8_0(): levelsroundf(x / d)(half away from zero, fromtrunc),d = amax / 127kept as a half.
The divisions must be the CPU's correctly rounded ones, while a
Vulkan division is only accurate to 2.5 ulp and the UHD 750 has no
doubles: q8_div() estimates the quotient of the
significands with a float division, then makes it exact from its integer
remainder, whose low 32 bits the 32-bit products give exactly (it is a
few divisors at most), and rounds to nearest even (zeros, denormals and
overflow handled).
Each subgroup then walks its rows' blocks: eight lanes a super-block
for the K-quants (16 values of each of two sub-blocks a lane, the scales
applied to the integer sums, the sums added across the eight lanes with
a clustered add, two steps of four super-blocks unrolled), one lane a
block for the blocks of 32 (32 blocks a step). Each block's float term
is the CPU's: d isum - dmin summs with
d = f16(d) y.d and dmin = f16(dmin) y.d (Q4_K,
Q5_K), (f16(d) y.d) isum (Q6_K),
isum (f16(d) f16(y.d)) (Q8_0),
(isum f16(d)) f16(y.d) (Q4_0),
(f16(d) f16(y.d)) isum (Q5_0), every variable
precise (no fused multiply-adds). A row's terms must be
added in order of k, one at a time: the terms of a step go to a row of
shared memory per output, and the output's own lane adds them in block
order (with 16-byte reads), so the serial additions cost one instruction
per term rather than a broadcast and an add per term and row in every
lane (which made the blocks of 32 up to 25 % slower than the float
products).
An RMS norm before the product is not folded into it (OpenCL's
mqn_<type>): the routing gives the group one op at a
time and holds no norm back, and the folding paid on OpenCL only by
saving a launch.
The K-quants have a second layout, the row shaders
(shaders/q8/q8kr.glsl, built as
q8kr_<type> for Q4_K, Q5_K and Q6_K), for a device
bound by the shader's arithmetic rather than by its memory (the UHD
750): there q8mv_<type> repeats a super-block's
header work (scales, mins, the two half floats) in each of its eight
lanes, rounds the activations again in every work-group, and pays shared
memory and barriers for the ordered sums. Two dispatches instead:
q8k_roundrounds the activation columns once, a subgroup of 32 per block of 256 (the rounding above), into records of 304 bytes in the group's scratch buffer: the 256 levels, the eight sums of each 32, the block's scale;q8kr_<type>gives each invocation one weight row (work-groups of 32, no shared memory, no barrier, no subgroup operation): per super-block, 16-byte loads of the block and of the record (Q6_K's blocks, aligned to 2 bytes only, as words from the aligned address before them), the eight sub-blocks' integer sums, the float term, and the row's sum in order of k as the CPU adds it. One, two or four columns per invocation share the weights' loads; MATMUL_ID takes one (slot, token) pair per z.
The results are the same bits as q8mv_<type>'s and
the CPU's. OKLM_VK_Q8_ROWS=1 takes the row shaders on any
device, 0 never; unset, Intel GPUs take them. On the RX
7900 XT the second dispatch and its barrier cost more than they save
(Q4_K 2048 x 2048: 9.9 against 5.8 us; 4096 x 12288: 57 against 50
us).
Policy: with OKLM_VK_Q8 unset
(q8_default()), MATMUL takes the 8-bit path where it was
measured not slower than the float products: on AMD GPUs, the K-quants
with rows of 1536 values or more and 4096 rows or more (any column count
up to 16), the blocks of 32 on the same shapes from two to eight
columns; on Intel GPUs, the K-quants at every shape (on the row shaders;
never with OKLM_VK_Q8_ROWS=0) and the blocks of 32 from 512
values and 512 rows; on other devices, never. MATMUL_ID decode steps
(q8_default_id(), against the mixture-of-experts group's
mvid_<type>) take it on AMD GPUs for experts of 512
rows or more, the blocks of 32 at any row length and the K-quants from
1536 values (eight ranks of 32 or 64 experts: 5 to 40 % faster, Q6_K
4096 x 1536 36.9 against 59.5 us; the K-quants' rows of 768 and 1024
values, the down products of small experts, 5 to 20 % slower), and on
Intel GPUs as MATMUL. OKLM_VK_Q8=1 takes every product the
shaders can run, OKLM_VK_Q8=0 none.
On the RX 7900 XT (RADV), decode products of one column, each on
another copy of the weights (test_vulkan_q8.c,
OKLM_VK_BENCH=1), the 8-bit path, the float one-column
products and the OpenCL backend's default kernels (its own 8-bit
mq_<type> for the K-quants, float
mv_<type> for the others):
| Type | K x N | 8-bit | float | OpenCL |
|---|---|---|---|---|
| Q4_K | 2048 x 8192 | 17.9 us (528 GB/s) | 18.6 us | 21.2 us |
| Q4_K | 4096 x 12288 | 47.6 us (595 GB/s) | 47.7 us | 51.8 us |
| Q4_K | 12288 x 4096 | 52.5 us (540 GB/s) | 56.1 us | 55.4 us |
| Q5_K | 4096 x 12288 | 57.3 us (604 GB/s) | 59.0 us | 61.1 us |
| Q6_K | 2048 x 8192 | 23.8 us (578 GB/s) | 25.4 us | 27.6 us |
| Q6_K | 4096 x 12288 | 64.9 us (637 GB/s) | 70.0 us | 69.0 us |
| Q6_K | 12288 x 4096 | 72.3 us (571 GB/s) | 76.3 us | 84.1 us |
| Q8_0 | 4096 x 12288 | 83.9 us (638 GB/s) | 82.4 us | 91.3 us |
| Q4_0 | 4096 x 12288 | 48.3 us (586 GB/s) | 46.5 us | 50.3 us |
| Q5_0 | 4096 x 12288 | 59.1 us (585 GB/s) | 55.1 us | 60.6 us |
The 8-bit products pay about 1.5 us more of fixed cost (the rounding before the first product, the work-group's prologue), so small products (1024 x 1024, 2048 x 512) are 20 to 40 % slower and left to the float products. With two to eight columns the 8-bit path is ahead on model shapes, by up to 40 % (Q4_K 4096 x 12288: 50.8 against 64.3 us with two columns, 154 against 224 us with eight; Q8_0 with four: 100 against 127 us).
On the UHD 750 (ANV) the float products and
q8mv_<type> are bound by the shader's arithmetic, not
by memory: the blocks of 32 run two to three times faster in 8 bits
(Q4_0 4096 x 4096: 570 against 1 904 us), the K-quants on
q8mv_<type> 15 to 50 % slower than in floats. The row
shaders read the K-quants' weights at 14 to 23 GB/s, where the weight
loads are most of the time left (Q6_K 2048 x 16384 without them: 0.5 of
1.9 ms). One column, each product on another copy of the weights
(2026-11-12,
OKLM_VKT_BENCH_IGPU=1 OKLM_VK_DEVICE=Intel):
| Type | K x N | row shaders | q8mv_<type> |
float |
|---|---|---|---|---|
| Q4_K | 2048 x 2048 | 122 us (19 GB/s) | 553 us | 483 us |
| Q4_K | 2048 x 16384 | 832 us (23 GB/s) | 4 152 us | 3 780 us |
| Q4_K | 8192 x 2048 | 442 us (21 GB/s) | 1 844 us | 1 828 us |
| Q4_K | 512 x 512 | 17 us | 54 us | |
| Q5_K | 2048 x 16384 | 1 009 us (23 GB/s) | 5 013 us | 4 204 us |
| Q6_K | 2048 x 16384 | 1 898 us (14 GB/s) | 5 586 us | 4 678 us |
| Q6_K | 8192 x 2048 | 804 us (17 GB/s) | 3 235 us | 2 144 us |
| Q6_K | 2048 x 128256 | 14.7 ms (15 GB/s) | 36.6 ms |
With 4 columns Q4_K 2048 x 16384 takes 1.44 ms against 13.6 ms in
floats, with 16 columns 5.6 against 54 ms; the indexed products of eight
ranks (Q4_K, 2048 x 1024 of 64 experts) 347 us against 1 883 us on
mvid_q4_k. Llama-3.2-1B Q4_K_M then decodes at 19.8 t/s on
the UHD 750 (6.4 before the row shaders; llama.cpp's Vulkan backend
15.6), 42 of a token's 50 ms in its products, whose 763 MiB of weights
go at 18 GB/s.
tests/core/test_vulkan_q8.c runs on both GPUs in about
two seconds each: with OKLM_VK_Q8=1, every type
bit-identical to the CPU's product (rows of one block, of a block count
that is not a multiple of a step, of 352 values for the blocks of 32,
one row, 5 and 16 columns over column groups, batches with broadcast
weights, then the product accumulated once more), on activations with
ties of magnitude of opposite signs, zero blocks, values over 22 binades
and rows scaled by up to 2^80 either way; indexed products of one token
with four slots, of three tokens sharing an activation row and of two
tokens with a row per slot; the K-quants once per layout
(OKLM_VK_Q8_ROWS=0 and 1) on each GPU; then
the default policy on model-sized products, exact where it takes the
8-bit path and within the float tolerance elsewhere, and
OKLM_VK_Q8=0. With OKLM_VK_BENCH=1 it times
the decode products, 8-bit against float (OKLM_VKT_CL=1:
and OpenCL's; OKLM_VKT_BENCH_TYPE,
OKLM_VKT_BENCH_COLS,
OKLM_VKT_BENCH_SHAPES=KxN,...,
OKLM_VKT_BENCH_IGPU for the integrated GPUs; the indexed
decode products of MoE shapes against the mixture-of-experts group's,
and OKLM_VKT_BENCH_NO_MM times those alone):
OKLM_VK_BENCH=1 OKLM_VKT_CL=1 OKLM_VKT_BENCH_ONLY=1 OKLM_VK_DEVICE=radv build/release/bin/tests/core/test_vulkan_q8Prompt products
vk_ops_gemm.c takes MATMUL past the one-column products'
16 columns (prompts, large batches) for every weight type
dequant.glsl decodes, with iparams[0] = 1
accumulating and the weights broadcast over dimensions 2 and 3 as on the
CPU. Activations and destination are F32, the activation columns 16-byte
aligned (anything else goes to the CPU); IQ2_S, IQ1_S and IQ1_M stay on
the CPU on Intel GPUs, as for the one-column products. Two kinds of
shaders, one module per weight type each
(shaders/gemm/<kind>_<type>.comp, the bodies in
gm.glsl and cm.glsl, what they share in
gemm.glsl):
cm_<type>, cooperative matrices (VK_KHR_cooperative_matrix), on a device that lists 16 x 16 x 16 blocks of F16 inputs with F32 accumulation at subgroup scope and has 16-bit floats and full subgroups (the RX 7900 XT; not the UHD 750). A first dispatch,gm_tohalf, rounds the activations to F16 once into a contiguous copy (scratch); a work-group of four subgroups of 64 (OKLM_VK_GEMM_SG=32takes 32) computes 128 weight rows by 128 columns, each subgroup a quarter, 4 x 4 accumulators. Per step of 32 values of k, 128 invocations dequantize one row's group each (F16 weights are copied as they are) and round it to F16 into shared memory, every invocation copies its share of the F16 activations, and each subgroup multiplies its 4 x 4 blocks twice; the next step's loads are in flight meanwhile. F32 weights do not take this path: their products stay F32, as on the CPU, which does not round their activations.cm_<type>with the specialization constantWIDE, for a product that asks for its activations at F32's precision (oklm_matmul_wide(),iparams[1] = 1: the products of the vision and audio encoders' blocks, below) with F16 weights:gm_tohalfwrites two F16 copies, the rounded values and what the rounding dropped, and each step multiplies the weight blocks by both. The other weight types of such a product takegm_<type>.gm_<type>, F32 tiles, everywhere else (and withOKLM_VK_COOPMAT=0): 64 invocations per tile of 64 rows by 64 columns, the weights dequantized into shared memory k-major, the activations beside them, 8 x 8 outputs per invocation, the next step loaded during the current one (OpenCL'sgm_<type>).
gm_tohalf and gm_reduce find their column
and batch by dividing a 32-bit linear index; indexing them by the
dispatch grid instead (x within a column, y the columns, z the batches)
changed nothing (a layer's seven Q4_K products at 512 columns: 19 420 to
19 450 t/s on Llama-3.2-1B's sizes and 3 280 to 3 300 on Qwen3-8B's
either way, 2026-11-12), unlike the 64-bit divisions of the element-wise
shader; not kept.
A product whose tiles do not fill the device is split over k: each
split writes its partial tile to scratch
(OKLM_VK_SCRATCH_GEMM) and gm_reduce adds the
splits in their order, then writes or accumulates. The F32 tiles split
up to 256 work-groups (64 on an integrated GPU), at least 4 steps per
split; the cooperative-matrix tiles, four times the work each, up to
128, at least 16 steps per split, and a product of 64 tiles or more only
when k reaches 8192 (OKLM_VK_GEMM_SPLIT=n forces n). Every
sum runs in a fixed order, so a product is deterministic and
accumulating it twice doubles it exactly.
Precision: the F32 tiles are within
test_vulkan_matmul.c's tolerance
(1e-5 sum |w x| + 1e-6 of a double sum over the dequantized
weights); the cooperative matrices round both inputs to F16, within
`2e-3 sum |w x|
- 1e-6
(test_opencl_wmma.c's F16 tolerance; the RX 7900 XT's worst is well inside it); a product that asks for its activations at F32's precision is within1e-6 sum |w x| + 1e-6` on either (below). BF16 weights beyond the F16 range (above 65504) would not survive the rounding; no model has them.
An encoder's products
A language model's prompt tolerates the rounding of its activations
to F16 (the figures under The model on
Vulkan); an encoder's blocks do not. Their activations reach tens
(80 in Qwen2-VL's), the terms of a product cancel, and 23 to 32 blocks
add the rounding up: with every product on cm_f16 the
embeddings were 2.6e-3 to 1.5e-2 from the CPU's (relative norm), where
OpenCL's F32 products give 1.4e-5 to 5.3e-5. Measured on Qwen2-VL 2B (RX
7900 XT, 2026-11-14, two pictures of 273 embeddings,
test_mm_public), with one group of a block's products left
on the rounded activations and the others not:
| Rounded to F16 | Embeddings from the CPU's |
|---|---|
| every product (before) | 1.5e-2, 1.1e-2 |
| the queries, keys and values | 1.3e-3, 2.5e-3 |
| the attention's output | 3.5e-3, 4.2e-3 |
| the MLP's first product | 1.8e-2, 1.2e-2 |
| the MLP's second product | 4.1e-3, 5.1e-3 |
none (oklm_matmul_wide() for the six) |
1.9e-5, 3.0e-5 |
The error enters with the activations: the units accumulate in F32
and every result is stored as F32. So the encoders ask for their
products through oklm_matmul_wide()
(OKLM_OP_MATMUL with iparams[1] = 1), which
the CPU and OpenCL run as any product and which
oklm_vk_gemm_tiles() runs, on a device with cooperative
matrices, in one of three ways (OKLM_VK_WIDE, read at each
product):
- unset: two F16 parts on the units for F16 weights
(every published encoder with F16 matrices).
gm_tohalfwrites the half nearest each activation and, in a second copy, what that rounding dropped (x - half(x), exact in F32, itself rounded to F16): about 22 bits of each value together.cm_f16withWIDEloads both tiles and, per 16 x 16 x 16 block, multiplies the weights by the first then by the second into the same F32 accumulator; the second tile has rows of 32 halves without padding, so the shader's shared memory is 32 768 bytes (a device with less takes the F32 tiles). The accumulators are also added into a second set every 4 steps (128 values of k) and cleared: one running F32 sum over the 2 560 to 10 240 terms of such a product alone left Qwen2-VL's embeddings at 1.7e-4 to 3.2e-4. The nearest half is made on the bits of the F32 (13 bits of mantissa rounded to nearest even): written as a conversion to F16 and back, RADV (Mesa 26.1) gives the value itself and the second copy is zero. Weights of another type (BF16, quantized: their own rounding to F16 would remain) take the F32 tiles. f32: every such product on the F32 tiles (gm_<type>).0: as any other product, the activations rounded to F16 (the behaviour before 2026-11-14).
F32 activations on the units themselves are not to be had: the RX 7900 XT (RADV, Mesa 26.1) lists its float configurations with F16 inputs alone (an F16 or an F32 result), the others with 8-bit integers.
The products of a language model do not set the flag and keep their
shader variant (WIDE 0, the same specialization as before),
their gm_tohalf values and their bits.
One block's six products with F16 weights at width 1 280 (5 120
inside the MLP), test_vulkan_gemm with
OKLM_VK_BENCH=1 OKLM_VKT_BENCH_WIDE=1 on the RX 7900
XT:
| Rows | Rounded (OKLM_VK_WIDE=0) |
Two F16 parts (default) | F32 tiles (OKLM_VK_WIDE=f32) |
|---|---|---|---|
| 1 092 (Qwen2-VL, 273 embeddings) | 1.42 ms | 3.25 ms (2.3 x) | 4.30 ms (3.0 x) |
| 1 500 (Whisper large-v3) | 1.53 ms | 3.48 ms (2.3 x) | 5.32 ms (3.5 x) |
What that is of an encoder and of a prompt, and the embeddings each setting gives, is in multimodal.md. An activation beyond the F16 range (above 65 504) is infinite in the first copy as it was; values under 2^-14 have no second part (an absolute error under 3e-8 each).
src/core/vulkan/vk_gemm.h offers the same products to
the other groups, oklm_vk_gemm_tiles(s, &args):
addresses and byte strides instead of tensors, and a grouped form for
the mixture-of-experts prompt path, a tile table read on the device (one
work-group row per entry: weight matrix e at w + e * wnb2,
product columns c0 .. c1 - 1, at most
oklm_vk_gemm_group_cols(), a tile's columns: 128 on
cooperative matrices, 64 on the F32 tiles; e < 0 an unused entry)
with column id tables (x_ids, d_ids: product
column c reads activation column x_ids[c], writes
destination column d_ids[c]).
tests/core/test_vulkan_gemm.c checks on every GPU, on
both paths where the device has cooperative matrices: every type (70
rows by 40 columns, split, and 130 by 150, partial tiles both ways); one
row by 17 columns, one step of k, 129 by 129, k = 4096 over one tile (16
splits), 200 by 300 unsplit and three forced splits, for F32, Q8_0, Q4_K
and IQ2_XS; the batch broadcasts and strided columns of the one-column
test; each product accumulated, which must double it exactly; that the
two paths differ for Q8_0 weights (the cooperative matrices ran) and
agree bit for bit for F32; the products that ask for their activations
at F32's precision (test_wide: activations of +-80, partial
tiles, a split product, a batch, strided columns, accumulated) within
1e-6 sum |w x| + 1e-6 for F16 weights by default and with
OKLM_VK_WIDE=f32, and for F32, BF16, Q8_0 and Q4_K weights,
where OKLM_VK_WIDE=0 must leave outputs beyond that bound
on a device with cooperative matrices (measured on the RX 7900 XT as a
fraction of sum |w x|: 8e-8 to 3.0e-7 with the two parts, 2e-8 to 1.4e-7
on the F32 tiles, 1.8e-5 to 7.4e-5 rounded); the products a convolution
is made of on a device (test_conv_views: one product a
position of the kernel, with the activations at F32's precision, over a
view of every stride-th row from that position on, the first written and
the others accumulated; kernels of 3 and 7, strides of 1 and 2, 96 to 1
280 channels, against the convolution in doubles, within 1.8e-7 of sum
|w x| on the RX 7900 XT and 5.6e-8 on the Intel UHD 750; the
element-wise ops over views of rows the vocoders' depthwise convolution
and Gemma 3's pool are made of, and a product into the rows after its
destination's first: multimodal.md,
speech.md); the
grouped form (three matrices, an unused entry, a single and a full
64-column entry, repeated and permuted column ids) for F32, Q4_K and
Q8_0. With OKLM_VK_BENCH=1 it times the seven products of a
Llama-3.2-1B and a Qwen3-8B layer with 512 columns for Q4_K, Q6_K, Q8_0
and F16 weights on the discrete GPUs (OKLM_VKT_CL=1: the
OpenCL backend's beside them; OKLM_VKT_BENCH_TYPE=Q4_K one
type):
OKLM_VK_BENCH=1 OKLM_VKT_CL=1 OKLM_VKT_BENCH_ONLY=1 OKLM_VK_DEVICE=radv build/release/bin/tests/core/test_vulkan_gemmOn the RX 7900 XT (RADV, 2026-11-12), the seven products of a layer at 512 columns as a prompt rate (512 tokens over the layers' product time) and the mean TFLOPS; OpenCL is OKLM's OpenCL backend with its defaults (8-bit matrix units for Q4_K, Q6_K and Q8_0, F32 tiles for F16):
| Weights | Model | Vulkan F32 tiles | Vulkan cooperative matrices | OpenCL |
|---|---|---|---|---|
| Q4_K | Llama-3.2-1B | 6300 t/s, 12.3 TFLOPS | 19800 to 20300 t/s, 38.5 to 39.4 | 18100 t/s, 35.3 |
| Q4_K | Qwen3-8B | 1000 t/s, 13.9 | 3290 to 3360 t/s, 45.7 to 46.7 | 3110 t/s, 43.2 |
| Q6_K | Llama-3.2-1B | 6000 t/s, 11.6 | 17800 t/s, 34.6 | 17300 t/s, 33.6 |
| Q6_K | Qwen3-8B | 920 t/s, 12.7 | 2920 t/s, 40.6 | 2810 t/s, 39.1 |
| Q8_0 | Llama-3.2-1B | 6400 t/s, 12.4 | 20100 t/s, 39.2 | 20600 t/s, 40.0 |
| Q8_0 | Qwen3-8B | 990 t/s, 13.7 | 3270 t/s, 45.4 | 3410 t/s, 47.3 |
| F16 | Llama-3.2-1B | 4700 t/s, 9.2 | 19600 t/s, 38.2 | 6600 t/s, 12.9 |
| F16 | Qwen3-8B | 760 t/s, 10.5 | 3290 t/s, 45.8 | 980 t/s, 13.5 |
The largest products (Qwen3-8B's gate and up, 4096 x 12288) reach 50
to 51 TFLOPS; the smallest (Llama-3.2-1B's keys and values, 2048 x 512,
16 tiles split in four) about 18. llama.cpp's Vulkan backend runs a
whole 512-token prompt (llama-bench pp512, Q4_K_M) at about
15700 t/s on Llama-3.2-1B and 2780 t/s on Qwen3-8B, attention and the
rest included; these products alone take 26 ms and 155 ms of such a
prompt. The display shares the GPU, so the small products vary by up to
a factor of two between runs; the large ones by a few percent. The F32
tiles run at OpenCL's F32 tiles' rate (12.7 and 13.6 TFLOPS on
Q4_K).
Mixture of experts
vk_ops_moe.c and shaders/moe/ run a MoE
layer's routing and its expert products: the decode step of a MoE model
(router, choice, weights, the three indexed products, the sum) stays on
the GPU, reads nothing back and replays.
| Op | Shader | Takes | Against the CPU |
|---|---|---|---|
| MOE_SELECT | moe_select |
F32 probabilities, up to 1024 experts, up to 64 chosen, an optional F32 selection bias, up to 64 expert groups | identical experts and order |
| MOE_WEIGHTS | moe_weights |
norm, scale | bit-identical |
| MOE_SUM | moe_sum |
F32, any row strides | bit-identical |
| MATMUL_ID | moe_group, mvid_<type>; prompts on
the prompt products' cm_<type> or
gm_<type> |
every weight type of the one-column products, F32 activations shared by the ranks of a token or one row per rank, any number of pairs | within the one-column products' tolerance; prompts on cooperative matrices within the prompt products' (F16 inputs) |
Left to the CPU: more than 1024 experts, 64 chosen or 64 groups;
indexed products with an expert table (w_experts, the CPU's
expert cache); IQ2_S, IQ1_S and IQ1_M weights on Intel GPUs (as the
one-column products); unaligned operands. The 8-bit products group is
tried first for MATMUL_ID.
moe_select: one work-group of 64 per token. The scores (probability plus bias, one rounded add) go to shared memory; with groups, each group sums its two best scores with the CPU's comparisons and counts the groups ranked before it, the bestn_group_usedkept. Then each competing expert counts the competing experts ranked before its own (higher score, or the same score and a lower index, stopping at K), and the expert of rank k writes slot k: the order is strict, so this is the CPU's K passes in one step. NaN scores are not ordered as on the CPU.moe_weights: one invocation per (rank, token); the token's sum in rank order and the product areprecise, the divisiondiv_rn()(norm/norm_common.glsl), so the weights are the CPU's bits.moe_sum: one invocation per output element, the ranks in order, each product rounded before it is added.MATMUL_ID, decode and small batches (no more pairs than experts): one work-group per (block of rows, pair), the pair's expert read from the
idstensor on the device. The loop is the one-column products' (each group of 32 weights dequantized once bydq_group()), with one change: an expert's rows are short (16 or 32 groups for rows of 512 or 1024), so the work-group of 64 is cut into segments of a power of two invocations at least the row's groups, each segment taking its own rows (Granite's down product: four rows per work-group instead of 16 busy invocations of 64).MATMUL_ID, more pairs than experts:
moe_group(one work-group) counts each expert's pairs, lays out one item per chunk of an expert's pairs and places the pairs by expert in the group's scratch buffer. The grid haspairs / chunk + expertsitems, the unused ones empty, so it depends on the shapes only, and a pair's result does not depend on the other pairs of its item. Two shaders take the items:- 16 pairs an expert or more on average (prompts): the prompt
products' tiles (
oklm_vk_gemm_tiles()'s grouped form, Prompt products). The items are its tile table (expert, first, end), a chunk a tile's columns (128 on cooperative matrices, 64 on the F32 tiles); the placed pairs are the destination columns, and the activation columns too unless the ranks share their token's row, wheremoe_groupalso writes each placed pair's token. Each tile is one expert's weights by its pairs: cooperative matrices with F16 inputs where the device has them (F32 weights keep the F32 tiles), as a dense layer's prompt products. It needs the pairs of a token adjacent in the destination (and in the activations with a row per rank), which the models' buffers are; other strides take the chunks below. - Fewer (a few tokens):
mvid_<type>takes one item (expert, first, count) per work-group, reading each weight group once for its NC pairs (NC: the average pairs per expert rounded up to 1, 2, 4, 8 or 16), F32 throughout.
OKLM_VK_MOE_TILES=0keeps every grouped product onmvid_<type>,1sends every one to the tiles (read at each product). The two meet at 8 pairs an expert (Tini-Cybersec-8B-A1B and OLMoE-1B-7B, prompts of 64 tokens: 1782 against 1868 and 2264 against 2212 t/s); at 16 the tiles are 1.5 times faster (128 tokens: 2218 against 3346 and 2719 against 4024), at 4 slower (32 tokens: 1435 against 1000 and 1861 against 1200), a tile costing the same whatever it holds.- 16 pairs an expert or more on average (prompts): the prompt
products' tiles (
Replay: the chosen experts never leave device memory and no pushed value depends on them: Granite-3.1-1B-A400M, OLMoE-1B-7B and GLM-4.7-Flash replay 381 to 383 of 384 decode steps (
oklm_vulkan_step_stats()).
Fusion. As on OpenCL, backend_vk.c holds back the first
ops of three patterns (two of a MoE layer, and the end of any residual
branch) and runs each pattern as one dispatch when the op that completes
it arrives; the group file does the dispatch:
| Ops | Fused dispatch | Against the ops one by one |
|---|---|---|
| SOFTMAX (no mask, at most 1024 values), MOE_SELECT on its output, MOE_WEIGHTS on both | moe_route |
bit-identical (probabilities, experts, weights) |
| MATMUL_ID gate, MATMUL_ID up (same weight type and layout, activations and experts; one column per pair; products the 8-bit group does not take), SWIGLU of the two in the gate output | mvid_<type> with its GLU constant set |
bit-identical (the SwiGLU output and the up output) |
MATMUL on mv_<type> (not accumulating, not one of
the 8-bit group's) or MOE_SUM, then optionally the SCALE of its output
in place, then the ADD of that output into a tensor of its shape, in
place (x = x + o or x = o + x, x apart from o
and from what the producer reads): the residual tail |
mv_<type> or moe_sum, given the
residual stream's address |
bit-identical (the output and the stream) |
- The hold. A softmax, a MATMUL_ID that the decode
path of
mvid_<type>takes, or a MATMUL or MOE_SUM whose shader could take a residual tail, is copied (the op and its tensors) instead of run; a MOE_SELECT on the held softmax's output, the up product matching a held gate product, or the SCALE in place of a held producer's output, joins it. The op after that either completes the pattern (the fused dispatch) or not: then the held ops run one by one, in order, as they would have, and the new op after them; operands the fused shader does not take (a prompt's grouped products, overlapping tensors, an op on the CPU) do the same. Every other entry point of the instance (upload, download, synchronize, flush, the stream calls,step_begin,step_end, a buffer free) runs the held ops first, so a read-back sees their results, an upload or a free cannot change what they read, and a decode step records them in their place: the fused steps replay as before (OLMoE: 381 of 384). Held ops belong to the thread that issued them only while no other thread uses the instance: contexts and encoders that share a device from threads of their own bracket each computation as a pass (oklm_backend_pass_begin(), server.md). A pass that lets another in between two of its blocks (oklm_backend_pass_yield()) first waits for its work (synchronize), which runs the held ops and submits the open command buffer: the pass let in finds neither. - Barriers. A fused dispatch declares every
destination of its ops as written and every source as read
(
oklm_vk_op_begin_fused()), the sources inside its own destinations left out (the write already waits for their earlier writers). moe_route. One work-group per token, at the softmax shader's work-group size for the row (64 or 128), so the max and the two-float sum reduce in the same order asnorm_softmax's and the probabilities have its bits; they are written out and kept in shared memory, then the choice runsmoe_select's ranking and invocation k computes the weight of rank k asmoe_weightsdoes. With no more experts than invocations (the usual case) each invocation reads its score once and keeps it for the three passes of the softmax, and the ranking reads sixteen scores of shared memory a step.- GLU. The fused variant of
mvid_<type>(specialization constant 4) has the plain product's layout and spec constants, and dequantizes the up rows' groups beside the gate rows' in the same loop: each sum is accumulated and reduced exactly as the plain product's, so both are the products' bits; it writes the up output andg / (1 + exp(-g)) * u(the element-wise shader's SwiGLU expression) where the in-place SwiGLU writes it. The activations are read once for both products, and two barriers per layer go away with the three ops. - Residual tail. The producer's invocation that
writes an output element also reads the residual stream's element,
multiplies its own value by the SCALE's factor when there is one, writes
it, and writes the sum to the stream: one rounded product and one
rounded add (
precise), as SCALE and ADD compute them, so the output and the stream have the bits of the three ops. The SCALE and the ADD, a dispatch and a barrier each, go away: Granite-3.1-1B-A400M has two tails a layer (the attention output's product, scaled, and the expert sum, scaled), OLMoE-1B-7B one (its expert sum; its attention output's product is the 8-bit group's on AMD GPUs), and the dense models whose output and down products are the 8-bit group's none. OKLM_VK_FUSE=0runs every op on its own;oklm_vulkan_step_stats()counts the fused routings (fused_route), gate / up / SwiGLU dispatches (fused_glu) and residual tails (fused_res).
tests/core/test_vulkan.c checks the residual tails on
both GPUs: a product of F32 and F16 weights (one and three columns) and
a MoE sum (one, two and five tokens), with and without the scale, the
add's operands in either order, each against the same ops with a wait
after each on the device bit for bit (the output and the stream), the
MoE sum's stream against the CPU's arithmetic too, the counter showing
the fusion ran; and that an accumulating product, a multiply after the
scale and a broadcast add are left op by op.
tests/core/test_vulkan_moe.c checks on both GPUs: the
choice at 1 to 1000 experts, with ties (coarse probabilities), a bias,
DeepSeek V3's 8 groups of 32 keeping 4, one expert per group, 64 chosen;
the fused routing (Granite's and OLMoE's shapes, DeepSeek V3's groups
and bias, 600 and 1000 experts, a prompt, in place or not, a held
softmax whose next op is another) and the fused gate / up / SwiGLU (the
15 types at decode, two tokens with a row per rank, Granite's and
OLMoE's sizes, a prompt left unfused), each against the same ops one by
one on the device bit for bit, the counters showing the fusion ran, and
against the CPU or a double reference; the weights normalized or not,
scaled or not; the sums at 1 to 8 ranks (all three bit for bit); the
indexed products of 15 types (F32, F16, BF16, Q4_0, Q5_1, Q8_0, Q2_K to
Q6_K, IQ4_NL, IQ4_XS, MXFP4, IQ2_XXS) at decode (8 of 16 experts, the
row shared) and two tokens with a row per rank, longer rows, then the
grouped path for six types (160 pairs over 8 experts, 18 pairs, most
pairs on one expert, an OLMoE-like prompt), each against a double sum
over the dequantized weights within 1e-5 sum |w x| + 1e-6;
the same six types on the prompt tiles (160 pairs with the row shared,
18 pairs with a row per rank, most of 300 pairs on one expert over two
tiles of rows, 270 pairs with a row per rank and 131 rows), on the F32
tiles at that tolerance and on the device's own (cooperative matrices
within 2e-3 sum |w x| + 1e-6), then the default policy at
1120 pairs over 64 experts. It runs in about 10 seconds on both GPUs.
With OKLM_VK_BENCH=1 it times each op of a decode step's
MoE layer (Q4_K experts, each op repeated with a barrier between runs)
on the discrete GPUs; on the RX 7900 XT:
| Sizes | select | weights | gate (or up) | down | sum |
|---|---|---|---|---|---|
| Granite-3.1-1B-A400M (1024, 512, 32 experts, 8 chosen) | 8.9 us | 7.9 us | 6.6 us | 5.7 us | 3.7 us |
| OLMoE-1B-7B (2048, 1024, 64, 8) | 10.4 us | 7.0 us | 13.9 us | 12.5 us | 3.3 us |
| GLM-4.7-Flash (2048, 1536, 64, 4) | 8.1 us | 5.0 us | 9.7 us | 13.4 us | 2.6 us |
Decode on the RX 7900 XT (bench_llama's generation test,
128 tokens, best of 2 to 4 runs;
oklm run ... -n 128 --temp 0 gives the same):
| Model (Q4_K_M) | Vulkan | OpenCL |
|---|---|---|
| Granite-3.1-1B-A400M | 294 to 296 t/s (278 to 279 with OKLM_VK_FUSE=0) |
270 to 307 t/s |
| OLMoE-1B-7B | 312 to 314 t/s (303 to 306 with OKLM_VK_FUSE=0) |
285 to 291 t/s |
OLMoE's gate and up products go to the 8-bit group on AMD GPUs (rows
of 2048, see the policy above), so only its routing is fused there; with
the float products and both fusions (OKLM_VK_Q8=0) it runs
at 324 to 325 t/s, but without the CPU's 8-bit results. | GLM-4.7-Flash
| 78 to 79 t/s | 72.5 to 73.5 t/s |
The products use F32 activations, as the one-column products do, where the CPU and OpenCL round them to 8 bits for quantized weights, so greedy output departs from theirs after a few tokens, as a dense model's does.
Prompts on the RX 7900 XT (bench_llama, 512 tokens in
one batch, the best of two or three runs, 2026-11-14), the grouped
products on mvid_<type>
(OKLM_VK_MOE_TILES=0, every prompt until that day) and on
the prompt tiles, and OpenCL's grouped tiles in the same session; decode
beside them (128 tokens, the same dispatches either way):
| Model | Pairs an expert | mvid_<type> |
Prompt tiles | OpenCL | Decode, Vulkan / OpenCL |
|---|---|---|---|---|---|
| Granite-3.1-1B-A400M Q4_K_M | 128 | 8365 t/s | 20500 | 17880 | 634 to 638 / 392 |
| OLMoE-1B-7B Q4_K_M | 64 | 3035 to 3074 | 9060 to 9190 | 8790 | 477 to 485 / 313 |
| Tini-Cybersec-8B-A1B Q4_K_M | 64 | 2512 to 2522 | 7720 to 7740 | 6900 to 6960 | 341 / 255 |
| gpt-oss-20b Q4_K_M (MXFP4 experts) | 64 | 844 | 2149, then 3738 | 1676 to 1681, then 3313 | 96, then 164 / 127 |
| Gemma 4 26B-A4B UD-Q4_K_M | 32 | 1304 | 2716 | 2224 | 106 / 72 |
gpt-oss-20b's second figures are with MXFP4's values computed in
registers (One-column products,
dq_fp4()), OpenCL's with its indexed products on the float
matrix units (opencl.md). Its
decode step was 10.34 ms, 6.67 of them the 72 indexed products (four
experts of 2880 x 2880 each, 92 us a product, 190 GB/s of weights),
measured by leaving each kind of op out of the step: the decoder read a
constant array once per weight. Without the array a product takes 34 us
(520 GB/s) and the step 6.10 ms: 2.43 ms the indexed products, 0.8 ms
the rest of the 24 expert layers (the router and its bias, the choice,
the softmax of the chosen scores, three gathers and three adds for the
biases, the gated unit, the sum: about twelve dispatches a layer, where
a model without expert biases has five), 2.9 ms the attention blocks and
the head. Tini-Cybersec-8B-A1B, whose experts are Q4_K and Q6_K, kept
its rates (7859 / 339.7 before, 7824 / 342.2 after) and was above
OpenCL's before: the cause was the type's, not the expert path's. A
segment of 32 invocations instead of 64 for gpt-oss's rows of 90 groups
(each invocation then takes 3 groups or 2, where it takes 2 or 1) gave
166.8 against 163.6 t/s and is not used: it changes the order of the
sums.
Where the time went on Tini (24 layers, 22 with experts: 2048 pairs
over 32 experts, gate and up 2048 x 1792 in Q4_K, down 1792 x 2048 in
Q4_K or Q6_K; each op timed with a wait after it): on
mvid_<type> an indexed product took 2.7 to 2.9 ms,
180 ms of a prompt's 220, 5.6 TFLOPS where the dense products of the
same prompt ran at 35 (2048 x 6144 by 512 columns in 0.37 ms); on the
tiles 0.65 to 0.73 ms, 44 ms of a prompt's 80, 21 to 23 TFLOPS. No op of
these prompts runs on the CPU but the token lookup, and nothing is read
back between dispatches. What the tiles still lose to a dense product is
the partly filled ones: a tile computes its 128 columns whatever it
holds, and 2048 pairs over 32 experts lay out 48 tiles (16 would hold
them, each expert's last tile being partial and 32 entries of the table
unused).
Hybrid layers, rotation and latent attention
vk_ops_ssm.c and shaders/ssm/ run the Gated
DeltaNet block (Qwen3-Next, Qwen3.5), the rotation around a quantized KV
cache, and the attention of heads wider than the attention group takes
(latent attention's 576, and the 512 of Gemma 4's blocks without window,
whose window blocks' heads of 256 are the attention group's).
| Op | Shader | Takes | Left to the CPU |
|---|---|---|---|
| SSM_CONV | ssm_conv |
F32, rows contiguous (views of wider columns), any K >= 2 | other types |
| GATED_DELTA | gated_delta |
F32, key heads up to 256, any value head; a decay per head or per
token (oklm_gated_delta_tok(), the published
qwen35 layout) |
key heads over 256 |
| HADAMARD | hadamard; hadamard_fold with a sign
vector, reordered heads or groups above 256
(oklm_hadamard_fold()) |
F32, groups of 2 to 4096, any strides of 4-byte elements; the signs F32 on the instance | other types; the second shader on a device with less than 16 KiB of shared memory |
| ATTENTION | mla_attn, mla_merge |
heads of 257 to 576 (a multiple of 8), F16 or F32 caches on 16 bytes, the values the keys' tensor or one with its strides; causal, bidirectional, window, soft-cap, block table, per-token tables | heads up to 256 the attention group does not take, quantized caches of wide heads |
ssm_conv: one invocation per channel over the tokens in order, the inputs before the first token read from the ring, then the ring takes the last K-1 inputs. The sums are the CPU's (precise, its order); the SiLU uses the device'sexp, so the outputs are within 1e-6 absolute plus 1e-5 relative, and the ring is bit-identical.gated_delta: the rows of a head's state are independent once a token's q and k are normalized, so a work-group takes up to 32 rows of one value head (four work-groups per head of 128) and keeps them in registers over every token: 32 invocations per row, each holding the columnslane,lane + 32, ... Per token the group loads q and k into shared memory, reduces their squares, sums each row'sS kover its 32 invocations, applies the rank-1 update and sumsS qinto the output. The state is read once and written once per call, so a prompt does not reread it per token. With a decay per token every invocation computes it from the token's projection (exp(factor * softplus(projection + bias)), the softplus linear above 20 as on the CPU), the key heads repeat over the value heads and the normalized query is divided bysqrt(dk); outputs and states are within 1e-5 absolute plus 1e-4 relative of the CPU's in both forms (test_vulkan_ssm, test_gated_delta).hadamard: one work-group of 64 per group, the butterflies of each stage in shared memory in the CPU's order, then the scale.hadamard_fold: the same with 16 KiB of shared memory (groups up to 4096), for the weights stored in a rotated basis (Ternary Bonsai 2, hybrid-layers-design.md): each element times its sign before the butterflies (a product's input) or after the scale (an embedding row turned back), and a row's value heads read in the order they were folded in. Against the CPU: 1e-6 absolute plus 1e-5 relative (test_vulkan_ssm, test_hadamard_fold).mla_attn: work-groups of 8 rows, a row a (query head, token) pair; the heads of a group that read one KV head share a work-group (latent attention's one shared head: 8 heads per work-group), with several tokens when the group is smaller in a prompt; a single token, or a batch with per-token tables (whose rows must share a table), puts one token in a work-group with as many heads of the group as fit (5 of GLM-4.7-Flash's 20). Keys go by tiles of 32: each invocation computes one (row, key) score over the whole head (8 halves per load), one invocation per row updates its running maximum and denominator, and each invocation adds the tile's weighted values into its dimensions of the 8 rows, so a value row is read once for 8 heads.- Key ranges: a work-group's n keys (from its rows' first visible key)
are cut into ranges of max(64, ceil(n / ns) rounded up to 32) keys, so
it takes at most ns ranges, and
mla_mergecombines their partials (scratch slotOKLM_VK_SCRATCH_SSM, ns per (token, head), so it keeps its size from one decode step to the next). With one token per work-group, ns comes from the head blocks alone (256 work-groups over them, at most 32: 32 for GLM-4.7-Flash, 16 for DeepSeek V3's 128 heads), so a query's ranges follow its own keys and its result is the same, bit for bit, decoded alone or with other sequences;test_vulkan_ssm.cchecks this at 150, 1500 and 4000 keys. Prompts (several tokens per work-group), and per-token batches whose partials would pass 128 MiB, take ns from their work-group count instead (up to 256 work-groups, 1 when there are more: no merge). The grid has the ranges of the longest work-group, min(ns, ceil(keys / 64)), dispatched indirectly; a work-group past its own ranges only marks its partials empty, and the merge skips empty partials. - Replay:
pos0, the cache's keys and the grid's range count are per-step data, and the merge runs whenever ns is above 1 (a constant of the model and the batch shape), so a latent attention decode step replays.
tests/core/test_vulkan_ssm.c checks each against the CPU
on both GPUs: the rotation at n = 2, 64, 128 and 256 (and in place)
within 1e-6 plus 1e-5 relative; the convolution then the recurrence over
a decode call and a prompt call continuing it (ring and state carried),
at heads of 128 (also 70 tokens), heads that are not multiples of the
shader's blocks (48 by 40, 64 by 200), and a key head of 512 that takes
the CPU fallback (its state written back): the ring bit-identical, the
outputs and the state within 1e-5 plus 1e-4 relative; latent attention
at 576 (16 and 20 heads decoding over 1 to 2000 keys, prompts of 37
tokens over 100 keys and 70 tokens of 128 heads over 1100 keys, an F32
cache, window with soft-cap, bidirectional, a block table, per-token
tables over two sequences) and heads of 320 and 512 with values of their
own, within 2e-6 plus 2e-4 relative; three sequences decoded together
against each alone (per-token tables and a block table), bit for bit;
twelve latent decode steps (a growing cache whose range count changes,
then three paged sequences) recorded twice and replayed ten times; and
decode timings at 20 and 128 heads. test_vulkan.c's
convolution check compares its outputs within the same 1e-6 plus 1e-5
relative, its ring bit for bit.
Norms, rotation and copies
The norm group (vk_ops_norm.c,
shaders/norm/, test_vulkan_norm.c) runs these
ops on the GPU; an operand set it does not take goes to the CPU
fallback.
| Op | Shader | Takes | Against the CPU |
|---|---|---|---|
| RMS_NORM, LAYER_NORM | norm_rows |
F32 rows (contiguous elements, any row strides), optional F32 weight and bias | within 1e-5 + 1e-5 relative |
| ROPE | norm_rope |
normal and NeoX pairs, partial rotation, frequency factors (with one
position a token or three), linear scaling, YaRN; one position a token,
or three with a band of pairs each (oklm_rope_parts(),
M-RoPE) |
within 2e-5 + 2e-4 relative |
| SOFTMAX | norm_softmax |
scale, optional F32 mask broadcast over rows; a fully masked row gives zeros | within 1e-7 + 1e-5 relative |
| GET_ROWS | norm_get_rows |
tables of F32, F16, BF16, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, IQ4_NL, Q4_K, Q5_K, Q6_K | bit-identical |
| SET_ROWS | norm_store_rows |
F32 rows into F32, F16, BF16, Q8_0, Q4_0, Q4_1, Q5_0, Q5_1, IQ4_NL | bit-identical |
| CPY (after the element-wise group) | norm_store_rows, norm_copy_rows |
F32, F16 or BF16 (any strides) into the six block quants above; rows of one type as bytes (one buffer copy when both are contiguous) | bit-identical |
| ARGMAX | norm_argmax |
F32 rows | identical indices |
| TOPK | norm_topk |
F32 rows, k up to 512, with or without the values | identical indices, order and values |
| CROSS_ENTROPY | norm_cross_entropy |
F32 logit rows, I32 targets | within 1e-6 + 1e-5 relative |
Left to the CPU: get_rows from the other quantized types (Q2_K, Q3_K, IQ4_XS, the IQ grid types, MXFP4, NVFP4, the ternary and low-bit types); copies into or out of the other quantized types and into or out of I32 other than byte copies; top-k beyond 512; rows whose elements are not contiguous for the row ops (as the CPU requires).
How the results stay close to the CPU, whose kernels are built without contraction, sum in double and call libm:
- Vulkan rounds an add, a subtract and a multiply correctly but allows
2.5 ulp for a division, a loose
exp, and an absolute error of 2^-11 forsinandcos; a compiler may also fuse a multiply and an add. Every expression that must match isprecise(never fused), divisions that feed bits (the quantizers' scales, the RoPE angle's frequency factor) go throughdiv_rn(), a correctly rounded division on the integer bits (the 27-bit quotient formed 8, 8, 8 and 2 bits at a time by 32-bit divisions: the devices divide 64-bit integers in software), andexp,sinandcosare the shaders' own (norm_common.glsl): within about one ulp, the angle reduced exactly by pi/2 in six 7-bit parts up to |x| = 2^17 pi / 2. - The block quantizers are the CPU's reference algorithms step by step
(the same comparisons,
roundfas rounding halves away from zero, the same truncating conversions, IQ4_NL's 15 candidate scales), so the KV cache rows written on the GPU are the CPU's bytes. Denormal inputs are an exception where the driver flushes them. - Norms, softmax and cross-entropy run one work-group per row (64 to
256 invocations by row length) and sum in two floats (a high part and
its rounding error), close to the CPU's double sums; the norm's scale is
the inverse square root refined by one Newton step. In
norm_rowseach invocation takes a contiguous run of the row (8, 16 or 32 values at a time in registers, the pipeline's second specialization constant), adds its squares in two floats, and the invocations' sums are added sixteen at a time in two rounds of shared memory (wg_sum_ff16()); the mean square of a row whose length is a power of two is a product by the inverse (div_n(), the same bits as the rounded division). - RoPE builds the angle as the CPU does, the position times theta_scale k times in a row, so the angle is the CPU's to the bit; the per-op constants (theta_scale, the YaRN range and magnitude) are computed on the host with the CPU's expressions.
- Argmax and top-k compare floats with both zeros equal and take the lower index on ties, as the CPU's scans; top-k finds the k-th largest key with four byte histograms, then sorts the k candidates (bitonic) in shared memory. A NaN is never chosen by argmax; NaN values in top-k rows are not ordered as on the CPU.
- Measured on both GPUs (the RX 7900 XT and the UHD 750 give the same bits), the float ops differ from the CPU by one or two ulp at most: the norms within 5e-7 absolute on values near 2, RoPE within 4e-7, softmax within one ulp, so the checks hold with margin.
- Speed, one op at a time with a wait at the end (RX 7900 XT / UHD 750), before the shaders were rewritten for latency: an RMS norm of a 4096 row 20 / 25 us, RoPE of 32 heads of 128 6 / 26 us, softmax of a 151936 row 0.41 / 0.73 ms, top-40 of it 0.70 / 1.04 ms. In a chain of dependent ops on the RX 7900 XT an RMS norm of a 2048 row now takes 3.6 us (10.4 before) and of an 8192 row 7.2 us (21.8): see Latency of a decode step. The vocabulary-sized rows are one work-group each, as on OpenCL.
- None of these ops takes a value that changes from token to token in its push constants: positions, row ids and slots are read from their device tensors, so a recorded decode step replays them as they are.
- An id or target outside its table (which the CPU refuses) gives a row of zeros (get_rows), no write (set_rows) or NaN (cross-entropy).
Attention
vk_ops_attention.c and shaders/attention/
run ATTENTION as oklm_attention() defines it: grouped-query
heads, pos0, sliding window, soft-cap, bidirectional
(encoders, more queries than keys), queries that see each other
(mutual, an image's rows in Gemma 3: on
attn_split, every query's last key the last query's and its
window's start its own), a span of keys a query
(oklm_attention_spans(), bidirectional: the windowed blocks
of Qwen2.5-VL's encoder and a text encoder's batch of several sequences,
each row over its own sequence's rows; on attn_tile only),
and the paged cache through its block table
(oklm_attention_paged(), any block size) or through
per-token metadata (oklm_attention_paged_tokens()). Cache
types: F32, F16, BF16, Q8_0, Q4_0, Q4_1, Q5_0, Q5_1 and IQ4_NL (a
Walsh-Hadamard-rotated cache is rotated by the caller, as on the CPU).
Heads up to 256, a multiple of 4 (of 32 for a block-quantized cache);
wider heads (latent attention's 576, the 512 of Gemma 4's blocks without
window) go on to the hybrid group. Attention sinks (gpt-oss: one score
per query head that joins its softmax without a value) are an F32 tensor
on the device whose address is a push constant: both shaders add a
head's sink where its output is normalized, exp(sink - max)
with the maximum taken over the scores and the sink (in
attn_split for one range, or where the ranges are combined;
in attn_tile at the end of the block's keys). The hybrid
group has no sinks: under heads wider than 256 the op runs on the CPU,
the sinks read back with the rest.
attn_split(decode, a few tokens, per-token metadata): one work-group of 128 per (key range, run of the query heads sharing a KV head, token). A run is the largest divisor of the group up to 8, so each key row is read once for all the heads of the run. One invocation per key computes the run's scores into shared memory, a key's scores side by side (two keys in flight when the range has more keys than invocations); a range of at most 64 keys (every range of a query of up to 1024 keys, so all of a chat's decoding) gives each key two invocations, half of the heads each, so a wave reads half of the queries. The maxima and sums are reduced with subgroup operations; for the values each invocation owns four dimensions and a share of the keys (four rows and their probabilities read before the sums). The visible keys are split into ranges of at most 512 keys, more of them (of 64 keys at least) while the work-groups are fewer than 128 on a discrete GPU (64 otherwise), at most 256 ranges (128K keys per query; beyond, the CPU). With several ranges each writes its (max, sum, output) to the group's scratch buffer, and the last work-group of a (run, token) to finish, found with an atomic counter that it resets afterwards, combines them in the same dispatch (flash-decoding without a second dispatch and its barrier).attn_tilewith a span of keys a query (ATT_SPANS, the table of first key and one past the last at the block table's address): a block walks the tiles from the first key any of its tokens sees to the last, each row masking the keys outside its own span as it masks those past its position in a prompt, so the windows of an encoder's block are one dispatch whatever their sizes and wherever they fall in the blocks of rows. Every cache type.attn_splitdoes not read the table: with fewer rows than half a block, or a block whose shared memory the device does not have (heads of 256 on the UHD 750), the op returnsOKLM_ERR_UNSUPPORTEDand the caller attends span by span (one dispatch a window, as before).attn_tile(prompts, when a block's rows are at least half used): a block of 32 query rows (16 for heads over 128, and on a discrete GPU for heads over 64;OKLM_VK_ATTN_ROWS=16or32asks for one), the heads of a run times tokens, walks tiles of 32 keys held as floats in shared memory (rows padded by one), any cache type dequantized on the way in: S = Q K^T in 4 x 2 register tiles (4 x 1 for heads over 128), masked; one invocation per row updates its maximum and sum; O = O f + P V in 4 x DO register tiles. (A variant with the queries and keys dimension-major and the softmax spread over 4 lanes per row measured 1.8 times slower on the 7900 XT, 1.5 times faster on the UHD 750.) Other tilings measured on the 512-token prompt layer of the table below (2026-11-12, the 7900 XT shared with other jobs, each beside the shader as it is in the same session) and not kept: shared memory as vectors of four dimensions, a dot product per load, 508 against 418 us (and 13.7 against 41.7 ms on the UHD 750, where prompt attention is 2 % of a prompt); eight rows an invocation, 694 against 418 us; then, best of 3 to 6 runs against 337 us: work-groups of 64 with 8 x 8 register tiles on adjacent floats, as the F32 prompt products, blocks of 64 rows by 64 keys in 53 KB of shared memory, 592 us (310 us of it the products, 137 us the tiles' loads, 190 us left with both and the exponentials removed); work-groups of 64 with blocks of 16 rows, 442 us; tiles of 16 keys, 297 us, within the runs' spread. Fewer shared-memory loads per product did not pay in any of them. What did, for heads of 128, is a smaller block: a work-group's shared memory (its queries, a tile of keys, the scores) bounds how many of them a compute unit runs at once, 37 KB with blocks of 32 rows and 27 KB with 16, and the Qwen3-8B prompt layer of the table went from 1 301 to 886 us with 16 (the 512-token prompt from 2 330 to 2 470 t/s). Heads of 64, at 21 KB, were slower with 16 rows (Llama-3.2-1B: 14 950 against 15 180 t/s).attn_common.glslholds what both share: four elements of a row of any cache type (16-bit reads for the block quants, whose blocks are aligned to 2 only), the block table, tanh, the window. A cache's halves (an F16 row, a block quant's scale) are converted by the device (unpackHalf2x16(), exact: every half is a float), not bycommon.glsl'sf16_to_f32(): its integer arithmetic, some twenty instructions per value, ran four times per load in the one invocation that walks a key row, about 25 ns per value, and was most of a decode dispatch (see the timings below).attn_splitkeeps its queries in shared memory four dimensions an element, read with one instruction.
Decode-step replay: pos0, the number of keys (the views'
ne[2]) and the number of ranges go to per-step data, and
the split grid is dispatched indirectly; the push constants hold only
addresses, strides, the batch's sizes and flags, so a decode step over a
growing cache replays (checked: six steps over 2041 to 2046 keys, the
ranges changing on the way, and six steps of three paged sequences
through per-token metadata; one recording and five replays each, no
replay given up). The scratch buffer is sized for at least 64 ranges,
then powers of two, so it seldom grows (a growth waits for the device
and ends the step's recording).
The scores, the softmax and the sums are in F32 and the query is not
rounded (the CPU's kernel quantizes it for its integer dot products over
a block-quantized cache, so those are checked against F32 attention over
the dequantized cache, as on OpenCL).
test_vulkan_attention.c checks against the CPU within 2e-6
absolute plus 2e-4 relative (OpenCL's tolerance) on both GPUs: decode at
heads of 32 to 256 and groups of 1 to 16 (7 included), prompts with
partial blocks and keys before the prompt, window and soft-cap,
bidirectional, a span of keys a query (test_spans: windows
of unequal sizes, of 1 to 100 rows, smaller than a tile and over
several, blocks of rows that straddle windows, one window over
everything, 2 to 16 heads, two a KV head, a soft-cap, spans that are not
windows, F32, F16, BF16 and Q8_0 caches, both block sizes; 5 rows
refused), every cache type, the paged cache (shuffled blocks, a block
size of 24, per-token metadata of three sequences), 8K keys, replayed
steps, prompts on both block sizes of the tiles
(OKLM_VK_ATTN_ROWS=16 and 32: heads of 64, 96
with a group of 7 and a window, 128, a Q8_0 cache with a soft-cap); the
device's conversion of halves on an F16 value row of chosen bit patterns
(zeros, the smallest and largest denormals and normals, random finite
halves) read back through one key, exactly the CPU's values, on
attn_split and on attn_tile; then it prints
the timings below.
Timings (2026-11-12, one layer, F16 cache, the mean of back-to-back
runs, in microseconds; OpenCL is OKLM's OpenCL backend on the same 7900
XT; the GPUs were shared with other jobs, so a few percent of noise;
"before": the halves converted by f16_to_f32(), the same
session):
| Shape | 7900 XT Vulkan | before | 7900 XT OpenCL | UHD 750 Vulkan | before |
|---|---|---|---|---|---|
| Llama-3.2-1B decode (32 heads of 64, 8 KV heads), 1 key | 5.8 to 6.3 | 8.5 to 8.9 | 7 | 14 | |
| the same, 48 keys | 7.3 | 10.6 to 10.8 | 17 | ||
| the same, 112 keys (two ranges) | 10.5 | 14.0 to 14.4 | 50 | ||
| the same, 2048 keys | 33 | 41 | 59 | 261 | 287 |
| the same, 8192 keys | 92 to 93 | 103 to 104 | 141 | 723 | 1065 |
| Qwen3-8B decode (32 heads of 128, 8 KV heads), 48 keys | 10.8 | 17.0 | 26 | ||
| the same, 112 keys | 15.5 | 22.2 to 22.5 | 68 | ||
| the same, 1000 keys | 19.6 | 26.6 | 346 | ||
| the same, 4096 keys | 68 to 69 | 78 | 145 | 729 | 1060 |
| OLMoE-1B-7B decode (16 heads of 128, 16 KV heads), 48 keys | 7.8 | 13.8 | 29 | ||
| the same, 112 keys | 10.5 | 16.7 | 68 | ||
| Llama-3.2-1B prompt, 512 tokens | 417 to 434 | 431 to 440 | 327 | 38000 | 41700 |
| Qwen3-8B prompt (32 heads of 128, 8 KV heads), 512 tokens | 808 to 817 | 909 | 68600 | 70800 |
Decode is bound by latency more than by bandwidth (2 MB of cache per
1K keys go at about 280 GB/s at 32K keys): each work-group waits on a
chain of dependent reads (query, keys, values, the partials for the
merge), and on what each invocation computes between them. Where a short
decode's time went before (the dispatch cut after each part, 48 keys;
Llama-3.2-1B, Qwen3-8B and OLMoE layers): an empty dispatch 1.6 us; the
per-step data and the queries into shared memory 0.7, 1.1 and 0.5 us;
the scores 4.0, 7.5 and 6.7 us (1.5, 4.1 and 1.0 us with the device's
conversion); the maxima, the exponentials and the sums about 1 us (two
shared-memory rounds); the values and the output 3.3, 5.5 and 5.1 us
(2.5, 4.2 and 3.6 us). llama.cpp's Vulkan backend takes 9.2 to 9.3 us
for the attention of a Qwen3-8B or OLMoE token of the same benchmark
(GGML_VK_PERF_LOGGER).
Measured on the same shapes and not kept (us, in the order Llama-3.2-1B 1, 48 and 112 keys, Qwen3-8B 48 and 112, OLMoE 48 and 112; the shader as it is: 6.2, 7.3, 10.5, 10.8, 15.5, 7.8, 10.5):
- a head's maximum and sum from subgroup operations alone when a short range's scores of one head sit in one subgroup (a pipeline of full subgroups of 64), the probabilities normalized at once, the key loads in batches of eight: 4.7, 6.2, 9.1, 10.5, 14.9, 6.7, 9.2 (one key load at a time: 5.0, 6.4, 9.4, 9.6, 14.3, 7.6, 10.4). At most 1 us of a dispatch, under 1 % of a token, for a second pipeline variant;
- on top of it, the value sums of a column added across its invocations with a clustered add (no shared-memory round): within 0.2 us;
- one range for up to 128 keys instead of two of 64 (no merge): 6.1, 7.3, 10.3, 10.8, 16.3, 7.7, 13.1.
Fallback
An op no group takes runs on the CPU backend over host copies of the
byte ranges its operands touch, then the destination range is written
back, and the source an op updates in place too (the convolution's ring
of SSM_CONV, the state of GATED_DELTA); it is logged once per op.
Operands in host memory (token embeddings) are used as they are. Every
op therefore works on this backend: the tiny model's greedy output
through --backend vulkan is the CPU's.
The model and the decode step
The model on Vulkan
--backend vulkan (or cp.backend = "vulkan",
"vulkan1" for the second instance) places a model exactly
as OpenCL does: the forward passes, the placement and the memory
planning go through the backend interface only. The last
--gpu-layers layers (by default as many as fit) and the
head are on the device, the first ones and the token lookup on the CPU;
the weights are copied once per model, the KV cache (contiguous or
paged, for every sequence of the context), the recurrent states of
hybrid layers and the activations live in device memory;
--gpu-stream streams the CPU layers for prompts through the
plain upload; contexts spread their layers over the other Vulkan
instances unless n_devices is 1. oklm run,
chat, serve, plan and the state
files (--kv-save) work the same way; an op no shader takes
yet runs on the CPU (see Fallback), so the output is the CPU's.
The fit (fit_layers,
oklm_context_mem_estimate(), the server's paged blocks)
reads the device-local heap and the largest allocation from the Vulkan
device (maxMemoryAllocationSize, 4 GiB on RADV and ANV).
The weights and the KV cache are split into several allocations when
they exceed it: the weights by runs of tensors (a run never straddles
two allocations; each tensor is addressed by itself), the cache by whole
layers, so a layer, the head and one layer's cache must each fit in one
allocation, not the whole model. OKLM_GPU_MAX_ALLOC_MB sets
a smaller limit (to test the split on a small model, on any device
backend).
Checked on both GPUs (tests/model/test_model.c,
test_vulkan; the CPU fallback taking every op no shader
takes yet): the tiny models of every architecture against their float64
references (F32 and F16 caches, a CPU layer before the device ones,
nothing on the device, routed experts on the CPU, a streamed CPU layer),
batches against single tokens, several sequences in one context
(contiguous and paged), argmax-only items, swapping a sequence out and
in; state files to and from device caches and hybrid states
(test_kv_file.c); layers spread over two instances of one
device (test_multi_device.c,
OKLM_VK_DEVICES=0,0).
The public models on the RX 7900 XT
(tests/model/test_infer_public.c, test_vulkan,
the float products: OKLM_VK_Q8=0,
OKLM_VK_COOPMAT=0), with OpenCL's bounds: against the
float64 reference SmolLM2-135M (F16 and seven quantized files) within
3.2e-5, Gemma 3 270M 3.2e-4, GPT-2 7.3e-4, the Gemma, Phi-3 and Phi-2
files within 2.5e-3, every greedy choice equal; against llama.cpp
(Qwen2.5, Qwen3 0.6B and 8B, GLM-4-9B, Llama-3.2-1B, TinyLlama, Gemma 1
to 3, Phi-3, OLMoE) its tokens, 153 to 160 of 160 choices on its tokens,
the continuations parting only at its near ties (gap at most 0.5). With
the cooperative matrices (the default), a prompt's activations are
rounded to F16: SmolLM2-135M then within 0.05 of the reference, its
greedy choices still equal.
Decode-step replay
The model marks a decode step (a batch of at most 8 tokens per
sequence, 64 in all, every device layer on the head's device, nothing
streamed) with the backend's optional step_begin(key) and
step_end(); the key hashes what makes two steps the same:
the context, the batch's sequences, rows and token counts, the logits
rows and the device layers. On Vulkan:
- The first step of a key is recorded into reusable command buffers (a graph, kept per key) as it runs, and every command is logged: the pipeline, the push bytes, the group counts, the buffers and offsets of copies and fills, the barrier.
- A later step with the same key is replayed. The forward pass runs on the host as usual and each op fills its push constants, but each command is compared with the log instead of being recorded, and the recorded command buffers are submitted again as they are: no command is recorded.
- A recording is cut into parts, command buffers of 16, 64, 256, ... commands (the eighth takes the rest), and each part is submitted as soon as its last command is recorded or, on a replay, matched: the device runs the step's first commands while the host goes through the following ones. With one command buffer submitted at the step's end the device waited for the whole forward pass, about 0.35 us per op on the host (see the profile). The barriers follow the ranges read and written across the parts as within one.
- A command that differs gives up the replay: the commands that matched since the last part submitted go into the ring, and the step runs on directly (the graph is recorded again on the key's next step). A host operation inside the step (an op on the CPU reading device data, a download, an upload of more than 64 KiB, a wait) ends the recording or the replay the same way, and the key is then run directly. Either way the results are the same; only the time differs.
- Which keys are kept. An instance keeps 16
recordings (
OKLM_VK_GRAPHS, 1 to 64); each gets its command buffers and its per-step data (256 KiB) when it is first recorded. While one is free, a new key is recorded at once. Once all are taken, a key seen for the first time runs directly and is remembered (the last 64 such keys); only a key seen before replaces the least recently used recording. A step whose shape does not come back (a speculative step's draft lengths) thus never pushes out the shapes that do. OKLM_VK_REPLAY=0runs every step directly;oklm_vulkan_step_stats()counts the steps recorded, replayed, broken and aborted, the command buffers of recordings submitted (parts), the recordings evicted, the new keys run directly, the barriers and the commands recorded without one; the instance logs them when it is released.
The values that change from token to token must reach the shaders through device memory, never through push constants or group counts, or the step never replays:
| Value | Where it changes | How it reaches the shader |
|---|---|---|
| positions (ROPE) | every token | the pos tensor, uploaded before the step (already an
operand); the three positions of an image's rows in a model with M-RoPE
follow the batch's one-part positions in it |
| the cache cells written | every token | contiguous caches on Vulkan are written with SET_ROWS through the
context's slots tensor (uploaded before the step) instead
of a CPY into a view at the position; paged caches through the block
tables, as before |
| the token's row of the embedding table | every token | GET_ROWS from the host's table: computed on the host and uploaded inside the step (a recorded copy from per-step data, rewritten on each replay) |
| the residual stream handed from CPU layers | every token | uploaded into x (as above) |
ATTENTION: pos0 (iparams[0]), the number
of keys (the key and value views' ne[2]) |
every token | per-step data (below) |
paged ATTENTION: pos0, max_keys
(iparams[5] of the multi-sequence form) |
every token | per-step data |
| a grid that follows the number of keys (split attention, latent attention's key ranges) | every token | oklm_vk_dispatch_indirect() |
Per-step data: an op calls
oklm_vk_step_data(s, bytes, &addr) (or
oklm_vk_step_u32(s, values, n)), writes the values at the
returned host pointer, pushes addr (a
uint64_t) instead of the values, and its shader reads them
with RefU32(addr + 4 * i).v. Within a step the data lives
in the graph, at the same address on every replay, and the host writes
the new values there before the part that reads them is submitted again
(so an op writes its values before it records the command that reads
them: the part may be submitted by that very command); outside a step it
lives in a small host-visible buffer of the command buffer being
recorded. An op may ask for up to 4 KiB per call to its group. A grid
that depends on such a value is dispatched with
oklm_vk_dispatch_indirect(s, p, &push, gx, gy, gz),
whose counts are per-step data too. Everything else an op pushes (tensor
addresses, strides, sizes of the batch, flags) is the same from token to
token in a decode step and stays in the push constants. An op that
pushes a per-token value directly is still correct: its steps only run
without replay (the comparison fails at its command), which
oklm_vulkan_step_stats() shows as aborted replays.
Observation
A context that records its activations (--observe, observe.md) copies the rows it wants
into staging memory on the device, one oklm_cpy() each:
commands of the step like the others, with the same addresses from token
to token, so the step is recorded and replayed with them, and the
staging memory is read back once after step_end(), with the
keys the attention rows need (oklm_vk_download_many(),
sixteen reads a wait at most). The staging memory is allocated before
the first step starts (model_observe_steps()).
A pass that has to read the device before it ends is not marked as a
step (forward_step() in src/model/model.c):
the lens's logits, the means' rows, parts taken out, more than eight
sequences in the batch. Nothing is recorded or replayed then, each op is
submitted as in a prompt's pass and each read waits for the queue. A
context that does not record is not concerned: the test is on the host,
once a pass.
Latency of a decode step
A decode step is a chain of a few hundred dispatches, most of them waiting for the one before (a memory barrier), so on a discrete GPU its time is the sum of each dispatch's latency plus the weights' bytes over the memory bandwidth. Measured on the RX 7900 XT (Llama-3.2-1B Q4_K_M, 203 commands and 163 barriers a token, 2026-11-12) by recording the step with one class of dispatch left out, or with every dispatch cut to one work-group:
| Part of a token | Before | After |
|---|---|---|
| whole step on the device (submission to completion) | 2 330 to 2 390 us | 1 930 us |
| host: the forward pass that checks the replay | 80 to 130 us | 90 us |
| every dispatch cut to one work-group (the latency alone) | 1 300 us | 760 us |
| the products' weights (800 MB) | 1 030 to 1 130 us, about 700 GB/s | the same |
| 33 RMS norms | 330 to 370 us | 165 us |
| 16 attention dispatches (128 keys at most) | 250 to 300 us | 300 us |
| 24 rotations, 32 cache writes | 140 and 160 us | 72 and 64 us |
| 16 SwiGLU | 150 us | 50 us |
| the barriers alone (no dispatch) | 28 us | 28 us |
What a dispatch's latency is made of, and the rules the decode shaders follow since:
- No 64-bit division. The devices divide 64-bit
integers in software: the five divisions that turned a 64-bit invocation
index into (element, row, i1, i2, i3) cost a work-group about 4 us, more
than the op itself. The shaders that divide their index take it in 32
bits (
GLOBAL_INDEX_1D_U32(),ROW_INDEX_U32();oklm_vk_dispatch_1d()refuses more than 2^31 invocations), and the element-wise shader skips the division when every operand is contiguous or has its rows at a constant stride (itsFLATvariants). A chain of dependent element-wise ops went from 5.5 to 1.6 us an op. - Loads cost, arithmetic does not. A load instruction
of a wave of 64 invocations takes about 0.1 us, whatever it reads, and
loads issued one after another are served one after another; an
instruction of arithmetic is a hundred times cheaper. So an invocation
reads little (the work is spread over the invocations of the work-group,
up to 256), reads everything it needs before computing (the loads are
then in flight together), and reads nothing twice.
norm_rowswas 256 invocations each reading 8 values one by one, twice, with a shared-memory round per halving of the reduction: 10 us; it is now two vector loads, the weights read beside them, and two shared-memory rounds: 3.6 us. - Cheap is not free in a long chain. A wave runs
about an instruction per nanosecond, so an invocation that does a few
thousand of them in a row adds microseconds to its dispatch whatever the
device has idle: decode attention's invocation per key converted the 64
or 128 halves of its key row with
f16_to_f32(), about 25 ns each, 4 to 7.5 us of a dispatch of 11 to 17 (Attention). A product's invocations convert two halves per group of weights and are bound by memory: there the same function costs nothing measurable. - A dispatch's floor is about 1.6 us on this device (a barrier, a pipeline, push constants and one short wave), so the number of dependent dispatches still counts once each is short.
Three more changes follow the same rules, each measured alone against
the tree before it (bench_llama, 128 tokens, Q4_K_M, the RX
7900 XT, best of 4, two rounds of interleaved runs, 2026-11-12):
| Change | Llama-3.2-1B | Qwen3-8B | Granite-3.1-1B-A400M | OLMoE-1B-7B |
|---|---|---|---|---|
| before | 465 to 466 t/s | 105.7 to 105.8 | 449 to 450 | 394 |
decode attention (attn_split): a key's scores side by
side in shared memory, two invocations per key for ranges of at most 64
keys, the probabilities read with the value rows |
476 to 478 | 107.5 to 107.7 | 460 to 461 | 407 to 409 |
MoE routing (moe_route): a score read once per
invocation, the ranking's scores sixteen a step; div_rn()
without a 64-bit division (the routing weights, the softmax) |
467 to 469 | 105.3 to 105.5 | 462 | 404 to 406 |
| the residual tail in its product or MoE sum (see Mixture of experts) | 463 to 469 | 105.3 to 105.7 | 496 to 500 | 399 to 401 |
| the three together | 472 to 482 | 107.2 to 107.3 | 527 to 532 | 423 to 424 |
The dense models' differences in the last two single rows are within the noise of the runs (about 0.5 %): they have no routing and no tail fused. Qwen3-8B with the three is 0.3 % under the attention change alone (107.2 to 107.3 against 107.4 to 107.8, repeated), most likely the host asking for every product whether it could take a tail.
Decode on the RX 7900 XT (bench_llama, 128 tokens,
Q4_K_M, best of 3 to 5, 2026-11-12; llama.cpp's Vulkan backend on the
same day):
| Model | Before | With the latency rules | With the three changes above | llama.cpp Vulkan |
|---|---|---|---|---|
| Llama-3.2-1B | 388 t/s | 471 t/s | 482 to 493 t/s | 481 t/s |
| Qwen3-8B | 87 t/s | 106 t/s | 107.4 to 107.7 t/s | 117 t/s |
| Granite-3.1-1B-A400M | 295 t/s | 446 t/s | 552 to 559 t/s | 514 t/s |
| OLMoE-1B-7B | 313 t/s | 393 t/s | 428 to 439 t/s | 481 t/s |
The last two columns are from one session, five runs of each
(llama.cpp llama-bench -ngl 99 -p 0 -n 128 -r 5); the tree
before the three changes ran at 476 to 483, 106, 461 to 464 and 399 to
404 in it. The same binaries ran 3 to 5 % slower in the session of the
table of changes above, earlier the same day (475, 107, 530 and 424 with
the three): compare runs of one session only.
What is left is Qwen3-8B (0.92 of llama.cpp) and OLMoE (about 0.9).
None of the three changes moved Qwen3-8B by more than 2 %, and neither
does running its products on the float group (109.7 with
OKLM_VK_Q8=0 against 107.4 in the same minutes; OLMoE 442
against 428, without the CPU's 8-bit results). The profile below says
where the rest was: the host's pass, which the device waited for (the
tail's check among its ops), and attention.
Profile of a decode step
Where a token's time goes on the RX 7900 XT (RADV),
bench_llama's generation test of 128 tokens, Q4_K_M,
2026-11-12, measured with the clock around the host's calls and by
recording the step with one class of dispatch left out (as the table of
the section above).
Host against device. A replayed step is the host's forward pass, which checks every command against the recording, then the device's work:
| Model | Commands (barriers) | Host pass | Of it in the backend | Step on the device | Read-back of the logits |
|---|---|---|---|---|---|
| Qwen3-8B | 525 (399) | 190 to 210 us | 180 us, 528 ops | 9 010 us | 30 us |
| OLMoE-1B-7B | 299 (227) | 127 to 150 us | 120 us, 349 ops | 2 220 us | 12 us |
| Llama-3.2-1B | 203 (163) | 95 us | 75 us, 204 ops | 1 920 us | 25 us |
| Granite-3.1-1B-A400M | 353 (293) | 210 us | 172 us, 548 ops | 1 640 us | 9 us |
The host pass costs about 0.35 us per op (the op's route, its push constants, the comparison with the recording), and with the recording submitted at the step's end the device was idle through all of it: 2 % of a Qwen3-8B token, 6 % of an OLMoE one, 11 % of a Granite one. Submitting the recording before the pass (wrong results, the per-step data being the previous token's; a bound on what the pass costs) gave 109.0 against 107.3 t/s on Qwen3-8B, 445 against 421 on OLMoE, 486 against 475 on Llama-3.2-1B and 586 against 528 on Granite. The recording cut into parts submitted as the pass reaches them (Decode-step replay) gets that without changing a result (best of 4, two alternated rounds each, llama.cpp's Vulkan backend in the same session):
| Model | One command buffer | Parts of 16, 32, 64, ... commands | llama.cpp Vulkan |
|---|---|---|---|
| Qwen3-8B | 107.2 to 107.5 t/s | 108.3 to 108.6 | 116.5 |
| OLMoE-1B-7B | 422 to 423 | 443 to 450 | 481 to 483 |
| Llama-3.2-1B | 476 to 478 | 485 to 489 | 484 |
| Granite-3.1-1B-A400M | 532 to 535 | 589 to 593 | 513 to 514 |
A first part of 8, 16, 32 or 64 commands: OLMoE 446 to 449, 443 to 450, 434 to 437 and 434 to 437 t/s, Llama-3.2-1B 491 to 496, 487, 482 to 487 and 481 to 484, Granite 587 to 594, 592 to 601, 594 to 600 and 586 to 590; 16 is kept. Each part four times the one before (16, 64, 256, ...: three command buffers for a step of 300 commands where doubling makes five) is what the backend does: measured after the attention change below, three alternated rounds, 637 to 640 against 616 to 628 t/s on Granite, 461 to 472 against 459 to 465 on OLMoE, 502 to 504 against 499 to 503 on Llama-3.2-1B, 111.3 against 110.9 on Qwen3-8B. Two parts only (16 commands, then the rest) leave the device idle while the host finishes its pass: Granite 586 to 588, OLMoE 459 to 460.
Attention. Leaving the attention dispatches out of
the recording took 647 us off a Qwen3-8B token (36 dispatches, 18 us
each) and 282 us off an OLMoE one (16, 17.6 us each), where llama.cpp's
flash attention takes 9.3 and 9.2 us
(GGML_VK_PERF_LOGGER=1): the largest difference per
dispatch of both models. Most of it was the conversion of the cache's
halves in the invocation that walks a key row (Attention); with the device's conversion (same
session, best of 4, two alternated rounds):
| Model | Before | Halves converted by the device | llama.cpp Vulkan |
|---|---|---|---|
| Qwen3-8B | 108.8 to 109.0 t/s | 111.3 to 111.7 | 116.5 |
| OLMoE-1B-7B | 443 to 449 | 464 to 465 | 481 to 483 |
| Llama-3.2-1B | 489 to 490 | 502 to 504 | 484 |
| Granite-3.1-1B-A400M | 590 to 592 | 631 to 633 | 513 to 514 |
Per kind of dispatch, Qwen3-8B (36 layers; a layer
is 13 or 16 dispatches: the q, k and v products are one dispatch of 6144
rows where the three tensors have one type, the gate and up products one
of 24576 rows, the residual adds are accumulating products). Time a
token loses when the kind is left out of the recording, before the two
changes (the step 9 010 us on the device) and after them (8 845 us from
the step's start), against llama.cpp's Vulkan backend on the same model
(GGML_VK_PERF_LOGGER=1, which times each op alone: 509
dispatches, 8 640 to 8 680 us a token under the logger, 8 570 without
it):
| Kind | Dispatches | Before | After | llama.cpp |
|---|---|---|---|---|
| products | 181 | 7 410 us | 7 500 us | 253 products, 7 254 us |
| attention | 36 | 647 us (18.0 each) | 440 us (12.2 each) | 335 us (9.3 each) |
| RMS norms (73 of a layer or the head, 72 of the queries and keys) | 145 | 384 us (2.6 each) | 410 us | 463 to 475 us (73, with their weights), the query and key norms with the rotations and cache writes below |
| rotations | 54 | 122 us (2.3 each) | 120 us | 430 to 445 us (72 dispatches: norm, weights, rotation, 36 of them with the cache write) |
| cache writes | 72 | 95 us (1.3 each) | 93 us | 91 to 103 us (36) |
| SwiGLU | 36 | 69 us (1.9 each) | 65 us | 87 to 89 us (2.4 each) |
| every dispatch cut to one work-group | 525 | 2 880 us | 2 590 us | |
| no barrier at all (wrong results) | 7 256 us | |||
| the products alone, and with no barrier between them | 181 | 7 424 and 6 608 us |
The products are 83 % of the token, and the other kinds together now
take less than llama.cpp's. Per product shape after the changes, the
8-bit group's dispatch, the float group's on the same step
(OKLM_VK_Q8=0) and llama.cpp's ops for the same
weights:
| Product | Per token | 8-bit (q8mv) |
Float (mv) |
llama.cpp |
|---|---|---|---|---|
| gate and up, 24576 x 4096 Q4_K (56.6 MB) | 36 | 86 us (660 GB/s) | 81 us | 2 x 40.6 us |
| down, 4096 x 12288 Q6_K (41.3 MB) | 18 | 71 us (580 GB/s) | 65.5 us | 65.2 us |
| q, o, and down in Q4_K: 4096 rows | 72 | 1 740 us together | 1 683 us | 1 765 us |
| q, k, v in one, 6144 x 4096 Q4_K | 18 | 24 us | 22 us | 26.5 us (three) |
| k or v alone, 1024 x 4096 (float group) | 36 | 2.9 us (Q4_K), 6 us (Q6_K) | 4.3 and 6.9 us | |
| head, 151936 x 4096 Q6_K (510 MB) | 1 | 752 us (678 GB/s) | 785 us | 701 to 712 us |
In the step, the float group's gate / up and down products are 5 to 9
% faster than the 8-bit group's, which the products timed alone do not
show (8-bit products): Qwen3-8B decodes at
113.8 to 114.0 t/s with OKLM_VK_Q8=0 against 111.3 to 111.7
(Llama-3.2-1B 503 to 509 against 500 to 504, OLMoE and Granite the
same), without the CPU's 8-bit results. Tried on the 8-bit shader and
not kept: four weight rows per subgroup instead of two, which halves the
share of the rounding (Qwen3-8B 111.2 to 111.4 against 111.1 to 111.5,
OLMoE 459 against 461 to 466); the device's conversion of the blocks'
half scales in every shader (f16_to_f32() as
unpackHalf2x16(): Qwen3-8B 111.1 to 111.2 against 111.3 to
111.6, OLMoE 465 to 466 against 470, Llama-3.2-1B 501 to 506 against 504
to 505, Granite 620 to 625 against 623 to 625: a product's invocations
are bound by memory, not by their arithmetic).
Per kind of dispatch, OLMoE-1B-7B (16 layers of 18 or 20 dispatches; the step 2 220 us before, 2 025 us after; llama.cpp 2 340 to 2 400 us under its logger, 2 080 without it). Leaving the routing out changes which experts the products read (the same ones every token, served by the device's cache), so the products' rows hold with the routing in place only:
| Kind | Dispatches | Before | After | llama.cpp |
|---|---|---|---|---|
| gate and up of the 8 chosen experts, 1024 x 2048 each (8-bit) | 32 | 720 us (22.5 each) | 593 us (18.5 each) | |
| q, k, v in one, 6144 x 2048 Q4_K (8-bit) | 8 | 117 us (14.7 each) | 3 x 7.0 us | |
| head, 50304 x 2048 Q6_K | 1 | 137 us | 138 to 200 us | |
| q, k, v, o alone, 2048 x 2048 (float) | 40 | 277 us (6.9 each) | 245 us (6.1 each) | 7.0 to 9.2 us each |
down of the 8 experts, 2048 x 1024 each (mvid) |
16 | 386 us (24.1 each) | 353 us (22.1 each) | 19.4 (Q4_K) and 24.4 us (Q6_K), the routing weights applied in them |
| router's product, 64 x 2048 F32 | 16 | 71 us (4.4 each) | 57 us (3.6 each) | 2.1 us each |
routing (moe_route) |
16 | 107 us (6.7 each) | 3.5 to 3.7 us each | |
| expert sum with the residual add | 16 | 66 us (4.1 each) | 46 us (2.9 each) | 1.9 us each |
| RMS norms | 65 | 208 us (3.2 each) | 167 us (2.6 each) | 308 to 316 us (4.7 to 4.9 each) |
| rotations | 24 | 86 us (3.6 each) | 57 us | 120 to 124 us (32, 16 of them with a cache write) |
| cache writes | 32 | 79 us (2.5 each) | 56 us | 27 to 33 us (16) |
| attention | 16 | 282 us (17.6 each) | 148 us (9.2 each) | 147 us (9.2 each) |
| SwiGLU | 16 | 60 us (3.7 each) | 43 us (2.7 each) | 27 us (1.7 each) |
| every dispatch cut to one work-group | 299 | 1 090 us | 990 us |
The read-back of the logits. The logits are copied to host memory by a command buffer of its own, submitted behind the step's last part. Waiting for the step before submitting it (the device idle for the round trip) measured 477 to 479 against 465 to 467 t/s on OLMoE, 513 to 515 against 503 on Llama-3.2-1B and 645 against 618 to 626 on Granite, but 108.7 to 109.1 against 111.1 on Qwen3-8B: the copy queued behind the step ends 125, 157, 124 and 79 us after the step would have ended alone, the copy on an idle device 61, 114, 62 and 265 us after it is submitted. Spinning on the fence instead of sleeping changes nothing (OLMoE 463 to 465 against 468 to 471, Qwen3-8B 111.1 for both), so it is not the host's wake-up. The cause is not found and the read-back is unchanged.
Where the two models stand (best of 4, two alternated rounds, llama.cpp in the same session):
| Model | At the start | Now | llama.cpp Vulkan |
|---|---|---|---|
| Qwen3-8B | 107.2 to 107.5 t/s | 111.5 to 111.6 | 116.5 to 116.8 (0.96) |
| OLMoE-1B-7B | 422 to 423 | 465 to 473 | 482 to 483 (0.97) |
| Llama-3.2-1B | 476 to 478 | 502 to 503 | 486 |
| Granite-3.1-1B-A400M | 532 to 535 | 637 to 638 | 514 |
What is left on Qwen3-8B, 390 us of a token of 8 970: the 8-bit
products against the float ones in the step (about 190 us; its cause in
q8mv is not the rounding's share nor the conversions, see
above), attention (12.2 against 9.3 us a dispatch, 105 us: the scores
and the values of a key range spread over more invocations than one per
key and one per four dimensions), the head (45 us). On OLMoE, 60 us of 2
130: the gate and up products, two 8-bit dispatches and a SwiGLU where
the float group runs one fused dispatch and llama.cpp two products of
18.5 us (an 8-bit gate / up / SwiGLU dispatch, rounding the activations
once, would be the equivalent), the routing and the router's product (3
and 1.5 us each over llama.cpp's).
Serving, several devices and the expert tier
The server. oklm serve --backend vulkan
runs what it runs on OpenCL: several requests at once
(--parallel), the contiguous or the paged cache
(--kv-mem), the shared prefix blocks, swapping to host
memory (--swap-mem), preemption, speculative steps. Each
scheduler step is one batch of the sequences that decode (and prompt
pieces); a batch of at most 8 tokens per sequence and 64 in all is a
decode step, marked for replay with a key from its sequences, their rows
and token counts. The scheduler puts the decode rows in sequence order
whatever the requests' ages, so the same requests make the same batch
from step to step: a composition is recorded once and replayed until a
request ends or arrives. With speculative decoding
(--spec-ngram, on by default) a sequence's row count varies
with its drafts, and each mix of draft lengths is a key of its own;
those that do not come back are run directly without replacing a
recording (see Decode-step replay). Measured with the SmolLM2-135M
server, 4 sequences and 8 requests of 48 tokens (2026-11-12, the RX 7900
XT), first while every step still broke at its first op on the CPU (the
counts show how keys are found):
| Recordings kept | Steps | Keys recorded | Evicted | New keys run directly | Steps whose key was kept |
|---|---|---|---|---|---|
| 4 | 95 | 26 | 22 | 20 | 49 (52 %) |
| 16 (default) | 96 | 20 | 4 | 11 | 65 (68 %) |
With 8 sequences and 12 requests, 16 recordings: 93 steps, 21 keys
recorded, 5 evicted, 11 run directly. Once the one-column products and
attention ran on the GPU, the same 4-sequence session ran 356 decode
steps: 12 recorded, 343 replayed, 1 replay given up, none evicted;
test_vulkan_serving 1030 steps: 66 recorded, 957 replayed,
2 evicted, 6 new keys run directly. Without the sequence order the
compositions came back rotated (the oldest request first) and the keys
did not repeat. test_server
(test_vulkan_serving) runs the server on the Vulkan device:
six requests on four sequences with both caches (the answers of one by
one), a paged cache too small for four requests, preemption with
swapping (the answers of each request alone), a shared system prompt
found by two requests at once; it prints the step counters.
Several devices. Layers spread over the Vulkan
instances as over the OpenCL ones (opencl.md): consecutive layer
ranges, the head on the selected instance, the residual stream handed
from one device to the next through the host; the devices are those of
OKLM_VK_DEVICES (0,1, radv,Intel,
all), a device listed twice giving two instances. A forward
pass of a spread context holds the pass lock of every device it uses,
taken in the order of their registration (server.md).
test_multi_device checks two instances of the RX 7900 XT
(0,0: the logits of one device exactly) and the RX 7900 XT
with the UHD 750 (all: the head on the first, the first
layers on the second: the tiny fixtures within 1e-4 of the first alone,
the element-wise shaders using each device's exp;
SmolLM2-135M with 10 layers streamed and random tokens: the Intel GPU
takes its Q8_0 products in 8 bits and the AMD one in floats (8-bit products), so it drifts up to 0.5 in a
logit over its 30 layers, 18 of its 20 best tokens the same; the check
asks for 18 and at most 1.5; with OKLM_VK_Q8=0 both use
floats: within 2e-5, 20 of 20), for the dense, GPT-2, MoE and DeepSeek
fixtures and SmolLM2-135M. Each GPU alone gives the CPU's greedy text on
SmolLM2-135M. A context spread over several devices runs its steps
without replay (the hand-over is a host operation). Vulkan and OpenCL
instances are not mixed: a context spreads over the instances of the
backend it selected, and a backend list such as
--backend vulkan,opencl takes the first one with a warning
saying so.
Weight streaming and the transfer queue. The
instance has a transfer queue when the device offers one: a
transfer-only family (a copy engine), else a second queue of the compute
family (RADV on the RX 7900 XT exposes the latter), with timeline
semaphores (Vulkan 1.2); OKLM_VK_XFER=0 turns it off.
stream_upload copies the host bytes into the slot's
host-visible staging (so the caller's memory is free at once) and
submits the copy on that queue, waiting for the compute timeline value
of the slot's last stream_release;
stream_release submits what is recorded and signals the
compute timeline after it; stream_acquire makes every later
compute submission wait for the slot's last copy on the transfer
timeline. A slot's staging grows to its largest batch of uploads and is
reused once they are done. A release inside a decode step ends its
recording or replay (a host operation); an acquire does not. Without a
transfer queue stream_upload is the plain upload.
--gpu-stream (layers streamed for long prompts) and the
expert tier use it.
The expert tier and the expert cache.
--expert-gpu places its slots on the Vulkan device as on
OpenCL (moe.md): its refills
go through the transfer queue while the device computes the following
layers, the products of the next step waiting for them;
--expert-cache reads the misses from the file beside it.
test_model (test_expert_gpu) runs the tier's
cases on three MoE fixtures (Qwen3-MoE, DeepSeek-V3, OLMoE) on the
Vulkan device: no expert held, half (filled, then refilled), all, half
from a usage file, half with the expert cache (0 and 4 loaders, both
read-aheads: the same bits); the device-only run's greedy tokens, logits
within 3e-6. download_many queues its copies into the
staging buffer and waits once.
auto
oklm_backend_select("auto") (no --backend,
OKLM_BACKEND unset or auto) takes, in
order:
- the first available backend of
OKLM_BACKEND_ORDER, a comma-separated list of backend names (vulkan,opencl,cpu,opencl,cpu), probing each named one; - else the registered backend of highest priority: OpenCL instances 10
less their number, the CPU 0, Vulkan instances from
oklm_vk_auto_priority().
A Vulkan instance's priority puts it where it is ahead of OpenCL
(oklm_vk_auto_priority() in
src/core/vulkan/backend_vk.h, which
OKLM_VK_AUTO there switches on): an integrated GPU 15 less
the instance number and a discrete GPU 20 less it, both above OpenCL; a
CPU implementation -1 less it, below the CPU. auto probes
Vulkan for that. Until 2026-11-15 an AMD discrete GPU ranked below
OpenCL, whose prompts were faster there when the rule was set; on the
table below only Qwen3-8B's prompt still is, by 3 %, and Vulkan decodes
18 to 26 % faster on both models, so the rule went.
OKLM_BACKEND_ORDER=opencl,vulkan,cpu gives the earlier
order. Measured with bench_llama, best of three runs (two
on the UHD 750), Q4_K_M, 2026-11-15, the machine otherwise idle:
| Device, model | Backend | Prompt (t/s) | Decode (t/s) |
|---|---|---|---|
| RX 7900 XT, Llama-3.2-1B | Vulkan | 16 235 (512) | 509 (128) |
| RX 7900 XT, Llama-3.2-1B | OpenCL | 14 861 (512) | 403 (128) |
| RX 7900 XT, Qwen3-8B | Vulkan | 2 663 (512) | 112 (128) |
| RX 7900 XT, Qwen3-8B | OpenCL | 2 747 (512) | 94.6 (128) |
| UHD 750, Llama-3.2-1B | Vulkan | 55 (128) | 22.1 (32) |
| UHD 750, Llama-3.2-1B | OpenCL | 8.9 (128) | 5.4 (32) |
Earlier, with the products on the CPU fallback (SmolLM2-135M Q8_0 on the RX 7900 XT, before the prompt and MoE products ran on the GPU), Vulkan ranked below the CPU:
| SmolLM2-135M Q8_0, RX 7900 XT | Prompt of 64 (t/s) | Decode of 16 (t/s) |
|---|---|---|
| OpenCL | 11 668 | 479 |
| Vulkan, every product on the CPU fallback | 454 | 19.8 |
| Vulkan, one-column products and attention on the GPU | 331 | 392 |
| CPU (i7-11700K) | 3 202 | 174 |
Verification
tests/core/test_vulkan.c and the group files
tests/core/test_vulkan_<group>.c run on every Vulkan
GPU present (tests/core/vk_test_util.h sets
OKLM_VK_DEVICES=all unless OKLM_VK_DEVICES or
OKLM_VK_DEVICE is set; OKLM_VK_DEVICES=none
skips them), here the RX 7900 XT through RADV and the UHD 750 through
ANV, and skip when there is none.
| Check | Result on both GPUs (and llvmpipe when named) |
|---|---|
| uploads and downloads at offsets, 70 MiB through the 64 MiB staging buffer | bit-identical |
| ADD, MUL (full, row and middle broadcasts), SCALE, in place, on a column slice | bit-identical to the CPU |
| SILU, GELU, GELU_ERF, SWIGLU, GEGLU, SOFTCAP (caps 2.5 and 30), SIGMOID | within 1e-6 absolute plus 1e-5 relative |
| CPY F32 / F16 / BF16 in every direction, on random values and the rounding cases (ties, the half range's edges, denormals, infinities, NaN), and a permuted view | bit-identical to the CPU |
| a dispatch of 16.8 million elements (more work-groups than the UHD 750's x limit) | exact |
| 300 dependent ops, more than one command buffer | exact |
| a product or a MoE sum with its residual tail (scale, add) as one dispatch | bit-identical to the ops one by one on the device |
| RMS norm on the GPU | within 1e-5 plus 1e-5 relative |
| get_rows from a host table (CPU fallback) | bit-identical |
| SSM_CONV on the GPU: its ring | bit-identical; its outputs within 1e-6 plus 1e-5 relative (the
device's exp in the SiLU) |
| one op per group (RMS norm, one- and 19-column F32 products, decode attention, a Q8_0 product, the MoE sum, the Hadamard rotation), through the fallback until their shaders land | as the CPU |
The transcendental functions are the shader's own: tanh
from its Taylor series near zero and from exp elsewhere,
erf from the complementary error function's Chebyshev fit
(fractional error below 1.2e-7).
Precision across drivers
Vulkan rounds a float add, subtract and multiply correctly but leaves
the built-in exp to the driver's precision, which differs
between vendors. Every shader whose results are checked against the
CPU's expf (the attention kernels' softmax, latent
attention, the gated delta's decay, the conv's SiLU, the element-wise
activations, the norms) uses exp_acc() from
shaders/common.glsl, within about one ulp, so the results
do not depend on the driver's exp (on the UHD 750,
deepseek3's several sequences against one per context went from 1.8e-5
to 2e-6).
Limits
A query's result does not depend on how many keys other queries of its batch see (the key ranges follow its own keys), but prompts go through the attention tiles and single tokens through the key ranges, two kernels whose sums round differently: batched and token-by-token evaluation agree within 2e-5 (1.1e-5 on the UHD 750, 3.6e-6 on the RX 7900 XT), not bit for bit as on OpenCL. Latent attention (heads over 256) gives a decoded query the same result alone or with other sequences, bit for bit, up to per-token batches whose partials stay within 128 MiB (about 90 tokens of GLM-4.7-Flash, 28 of DeepSeek V3); larger batches split by their work-group count.
The ops of a group not yet written for Vulkan take the CPU fallback, which copies their operands each time (slow, but correct); the group subsections above list what each group runs on the GPU and what it leaves to the CPU.
A step replays only once every op of its decode path runs on the GPU and reads its per-token values from per-step data; until then each step is recorded, breaks at its first op on the CPU and runs on directly.
autopicks Vulkan on every GPU it finds, OpenCL where Vulkan is not there (auto); Vulkan and OpenCL instances are not mixed in one context.NVIDIA is untested (ADR-0023, V4); the Windows build has not been run.