OKLMdocumentation v1.0.3
On this page

Vulkan backend

How OKLM runs on a GPU through Vulkan (ADR-0023): how the API is loaded, how devices are chosen, how tensors are addressed, how shaders are built and dispatched, which file holds which ops, the fallback to the CPU, and how it is tested.

Status: working on AMD (RADV) and Intel (ANV); NVIDIA untested. Steps V1 to V3 of the roadmap's GPU backend section are done: Vulkan is the main GPU backend and OpenCL the fallback. Which of the two auto takes on a device is in auto.

Using it

oklm run -m model.gguf -p "Hello" --backend vulkan
OKLM_BACKEND=vulkan oklm chat -m model.gguf
OKLM_VK_DEVICE=Intel oklm run -m model.gguf -p "Hello" --backend vulkan

From C, oklm_backend_get("vulkan") probes the backend on first use (as opencl does) and returns NULL without a usable device; oklm_backend_select("vulkan") then gives the CPU with a warning. oklm_vulkan_device_count() and oklm_vulkan_device_info() describe the instances. The backend registers below the CPU's priority, so auto keeps choosing OpenCL or the CPU until Vulkan reaches parity (auto); OKLM_BACKEND_ORDER=vulkan,opencl,cpu makes auto take it before.

Building

The shaders are GLSL compute shaders compiled to SPIR-V at build time by glslc (Debian's glslc package, or shaderc), targeting Vulkan 1.2, and embedded in the library. Without glslc the library is built without the Vulkan backend (OKLM_HAVE_VULKAN undefined, src/core/vulkan/vk_none.c in its place), as is a build with FORCE_NO_VULKAN=1; GLSLC=path names another compiler. make show-config prints HAVE_VULKAN.

Nothing of Vulkan is needed otherwise: no SDK, no headers, no library to link. The MinGW build (make MINGW=1) embeds the same SPIR-V (the compiler runs on the build host) and loads vulkan-1.dll at run time; it builds, but has not been run on Windows.

Runtime loading

src/core/vulkan/vk_loader.h declares the part of the Vulkan 1.3 API the backend uses (types, constants, structures with the layouts of vulkan_core.h), and vk_loader.c opens the loader with dlopen (libvulkan.so.1; vulkan-1.dll with LoadLibrary on Windows), takes vkGetInstanceProcAddr from it and every other entry point through vkGetInstanceProcAddr and vkGetDeviceProcAddr. OKLM_VULKAN_LIB names another loader (empty: Vulkan disabled). A sanitizer build loads no driver unless OKLM_VULKAN_LIB names one, as with OpenCL.

Devices

The probe creates one Vulkan instance (API 1.3), lists the physical devices and logs each one: name, driver and its version, Vulkan version, memory, kind (discrete, integrated, CPU implementation), subgroup sizes, and whether it has cooperative matrices, integer dot products (and whether they are accelerated), buffer device addresses, 16-bit floats, 8-bit integers:

Vulkan device 0: AMD Radeon RX 7900 XT (RADV NAVI31) (radv Mesa 26.1.6-1, Vulkan 1.4.354), 20.0 GiB, discrete; subgroup 64 (32-64), cooperative matrix, integer dot product (accelerated), buffer device address, fp16, int8
Vulkan device 1: Intel(R) Graphics (RKL GT1) (Intel open-source Mesa driver Mesa 26.1.6-1, Vulkan 1.4.354), 93.5 GiB, integrated (shares the host's memory); subgroup 32 (8-32), integer dot product (accelerated), buffer device address, fp16, int8
Vulkan device 2: llvmpipe (LLVM 21.1.8, 256 bits) (llvmpipe Mesa 26.1.6-1 (LLVM 21.1.8), Vulkan 1.4.354), 124.6 GiB, CPU implementation; subgroup 8 (8-8), cooperative matrix, integer dot product, buffer device address, fp16, int8

Each chosen device becomes a backend instance: vulkan, vulkan1, ...

  • OKLM_VK_DEVICES: a comma-separated list of indexes or name substrings (of the device or the driver name: radv, Intel), one instance each (a device listed twice gives two independent instances), or all: every GPU, discrete ones first.
  • OKLM_VK_DEVICE: one device, when OKLM_VK_DEVICES is unset.
  • Neither: every discrete GPU; integrated (or virtual) GPUs only when there is no discrete one, the rule of the OpenCL backend.
  • A CPU implementation (llvmpipe) is taken only when named.

A device needs Vulkan 1.2, buffer device addresses, 64-bit integers and 16-bit storage in shaders, and a compute queue; one without is logged and left out. The logical device enables, where the device has them, 16-bit floats, 8-bit and 16-bit integers, 8-bit storage, subgroup size control and full subgroups, integer dot products, cooperative matrices (VK_KHR_cooperative_matrix) and VK_EXT_memory_budget (the free memory the backend reports), so the groups of ops can use them without touching device creation. The queue is a compute-only family when there is one.

Addressing: buffer device addresses

Every buffer is created with a buffer device address (Vulkan 1.2), and a tensor's data is that address, plus a per-instance bias (k << 52 for instance k) that keeps two instances' ranges apart. A shader gets the addresses of its tensors and their byte strides in its push constants and reads and writes through GL_EXT_buffer_reference; there are no descriptor sets.

The other scheme, descriptors with fake addresses mapped back to (buffer, offset) at dispatch as the OpenCL backend does, was left aside:

  • a view's pointer arithmetic carries straight into the shader, with no translation at dispatch (the translation is kept only for copies);
  • every pipeline shares one pipeline layout (one push-constant range), with no descriptor set layouts, pools or updates, and an op may take any number of tensors;
  • a recorded dispatch holds its addresses in its push constants, which suits the decode step recorded once and replayed (V2d).

Buffer device addresses are core in Vulkan 1.2 and required by 1.3: RADV, ANV and NVIDIA's driver have them. The cost is 64-bit integers in shaders (present on all three) and that an element must sit at an address aligned to its type, which tensors do. Both local GPUs run every test this way.

Memory and copies

Each allocation is a device-local buffer with its own memory (on RADV one allocation is at most 4 GiB, reported as the largest buffer). An upload of up to 64 KiB (positions, cache cells, a token's activations) is copied into a host-visible buffer of the command buffer being recorded and recorded as a copy from there: it is not waited for, and the host's memory can be reused at once. Larger uploads, and downloads, go through a host-visible staging buffer (grown to 64 MiB, larger copies in pieces) and wait for their copy; download_many queues its copies together and waits once.

Commands are recorded in call order into a ring of four command buffers. A memory barrier goes before a command only when it is needed: an op's first command gets one when the op reads or writes a byte range written since the last barrier, or writes one read since then (its tensors' spans, and the scratch buffers it asked for); every further command of the same op gets one (a group chains its dispatches through scratch freely). The Q, K and V products of a layer, or its gate and up products, thus run without a barrier between them. The ops that update a source in place (SSM_CONV, GATED_DELTA) and commands outside an op get one as before. A command buffer is submitted when a copy needs the host to wait, on synchronize or flush, or after 64 commands (between ops), and recording continues in the next one. A freed buffer is destroyed once the device is idle (the next wait), not when it is freed; when 64 of them wait, or 256 MiB of them (a model's weights, a context's cache), the free itself waits for the device and destroys them, so that what is allocated next finds the memory. Before 2026-11-14 only the count did: a model of 18.5 GiB freed and placed again otherwise in the same process (Qwen2.5-VL 32B, 61 of 64 layers then 32 of 64 with a paged cache, test_mm_public) was allocated beside its own freed weights, past the card's 20 GiB: the context failed once, and once ran its prompt at 11 rows/s where it takes 42.

Shaders and pipelines

src/core/vulkan/
  vk_loader.h, vk_loader.c     the API's declarations and its loading
  vk_device.c                  instance, device listing and choice, device creation
  vk_runtime.c                 buffers, copies, command buffers, pipelines, dispatch, push constants
  backend_vk.c                 the backend interface, the routing of each op, the CPU fallback, the probe
  vk_internal.h                the internal API the op files use
  vk_ops_<group>.c             one host file per group of ops
  shaders/common.glsl          what every shader includes
  shaders/<group>/<name>.comp  the shaders of each group

The build compiles every shaders/<group>/<name>.comp it finds (glslc --target-env=vulkan1.2 -O -Werror, with shaders/ on the include path) to a C initializer in $(BUILD)/gen/vk/, and writes a table of them by name ($(BUILD)/gen/vk_shaders.inc) that vk_runtime.c includes; adding a shader needs no Makefile edit. <name> is the pipeline's name and must be unique across the groups (the build stops on a duplicate).

common.glsl declares the buffer references (RefF32, RefU32, RefU16), load_f() and store_f() for F32, F16 and BF16 elements, the conversions done on the integer bits (F32 to F16 and BF16 with round to nearest even, as the CPU does, whatever the device's float modes), and GLOBAL_INDEX_1D().

oklm_vk_pipeline_get(s, name, spec) builds a pipeline on first use and caches it by name and variant: up to 8 specialization constants (32-bit, ids 0 to 7; by convention id 0 is the work-group size, layout(local_size_x_id = 0) in;), a required subgroup size, full subgroups. A shader's parameters are one push-constant block of scalar members (uint64_t, uint, int, float) filled in the same order with oklm_vk_push_*() (oklm_vk_push_addr() for a tensor, which marks the block failed when the tensor is not on the instance; oklm_vk_push_u32_checked() for a size that must fit 32 bits). The block is at most 256 bytes, or the device's limit. oklm_vk_dispatch() records a dispatch, oklm_vk_dispatch_1d() spreads n invocations over the work group grid (the groups beyond the device's x limit, 65535 on the UHD 750, in y). oklm_vk_copy(), oklm_vk_fill() and oklm_vk_scratch() (a scratch buffer per group) complete the set; vk_internal.h documents each.

Groups of ops

backend_vk.c's switch routes every op to its group's entry point, which returns 1 for what it does not take; the next group in the route, else the CPU fallback, then runs it. A group adds shaders in its own host file and directory and never edits the switch.

Group Host file, shaders, test Ops Roadmap
element-wise vk_ops_ew.c, shaders/ew/, test_vulkan.c ADD, MUL, SCALE, SILU, GELU, GELU_ERF, SWIGLU, GEGLU, SIGGLU, SOFTCAP, SIGMOID, CPY between float types V1 (done)
norm vk_ops_norm.c, shaders/norm/, test_vulkan_norm.c RMS_NORM, LAYER_NORM, ROPE, SOFTMAX, GET_ROWS, SET_ROWS, CPY into quantized types and byte copies (after the element-wise group), ARGMAX, TOPK, CROSS_ENTROPY V2a
matmul vk_ops_matmul.c, shaders/matmul/, test_vulkan_matmul.c MATMUL with one activation column, every weight type V2b
attention vk_ops_attention.c, shaders/attention/, test_vulkan_attention.c ATTENTION V2c
gemm vk_ops_gemm.c, shaders/gemm/, test_vulkan_gemm.c MATMUL with several columns (after matmul) V3a
q8 vk_ops_q8.c, shaders/q8/, test_vulkan_q8.c MATMUL and MATMUL_ID on integer dot products (tried first) V3b
moe vk_ops_moe.c, shaders/moe/, test_vulkan_moe.c MATMUL_ID (after q8), MOE_SELECT, MOE_WEIGHTS, MOE_SUM; the fused routing and gate / up / SwiGLU (held by backend_vk.c) V3c
ssm vk_ops_ssm.c, shaders/ssm/, test_vulkan_ssm.c SSM_CONV, GATED_DELTA, HADAMARD, latent attention (ATTENTION after the attention group) V3d

Adding an op: a shader in the group's directory; in the group's host file, a function that checks the operands (returning 1 for what the shader does not handle), fills the push block, gets the pipeline and dispatches; a check against the CPU in the group's test file.

Element-wise ops

vk_ops_ew.c runs every element-wise op on one shader (shaders/ew/ew.comp), one invocation per element of the destination, 4-D with any strides, the second source broadcast. Indexing an element in general takes 64-bit divisions of the invocation's linear index by the sizes, which GPUs emulate: they were most of a large op's time (the SwiGLU of a 512-token prompt of Llama-3.2-1B, 4.2 million elements: 834 us on the RX 7900 XT). Two variants of the shader (its specialization constant 1) index without them, when the destination and the sources have one shape:

  • contiguous tensors: element i at i times the element size in each;
  • rows at a constant stride (a view of half of wider rows: the gate and up outputs of a fused gate / up product), rows of 128 elements or more: the dispatch has x over a row's elements and y over the rows.

The same arithmetic per element, so the same bits. The SwiGLU above then takes 58 us, and a 512-token prompt on the RX 7900 XT goes from 11 020 to 15 150 t/s on Llama-3.2-1B and from 1 970 to 2 340 t/s on Qwen3-8B (Q4_K_M, 2026-11-12); a broadcast, a permuted view or shorter strided rows keep the general indexing, and OKLM_VK_EW_FLAT=0 gives it to every op. test_vulkan.c checks the row-strided variant on the halves of rows of 600 over two batch dimensions (a sum and an F16 copy bit for bit, the gated unit in place within the activations' tolerance, the other half untouched).

One-column products

vk_ops_matmul.c takes MATMUL with up to 16 activation columns (decode, and the small batches of speculative decoding or several sequences) for every weight type OKLM computes: F32, F16, BF16, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q2_K to Q6_K, IQ4_NL, IQ4_XS, MXFP4, NVFP4, TQ1_0, TQ2_0, Q1_0, Q2_0, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ1_S, IQ1_M, and PQ2_0 and PTQ1_0 (types 142 and 143 of PrismML's llama.cpp fork: Q2_0's and TQ1_0's codes with a scale per 128, the prompt products and the indexed ones too). Activations and destination are F32 (columns 16-byte aligned), iparams[0] = 1 adds the product to the destination (oklm_matmul_acc()), and the weights broadcast over dimensions 2 and 3 as on the CPU. A product of more than 16 columns returns 1, so the routing tries the prompt products (vk_ops_gemm.c) next: past 16 columns the accumulators no longer fit the registers and a tile that reuses each dequantized weight for many columns is the faster shape.

The shader (shaders/matmul/mv.glsl, built once per weight type as mv_<type>, shaders/matmul/mv_<type>.comp defining the type) gives each work-group of 64 invocations two weight rows (one column) or one row (2 to 16 columns: 2, 4, 8 or 16 per pass, the columns past the product's skipped). The invocations take the row's groups of 32 weights in turn, adjacent invocations reading adjacent blocks; each group is dequantized and used at once for every column (only one group of weights is live in registers: holding several rows' groups made ANV's compiler spill and take minutes), so a weight block is read once per product. The partial sums are reduced by a shared-memory tree in a fixed order, so accumulating a product twice doubles it exactly. The columns per pass and the rows per work-group are specialization constants: one pipeline per type and combination, built on first use. The type is fixed at compile time rather than by a specialization constant: a module holding all 28 decoders is 370 KB of SPIR-V, and the drivers took seconds per pipeline over it (minutes for the test's variants on ANV); one type's module is 17 to 66 KB.

shaders/dequant.glsl is the one place the block layout and the dequantization of every weight type live; every shader that reads weights (these products, the prompt products, the 8-bit products, the mixture-of-experts products) includes it after common.glsl:

#include "common.glsl"
#include "dequant.glsl"
const uint TYPE = 12u;  // an oklm_dtype code (here Q4_K), from the .comp's #define
...
float v[32];
dq_group(TYPE, row_address, j, v);  // weights 32 j .. 32 j + 31 of the row

dq_group() computes what the OpenCL dequantizers (dq_<type> in kernels.cl) and oklm_dequantize_row() compute. Give it a compile-time constant as the type (one .comp per type defining it, as shaders/matmul/mv_<type>.comp do): the compiler then keeps only that type's code and tables. A specialization constant works too, but leaves every decoder in the module for the driver to compile through. The row address must be a multiple of DQ_ALIGN(type) (16 for the float types, Q4_K and Q5_K, which are read with 16-byte loads; 4 for Q2_K, IQ4_XS, NVFP4, IQ1_M; 1 for MXFP4; 2 for the others), which the host checks on the tensor's address and strides before choosing a shader (dq_align() in oklm_vk_dq_align(), src/core/vulkan/vk_dequant.h); the file's header lists every type's block size, bytes and alignment. The type codes are T_* (OKLM's). The data is handled with 32-bit integers only (64-bit integers serve the addresses): Intel's Gen12 emulates 64-bit integers.

A code is never the index of a constant array. The two 16-entry codebooks (IQ4_NL's and IQ4_XS's grid, the E2M1 values of MXFP4 and NVFP4) are computed from constants in registers: dq_iq4nl() holds the grid as signed bytes in four words and sign-extends the code's byte, dq_fp4() reads the magnitude from the 4-bit fields of one word and takes bit 3 as the sign, both to the integer the table held, so the weights are the same bits. An array indexed by the code is a memory load per weight on RADV, about twice the time of a product: with the arrays gpt-oss-20b (MXFP4 experts) decoded at 96.7 t/s and SmolLM2-135M IQ4_XS at 391, without them at 163.9 and at 650 to 707 (RX 7900 XT, 2026-11-14; Mixture of experts). The shaders of an IQ4_NL cache take the grid the same way (iq4nl_grid() in common.glsl, for norm_store_rows, norm_get_rows and the attention's cache rows, which each held a constant array): a row's quantization bisects the grid for each of a block's 32 values at sixteen scales, some three thousand reads a block, one invocation a block. Llama-3.2-1B Q4_K_M with an IQ4_NL cache decoded at 93.5 t/s with the arrays, the same at 64, 256 and 1024 tokens of context (the writes, not the attention's reads), and at 154.5 without them; its prompt of 512 tokens went from 11 900 to 12 190 t/s to 12 610 (a Q4_0 cache: 447 to 456 and 15 230 to 15 370; F16: 470 to 483 and 15 680 to 16 110; 1024 tokens decoded, the best of three runs, 2026-11-14). The bytes written and the values read are unchanged (test_vulkan_norm, test_vulkan_attention, on the RX 7900 XT and the Intel UHD 750).

The grid IQ types (IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ1_S, IQ1_M) read their codebooks (the tables of src/core/iq_grids.inc, 33 KB) from a device buffer each instance uploads once, oklm_vk_iq_grids(s): the shader sets the global dq_grids to that address (passed in its push constants) before decoding, the tables at the DQ_GRID_* word offsets. Constant arrays in the shader were copied into every invocation's memory by the drivers. vk_dequant.h declares oklm_vk_dq_align(), oklm_vk_dq_needs_grids() and oklm_vk_iq_grids() for the other groups.

On Intel GPUs IQ2_S, IQ1_S and IQ1_M are left to the CPU fallback: on the UHD 750 (ANV, Mesa 26.1) their pipelines take over 30 seconds each to build and then lose the device or give wrong results, while the same shaders pass on RADV and llvmpipe. Every other type runs on both GPUs.

tests/core/test_vulkan_matmul.c checks every type on every GPU against a double sum over the dequantized weights (within 1e-5 sum |w x| + 1e-6, test_opencl.c's matmul tolerance): one and 7 columns (2, 3 and 16 too for F32, Q8_0, Q4_K, Q6_K, IQ2_XS), row counts that are not multiples of the work-group's rows, rows of one block and of several hundred groups; then the accumulated product, which must double every value exactly (IQ2_S, IQ1_S and IQ1_M skipped on Intel, IQ3_S's pipelines there limited to the decode one, ANV building each in about 10 seconds before its cache keeps them); weights broadcast over each batch dimension, strided activation columns, F16 denormal weights, and products of 17 and 40 F32 columns (past the group). With OKLM_VK_BENCH=1 it also measures, on the discrete GPUs, the decode product's weight bandwidth for Q4_K, Q6_K, Q8_0 and F16 at Llama-3.2-1B and Qwen3-8B sizes, each product on another copy of the weights (320 MB in all, past the 7900 XT's 80 MB Infinity Cache, which otherwise serves the weights of repeated products at up to 1.5 TB/s); OKLM_VKT_CL=1 adds the OpenCL backend's on the same shapes, OKLM_VKT_BENCH_ONLY=1 skips the checks:

OKLM_VK_BENCH=1 OKLM_VKT_CL=1 OKLM_VKT_BENCH_ONLY=1 OKLM_VK_DEVICE=radv build/release/bin/tests/core/test_vulkan_matmul

On the RX 7900 XT (RADV), weight bytes read per second, Vulkan and the OpenCL backend's decode kernels on the same shapes:

Type K x N MB Vulkan OpenCL
Q4_K 2048 x 8192 9.4 494 GB/s (19.1 us) 455 GB/s
Q4_K 8192 x 2048 9.4 417 GB/s 340 GB/s
Q4_K 4096 x 12288 28.3 558 GB/s (50.8 us) 514 GB/s
Q4_K 12288 x 4096 28.3 514 GB/s 512 GB/s
Q6_K 2048 x 8192 13.8 533 GB/s 484 GB/s
Q6_K 8192 x 2048 13.8 488 GB/s 421 GB/s
Q6_K 4096 x 12288 41.3 603 GB/s 583 GB/s
Q6_K 12288 x 4096 41.3 534 GB/s 475 GB/s
Q8_0 2048 x 8192 17.8 592 GB/s 520 GB/s
Q8_0 8192 x 2048 17.8 587 GB/s 521 GB/s
Q8_0 4096 x 12288 53.5 659 GB/s 584 GB/s
Q8_0 12288 x 4096 53.5 611 GB/s 562 GB/s
F16 2048 x 8192 33.6 449 to 480 GB/s 537 GB/s
F16 8192 x 2048 33.6 578 to 600 GB/s 623 GB/s
F16 4096 x 12288 100.7 526 to 590 GB/s 651 GB/s
F16 12288 x 4096 100.7 558 to 640 GB/s 619 GB/s

Every product pays about 5.7 us of dispatch and barrier besides its reads (the intercept of the two sizes); past it the Q8_0 product streams at about 730 GB/s, Q4_K and F16 near 680 GB/s, Q6_K near 600 GB/s. F32 and BF16 weights, read with plain 16-byte loads, reach 655 to 700 GB/s on these shapes: the shader's ceiling, near the 7900 XT's practical bandwidth. F16 varies between runs over the range shown and stays a little behind the OpenCL kernel; the quantized types are ahead of it.

8-bit products

vk_ops_q8.c is tried before the float products. It takes MATMUL of up to 16 activation columns and the decode steps of MATMUL_ID (up to 64 (slot, token) pairs) with Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 or Q5_0 weights and gives the CPU's products bit for bit: the activations are rounded to 8 bits as the CPU rounds them, each block's integer sums are formed with the 4-way 8-bit dot products of VK_KHR_shader_integer_dot_product (dotPacked4x8EXT, exact), and the float terms are formed and added as the CPU's dot products form and add them. A greedy decode on the 8-bit path therefore follows the CPU's tokens, which the float products only approach. It needs integer dot products and pipelines of 32-wide subgroups with arithmetic, clustered, ballot and shuffle operations (both local GPUs have them, the 4x8-bit packed product accelerated on both); elsewhere it returns 1 and the float groups run.

The shader (shaders/q8/q8mv.glsl, built per weight type as q8mv_<type>) gives each work-group of 256 invocations (eight subgroups of 32) 16 weight rows, two per subgroup. The work-group first rounds its activation columns (one to four of them, as shared memory allows) into shared memory, so no launch is added:

  • Q4_K, Q5_K, Q6_K: blocks of 256 as oklm_quantize_row_q8_k() (a subgroup per block, eight values a lane): the largest magnitude and, on ties, the first one gives the scale's sign; levels min(roundEven(iscale x), 127) with iscale = -127 / max; the block's scale 1 / iscale; the sums of each 32 levels for the mins term;
  • Q8_0, Q4_0, Q5_0: blocks of 32 as oklm_quantize_row_q8_0(): levels roundf(x / d) (half away from zero, from trunc), d = amax / 127 kept as a half.

The divisions must be the CPU's correctly rounded ones, while a Vulkan division is only accurate to 2.5 ulp and the UHD 750 has no doubles: q8_div() estimates the quotient of the significands with a float division, then makes it exact from its integer remainder, whose low 32 bits the 32-bit products give exactly (it is a few divisors at most), and rounds to nearest even (zeros, denormals and overflow handled).

Each subgroup then walks its rows' blocks: eight lanes a super-block for the K-quants (16 values of each of two sub-blocks a lane, the scales applied to the integer sums, the sums added across the eight lanes with a clustered add, two steps of four super-blocks unrolled), one lane a block for the blocks of 32 (32 blocks a step). Each block's float term is the CPU's: d isum - dmin summs with d = f16(d) y.d and dmin = f16(dmin) y.d (Q4_K, Q5_K), (f16(d) y.d) isum (Q6_K), isum (f16(d) f16(y.d)) (Q8_0), (isum f16(d)) f16(y.d) (Q4_0), (f16(d) f16(y.d)) isum (Q5_0), every variable precise (no fused multiply-adds). A row's terms must be added in order of k, one at a time: the terms of a step go to a row of shared memory per output, and the output's own lane adds them in block order (with 16-byte reads), so the serial additions cost one instruction per term rather than a broadcast and an add per term and row in every lane (which made the blocks of 32 up to 25 % slower than the float products).

An RMS norm before the product is not folded into it (OpenCL's mqn_<type>): the routing gives the group one op at a time and holds no norm back, and the folding paid on OpenCL only by saving a launch.

The K-quants have a second layout, the row shaders (shaders/q8/q8kr.glsl, built as q8kr_<type> for Q4_K, Q5_K and Q6_K), for a device bound by the shader's arithmetic rather than by its memory (the UHD 750): there q8mv_<type> repeats a super-block's header work (scales, mins, the two half floats) in each of its eight lanes, rounds the activations again in every work-group, and pays shared memory and barriers for the ordered sums. Two dispatches instead:

  • q8k_round rounds the activation columns once, a subgroup of 32 per block of 256 (the rounding above), into records of 304 bytes in the group's scratch buffer: the 256 levels, the eight sums of each 32, the block's scale;
  • q8kr_<type> gives each invocation one weight row (work-groups of 32, no shared memory, no barrier, no subgroup operation): per super-block, 16-byte loads of the block and of the record (Q6_K's blocks, aligned to 2 bytes only, as words from the aligned address before them), the eight sub-blocks' integer sums, the float term, and the row's sum in order of k as the CPU adds it. One, two or four columns per invocation share the weights' loads; MATMUL_ID takes one (slot, token) pair per z.

The results are the same bits as q8mv_<type>'s and the CPU's. OKLM_VK_Q8_ROWS=1 takes the row shaders on any device, 0 never; unset, Intel GPUs take them. On the RX 7900 XT the second dispatch and its barrier cost more than they save (Q4_K 2048 x 2048: 9.9 against 5.8 us; 4096 x 12288: 57 against 50 us).

Policy: with OKLM_VK_Q8 unset (q8_default()), MATMUL takes the 8-bit path where it was measured not slower than the float products: on AMD GPUs, the K-quants with rows of 1536 values or more and 4096 rows or more (any column count up to 16), the blocks of 32 on the same shapes from two to eight columns; on Intel GPUs, the K-quants at every shape (on the row shaders; never with OKLM_VK_Q8_ROWS=0) and the blocks of 32 from 512 values and 512 rows; on other devices, never. MATMUL_ID decode steps (q8_default_id(), against the mixture-of-experts group's mvid_<type>) take it on AMD GPUs for experts of 512 rows or more, the blocks of 32 at any row length and the K-quants from 1536 values (eight ranks of 32 or 64 experts: 5 to 40 % faster, Q6_K 4096 x 1536 36.9 against 59.5 us; the K-quants' rows of 768 and 1024 values, the down products of small experts, 5 to 20 % slower), and on Intel GPUs as MATMUL. OKLM_VK_Q8=1 takes every product the shaders can run, OKLM_VK_Q8=0 none.

On the RX 7900 XT (RADV), decode products of one column, each on another copy of the weights (test_vulkan_q8.c, OKLM_VK_BENCH=1), the 8-bit path, the float one-column products and the OpenCL backend's default kernels (its own 8-bit mq_<type> for the K-quants, float mv_<type> for the others):

Type K x N 8-bit float OpenCL
Q4_K 2048 x 8192 17.9 us (528 GB/s) 18.6 us 21.2 us
Q4_K 4096 x 12288 47.6 us (595 GB/s) 47.7 us 51.8 us
Q4_K 12288 x 4096 52.5 us (540 GB/s) 56.1 us 55.4 us
Q5_K 4096 x 12288 57.3 us (604 GB/s) 59.0 us 61.1 us
Q6_K 2048 x 8192 23.8 us (578 GB/s) 25.4 us 27.6 us
Q6_K 4096 x 12288 64.9 us (637 GB/s) 70.0 us 69.0 us
Q6_K 12288 x 4096 72.3 us (571 GB/s) 76.3 us 84.1 us
Q8_0 4096 x 12288 83.9 us (638 GB/s) 82.4 us 91.3 us
Q4_0 4096 x 12288 48.3 us (586 GB/s) 46.5 us 50.3 us
Q5_0 4096 x 12288 59.1 us (585 GB/s) 55.1 us 60.6 us

The 8-bit products pay about 1.5 us more of fixed cost (the rounding before the first product, the work-group's prologue), so small products (1024 x 1024, 2048 x 512) are 20 to 40 % slower and left to the float products. With two to eight columns the 8-bit path is ahead on model shapes, by up to 40 % (Q4_K 4096 x 12288: 50.8 against 64.3 us with two columns, 154 against 224 us with eight; Q8_0 with four: 100 against 127 us).

On the UHD 750 (ANV) the float products and q8mv_<type> are bound by the shader's arithmetic, not by memory: the blocks of 32 run two to three times faster in 8 bits (Q4_0 4096 x 4096: 570 against 1 904 us), the K-quants on q8mv_<type> 15 to 50 % slower than in floats. The row shaders read the K-quants' weights at 14 to 23 GB/s, where the weight loads are most of the time left (Q6_K 2048 x 16384 without them: 0.5 of 1.9 ms). One column, each product on another copy of the weights (2026-11-12, OKLM_VKT_BENCH_IGPU=1 OKLM_VK_DEVICE=Intel):

Type K x N row shaders q8mv_<type> float
Q4_K 2048 x 2048 122 us (19 GB/s) 553 us 483 us
Q4_K 2048 x 16384 832 us (23 GB/s) 4 152 us 3 780 us
Q4_K 8192 x 2048 442 us (21 GB/s) 1 844 us 1 828 us
Q4_K 512 x 512 17 us 54 us
Q5_K 2048 x 16384 1 009 us (23 GB/s) 5 013 us 4 204 us
Q6_K 2048 x 16384 1 898 us (14 GB/s) 5 586 us 4 678 us
Q6_K 8192 x 2048 804 us (17 GB/s) 3 235 us 2 144 us
Q6_K 2048 x 128256 14.7 ms (15 GB/s) 36.6 ms

With 4 columns Q4_K 2048 x 16384 takes 1.44 ms against 13.6 ms in floats, with 16 columns 5.6 against 54 ms; the indexed products of eight ranks (Q4_K, 2048 x 1024 of 64 experts) 347 us against 1 883 us on mvid_q4_k. Llama-3.2-1B Q4_K_M then decodes at 19.8 t/s on the UHD 750 (6.4 before the row shaders; llama.cpp's Vulkan backend 15.6), 42 of a token's 50 ms in its products, whose 763 MiB of weights go at 18 GB/s.

tests/core/test_vulkan_q8.c runs on both GPUs in about two seconds each: with OKLM_VK_Q8=1, every type bit-identical to the CPU's product (rows of one block, of a block count that is not a multiple of a step, of 352 values for the blocks of 32, one row, 5 and 16 columns over column groups, batches with broadcast weights, then the product accumulated once more), on activations with ties of magnitude of opposite signs, zero blocks, values over 22 binades and rows scaled by up to 2^80 either way; indexed products of one token with four slots, of three tokens sharing an activation row and of two tokens with a row per slot; the K-quants once per layout (OKLM_VK_Q8_ROWS=0 and 1) on each GPU; then the default policy on model-sized products, exact where it takes the 8-bit path and within the float tolerance elsewhere, and OKLM_VK_Q8=0. With OKLM_VK_BENCH=1 it times the decode products, 8-bit against float (OKLM_VKT_CL=1: and OpenCL's; OKLM_VKT_BENCH_TYPE, OKLM_VKT_BENCH_COLS, OKLM_VKT_BENCH_SHAPES=KxN,..., OKLM_VKT_BENCH_IGPU for the integrated GPUs; the indexed decode products of MoE shapes against the mixture-of-experts group's, and OKLM_VKT_BENCH_NO_MM times those alone):

OKLM_VK_BENCH=1 OKLM_VKT_CL=1 OKLM_VKT_BENCH_ONLY=1 OKLM_VK_DEVICE=radv build/release/bin/tests/core/test_vulkan_q8

Prompt products

vk_ops_gemm.c takes MATMUL past the one-column products' 16 columns (prompts, large batches) for every weight type dequant.glsl decodes, with iparams[0] = 1 accumulating and the weights broadcast over dimensions 2 and 3 as on the CPU. Activations and destination are F32, the activation columns 16-byte aligned (anything else goes to the CPU); IQ2_S, IQ1_S and IQ1_M stay on the CPU on Intel GPUs, as for the one-column products. Two kinds of shaders, one module per weight type each (shaders/gemm/<kind>_<type>.comp, the bodies in gm.glsl and cm.glsl, what they share in gemm.glsl):

  • cm_<type>, cooperative matrices (VK_KHR_cooperative_matrix), on a device that lists 16 x 16 x 16 blocks of F16 inputs with F32 accumulation at subgroup scope and has 16-bit floats and full subgroups (the RX 7900 XT; not the UHD 750). A first dispatch, gm_tohalf, rounds the activations to F16 once into a contiguous copy (scratch); a work-group of four subgroups of 64 (OKLM_VK_GEMM_SG=32 takes 32) computes 128 weight rows by 128 columns, each subgroup a quarter, 4 x 4 accumulators. Per step of 32 values of k, 128 invocations dequantize one row's group each (F16 weights are copied as they are) and round it to F16 into shared memory, every invocation copies its share of the F16 activations, and each subgroup multiplies its 4 x 4 blocks twice; the next step's loads are in flight meanwhile. F32 weights do not take this path: their products stay F32, as on the CPU, which does not round their activations.
  • cm_<type> with the specialization constant WIDE, for a product that asks for its activations at F32's precision (oklm_matmul_wide(), iparams[1] = 1: the products of the vision and audio encoders' blocks, below) with F16 weights: gm_tohalf writes two F16 copies, the rounded values and what the rounding dropped, and each step multiplies the weight blocks by both. The other weight types of such a product take gm_<type>.
  • gm_<type>, F32 tiles, everywhere else (and with OKLM_VK_COOPMAT=0): 64 invocations per tile of 64 rows by 64 columns, the weights dequantized into shared memory k-major, the activations beside them, 8 x 8 outputs per invocation, the next step loaded during the current one (OpenCL's gm_<type>).

gm_tohalf and gm_reduce find their column and batch by dividing a 32-bit linear index; indexing them by the dispatch grid instead (x within a column, y the columns, z the batches) changed nothing (a layer's seven Q4_K products at 512 columns: 19 420 to 19 450 t/s on Llama-3.2-1B's sizes and 3 280 to 3 300 on Qwen3-8B's either way, 2026-11-12), unlike the 64-bit divisions of the element-wise shader; not kept.

A product whose tiles do not fill the device is split over k: each split writes its partial tile to scratch (OKLM_VK_SCRATCH_GEMM) and gm_reduce adds the splits in their order, then writes or accumulates. The F32 tiles split up to 256 work-groups (64 on an integrated GPU), at least 4 steps per split; the cooperative-matrix tiles, four times the work each, up to 128, at least 16 steps per split, and a product of 64 tiles or more only when k reaches 8192 (OKLM_VK_GEMM_SPLIT=n forces n). Every sum runs in a fixed order, so a product is deterministic and accumulating it twice doubles it exactly.

Precision: the F32 tiles are within test_vulkan_matmul.c's tolerance (1e-5 sum |w x| + 1e-6 of a double sum over the dequantized weights); the cooperative matrices round both inputs to F16, within `2e-3 sum |w x|

  • 1e-6 (test_opencl_wmma.c's F16 tolerance; the RX 7900 XT's worst is well inside it); a product that asks for its activations at F32's precision is within 1e-6 sum |w x| + 1e-6` on either (below). BF16 weights beyond the F16 range (above 65504) would not survive the rounding; no model has them.

An encoder's products

A language model's prompt tolerates the rounding of its activations to F16 (the figures under The model on Vulkan); an encoder's blocks do not. Their activations reach tens (80 in Qwen2-VL's), the terms of a product cancel, and 23 to 32 blocks add the rounding up: with every product on cm_f16 the embeddings were 2.6e-3 to 1.5e-2 from the CPU's (relative norm), where OpenCL's F32 products give 1.4e-5 to 5.3e-5. Measured on Qwen2-VL 2B (RX 7900 XT, 2026-11-14, two pictures of 273 embeddings, test_mm_public), with one group of a block's products left on the rounded activations and the others not:

Rounded to F16 Embeddings from the CPU's
every product (before) 1.5e-2, 1.1e-2
the queries, keys and values 1.3e-3, 2.5e-3
the attention's output 3.5e-3, 4.2e-3
the MLP's first product 1.8e-2, 1.2e-2
the MLP's second product 4.1e-3, 5.1e-3
none (oklm_matmul_wide() for the six) 1.9e-5, 3.0e-5

The error enters with the activations: the units accumulate in F32 and every result is stored as F32. So the encoders ask for their products through oklm_matmul_wide() (OKLM_OP_MATMUL with iparams[1] = 1), which the CPU and OpenCL run as any product and which oklm_vk_gemm_tiles() runs, on a device with cooperative matrices, in one of three ways (OKLM_VK_WIDE, read at each product):

  • unset: two F16 parts on the units for F16 weights (every published encoder with F16 matrices). gm_tohalf writes the half nearest each activation and, in a second copy, what that rounding dropped (x - half(x), exact in F32, itself rounded to F16): about 22 bits of each value together. cm_f16 with WIDE loads both tiles and, per 16 x 16 x 16 block, multiplies the weights by the first then by the second into the same F32 accumulator; the second tile has rows of 32 halves without padding, so the shader's shared memory is 32 768 bytes (a device with less takes the F32 tiles). The accumulators are also added into a second set every 4 steps (128 values of k) and cleared: one running F32 sum over the 2 560 to 10 240 terms of such a product alone left Qwen2-VL's embeddings at 1.7e-4 to 3.2e-4. The nearest half is made on the bits of the F32 (13 bits of mantissa rounded to nearest even): written as a conversion to F16 and back, RADV (Mesa 26.1) gives the value itself and the second copy is zero. Weights of another type (BF16, quantized: their own rounding to F16 would remain) take the F32 tiles.
  • f32: every such product on the F32 tiles (gm_<type>).
  • 0: as any other product, the activations rounded to F16 (the behaviour before 2026-11-14).

F32 activations on the units themselves are not to be had: the RX 7900 XT (RADV, Mesa 26.1) lists its float configurations with F16 inputs alone (an F16 or an F32 result), the others with 8-bit integers.

The products of a language model do not set the flag and keep their shader variant (WIDE 0, the same specialization as before), their gm_tohalf values and their bits.

One block's six products with F16 weights at width 1 280 (5 120 inside the MLP), test_vulkan_gemm with OKLM_VK_BENCH=1 OKLM_VKT_BENCH_WIDE=1 on the RX 7900 XT:

Rows Rounded (OKLM_VK_WIDE=0) Two F16 parts (default) F32 tiles (OKLM_VK_WIDE=f32)
1 092 (Qwen2-VL, 273 embeddings) 1.42 ms 3.25 ms (2.3 x) 4.30 ms (3.0 x)
1 500 (Whisper large-v3) 1.53 ms 3.48 ms (2.3 x) 5.32 ms (3.5 x)

What that is of an encoder and of a prompt, and the embeddings each setting gives, is in multimodal.md. An activation beyond the F16 range (above 65 504) is infinite in the first copy as it was; values under 2^-14 have no second part (an absolute error under 3e-8 each).

src/core/vulkan/vk_gemm.h offers the same products to the other groups, oklm_vk_gemm_tiles(s, &args): addresses and byte strides instead of tensors, and a grouped form for the mixture-of-experts prompt path, a tile table read on the device (one work-group row per entry: weight matrix e at w + e * wnb2, product columns c0 .. c1 - 1, at most oklm_vk_gemm_group_cols(), a tile's columns: 128 on cooperative matrices, 64 on the F32 tiles; e < 0 an unused entry) with column id tables (x_ids, d_ids: product column c reads activation column x_ids[c], writes destination column d_ids[c]).

tests/core/test_vulkan_gemm.c checks on every GPU, on both paths where the device has cooperative matrices: every type (70 rows by 40 columns, split, and 130 by 150, partial tiles both ways); one row by 17 columns, one step of k, 129 by 129, k = 4096 over one tile (16 splits), 200 by 300 unsplit and three forced splits, for F32, Q8_0, Q4_K and IQ2_XS; the batch broadcasts and strided columns of the one-column test; each product accumulated, which must double it exactly; that the two paths differ for Q8_0 weights (the cooperative matrices ran) and agree bit for bit for F32; the products that ask for their activations at F32's precision (test_wide: activations of +-80, partial tiles, a split product, a batch, strided columns, accumulated) within 1e-6 sum |w x| + 1e-6 for F16 weights by default and with OKLM_VK_WIDE=f32, and for F32, BF16, Q8_0 and Q4_K weights, where OKLM_VK_WIDE=0 must leave outputs beyond that bound on a device with cooperative matrices (measured on the RX 7900 XT as a fraction of sum |w x|: 8e-8 to 3.0e-7 with the two parts, 2e-8 to 1.4e-7 on the F32 tiles, 1.8e-5 to 7.4e-5 rounded); the products a convolution is made of on a device (test_conv_views: one product a position of the kernel, with the activations at F32's precision, over a view of every stride-th row from that position on, the first written and the others accumulated; kernels of 3 and 7, strides of 1 and 2, 96 to 1 280 channels, against the convolution in doubles, within 1.8e-7 of sum |w x| on the RX 7900 XT and 5.6e-8 on the Intel UHD 750; the element-wise ops over views of rows the vocoders' depthwise convolution and Gemma 3's pool are made of, and a product into the rows after its destination's first: multimodal.md, speech.md); the grouped form (three matrices, an unused entry, a single and a full 64-column entry, repeated and permuted column ids) for F32, Q4_K and Q8_0. With OKLM_VK_BENCH=1 it times the seven products of a Llama-3.2-1B and a Qwen3-8B layer with 512 columns for Q4_K, Q6_K, Q8_0 and F16 weights on the discrete GPUs (OKLM_VKT_CL=1: the OpenCL backend's beside them; OKLM_VKT_BENCH_TYPE=Q4_K one type):

OKLM_VK_BENCH=1 OKLM_VKT_CL=1 OKLM_VKT_BENCH_ONLY=1 OKLM_VK_DEVICE=radv build/release/bin/tests/core/test_vulkan_gemm

On the RX 7900 XT (RADV, 2026-11-12), the seven products of a layer at 512 columns as a prompt rate (512 tokens over the layers' product time) and the mean TFLOPS; OpenCL is OKLM's OpenCL backend with its defaults (8-bit matrix units for Q4_K, Q6_K and Q8_0, F32 tiles for F16):

Weights Model Vulkan F32 tiles Vulkan cooperative matrices OpenCL
Q4_K Llama-3.2-1B 6300 t/s, 12.3 TFLOPS 19800 to 20300 t/s, 38.5 to 39.4 18100 t/s, 35.3
Q4_K Qwen3-8B 1000 t/s, 13.9 3290 to 3360 t/s, 45.7 to 46.7 3110 t/s, 43.2
Q6_K Llama-3.2-1B 6000 t/s, 11.6 17800 t/s, 34.6 17300 t/s, 33.6
Q6_K Qwen3-8B 920 t/s, 12.7 2920 t/s, 40.6 2810 t/s, 39.1
Q8_0 Llama-3.2-1B 6400 t/s, 12.4 20100 t/s, 39.2 20600 t/s, 40.0
Q8_0 Qwen3-8B 990 t/s, 13.7 3270 t/s, 45.4 3410 t/s, 47.3
F16 Llama-3.2-1B 4700 t/s, 9.2 19600 t/s, 38.2 6600 t/s, 12.9
F16 Qwen3-8B 760 t/s, 10.5 3290 t/s, 45.8 980 t/s, 13.5

The largest products (Qwen3-8B's gate and up, 4096 x 12288) reach 50 to 51 TFLOPS; the smallest (Llama-3.2-1B's keys and values, 2048 x 512, 16 tiles split in four) about 18. llama.cpp's Vulkan backend runs a whole 512-token prompt (llama-bench pp512, Q4_K_M) at about 15700 t/s on Llama-3.2-1B and 2780 t/s on Qwen3-8B, attention and the rest included; these products alone take 26 ms and 155 ms of such a prompt. The display shares the GPU, so the small products vary by up to a factor of two between runs; the large ones by a few percent. The F32 tiles run at OpenCL's F32 tiles' rate (12.7 and 13.6 TFLOPS on Q4_K).

Mixture of experts

vk_ops_moe.c and shaders/moe/ run a MoE layer's routing and its expert products: the decode step of a MoE model (router, choice, weights, the three indexed products, the sum) stays on the GPU, reads nothing back and replays.

Op Shader Takes Against the CPU
MOE_SELECT moe_select F32 probabilities, up to 1024 experts, up to 64 chosen, an optional F32 selection bias, up to 64 expert groups identical experts and order
MOE_WEIGHTS moe_weights norm, scale bit-identical
MOE_SUM moe_sum F32, any row strides bit-identical
MATMUL_ID moe_group, mvid_<type>; prompts on the prompt products' cm_<type> or gm_<type> every weight type of the one-column products, F32 activations shared by the ranks of a token or one row per rank, any number of pairs within the one-column products' tolerance; prompts on cooperative matrices within the prompt products' (F16 inputs)

Left to the CPU: more than 1024 experts, 64 chosen or 64 groups; indexed products with an expert table (w_experts, the CPU's expert cache); IQ2_S, IQ1_S and IQ1_M weights on Intel GPUs (as the one-column products); unaligned operands. The 8-bit products group is tried first for MATMUL_ID.

  • moe_select: one work-group of 64 per token. The scores (probability plus bias, one rounded add) go to shared memory; with groups, each group sums its two best scores with the CPU's comparisons and counts the groups ranked before it, the best n_group_used kept. Then each competing expert counts the competing experts ranked before its own (higher score, or the same score and a lower index, stopping at K), and the expert of rank k writes slot k: the order is strict, so this is the CPU's K passes in one step. NaN scores are not ordered as on the CPU.

  • moe_weights: one invocation per (rank, token); the token's sum in rank order and the product are precise, the division div_rn() (norm/norm_common.glsl), so the weights are the CPU's bits.

  • moe_sum: one invocation per output element, the ranks in order, each product rounded before it is added.

  • MATMUL_ID, decode and small batches (no more pairs than experts): one work-group per (block of rows, pair), the pair's expert read from the ids tensor on the device. The loop is the one-column products' (each group of 32 weights dequantized once by dq_group()), with one change: an expert's rows are short (16 or 32 groups for rows of 512 or 1024), so the work-group of 64 is cut into segments of a power of two invocations at least the row's groups, each segment taking its own rows (Granite's down product: four rows per work-group instead of 16 busy invocations of 64).

  • MATMUL_ID, more pairs than experts: moe_group (one work-group) counts each expert's pairs, lays out one item per chunk of an expert's pairs and places the pairs by expert in the group's scratch buffer. The grid has pairs / chunk + experts items, the unused ones empty, so it depends on the shapes only, and a pair's result does not depend on the other pairs of its item. Two shaders take the items:

    • 16 pairs an expert or more on average (prompts): the prompt products' tiles (oklm_vk_gemm_tiles()'s grouped form, Prompt products). The items are its tile table (expert, first, end), a chunk a tile's columns (128 on cooperative matrices, 64 on the F32 tiles); the placed pairs are the destination columns, and the activation columns too unless the ranks share their token's row, where moe_group also writes each placed pair's token. Each tile is one expert's weights by its pairs: cooperative matrices with F16 inputs where the device has them (F32 weights keep the F32 tiles), as a dense layer's prompt products. It needs the pairs of a token adjacent in the destination (and in the activations with a row per rank), which the models' buffers are; other strides take the chunks below.
    • Fewer (a few tokens): mvid_<type> takes one item (expert, first, count) per work-group, reading each weight group once for its NC pairs (NC: the average pairs per expert rounded up to 1, 2, 4, 8 or 16), F32 throughout.

    OKLM_VK_MOE_TILES=0 keeps every grouped product on mvid_<type>, 1 sends every one to the tiles (read at each product). The two meet at 8 pairs an expert (Tini-Cybersec-8B-A1B and OLMoE-1B-7B, prompts of 64 tokens: 1782 against 1868 and 2264 against 2212 t/s); at 16 the tiles are 1.5 times faster (128 tokens: 2218 against 3346 and 2719 against 4024), at 4 slower (32 tokens: 1435 against 1000 and 1861 against 1200), a tile costing the same whatever it holds.

  • Replay: the chosen experts never leave device memory and no pushed value depends on them: Granite-3.1-1B-A400M, OLMoE-1B-7B and GLM-4.7-Flash replay 381 to 383 of 384 decode steps (oklm_vulkan_step_stats()).

Fusion. As on OpenCL, backend_vk.c holds back the first ops of three patterns (two of a MoE layer, and the end of any residual branch) and runs each pattern as one dispatch when the op that completes it arrives; the group file does the dispatch:

Ops Fused dispatch Against the ops one by one
SOFTMAX (no mask, at most 1024 values), MOE_SELECT on its output, MOE_WEIGHTS on both moe_route bit-identical (probabilities, experts, weights)
MATMUL_ID gate, MATMUL_ID up (same weight type and layout, activations and experts; one column per pair; products the 8-bit group does not take), SWIGLU of the two in the gate output mvid_<type> with its GLU constant set bit-identical (the SwiGLU output and the up output)
MATMUL on mv_<type> (not accumulating, not one of the 8-bit group's) or MOE_SUM, then optionally the SCALE of its output in place, then the ADD of that output into a tensor of its shape, in place (x = x + o or x = o + x, x apart from o and from what the producer reads): the residual tail mv_<type> or moe_sum, given the residual stream's address bit-identical (the output and the stream)
  • The hold. A softmax, a MATMUL_ID that the decode path of mvid_<type> takes, or a MATMUL or MOE_SUM whose shader could take a residual tail, is copied (the op and its tensors) instead of run; a MOE_SELECT on the held softmax's output, the up product matching a held gate product, or the SCALE in place of a held producer's output, joins it. The op after that either completes the pattern (the fused dispatch) or not: then the held ops run one by one, in order, as they would have, and the new op after them; operands the fused shader does not take (a prompt's grouped products, overlapping tensors, an op on the CPU) do the same. Every other entry point of the instance (upload, download, synchronize, flush, the stream calls, step_begin, step_end, a buffer free) runs the held ops first, so a read-back sees their results, an upload or a free cannot change what they read, and a decode step records them in their place: the fused steps replay as before (OLMoE: 381 of 384). Held ops belong to the thread that issued them only while no other thread uses the instance: contexts and encoders that share a device from threads of their own bracket each computation as a pass (oklm_backend_pass_begin(), server.md). A pass that lets another in between two of its blocks (oklm_backend_pass_yield()) first waits for its work (synchronize), which runs the held ops and submits the open command buffer: the pass let in finds neither.
  • Barriers. A fused dispatch declares every destination of its ops as written and every source as read (oklm_vk_op_begin_fused()), the sources inside its own destinations left out (the write already waits for their earlier writers).
  • moe_route. One work-group per token, at the softmax shader's work-group size for the row (64 or 128), so the max and the two-float sum reduce in the same order as norm_softmax's and the probabilities have its bits; they are written out and kept in shared memory, then the choice runs moe_select's ranking and invocation k computes the weight of rank k as moe_weights does. With no more experts than invocations (the usual case) each invocation reads its score once and keeps it for the three passes of the softmax, and the ranking reads sixteen scores of shared memory a step.
  • GLU. The fused variant of mvid_<type> (specialization constant 4) has the plain product's layout and spec constants, and dequantizes the up rows' groups beside the gate rows' in the same loop: each sum is accumulated and reduced exactly as the plain product's, so both are the products' bits; it writes the up output and g / (1 + exp(-g)) * u (the element-wise shader's SwiGLU expression) where the in-place SwiGLU writes it. The activations are read once for both products, and two barriers per layer go away with the three ops.
  • Residual tail. The producer's invocation that writes an output element also reads the residual stream's element, multiplies its own value by the SCALE's factor when there is one, writes it, and writes the sum to the stream: one rounded product and one rounded add (precise), as SCALE and ADD compute them, so the output and the stream have the bits of the three ops. The SCALE and the ADD, a dispatch and a barrier each, go away: Granite-3.1-1B-A400M has two tails a layer (the attention output's product, scaled, and the expert sum, scaled), OLMoE-1B-7B one (its expert sum; its attention output's product is the 8-bit group's on AMD GPUs), and the dense models whose output and down products are the 8-bit group's none.
  • OKLM_VK_FUSE=0 runs every op on its own; oklm_vulkan_step_stats() counts the fused routings (fused_route), gate / up / SwiGLU dispatches (fused_glu) and residual tails (fused_res).

tests/core/test_vulkan.c checks the residual tails on both GPUs: a product of F32 and F16 weights (one and three columns) and a MoE sum (one, two and five tokens), with and without the scale, the add's operands in either order, each against the same ops with a wait after each on the device bit for bit (the output and the stream), the MoE sum's stream against the CPU's arithmetic too, the counter showing the fusion ran; and that an accumulating product, a multiply after the scale and a broadcast add are left op by op.

tests/core/test_vulkan_moe.c checks on both GPUs: the choice at 1 to 1000 experts, with ties (coarse probabilities), a bias, DeepSeek V3's 8 groups of 32 keeping 4, one expert per group, 64 chosen; the fused routing (Granite's and OLMoE's shapes, DeepSeek V3's groups and bias, 600 and 1000 experts, a prompt, in place or not, a held softmax whose next op is another) and the fused gate / up / SwiGLU (the 15 types at decode, two tokens with a row per rank, Granite's and OLMoE's sizes, a prompt left unfused), each against the same ops one by one on the device bit for bit, the counters showing the fusion ran, and against the CPU or a double reference; the weights normalized or not, scaled or not; the sums at 1 to 8 ranks (all three bit for bit); the indexed products of 15 types (F32, F16, BF16, Q4_0, Q5_1, Q8_0, Q2_K to Q6_K, IQ4_NL, IQ4_XS, MXFP4, IQ2_XXS) at decode (8 of 16 experts, the row shared) and two tokens with a row per rank, longer rows, then the grouped path for six types (160 pairs over 8 experts, 18 pairs, most pairs on one expert, an OLMoE-like prompt), each against a double sum over the dequantized weights within 1e-5 sum |w x| + 1e-6; the same six types on the prompt tiles (160 pairs with the row shared, 18 pairs with a row per rank, most of 300 pairs on one expert over two tiles of rows, 270 pairs with a row per rank and 131 rows), on the F32 tiles at that tolerance and on the device's own (cooperative matrices within 2e-3 sum |w x| + 1e-6), then the default policy at 1120 pairs over 64 experts. It runs in about 10 seconds on both GPUs. With OKLM_VK_BENCH=1 it times each op of a decode step's MoE layer (Q4_K experts, each op repeated with a barrier between runs) on the discrete GPUs; on the RX 7900 XT:

Sizes select weights gate (or up) down sum
Granite-3.1-1B-A400M (1024, 512, 32 experts, 8 chosen) 8.9 us 7.9 us 6.6 us 5.7 us 3.7 us
OLMoE-1B-7B (2048, 1024, 64, 8) 10.4 us 7.0 us 13.9 us 12.5 us 3.3 us
GLM-4.7-Flash (2048, 1536, 64, 4) 8.1 us 5.0 us 9.7 us 13.4 us 2.6 us

Decode on the RX 7900 XT (bench_llama's generation test, 128 tokens, best of 2 to 4 runs; oklm run ... -n 128 --temp 0 gives the same):

Model (Q4_K_M) Vulkan OpenCL
Granite-3.1-1B-A400M 294 to 296 t/s (278 to 279 with OKLM_VK_FUSE=0) 270 to 307 t/s
OLMoE-1B-7B 312 to 314 t/s (303 to 306 with OKLM_VK_FUSE=0) 285 to 291 t/s

OLMoE's gate and up products go to the 8-bit group on AMD GPUs (rows of 2048, see the policy above), so only its routing is fused there; with the float products and both fusions (OKLM_VK_Q8=0) it runs at 324 to 325 t/s, but without the CPU's 8-bit results. | GLM-4.7-Flash | 78 to 79 t/s | 72.5 to 73.5 t/s |

The products use F32 activations, as the one-column products do, where the CPU and OpenCL round them to 8 bits for quantized weights, so greedy output departs from theirs after a few tokens, as a dense model's does.

Prompts on the RX 7900 XT (bench_llama, 512 tokens in one batch, the best of two or three runs, 2026-11-14), the grouped products on mvid_<type> (OKLM_VK_MOE_TILES=0, every prompt until that day) and on the prompt tiles, and OpenCL's grouped tiles in the same session; decode beside them (128 tokens, the same dispatches either way):

Model Pairs an expert mvid_<type> Prompt tiles OpenCL Decode, Vulkan / OpenCL
Granite-3.1-1B-A400M Q4_K_M 128 8365 t/s 20500 17880 634 to 638 / 392
OLMoE-1B-7B Q4_K_M 64 3035 to 3074 9060 to 9190 8790 477 to 485 / 313
Tini-Cybersec-8B-A1B Q4_K_M 64 2512 to 2522 7720 to 7740 6900 to 6960 341 / 255
gpt-oss-20b Q4_K_M (MXFP4 experts) 64 844 2149, then 3738 1676 to 1681, then 3313 96, then 164 / 127
Gemma 4 26B-A4B UD-Q4_K_M 32 1304 2716 2224 106 / 72

gpt-oss-20b's second figures are with MXFP4's values computed in registers (One-column products, dq_fp4()), OpenCL's with its indexed products on the float matrix units (opencl.md). Its decode step was 10.34 ms, 6.67 of them the 72 indexed products (four experts of 2880 x 2880 each, 92 us a product, 190 GB/s of weights), measured by leaving each kind of op out of the step: the decoder read a constant array once per weight. Without the array a product takes 34 us (520 GB/s) and the step 6.10 ms: 2.43 ms the indexed products, 0.8 ms the rest of the 24 expert layers (the router and its bias, the choice, the softmax of the chosen scores, three gathers and three adds for the biases, the gated unit, the sum: about twelve dispatches a layer, where a model without expert biases has five), 2.9 ms the attention blocks and the head. Tini-Cybersec-8B-A1B, whose experts are Q4_K and Q6_K, kept its rates (7859 / 339.7 before, 7824 / 342.2 after) and was above OpenCL's before: the cause was the type's, not the expert path's. A segment of 32 invocations instead of 64 for gpt-oss's rows of 90 groups (each invocation then takes 3 groups or 2, where it takes 2 or 1) gave 166.8 against 163.6 t/s and is not used: it changes the order of the sums.

Where the time went on Tini (24 layers, 22 with experts: 2048 pairs over 32 experts, gate and up 2048 x 1792 in Q4_K, down 1792 x 2048 in Q4_K or Q6_K; each op timed with a wait after it): on mvid_<type> an indexed product took 2.7 to 2.9 ms, 180 ms of a prompt's 220, 5.6 TFLOPS where the dense products of the same prompt ran at 35 (2048 x 6144 by 512 columns in 0.37 ms); on the tiles 0.65 to 0.73 ms, 44 ms of a prompt's 80, 21 to 23 TFLOPS. No op of these prompts runs on the CPU but the token lookup, and nothing is read back between dispatches. What the tiles still lose to a dense product is the partly filled ones: a tile computes its 128 columns whatever it holds, and 2048 pairs over 32 experts lay out 48 tiles (16 would hold them, each expert's last tile being partial and 32 entries of the table unused).

Hybrid layers, rotation and latent attention

vk_ops_ssm.c and shaders/ssm/ run the Gated DeltaNet block (Qwen3-Next, Qwen3.5), the rotation around a quantized KV cache, and the attention of heads wider than the attention group takes (latent attention's 576, and the 512 of Gemma 4's blocks without window, whose window blocks' heads of 256 are the attention group's).

Op Shader Takes Left to the CPU
SSM_CONV ssm_conv F32, rows contiguous (views of wider columns), any K >= 2 other types
GATED_DELTA gated_delta F32, key heads up to 256, any value head; a decay per head or per token (oklm_gated_delta_tok(), the published qwen35 layout) key heads over 256
HADAMARD hadamard; hadamard_fold with a sign vector, reordered heads or groups above 256 (oklm_hadamard_fold()) F32, groups of 2 to 4096, any strides of 4-byte elements; the signs F32 on the instance other types; the second shader on a device with less than 16 KiB of shared memory
ATTENTION mla_attn, mla_merge heads of 257 to 576 (a multiple of 8), F16 or F32 caches on 16 bytes, the values the keys' tensor or one with its strides; causal, bidirectional, window, soft-cap, block table, per-token tables heads up to 256 the attention group does not take, quantized caches of wide heads
  • ssm_conv: one invocation per channel over the tokens in order, the inputs before the first token read from the ring, then the ring takes the last K-1 inputs. The sums are the CPU's (precise, its order); the SiLU uses the device's exp, so the outputs are within 1e-6 absolute plus 1e-5 relative, and the ring is bit-identical.
  • gated_delta: the rows of a head's state are independent once a token's q and k are normalized, so a work-group takes up to 32 rows of one value head (four work-groups per head of 128) and keeps them in registers over every token: 32 invocations per row, each holding the columns lane, lane + 32, ... Per token the group loads q and k into shared memory, reduces their squares, sums each row's S k over its 32 invocations, applies the rank-1 update and sums S q into the output. The state is read once and written once per call, so a prompt does not reread it per token. With a decay per token every invocation computes it from the token's projection (exp(factor * softplus(projection + bias)), the softplus linear above 20 as on the CPU), the key heads repeat over the value heads and the normalized query is divided by sqrt(dk); outputs and states are within 1e-5 absolute plus 1e-4 relative of the CPU's in both forms (test_vulkan_ssm, test_gated_delta).
  • hadamard: one work-group of 64 per group, the butterflies of each stage in shared memory in the CPU's order, then the scale.
  • hadamard_fold: the same with 16 KiB of shared memory (groups up to 4096), for the weights stored in a rotated basis (Ternary Bonsai 2, hybrid-layers-design.md): each element times its sign before the butterflies (a product's input) or after the scale (an embedding row turned back), and a row's value heads read in the order they were folded in. Against the CPU: 1e-6 absolute plus 1e-5 relative (test_vulkan_ssm, test_hadamard_fold).
  • mla_attn: work-groups of 8 rows, a row a (query head, token) pair; the heads of a group that read one KV head share a work-group (latent attention's one shared head: 8 heads per work-group), with several tokens when the group is smaller in a prompt; a single token, or a batch with per-token tables (whose rows must share a table), puts one token in a work-group with as many heads of the group as fit (5 of GLM-4.7-Flash's 20). Keys go by tiles of 32: each invocation computes one (row, key) score over the whole head (8 halves per load), one invocation per row updates its running maximum and denominator, and each invocation adds the tile's weighted values into its dimensions of the 8 rows, so a value row is read once for 8 heads.
  • Key ranges: a work-group's n keys (from its rows' first visible key) are cut into ranges of max(64, ceil(n / ns) rounded up to 32) keys, so it takes at most ns ranges, and mla_merge combines their partials (scratch slot OKLM_VK_SCRATCH_SSM, ns per (token, head), so it keeps its size from one decode step to the next). With one token per work-group, ns comes from the head blocks alone (256 work-groups over them, at most 32: 32 for GLM-4.7-Flash, 16 for DeepSeek V3's 128 heads), so a query's ranges follow its own keys and its result is the same, bit for bit, decoded alone or with other sequences; test_vulkan_ssm.c checks this at 150, 1500 and 4000 keys. Prompts (several tokens per work-group), and per-token batches whose partials would pass 128 MiB, take ns from their work-group count instead (up to 256 work-groups, 1 when there are more: no merge). The grid has the ranges of the longest work-group, min(ns, ceil(keys / 64)), dispatched indirectly; a work-group past its own ranges only marks its partials empty, and the merge skips empty partials.
  • Replay: pos0, the cache's keys and the grid's range count are per-step data, and the merge runs whenever ns is above 1 (a constant of the model and the batch shape), so a latent attention decode step replays.

tests/core/test_vulkan_ssm.c checks each against the CPU on both GPUs: the rotation at n = 2, 64, 128 and 256 (and in place) within 1e-6 plus 1e-5 relative; the convolution then the recurrence over a decode call and a prompt call continuing it (ring and state carried), at heads of 128 (also 70 tokens), heads that are not multiples of the shader's blocks (48 by 40, 64 by 200), and a key head of 512 that takes the CPU fallback (its state written back): the ring bit-identical, the outputs and the state within 1e-5 plus 1e-4 relative; latent attention at 576 (16 and 20 heads decoding over 1 to 2000 keys, prompts of 37 tokens over 100 keys and 70 tokens of 128 heads over 1100 keys, an F32 cache, window with soft-cap, bidirectional, a block table, per-token tables over two sequences) and heads of 320 and 512 with values of their own, within 2e-6 plus 2e-4 relative; three sequences decoded together against each alone (per-token tables and a block table), bit for bit; twelve latent decode steps (a growing cache whose range count changes, then three paged sequences) recorded twice and replayed ten times; and decode timings at 20 and 128 heads. test_vulkan.c's convolution check compares its outputs within the same 1e-6 plus 1e-5 relative, its ring bit for bit.

Norms, rotation and copies

The norm group (vk_ops_norm.c, shaders/norm/, test_vulkan_norm.c) runs these ops on the GPU; an operand set it does not take goes to the CPU fallback.

Op Shader Takes Against the CPU
RMS_NORM, LAYER_NORM norm_rows F32 rows (contiguous elements, any row strides), optional F32 weight and bias within 1e-5 + 1e-5 relative
ROPE norm_rope normal and NeoX pairs, partial rotation, frequency factors (with one position a token or three), linear scaling, YaRN; one position a token, or three with a band of pairs each (oklm_rope_parts(), M-RoPE) within 2e-5 + 2e-4 relative
SOFTMAX norm_softmax scale, optional F32 mask broadcast over rows; a fully masked row gives zeros within 1e-7 + 1e-5 relative
GET_ROWS norm_get_rows tables of F32, F16, BF16, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, IQ4_NL, Q4_K, Q5_K, Q6_K bit-identical
SET_ROWS norm_store_rows F32 rows into F32, F16, BF16, Q8_0, Q4_0, Q4_1, Q5_0, Q5_1, IQ4_NL bit-identical
CPY (after the element-wise group) norm_store_rows, norm_copy_rows F32, F16 or BF16 (any strides) into the six block quants above; rows of one type as bytes (one buffer copy when both are contiguous) bit-identical
ARGMAX norm_argmax F32 rows identical indices
TOPK norm_topk F32 rows, k up to 512, with or without the values identical indices, order and values
CROSS_ENTROPY norm_cross_entropy F32 logit rows, I32 targets within 1e-6 + 1e-5 relative

Left to the CPU: get_rows from the other quantized types (Q2_K, Q3_K, IQ4_XS, the IQ grid types, MXFP4, NVFP4, the ternary and low-bit types); copies into or out of the other quantized types and into or out of I32 other than byte copies; top-k beyond 512; rows whose elements are not contiguous for the row ops (as the CPU requires).

How the results stay close to the CPU, whose kernels are built without contraction, sum in double and call libm:

  • Vulkan rounds an add, a subtract and a multiply correctly but allows 2.5 ulp for a division, a loose exp, and an absolute error of 2^-11 for sin and cos; a compiler may also fuse a multiply and an add. Every expression that must match is precise (never fused), divisions that feed bits (the quantizers' scales, the RoPE angle's frequency factor) go through div_rn(), a correctly rounded division on the integer bits (the 27-bit quotient formed 8, 8, 8 and 2 bits at a time by 32-bit divisions: the devices divide 64-bit integers in software), and exp, sin and cos are the shaders' own (norm_common.glsl): within about one ulp, the angle reduced exactly by pi/2 in six 7-bit parts up to |x| = 2^17 pi / 2.
  • The block quantizers are the CPU's reference algorithms step by step (the same comparisons, roundf as rounding halves away from zero, the same truncating conversions, IQ4_NL's 15 candidate scales), so the KV cache rows written on the GPU are the CPU's bytes. Denormal inputs are an exception where the driver flushes them.
  • Norms, softmax and cross-entropy run one work-group per row (64 to 256 invocations by row length) and sum in two floats (a high part and its rounding error), close to the CPU's double sums; the norm's scale is the inverse square root refined by one Newton step. In norm_rows each invocation takes a contiguous run of the row (8, 16 or 32 values at a time in registers, the pipeline's second specialization constant), adds its squares in two floats, and the invocations' sums are added sixteen at a time in two rounds of shared memory (wg_sum_ff16()); the mean square of a row whose length is a power of two is a product by the inverse (div_n(), the same bits as the rounded division).
  • RoPE builds the angle as the CPU does, the position times theta_scale k times in a row, so the angle is the CPU's to the bit; the per-op constants (theta_scale, the YaRN range and magnitude) are computed on the host with the CPU's expressions.
  • Argmax and top-k compare floats with both zeros equal and take the lower index on ties, as the CPU's scans; top-k finds the k-th largest key with four byte histograms, then sorts the k candidates (bitonic) in shared memory. A NaN is never chosen by argmax; NaN values in top-k rows are not ordered as on the CPU.
  • Measured on both GPUs (the RX 7900 XT and the UHD 750 give the same bits), the float ops differ from the CPU by one or two ulp at most: the norms within 5e-7 absolute on values near 2, RoPE within 4e-7, softmax within one ulp, so the checks hold with margin.
  • Speed, one op at a time with a wait at the end (RX 7900 XT / UHD 750), before the shaders were rewritten for latency: an RMS norm of a 4096 row 20 / 25 us, RoPE of 32 heads of 128 6 / 26 us, softmax of a 151936 row 0.41 / 0.73 ms, top-40 of it 0.70 / 1.04 ms. In a chain of dependent ops on the RX 7900 XT an RMS norm of a 2048 row now takes 3.6 us (10.4 before) and of an 8192 row 7.2 us (21.8): see Latency of a decode step. The vocabulary-sized rows are one work-group each, as on OpenCL.
  • None of these ops takes a value that changes from token to token in its push constants: positions, row ids and slots are read from their device tensors, so a recorded decode step replays them as they are.
  • An id or target outside its table (which the CPU refuses) gives a row of zeros (get_rows), no write (set_rows) or NaN (cross-entropy).

Attention

vk_ops_attention.c and shaders/attention/ run ATTENTION as oklm_attention() defines it: grouped-query heads, pos0, sliding window, soft-cap, bidirectional (encoders, more queries than keys), queries that see each other (mutual, an image's rows in Gemma 3: on attn_split, every query's last key the last query's and its window's start its own), a span of keys a query (oklm_attention_spans(), bidirectional: the windowed blocks of Qwen2.5-VL's encoder and a text encoder's batch of several sequences, each row over its own sequence's rows; on attn_tile only), and the paged cache through its block table (oklm_attention_paged(), any block size) or through per-token metadata (oklm_attention_paged_tokens()). Cache types: F32, F16, BF16, Q8_0, Q4_0, Q4_1, Q5_0, Q5_1 and IQ4_NL (a Walsh-Hadamard-rotated cache is rotated by the caller, as on the CPU). Heads up to 256, a multiple of 4 (of 32 for a block-quantized cache); wider heads (latent attention's 576, the 512 of Gemma 4's blocks without window) go on to the hybrid group. Attention sinks (gpt-oss: one score per query head that joins its softmax without a value) are an F32 tensor on the device whose address is a push constant: both shaders add a head's sink where its output is normalized, exp(sink - max) with the maximum taken over the scores and the sink (in attn_split for one range, or where the ranges are combined; in attn_tile at the end of the block's keys). The hybrid group has no sinks: under heads wider than 256 the op runs on the CPU, the sinks read back with the rest.

  • attn_split (decode, a few tokens, per-token metadata): one work-group of 128 per (key range, run of the query heads sharing a KV head, token). A run is the largest divisor of the group up to 8, so each key row is read once for all the heads of the run. One invocation per key computes the run's scores into shared memory, a key's scores side by side (two keys in flight when the range has more keys than invocations); a range of at most 64 keys (every range of a query of up to 1024 keys, so all of a chat's decoding) gives each key two invocations, half of the heads each, so a wave reads half of the queries. The maxima and sums are reduced with subgroup operations; for the values each invocation owns four dimensions and a share of the keys (four rows and their probabilities read before the sums). The visible keys are split into ranges of at most 512 keys, more of them (of 64 keys at least) while the work-groups are fewer than 128 on a discrete GPU (64 otherwise), at most 256 ranges (128K keys per query; beyond, the CPU). With several ranges each writes its (max, sum, output) to the group's scratch buffer, and the last work-group of a (run, token) to finish, found with an atomic counter that it resets afterwards, combines them in the same dispatch (flash-decoding without a second dispatch and its barrier).
  • attn_tile with a span of keys a query (ATT_SPANS, the table of first key and one past the last at the block table's address): a block walks the tiles from the first key any of its tokens sees to the last, each row masking the keys outside its own span as it masks those past its position in a prompt, so the windows of an encoder's block are one dispatch whatever their sizes and wherever they fall in the blocks of rows. Every cache type. attn_split does not read the table: with fewer rows than half a block, or a block whose shared memory the device does not have (heads of 256 on the UHD 750), the op returns OKLM_ERR_UNSUPPORTED and the caller attends span by span (one dispatch a window, as before).
  • attn_tile (prompts, when a block's rows are at least half used): a block of 32 query rows (16 for heads over 128, and on a discrete GPU for heads over 64; OKLM_VK_ATTN_ROWS=16 or 32 asks for one), the heads of a run times tokens, walks tiles of 32 keys held as floats in shared memory (rows padded by one), any cache type dequantized on the way in: S = Q K^T in 4 x 2 register tiles (4 x 1 for heads over 128), masked; one invocation per row updates its maximum and sum; O = O f + P V in 4 x DO register tiles. (A variant with the queries and keys dimension-major and the softmax spread over 4 lanes per row measured 1.8 times slower on the 7900 XT, 1.5 times faster on the UHD 750.) Other tilings measured on the 512-token prompt layer of the table below (2026-11-12, the 7900 XT shared with other jobs, each beside the shader as it is in the same session) and not kept: shared memory as vectors of four dimensions, a dot product per load, 508 against 418 us (and 13.7 against 41.7 ms on the UHD 750, where prompt attention is 2 % of a prompt); eight rows an invocation, 694 against 418 us; then, best of 3 to 6 runs against 337 us: work-groups of 64 with 8 x 8 register tiles on adjacent floats, as the F32 prompt products, blocks of 64 rows by 64 keys in 53 KB of shared memory, 592 us (310 us of it the products, 137 us the tiles' loads, 190 us left with both and the exponentials removed); work-groups of 64 with blocks of 16 rows, 442 us; tiles of 16 keys, 297 us, within the runs' spread. Fewer shared-memory loads per product did not pay in any of them. What did, for heads of 128, is a smaller block: a work-group's shared memory (its queries, a tile of keys, the scores) bounds how many of them a compute unit runs at once, 37 KB with blocks of 32 rows and 27 KB with 16, and the Qwen3-8B prompt layer of the table went from 1 301 to 886 us with 16 (the 512-token prompt from 2 330 to 2 470 t/s). Heads of 64, at 21 KB, were slower with 16 rows (Llama-3.2-1B: 14 950 against 15 180 t/s).
  • attn_common.glsl holds what both share: four elements of a row of any cache type (16-bit reads for the block quants, whose blocks are aligned to 2 only), the block table, tanh, the window. A cache's halves (an F16 row, a block quant's scale) are converted by the device (unpackHalf2x16(), exact: every half is a float), not by common.glsl's f16_to_f32(): its integer arithmetic, some twenty instructions per value, ran four times per load in the one invocation that walks a key row, about 25 ns per value, and was most of a decode dispatch (see the timings below). attn_split keeps its queries in shared memory four dimensions an element, read with one instruction.

Decode-step replay: pos0, the number of keys (the views' ne[2]) and the number of ranges go to per-step data, and the split grid is dispatched indirectly; the push constants hold only addresses, strides, the batch's sizes and flags, so a decode step over a growing cache replays (checked: six steps over 2041 to 2046 keys, the ranges changing on the way, and six steps of three paged sequences through per-token metadata; one recording and five replays each, no replay given up). The scratch buffer is sized for at least 64 ranges, then powers of two, so it seldom grows (a growth waits for the device and ends the step's recording).

The scores, the softmax and the sums are in F32 and the query is not rounded (the CPU's kernel quantizes it for its integer dot products over a block-quantized cache, so those are checked against F32 attention over the dequantized cache, as on OpenCL). test_vulkan_attention.c checks against the CPU within 2e-6 absolute plus 2e-4 relative (OpenCL's tolerance) on both GPUs: decode at heads of 32 to 256 and groups of 1 to 16 (7 included), prompts with partial blocks and keys before the prompt, window and soft-cap, bidirectional, a span of keys a query (test_spans: windows of unequal sizes, of 1 to 100 rows, smaller than a tile and over several, blocks of rows that straddle windows, one window over everything, 2 to 16 heads, two a KV head, a soft-cap, spans that are not windows, F32, F16, BF16 and Q8_0 caches, both block sizes; 5 rows refused), every cache type, the paged cache (shuffled blocks, a block size of 24, per-token metadata of three sequences), 8K keys, replayed steps, prompts on both block sizes of the tiles (OKLM_VK_ATTN_ROWS=16 and 32: heads of 64, 96 with a group of 7 and a window, 128, a Q8_0 cache with a soft-cap); the device's conversion of halves on an F16 value row of chosen bit patterns (zeros, the smallest and largest denormals and normals, random finite halves) read back through one key, exactly the CPU's values, on attn_split and on attn_tile; then it prints the timings below.

Timings (2026-11-12, one layer, F16 cache, the mean of back-to-back runs, in microseconds; OpenCL is OKLM's OpenCL backend on the same 7900 XT; the GPUs were shared with other jobs, so a few percent of noise; "before": the halves converted by f16_to_f32(), the same session):

Shape 7900 XT Vulkan before 7900 XT OpenCL UHD 750 Vulkan before
Llama-3.2-1B decode (32 heads of 64, 8 KV heads), 1 key 5.8 to 6.3 8.5 to 8.9 7 14
the same, 48 keys 7.3 10.6 to 10.8 17
the same, 112 keys (two ranges) 10.5 14.0 to 14.4 50
the same, 2048 keys 33 41 59 261 287
the same, 8192 keys 92 to 93 103 to 104 141 723 1065
Qwen3-8B decode (32 heads of 128, 8 KV heads), 48 keys 10.8 17.0 26
the same, 112 keys 15.5 22.2 to 22.5 68
the same, 1000 keys 19.6 26.6 346
the same, 4096 keys 68 to 69 78 145 729 1060
OLMoE-1B-7B decode (16 heads of 128, 16 KV heads), 48 keys 7.8 13.8 29
the same, 112 keys 10.5 16.7 68
Llama-3.2-1B prompt, 512 tokens 417 to 434 431 to 440 327 38000 41700
Qwen3-8B prompt (32 heads of 128, 8 KV heads), 512 tokens 808 to 817 909 68600 70800

Decode is bound by latency more than by bandwidth (2 MB of cache per 1K keys go at about 280 GB/s at 32K keys): each work-group waits on a chain of dependent reads (query, keys, values, the partials for the merge), and on what each invocation computes between them. Where a short decode's time went before (the dispatch cut after each part, 48 keys; Llama-3.2-1B, Qwen3-8B and OLMoE layers): an empty dispatch 1.6 us; the per-step data and the queries into shared memory 0.7, 1.1 and 0.5 us; the scores 4.0, 7.5 and 6.7 us (1.5, 4.1 and 1.0 us with the device's conversion); the maxima, the exponentials and the sums about 1 us (two shared-memory rounds); the values and the output 3.3, 5.5 and 5.1 us (2.5, 4.2 and 3.6 us). llama.cpp's Vulkan backend takes 9.2 to 9.3 us for the attention of a Qwen3-8B or OLMoE token of the same benchmark (GGML_VK_PERF_LOGGER).

Measured on the same shapes and not kept (us, in the order Llama-3.2-1B 1, 48 and 112 keys, Qwen3-8B 48 and 112, OLMoE 48 and 112; the shader as it is: 6.2, 7.3, 10.5, 10.8, 15.5, 7.8, 10.5):

  • a head's maximum and sum from subgroup operations alone when a short range's scores of one head sit in one subgroup (a pipeline of full subgroups of 64), the probabilities normalized at once, the key loads in batches of eight: 4.7, 6.2, 9.1, 10.5, 14.9, 6.7, 9.2 (one key load at a time: 5.0, 6.4, 9.4, 9.6, 14.3, 7.6, 10.4). At most 1 us of a dispatch, under 1 % of a token, for a second pipeline variant;
  • on top of it, the value sums of a column added across its invocations with a clustered add (no shared-memory round): within 0.2 us;
  • one range for up to 128 keys instead of two of 64 (no merge): 6.1, 7.3, 10.3, 10.8, 16.3, 7.7, 13.1.

Fallback

An op no group takes runs on the CPU backend over host copies of the byte ranges its operands touch, then the destination range is written back, and the source an op updates in place too (the convolution's ring of SSM_CONV, the state of GATED_DELTA); it is logged once per op. Operands in host memory (token embeddings) are used as they are. Every op therefore works on this backend: the tiny model's greedy output through --backend vulkan is the CPU's.

The model and the decode step

The model on Vulkan

--backend vulkan (or cp.backend = "vulkan", "vulkan1" for the second instance) places a model exactly as OpenCL does: the forward passes, the placement and the memory planning go through the backend interface only. The last --gpu-layers layers (by default as many as fit) and the head are on the device, the first ones and the token lookup on the CPU; the weights are copied once per model, the KV cache (contiguous or paged, for every sequence of the context), the recurrent states of hybrid layers and the activations live in device memory; --gpu-stream streams the CPU layers for prompts through the plain upload; contexts spread their layers over the other Vulkan instances unless n_devices is 1. oklm run, chat, serve, plan and the state files (--kv-save) work the same way; an op no shader takes yet runs on the CPU (see Fallback), so the output is the CPU's.

The fit (fit_layers, oklm_context_mem_estimate(), the server's paged blocks) reads the device-local heap and the largest allocation from the Vulkan device (maxMemoryAllocationSize, 4 GiB on RADV and ANV). The weights and the KV cache are split into several allocations when they exceed it: the weights by runs of tensors (a run never straddles two allocations; each tensor is addressed by itself), the cache by whole layers, so a layer, the head and one layer's cache must each fit in one allocation, not the whole model. OKLM_GPU_MAX_ALLOC_MB sets a smaller limit (to test the split on a small model, on any device backend).

Checked on both GPUs (tests/model/test_model.c, test_vulkan; the CPU fallback taking every op no shader takes yet): the tiny models of every architecture against their float64 references (F32 and F16 caches, a CPU layer before the device ones, nothing on the device, routed experts on the CPU, a streamed CPU layer), batches against single tokens, several sequences in one context (contiguous and paged), argmax-only items, swapping a sequence out and in; state files to and from device caches and hybrid states (test_kv_file.c); layers spread over two instances of one device (test_multi_device.c, OKLM_VK_DEVICES=0,0).

The public models on the RX 7900 XT (tests/model/test_infer_public.c, test_vulkan, the float products: OKLM_VK_Q8=0, OKLM_VK_COOPMAT=0), with OpenCL's bounds: against the float64 reference SmolLM2-135M (F16 and seven quantized files) within 3.2e-5, Gemma 3 270M 3.2e-4, GPT-2 7.3e-4, the Gemma, Phi-3 and Phi-2 files within 2.5e-3, every greedy choice equal; against llama.cpp (Qwen2.5, Qwen3 0.6B and 8B, GLM-4-9B, Llama-3.2-1B, TinyLlama, Gemma 1 to 3, Phi-3, OLMoE) its tokens, 153 to 160 of 160 choices on its tokens, the continuations parting only at its near ties (gap at most 0.5). With the cooperative matrices (the default), a prompt's activations are rounded to F16: SmolLM2-135M then within 0.05 of the reference, its greedy choices still equal.

Decode-step replay

The model marks a decode step (a batch of at most 8 tokens per sequence, 64 in all, every device layer on the head's device, nothing streamed) with the backend's optional step_begin(key) and step_end(); the key hashes what makes two steps the same: the context, the batch's sequences, rows and token counts, the logits rows and the device layers. On Vulkan:

  • The first step of a key is recorded into reusable command buffers (a graph, kept per key) as it runs, and every command is logged: the pipeline, the push bytes, the group counts, the buffers and offsets of copies and fills, the barrier.
  • A later step with the same key is replayed. The forward pass runs on the host as usual and each op fills its push constants, but each command is compared with the log instead of being recorded, and the recorded command buffers are submitted again as they are: no command is recorded.
  • A recording is cut into parts, command buffers of 16, 64, 256, ... commands (the eighth takes the rest), and each part is submitted as soon as its last command is recorded or, on a replay, matched: the device runs the step's first commands while the host goes through the following ones. With one command buffer submitted at the step's end the device waited for the whole forward pass, about 0.35 us per op on the host (see the profile). The barriers follow the ranges read and written across the parts as within one.
  • A command that differs gives up the replay: the commands that matched since the last part submitted go into the ring, and the step runs on directly (the graph is recorded again on the key's next step). A host operation inside the step (an op on the CPU reading device data, a download, an upload of more than 64 KiB, a wait) ends the recording or the replay the same way, and the key is then run directly. Either way the results are the same; only the time differs.
  • Which keys are kept. An instance keeps 16 recordings (OKLM_VK_GRAPHS, 1 to 64); each gets its command buffers and its per-step data (256 KiB) when it is first recorded. While one is free, a new key is recorded at once. Once all are taken, a key seen for the first time runs directly and is remembered (the last 64 such keys); only a key seen before replaces the least recently used recording. A step whose shape does not come back (a speculative step's draft lengths) thus never pushes out the shapes that do.
  • OKLM_VK_REPLAY=0 runs every step directly; oklm_vulkan_step_stats() counts the steps recorded, replayed, broken and aborted, the command buffers of recordings submitted (parts), the recordings evicted, the new keys run directly, the barriers and the commands recorded without one; the instance logs them when it is released.

The values that change from token to token must reach the shaders through device memory, never through push constants or group counts, or the step never replays:

Value Where it changes How it reaches the shader
positions (ROPE) every token the pos tensor, uploaded before the step (already an operand); the three positions of an image's rows in a model with M-RoPE follow the batch's one-part positions in it
the cache cells written every token contiguous caches on Vulkan are written with SET_ROWS through the context's slots tensor (uploaded before the step) instead of a CPY into a view at the position; paged caches through the block tables, as before
the token's row of the embedding table every token GET_ROWS from the host's table: computed on the host and uploaded inside the step (a recorded copy from per-step data, rewritten on each replay)
the residual stream handed from CPU layers every token uploaded into x (as above)
ATTENTION: pos0 (iparams[0]), the number of keys (the key and value views' ne[2]) every token per-step data (below)
paged ATTENTION: pos0, max_keys (iparams[5] of the multi-sequence form) every token per-step data
a grid that follows the number of keys (split attention, latent attention's key ranges) every token oklm_vk_dispatch_indirect()

Per-step data: an op calls oklm_vk_step_data(s, bytes, &addr) (or oklm_vk_step_u32(s, values, n)), writes the values at the returned host pointer, pushes addr (a uint64_t) instead of the values, and its shader reads them with RefU32(addr + 4 * i).v. Within a step the data lives in the graph, at the same address on every replay, and the host writes the new values there before the part that reads them is submitted again (so an op writes its values before it records the command that reads them: the part may be submitted by that very command); outside a step it lives in a small host-visible buffer of the command buffer being recorded. An op may ask for up to 4 KiB per call to its group. A grid that depends on such a value is dispatched with oklm_vk_dispatch_indirect(s, p, &push, gx, gy, gz), whose counts are per-step data too. Everything else an op pushes (tensor addresses, strides, sizes of the batch, flags) is the same from token to token in a decode step and stays in the push constants. An op that pushes a per-token value directly is still correct: its steps only run without replay (the comparison fails at its command), which oklm_vulkan_step_stats() shows as aborted replays.

Observation

A context that records its activations (--observe, observe.md) copies the rows it wants into staging memory on the device, one oklm_cpy() each: commands of the step like the others, with the same addresses from token to token, so the step is recorded and replayed with them, and the staging memory is read back once after step_end(), with the keys the attention rows need (oklm_vk_download_many(), sixteen reads a wait at most). The staging memory is allocated before the first step starts (model_observe_steps()).

A pass that has to read the device before it ends is not marked as a step (forward_step() in src/model/model.c): the lens's logits, the means' rows, parts taken out, more than eight sequences in the batch. Nothing is recorded or replayed then, each op is submitted as in a prompt's pass and each read waits for the queue. A context that does not record is not concerned: the test is on the host, once a pass.

Latency of a decode step

A decode step is a chain of a few hundred dispatches, most of them waiting for the one before (a memory barrier), so on a discrete GPU its time is the sum of each dispatch's latency plus the weights' bytes over the memory bandwidth. Measured on the RX 7900 XT (Llama-3.2-1B Q4_K_M, 203 commands and 163 barriers a token, 2026-11-12) by recording the step with one class of dispatch left out, or with every dispatch cut to one work-group:

Part of a token Before After
whole step on the device (submission to completion) 2 330 to 2 390 us 1 930 us
host: the forward pass that checks the replay 80 to 130 us 90 us
every dispatch cut to one work-group (the latency alone) 1 300 us 760 us
the products' weights (800 MB) 1 030 to 1 130 us, about 700 GB/s the same
33 RMS norms 330 to 370 us 165 us
16 attention dispatches (128 keys at most) 250 to 300 us 300 us
24 rotations, 32 cache writes 140 and 160 us 72 and 64 us
16 SwiGLU 150 us 50 us
the barriers alone (no dispatch) 28 us 28 us

What a dispatch's latency is made of, and the rules the decode shaders follow since:

  • No 64-bit division. The devices divide 64-bit integers in software: the five divisions that turned a 64-bit invocation index into (element, row, i1, i2, i3) cost a work-group about 4 us, more than the op itself. The shaders that divide their index take it in 32 bits (GLOBAL_INDEX_1D_U32(), ROW_INDEX_U32(); oklm_vk_dispatch_1d() refuses more than 2^31 invocations), and the element-wise shader skips the division when every operand is contiguous or has its rows at a constant stride (its FLAT variants). A chain of dependent element-wise ops went from 5.5 to 1.6 us an op.
  • Loads cost, arithmetic does not. A load instruction of a wave of 64 invocations takes about 0.1 us, whatever it reads, and loads issued one after another are served one after another; an instruction of arithmetic is a hundred times cheaper. So an invocation reads little (the work is spread over the invocations of the work-group, up to 256), reads everything it needs before computing (the loads are then in flight together), and reads nothing twice. norm_rows was 256 invocations each reading 8 values one by one, twice, with a shared-memory round per halving of the reduction: 10 us; it is now two vector loads, the weights read beside them, and two shared-memory rounds: 3.6 us.
  • Cheap is not free in a long chain. A wave runs about an instruction per nanosecond, so an invocation that does a few thousand of them in a row adds microseconds to its dispatch whatever the device has idle: decode attention's invocation per key converted the 64 or 128 halves of its key row with f16_to_f32(), about 25 ns each, 4 to 7.5 us of a dispatch of 11 to 17 (Attention). A product's invocations convert two halves per group of weights and are bound by memory: there the same function costs nothing measurable.
  • A dispatch's floor is about 1.6 us on this device (a barrier, a pipeline, push constants and one short wave), so the number of dependent dispatches still counts once each is short.

Three more changes follow the same rules, each measured alone against the tree before it (bench_llama, 128 tokens, Q4_K_M, the RX 7900 XT, best of 4, two rounds of interleaved runs, 2026-11-12):

Change Llama-3.2-1B Qwen3-8B Granite-3.1-1B-A400M OLMoE-1B-7B
before 465 to 466 t/s 105.7 to 105.8 449 to 450 394
decode attention (attn_split): a key's scores side by side in shared memory, two invocations per key for ranges of at most 64 keys, the probabilities read with the value rows 476 to 478 107.5 to 107.7 460 to 461 407 to 409
MoE routing (moe_route): a score read once per invocation, the ranking's scores sixteen a step; div_rn() without a 64-bit division (the routing weights, the softmax) 467 to 469 105.3 to 105.5 462 404 to 406
the residual tail in its product or MoE sum (see Mixture of experts) 463 to 469 105.3 to 105.7 496 to 500 399 to 401
the three together 472 to 482 107.2 to 107.3 527 to 532 423 to 424

The dense models' differences in the last two single rows are within the noise of the runs (about 0.5 %): they have no routing and no tail fused. Qwen3-8B with the three is 0.3 % under the attention change alone (107.2 to 107.3 against 107.4 to 107.8, repeated), most likely the host asking for every product whether it could take a tail.

Decode on the RX 7900 XT (bench_llama, 128 tokens, Q4_K_M, best of 3 to 5, 2026-11-12; llama.cpp's Vulkan backend on the same day):

Model Before With the latency rules With the three changes above llama.cpp Vulkan
Llama-3.2-1B 388 t/s 471 t/s 482 to 493 t/s 481 t/s
Qwen3-8B 87 t/s 106 t/s 107.4 to 107.7 t/s 117 t/s
Granite-3.1-1B-A400M 295 t/s 446 t/s 552 to 559 t/s 514 t/s
OLMoE-1B-7B 313 t/s 393 t/s 428 to 439 t/s 481 t/s

The last two columns are from one session, five runs of each (llama.cpp llama-bench -ngl 99 -p 0 -n 128 -r 5); the tree before the three changes ran at 476 to 483, 106, 461 to 464 and 399 to 404 in it. The same binaries ran 3 to 5 % slower in the session of the table of changes above, earlier the same day (475, 107, 530 and 424 with the three): compare runs of one session only.

What is left is Qwen3-8B (0.92 of llama.cpp) and OLMoE (about 0.9). None of the three changes moved Qwen3-8B by more than 2 %, and neither does running its products on the float group (109.7 with OKLM_VK_Q8=0 against 107.4 in the same minutes; OLMoE 442 against 428, without the CPU's 8-bit results). The profile below says where the rest was: the host's pass, which the device waited for (the tail's check among its ops), and attention.

Profile of a decode step

Where a token's time goes on the RX 7900 XT (RADV), bench_llama's generation test of 128 tokens, Q4_K_M, 2026-11-12, measured with the clock around the host's calls and by recording the step with one class of dispatch left out (as the table of the section above).

Host against device. A replayed step is the host's forward pass, which checks every command against the recording, then the device's work:

Model Commands (barriers) Host pass Of it in the backend Step on the device Read-back of the logits
Qwen3-8B 525 (399) 190 to 210 us 180 us, 528 ops 9 010 us 30 us
OLMoE-1B-7B 299 (227) 127 to 150 us 120 us, 349 ops 2 220 us 12 us
Llama-3.2-1B 203 (163) 95 us 75 us, 204 ops 1 920 us 25 us
Granite-3.1-1B-A400M 353 (293) 210 us 172 us, 548 ops 1 640 us 9 us

The host pass costs about 0.35 us per op (the op's route, its push constants, the comparison with the recording), and with the recording submitted at the step's end the device was idle through all of it: 2 % of a Qwen3-8B token, 6 % of an OLMoE one, 11 % of a Granite one. Submitting the recording before the pass (wrong results, the per-step data being the previous token's; a bound on what the pass costs) gave 109.0 against 107.3 t/s on Qwen3-8B, 445 against 421 on OLMoE, 486 against 475 on Llama-3.2-1B and 586 against 528 on Granite. The recording cut into parts submitted as the pass reaches them (Decode-step replay) gets that without changing a result (best of 4, two alternated rounds each, llama.cpp's Vulkan backend in the same session):

Model One command buffer Parts of 16, 32, 64, ... commands llama.cpp Vulkan
Qwen3-8B 107.2 to 107.5 t/s 108.3 to 108.6 116.5
OLMoE-1B-7B 422 to 423 443 to 450 481 to 483
Llama-3.2-1B 476 to 478 485 to 489 484
Granite-3.1-1B-A400M 532 to 535 589 to 593 513 to 514

A first part of 8, 16, 32 or 64 commands: OLMoE 446 to 449, 443 to 450, 434 to 437 and 434 to 437 t/s, Llama-3.2-1B 491 to 496, 487, 482 to 487 and 481 to 484, Granite 587 to 594, 592 to 601, 594 to 600 and 586 to 590; 16 is kept. Each part four times the one before (16, 64, 256, ...: three command buffers for a step of 300 commands where doubling makes five) is what the backend does: measured after the attention change below, three alternated rounds, 637 to 640 against 616 to 628 t/s on Granite, 461 to 472 against 459 to 465 on OLMoE, 502 to 504 against 499 to 503 on Llama-3.2-1B, 111.3 against 110.9 on Qwen3-8B. Two parts only (16 commands, then the rest) leave the device idle while the host finishes its pass: Granite 586 to 588, OLMoE 459 to 460.

Attention. Leaving the attention dispatches out of the recording took 647 us off a Qwen3-8B token (36 dispatches, 18 us each) and 282 us off an OLMoE one (16, 17.6 us each), where llama.cpp's flash attention takes 9.3 and 9.2 us (GGML_VK_PERF_LOGGER=1): the largest difference per dispatch of both models. Most of it was the conversion of the cache's halves in the invocation that walks a key row (Attention); with the device's conversion (same session, best of 4, two alternated rounds):

Model Before Halves converted by the device llama.cpp Vulkan
Qwen3-8B 108.8 to 109.0 t/s 111.3 to 111.7 116.5
OLMoE-1B-7B 443 to 449 464 to 465 481 to 483
Llama-3.2-1B 489 to 490 502 to 504 484
Granite-3.1-1B-A400M 590 to 592 631 to 633 513 to 514

Per kind of dispatch, Qwen3-8B (36 layers; a layer is 13 or 16 dispatches: the q, k and v products are one dispatch of 6144 rows where the three tensors have one type, the gate and up products one of 24576 rows, the residual adds are accumulating products). Time a token loses when the kind is left out of the recording, before the two changes (the step 9 010 us on the device) and after them (8 845 us from the step's start), against llama.cpp's Vulkan backend on the same model (GGML_VK_PERF_LOGGER=1, which times each op alone: 509 dispatches, 8 640 to 8 680 us a token under the logger, 8 570 without it):

Kind Dispatches Before After llama.cpp
products 181 7 410 us 7 500 us 253 products, 7 254 us
attention 36 647 us (18.0 each) 440 us (12.2 each) 335 us (9.3 each)
RMS norms (73 of a layer or the head, 72 of the queries and keys) 145 384 us (2.6 each) 410 us 463 to 475 us (73, with their weights), the query and key norms with the rotations and cache writes below
rotations 54 122 us (2.3 each) 120 us 430 to 445 us (72 dispatches: norm, weights, rotation, 36 of them with the cache write)
cache writes 72 95 us (1.3 each) 93 us 91 to 103 us (36)
SwiGLU 36 69 us (1.9 each) 65 us 87 to 89 us (2.4 each)
every dispatch cut to one work-group 525 2 880 us 2 590 us
no barrier at all (wrong results) 7 256 us
the products alone, and with no barrier between them 181 7 424 and 6 608 us

The products are 83 % of the token, and the other kinds together now take less than llama.cpp's. Per product shape after the changes, the 8-bit group's dispatch, the float group's on the same step (OKLM_VK_Q8=0) and llama.cpp's ops for the same weights:

Product Per token 8-bit (q8mv) Float (mv) llama.cpp
gate and up, 24576 x 4096 Q4_K (56.6 MB) 36 86 us (660 GB/s) 81 us 2 x 40.6 us
down, 4096 x 12288 Q6_K (41.3 MB) 18 71 us (580 GB/s) 65.5 us 65.2 us
q, o, and down in Q4_K: 4096 rows 72 1 740 us together 1 683 us 1 765 us
q, k, v in one, 6144 x 4096 Q4_K 18 24 us 22 us 26.5 us (three)
k or v alone, 1024 x 4096 (float group) 36 2.9 us (Q4_K), 6 us (Q6_K) 4.3 and 6.9 us
head, 151936 x 4096 Q6_K (510 MB) 1 752 us (678 GB/s) 785 us 701 to 712 us

In the step, the float group's gate / up and down products are 5 to 9 % faster than the 8-bit group's, which the products timed alone do not show (8-bit products): Qwen3-8B decodes at 113.8 to 114.0 t/s with OKLM_VK_Q8=0 against 111.3 to 111.7 (Llama-3.2-1B 503 to 509 against 500 to 504, OLMoE and Granite the same), without the CPU's 8-bit results. Tried on the 8-bit shader and not kept: four weight rows per subgroup instead of two, which halves the share of the rounding (Qwen3-8B 111.2 to 111.4 against 111.1 to 111.5, OLMoE 459 against 461 to 466); the device's conversion of the blocks' half scales in every shader (f16_to_f32() as unpackHalf2x16(): Qwen3-8B 111.1 to 111.2 against 111.3 to 111.6, OLMoE 465 to 466 against 470, Llama-3.2-1B 501 to 506 against 504 to 505, Granite 620 to 625 against 623 to 625: a product's invocations are bound by memory, not by their arithmetic).

Per kind of dispatch, OLMoE-1B-7B (16 layers of 18 or 20 dispatches; the step 2 220 us before, 2 025 us after; llama.cpp 2 340 to 2 400 us under its logger, 2 080 without it). Leaving the routing out changes which experts the products read (the same ones every token, served by the device's cache), so the products' rows hold with the routing in place only:

Kind Dispatches Before After llama.cpp
gate and up of the 8 chosen experts, 1024 x 2048 each (8-bit) 32 720 us (22.5 each) 593 us (18.5 each)
q, k, v in one, 6144 x 2048 Q4_K (8-bit) 8 117 us (14.7 each) 3 x 7.0 us
head, 50304 x 2048 Q6_K 1 137 us 138 to 200 us
q, k, v, o alone, 2048 x 2048 (float) 40 277 us (6.9 each) 245 us (6.1 each) 7.0 to 9.2 us each
down of the 8 experts, 2048 x 1024 each (mvid) 16 386 us (24.1 each) 353 us (22.1 each) 19.4 (Q4_K) and 24.4 us (Q6_K), the routing weights applied in them
router's product, 64 x 2048 F32 16 71 us (4.4 each) 57 us (3.6 each) 2.1 us each
routing (moe_route) 16 107 us (6.7 each) 3.5 to 3.7 us each
expert sum with the residual add 16 66 us (4.1 each) 46 us (2.9 each) 1.9 us each
RMS norms 65 208 us (3.2 each) 167 us (2.6 each) 308 to 316 us (4.7 to 4.9 each)
rotations 24 86 us (3.6 each) 57 us 120 to 124 us (32, 16 of them with a cache write)
cache writes 32 79 us (2.5 each) 56 us 27 to 33 us (16)
attention 16 282 us (17.6 each) 148 us (9.2 each) 147 us (9.2 each)
SwiGLU 16 60 us (3.7 each) 43 us (2.7 each) 27 us (1.7 each)
every dispatch cut to one work-group 299 1 090 us 990 us

The read-back of the logits. The logits are copied to host memory by a command buffer of its own, submitted behind the step's last part. Waiting for the step before submitting it (the device idle for the round trip) measured 477 to 479 against 465 to 467 t/s on OLMoE, 513 to 515 against 503 on Llama-3.2-1B and 645 against 618 to 626 on Granite, but 108.7 to 109.1 against 111.1 on Qwen3-8B: the copy queued behind the step ends 125, 157, 124 and 79 us after the step would have ended alone, the copy on an idle device 61, 114, 62 and 265 us after it is submitted. Spinning on the fence instead of sleeping changes nothing (OLMoE 463 to 465 against 468 to 471, Qwen3-8B 111.1 for both), so it is not the host's wake-up. The cause is not found and the read-back is unchanged.

Where the two models stand (best of 4, two alternated rounds, llama.cpp in the same session):

Model At the start Now llama.cpp Vulkan
Qwen3-8B 107.2 to 107.5 t/s 111.5 to 111.6 116.5 to 116.8 (0.96)
OLMoE-1B-7B 422 to 423 465 to 473 482 to 483 (0.97)
Llama-3.2-1B 476 to 478 502 to 503 486
Granite-3.1-1B-A400M 532 to 535 637 to 638 514

What is left on Qwen3-8B, 390 us of a token of 8 970: the 8-bit products against the float ones in the step (about 190 us; its cause in q8mv is not the rounding's share nor the conversions, see above), attention (12.2 against 9.3 us a dispatch, 105 us: the scores and the values of a key range spread over more invocations than one per key and one per four dimensions), the head (45 us). On OLMoE, 60 us of 2 130: the gate and up products, two 8-bit dispatches and a SwiGLU where the float group runs one fused dispatch and llama.cpp two products of 18.5 us (an 8-bit gate / up / SwiGLU dispatch, rounding the activations once, would be the equivalent), the routing and the router's product (3 and 1.5 us each over llama.cpp's).

Serving, several devices and the expert tier

The server. oklm serve --backend vulkan runs what it runs on OpenCL: several requests at once (--parallel), the contiguous or the paged cache (--kv-mem), the shared prefix blocks, swapping to host memory (--swap-mem), preemption, speculative steps. Each scheduler step is one batch of the sequences that decode (and prompt pieces); a batch of at most 8 tokens per sequence and 64 in all is a decode step, marked for replay with a key from its sequences, their rows and token counts. The scheduler puts the decode rows in sequence order whatever the requests' ages, so the same requests make the same batch from step to step: a composition is recorded once and replayed until a request ends or arrives. With speculative decoding (--spec-ngram, on by default) a sequence's row count varies with its drafts, and each mix of draft lengths is a key of its own; those that do not come back are run directly without replacing a recording (see Decode-step replay). Measured with the SmolLM2-135M server, 4 sequences and 8 requests of 48 tokens (2026-11-12, the RX 7900 XT), first while every step still broke at its first op on the CPU (the counts show how keys are found):

Recordings kept Steps Keys recorded Evicted New keys run directly Steps whose key was kept
4 95 26 22 20 49 (52 %)
16 (default) 96 20 4 11 65 (68 %)

With 8 sequences and 12 requests, 16 recordings: 93 steps, 21 keys recorded, 5 evicted, 11 run directly. Once the one-column products and attention ran on the GPU, the same 4-sequence session ran 356 decode steps: 12 recorded, 343 replayed, 1 replay given up, none evicted; test_vulkan_serving 1030 steps: 66 recorded, 957 replayed, 2 evicted, 6 new keys run directly. Without the sequence order the compositions came back rotated (the oldest request first) and the keys did not repeat. test_server (test_vulkan_serving) runs the server on the Vulkan device: six requests on four sequences with both caches (the answers of one by one), a paged cache too small for four requests, preemption with swapping (the answers of each request alone), a shared system prompt found by two requests at once; it prints the step counters.

Several devices. Layers spread over the Vulkan instances as over the OpenCL ones (opencl.md): consecutive layer ranges, the head on the selected instance, the residual stream handed from one device to the next through the host; the devices are those of OKLM_VK_DEVICES (0,1, radv,Intel, all), a device listed twice giving two instances. A forward pass of a spread context holds the pass lock of every device it uses, taken in the order of their registration (server.md). test_multi_device checks two instances of the RX 7900 XT (0,0: the logits of one device exactly) and the RX 7900 XT with the UHD 750 (all: the head on the first, the first layers on the second: the tiny fixtures within 1e-4 of the first alone, the element-wise shaders using each device's exp; SmolLM2-135M with 10 layers streamed and random tokens: the Intel GPU takes its Q8_0 products in 8 bits and the AMD one in floats (8-bit products), so it drifts up to 0.5 in a logit over its 30 layers, 18 of its 20 best tokens the same; the check asks for 18 and at most 1.5; with OKLM_VK_Q8=0 both use floats: within 2e-5, 20 of 20), for the dense, GPT-2, MoE and DeepSeek fixtures and SmolLM2-135M. Each GPU alone gives the CPU's greedy text on SmolLM2-135M. A context spread over several devices runs its steps without replay (the hand-over is a host operation). Vulkan and OpenCL instances are not mixed: a context spreads over the instances of the backend it selected, and a backend list such as --backend vulkan,opencl takes the first one with a warning saying so.

Weight streaming and the transfer queue. The instance has a transfer queue when the device offers one: a transfer-only family (a copy engine), else a second queue of the compute family (RADV on the RX 7900 XT exposes the latter), with timeline semaphores (Vulkan 1.2); OKLM_VK_XFER=0 turns it off. stream_upload copies the host bytes into the slot's host-visible staging (so the caller's memory is free at once) and submits the copy on that queue, waiting for the compute timeline value of the slot's last stream_release; stream_release submits what is recorded and signals the compute timeline after it; stream_acquire makes every later compute submission wait for the slot's last copy on the transfer timeline. A slot's staging grows to its largest batch of uploads and is reused once they are done. A release inside a decode step ends its recording or replay (a host operation); an acquire does not. Without a transfer queue stream_upload is the plain upload. --gpu-stream (layers streamed for long prompts) and the expert tier use it.

The expert tier and the expert cache. --expert-gpu places its slots on the Vulkan device as on OpenCL (moe.md): its refills go through the transfer queue while the device computes the following layers, the products of the next step waiting for them; --expert-cache reads the misses from the file beside it. test_model (test_expert_gpu) runs the tier's cases on three MoE fixtures (Qwen3-MoE, DeepSeek-V3, OLMoE) on the Vulkan device: no expert held, half (filled, then refilled), all, half from a usage file, half with the expert cache (0 and 4 loaders, both read-aheads: the same bits); the device-only run's greedy tokens, logits within 3e-6. download_many queues its copies into the staging buffer and waits once.

auto

oklm_backend_select("auto") (no --backend, OKLM_BACKEND unset or auto) takes, in order:

  1. the first available backend of OKLM_BACKEND_ORDER, a comma-separated list of backend names (vulkan,opencl,cpu, opencl,cpu), probing each named one;
  2. else the registered backend of highest priority: OpenCL instances 10 less their number, the CPU 0, Vulkan instances from oklm_vk_auto_priority().

A Vulkan instance's priority puts it where it is ahead of OpenCL (oklm_vk_auto_priority() in src/core/vulkan/backend_vk.h, which OKLM_VK_AUTO there switches on): an integrated GPU 15 less the instance number and a discrete GPU 20 less it, both above OpenCL; a CPU implementation -1 less it, below the CPU. auto probes Vulkan for that. Until 2026-11-15 an AMD discrete GPU ranked below OpenCL, whose prompts were faster there when the rule was set; on the table below only Qwen3-8B's prompt still is, by 3 %, and Vulkan decodes 18 to 26 % faster on both models, so the rule went. OKLM_BACKEND_ORDER=opencl,vulkan,cpu gives the earlier order. Measured with bench_llama, best of three runs (two on the UHD 750), Q4_K_M, 2026-11-15, the machine otherwise idle:

Device, model Backend Prompt (t/s) Decode (t/s)
RX 7900 XT, Llama-3.2-1B Vulkan 16 235 (512) 509 (128)
RX 7900 XT, Llama-3.2-1B OpenCL 14 861 (512) 403 (128)
RX 7900 XT, Qwen3-8B Vulkan 2 663 (512) 112 (128)
RX 7900 XT, Qwen3-8B OpenCL 2 747 (512) 94.6 (128)
UHD 750, Llama-3.2-1B Vulkan 55 (128) 22.1 (32)
UHD 750, Llama-3.2-1B OpenCL 8.9 (128) 5.4 (32)

Earlier, with the products on the CPU fallback (SmolLM2-135M Q8_0 on the RX 7900 XT, before the prompt and MoE products ran on the GPU), Vulkan ranked below the CPU:

SmolLM2-135M Q8_0, RX 7900 XT Prompt of 64 (t/s) Decode of 16 (t/s)
OpenCL 11 668 479
Vulkan, every product on the CPU fallback 454 19.8
Vulkan, one-column products and attention on the GPU 331 392
CPU (i7-11700K) 3 202 174

Verification

tests/core/test_vulkan.c and the group files tests/core/test_vulkan_<group>.c run on every Vulkan GPU present (tests/core/vk_test_util.h sets OKLM_VK_DEVICES=all unless OKLM_VK_DEVICES or OKLM_VK_DEVICE is set; OKLM_VK_DEVICES=none skips them), here the RX 7900 XT through RADV and the UHD 750 through ANV, and skip when there is none.

Check Result on both GPUs (and llvmpipe when named)
uploads and downloads at offsets, 70 MiB through the 64 MiB staging buffer bit-identical
ADD, MUL (full, row and middle broadcasts), SCALE, in place, on a column slice bit-identical to the CPU
SILU, GELU, GELU_ERF, SWIGLU, GEGLU, SOFTCAP (caps 2.5 and 30), SIGMOID within 1e-6 absolute plus 1e-5 relative
CPY F32 / F16 / BF16 in every direction, on random values and the rounding cases (ties, the half range's edges, denormals, infinities, NaN), and a permuted view bit-identical to the CPU
a dispatch of 16.8 million elements (more work-groups than the UHD 750's x limit) exact
300 dependent ops, more than one command buffer exact
a product or a MoE sum with its residual tail (scale, add) as one dispatch bit-identical to the ops one by one on the device
RMS norm on the GPU within 1e-5 plus 1e-5 relative
get_rows from a host table (CPU fallback) bit-identical
SSM_CONV on the GPU: its ring bit-identical; its outputs within 1e-6 plus 1e-5 relative (the device's exp in the SiLU)
one op per group (RMS norm, one- and 19-column F32 products, decode attention, a Q8_0 product, the MoE sum, the Hadamard rotation), through the fallback until their shaders land as the CPU

The transcendental functions are the shader's own: tanh from its Taylor series near zero and from exp elsewhere, erf from the complementary error function's Chebyshev fit (fractional error below 1.2e-7).

Precision across drivers

Vulkan rounds a float add, subtract and multiply correctly but leaves the built-in exp to the driver's precision, which differs between vendors. Every shader whose results are checked against the CPU's expf (the attention kernels' softmax, latent attention, the gated delta's decay, the conv's SiLU, the element-wise activations, the norms) uses exp_acc() from shaders/common.glsl, within about one ulp, so the results do not depend on the driver's exp (on the UHD 750, deepseek3's several sequences against one per context went from 1.8e-5 to 2e-6).

Limits

  • A query's result does not depend on how many keys other queries of its batch see (the key ranges follow its own keys), but prompts go through the attention tiles and single tokens through the key ranges, two kernels whose sums round differently: batched and token-by-token evaluation agree within 2e-5 (1.1e-5 on the UHD 750, 3.6e-6 on the RX 7900 XT), not bit for bit as on OpenCL. Latent attention (heads over 256) gives a decoded query the same result alone or with other sequences, bit for bit, up to per-token batches whose partials stay within 128 MiB (about 90 tokens of GLM-4.7-Flash, 28 of DeepSeek V3); larger batches split by their work-group count.

  • The ops of a group not yet written for Vulkan take the CPU fallback, which copies their operands each time (slow, but correct); the group subsections above list what each group runs on the GPU and what it leaves to the CPU.

  • A step replays only once every op of its decode path runs on the GPU and reads its per-token values from per-step data; until then each step is recorded, breaks at its first op on the CPU and runs on directly.

  • auto picks Vulkan on every GPU it finds, OpenCL where Vulkan is not there (auto); Vulkan and OpenCL instances are not mixed in one context.

  • NVIDIA is untested (ADR-0023, V4); the Windows build has not been run.