ADR-0023: Vulkan as the main GPU backend
Records how OKLM runs on AMD, NVIDIA and Intel GPUs: which API carries the fast path, what stays of the OpenCL backend, how a decode step avoids its per-launch cost, and how the vendors OKLM cannot test here are covered.
Status: accepted (2026-11-11), after the measurements below; implemented in the steps of the roadmap's "GPU backend for AMD, NVIDIA and Intel" section (V1 to V4).
Context
OKLM's GPU backend is OpenCL (ADR-0017). It was written and tuned on one device, a Radeon RX 7900 XT (gfx1100, ROCm's OpenCL 2.0), and its fast paths (8-bit products, the matrix units) are AMD builtins behind a device-name check. OKLM is to support AMD, NVIDIA and Intel GPUs at least.
Measured on this machine (2026-11-11; the 7900 XT, and the Core i7-11700K's UHD 750 through Intel's compute runtime and Mesa's ANV; llama.cpp at the pinned commit as the reference, decoding 128 tokens, two runs each):
| GPU, model | llama.cpp HIP | llama.cpp Vulkan | OKLM OpenCL |
|---|---|---|---|
| 7900 XT, Granite-3.1-1B-A400M Q4_K_M | 355 | 515 | 301 |
| 7900 XT, OLMoE-1B-7B Q4_K_M | 355 | 480 | 285 |
| 7900 XT, Llama-3.2-1B Q4_K_M | 405 | 482 | 380 |
| 7900 XT, Qwen3-8B Q4_K_M | 105 | 117 | 86 to 95 |
| UHD 750, Llama-3.2-1B Q4_K_M | - | 15.5 | 4.7 |
Prompts of 512 tokens on the 7900 XT: Vulkan level with HIP or 27 to 32 % faster; on the UHD 750, 294 t/s for Vulkan against 8.8 for OKLM's OpenCL.
What the measurements and the drivers show:
- A decode step pays for its launches. On the 7900 XT
a Granite token is about 2.4 ms of kernels and 3.5 ms in all: the GPU
waits about 2.6 microseconds between each of about 420 dependent
kernels, not on the host (0.46 ms of launch calls a token, overlapped).
llama.cpp's HIP build replays a token as a graph: 12 to 16 % on these
models (measured with
GGML_CUDA_DISABLE_GRAPHS). - OpenCL cannot replay a step on these drivers.
Neither ROCm's OpenCL nor Intel's compute runtime exposes
cl_khr_command_buffer; nothing published says NVIDIA's does. Core OpenCL launches every kernel alone. - Vulkan records once and replays natively, with no
extension, and exposes the fast hardware paths through standard
extensions on every vendor:
VK_KHR_cooperative_matrix(the 7900 XT's WMMA, NVIDIA's tensor cores, Intel's XMX) andVK_KHR_shader_integer_dot_product(8-bit dot products, present on both GPUs here). On the 7900 XT, llama.cpp's Vulkan backend beats its own HIP build. - HIP would be an AMD path (NVIDIA through HIP-on-CUDA, with the CUDA toolkit), with a second kernel source in C++; on this GPU it is slower than Vulkan.
- OKLM's OpenCL kernels are portable: every device and model test passes on Intel's driver (the first run on a non-AMD driver), at a third of Vulkan's decode speed there, its tuning being the Radeon's.
- NVIDIA: no device here, nor planned; tests can be run by someone else.
Decision
- Vulkan is OKLM's main GPU backend
(
--backend vulkan, andautoonce it reaches parity): one API on AMD, NVIDIA and Intel (and Android, and Apple through MoltenVK later), one kernel source. - Runtime loading, as OpenCL. The Vulkan loader is
opened with
dlopen(libvulkan.so.1,vulkan-1.dll); OKLM declares the types and entry points it uses, so no Vulkan SDK is needed to build the library or to run where Vulkan is absent. The host code is C17. - Kernels in GLSL, compiled to SPIR-V at build time
with
glslcand embedded in the binary (askernels.clis today); a build withoutglslcbuilds without the Vulkan backend. No compiler at run time. Specialization constants select the variants (types, tile sizes, the cooperative-matrix and integer-dot paths) where a device offers them. - The same op contract. The backend implements
oklm_backend_iface(buffers with fake addresses mapped back at dispatch, as ADR-0017;compute(op); the CPU fallback per op for what it does not take), so the forward passes, the scheduler and the tests do not change. Each op's results are checked against the CPU kernels, as the OpenCL ones are. - A decode step is recorded once and replayed. The model marks a step's start and end (a new backend entry, optional: other backends ignore it); the values that change from token to token (positions, KV lengths, cache rows, the sequences of a batch) are read by the kernels from small device buffers or push constants, not baked into the recorded dispatches. A step that cannot be replayed (an op on the host, a changed shape) is recorded again or run directly.
- OpenCL stays, as the fallback where Vulkan is absent or poor, and for the devices it already serves; it is not removed. The CPU stays the fallback per op for both.
- No HIP or CUDA backend. Revisited only if Vulkan falls far short of a vendor's native API on that vendor's hardware.
- Testing. AMD (RADV) and Intel (ANV) are tested here
on every change (
make testruns the Vulkan suite on each Vulkan GPU present). NVIDIA is covered by a self-contained script (tools/gpu/report.sh: build, the device tests, the public models against the CPU, a speed table, one report file) for an outside tester; until a report comes back, the documentation says NVIDIA is untested.
Consequences
- A second backend to maintain beside OpenCL, with its kernels in another language (GLSL); new ops are written for Vulkan first, and for OpenCL when a device needs them there.
- The build gains an optional dependency (
glslc, packaged asglslcor inshaderc); MinGW builds embed the same SPIR-V. - The step replay asks the forward passes to keep a decode step's shapes fixed and its varying values in device memory: kernels written for OpenCL with those values as arguments get them from a buffer instead.
- Speed targets per vendor come from llama.cpp's Vulkan backend on the same device, measured as above.