ROCm/HIP Training Backend — Implementation Plan
How to work through this plan
This file describes a multi-phase plan to add ROCm/HIP support to OKLM, a pure C framework.
- Treat each phase as a discrete task. Do not skip phases.
- Do not start the next phase until the current phase meets its acceptance criteria.
- At every 🛑 Decision point, stop and wait for explicit user approval.
- R0 (the audit) is mandatory and blocking. Do not write any new code until R0 is complete and the user has approved its findings.
- After each phase, report using the format in Reporting format.
Status: R0 done (2026-11-04: rocm-design.md, the gap analysis's ROCm
section); R1 done with rocBLAS in hipBLASLt's place (hipBLASLt measured
no faster than OKLM's OpenCL product kernel; rocBLAS approved as the
plan's fallback); R2 done (--backend rocm: the trainer's
large products through rocBLAS on shared virtual memory, 1.26 to 1.43
times a step); R3 to R5 and R7 done (LoRA in the trainer on the CPU,
opencl and rocm, adapter files in llama.cpp's
format, oklm train --lora, oklm lora-merge,
--lora on run / chat /
ppl / serve: ADR-0020, rocm-design.md); R6
done (the baseline: LoRA 1.3 to 1.6 times a full step, Llama-3.2-1B
trained with LoRA in 344 ms a step on rocm; rocBLAS's
default solutions kept; no custom kernel needed: rocm-design.md).
Every phase done; the 27B path (a quantized frozen model) split out.
Before starting (OKLM)
What this plan needs settled before its R0 can start, how its names map onto OKLM's, and what already exists. The plan was written for a generic pure-C framework with an OpenCL path; OKLM is that framework, and much of what the plan asks R0 to look for exists already.
Phases. The plan's phases are numbered R0 to R6 here, as the other specifications number theirs (S, P, G, C): R0 is the plan's Phase 0 (the audit), R1 to R6 its Phases 1 to 6. R7, added for OKLM, takes LoRA from validated GEMMs to the trainer, files and models end to end.
Names.
| The plan says | OKLM |
|---|---|
AUDIT.md at the repository root |
the "ROCm" section of gap-analysis.md and a design note,
docs/rocm-design.md, as for the other specifications; an
ADR when R0 decides how HIP is loaded and where the backend lives |
PERF.md |
the "Speed" section of training.md and benchmarks.md |
test_hip_gemm.c, test_hip_lora.c |
tests under tests/train/ built by the Makefile with the
other tests (oklm_test.h), which pass, saying so, on a
machine without a HIP device |
backend_hip.c |
where R0 decides: next to src/train/train_cl.c (a
training-only device path, as the plan intends) or under
src/core/hip/ beside the OpenCL backend |
USE_HIP / USE_OPENCL |
OKLM has no USE_* switches: OpenCL is loaded at run
time (src/core/opencl/cl_loader.c), so one binary runs with
or without it, and the backend is chosen at run time (`--backend
cpu |
| "the existing GEMM wrapper" | src/train/gemm_train.c on the CPU, the tiled product
kernel of src/train/train_kernels.cl on OpenCL |
| "CI or a pre-commit hook" | make test (per commit), make asan-test /
make tsan-test (daily) |
Hardware, as measured on 2026-11-02
(rocminfo, hipcc --version): an AMD Radeon RX
7900 XT, gfx1100, with 20 GB of memory (20 955 136 KB), not
the 7900 XTX's 24 GB the plan assumed. ROCm 7.2.4, HIP 6.4, hipBLASLt
installed (/opt/rocm/include/hipblaslt/hipblaslt.h,
/opt/rocm/lib/libhipblaslt.so). Host: i7-11700K, 128
GB.
What OKLM already does. R0 measures each part against the plan rather than assuming:
| The plan asks for | OKLM today | Where |
|---|---|---|
| A backend abstraction | oklm_backend_iface: buffers, upload / download,
supports / compute of an op descriptor,
synchronize, streams, memory query; CPU and OpenCL instances
(ADR-0017) |
include/oklm/core/backend.h, src/core/backend.c, src/core/opencl/backend_cl.c |
| Device memory API | the backend's buffers; the trainer's own device memory handed out as fake addresses, so its pointer arithmetic is the same on the CPU and the device | src/train/train_cl.c, training.md |
| GPU training | the whole trainer on OpenCL: parameters, gradients, AdamW moments and activations on the device, one launcher per kernel, deterministic (no atomics), within 1e-4 relative of the CPU | src/train/train_cl.c, train_kernels.cl |
| GEMM on the device | one tiled product kernel (64 x 64 outputs per work group, operands through strides, batched); on this gfx1100 the products are three quarters of a training step and bound by their loads | training.md |
| CPU reference GEMM | gemm_train.c: F32, scalar, AVX2 and AVX-512 giving the
same bits |
src/train/gemm_train.c |
| Layout | row-major throughout (hipBLASLt is column-major: the plan's operand swap applies) | include/oklm/core/tensor.h |
| Training code | forward and backward of GPT-2 and Llama-family blocks (Qwen's included), losses, AdamW, gradient clipping, checkpoints (ADR-0008), fine-tuning published models, reinforcement learning | src/train/, training.md |
| LoRA / adapters | none: oklm train fine-tunes every
parameter |
|
| Numerical test harness | oklm_test.h; every gradient against finite differences;
device against CPU at TOL_TRAIN_DEVICE; GPT-2 fine-tuning
against llm.c (a test-only oracle) |
tests/train/, adr/0015-test-references.md |
| Run-time loading of a GPU library | cl_loader.c opens the OpenCL ICD loader at run
time |
src/core/opencl/cl_loader.c |
| Speed to beat | on this GPU through OpenCL: the 4.95M-parameter Llama at 94 000 tokens/s (16 x 256), SmolLM2-135M fine-tuning at 4 240 tokens/s, 17 to 20 times the CPU | training.md |
So the plan's premise changes: OKLM already trains on this GPU through OpenCL, with its own product kernel. What HIP brings is hipBLASLt's products (three quarters of a step today) and LoRA, which OKLM lacks on every backend. R0 states how much of each is worth it: R1's GEMM against the OpenCL kernel on the same shapes is the first number, and the plan goes on to R2 only if hipBLASLt is clearly faster or LoRA needs it.
Decisions R0 takes (with the design note):
- How HIP is linked. Loaded at run time like OpenCL
(
libamdhip64,libhipblasltopened when present, so oneoklmbinary runs on machines without ROCm, and the MinGW build stays as it is), or linked behind a build option (make HIP=1). The pure-C constraint holds either way: the C HIP API and hipBLASLt's C header, the system compiler with__HIP_PLATFORM_AMD__. - Where it sits. A third device of the trainer next
to
train_cl.c, using hipBLASLt for the products and the existing OpenCL kernels (or CPU) for the rest; or a HIP instance ofoklm_backend_iface, which inference could use later. The plan says training only; the backend interface is the reusable seam either way. - The kernels that are not products (norms, softmax,
RoPE, AdamW, losses: about a quarter of a step). The plan forbids HIP
device kernels in C, so either the data goes through the OpenCL kernels
on the same GPU (two runtimes on one device: R0 checks that buffers can
be shared or must be copied), or those steps stay on OpenCL and only the
products move, or the plan's last decision point (a
.hipunit with anextern "C"interface) is taken. - LoRA's conventions. The plan's own A / B naming is
the reverse of the paper's; OKLM has none yet, so it takes the paper's
(A down to the rank, B up, B initialized to zero,
scaling = alpha / rank) and the plan's formulas are rewritten in it at R3. Which projections get adapters (Q, K, V, O and the MLP), and how adapters are saved: llama.cpp's GGUF LoRA format (sollama.cppreads them, andoklm run --loracan apply or merge them) or OKLM's checkpoint format. - The 27B target (since 2026-11-04 Qwen3-8B, the 27B being too large for this GPU: roadmap.md, item 18 for the path). The default models first made an abliterated Qwen3.8-27B, made by OKLM from the official BF16 safetensors, the base for security training. In F16 it is 54 GB; in 20 GB it fits only frozen and quantized (Q4_K about 16 GB, QLoRA-style), with the products of the frozen weights dequantized by tiles before hipBLASLt (or done by OKLM's quantized OpenCL matrix kernels), activations checkpointed, and possibly layers kept in host memory (128 GB). R0 says whether this plan's F32 path reaches it or where it stops, and what R7 must add.
Owned elsewhere, not built twice. The trainer, its
checkpoints and oklm train are training.md's; the OpenCL backend is ADR-0017's
and is not modified (the plan's first rule); the default models and the
security training runs are the roadmap's "Default models".
Project context
| Item | Value |
|---|---|
| Framework language | Pure C (no C++, no PyTorch, no Python bindings) |
| Existing compute path | OpenCL. Working, with benchmarks close to llama.cpp. Do not modify or replace without explicit approval. |
| New compute path | ROCm/HIP, used only for training / LoRA fine-tuning, unless R0 shows existing code can be reused |
| Target GPU | AMD Radeon RX 7900 XT (Navi 31, gfx1100), ROCm 7.2.4,
HIP 6.4 |
| Host | i7-11700K, 128 GB DDR4-3200, 2 TB NVMe |
| GPU memory | 20 GB (measured with rocminfo on 2026-11-02; the plan
first assumed the 7900 XTX's 24 GB) |
| API constraint | Host code uses the C-compatible HIP API
(hip/hip_runtime.h, hipblaslt/hipblaslt.h).
Only .c files. |
Core principles
- Audit before adding. The default outcome of R0 may be "reuse existing code" rather than "write new HIP code."
- The HIP backend is additive. Add new code only where existing code cannot be extended.
- Correctness before performance. Every phase has a numerical validation gate.
- Use hipBLASLt for GEMM. Do not write custom GEMM kernels.
- Do not port ggml's HIP backend. It is inference-oriented and does not fit training.
- Do not touch Vulkan. It does not help training.
- Do not use HIPRTC in the first iteration.
- Prefer GEMM composition over fused kernels until correctness is
proven. Fold scalar factors into GEMM
alphawhere possible so no custom kernels are needed.
Execution order
| Phase | Name | Notes |
|---|---|---|
| R0 | Audit existing framework | Blocking. Read-only except its documents. |
| R1 | Environment + minimal GEMM | Its speed against the OpenCL product kernel decides whether R2 follows |
| R2 | Backend abstraction layer | Decision point before editing |
| R3 | LoRA forward | |
| R4 | LoRA backward | Highest risk |
| R5 | Numerical validation harness | |
| R6 | Performance baseline | |
| R7 | LoRA end to end (OKLM) | The trainer, adapter files, oklm train and
oklm run |
R0 — Audit existing framework (blocking, read-only)
Goal: Produce a precise map of what already exists, so later phases reuse, extend or replace rather than duplicate. The output is a written report, not code.
Absolute rule: Do not edit, create or delete any
file except the audit's documents: the "ROCm" section of
docs/gap-analysis.md and
docs/rocm-design.md.
Tasks
- Inventory the repository layout. List top-level directories with a one-line role for each.
- Identify existing abstractions. Search for anything
resembling:
- a compute backend interface (function pointer tables,
structof ops,#ifdef USE_*switches, runtime dispatch); - memory management wrappers (device alloc / free / memcpy);
- GEMM or matrix-op wrappers, even if OpenCL-only today;
- kernel loading / compilation paths;
- error handling and logging conventions.
- a compute backend interface (function pointer tables,
- Map the OpenCL path in detail.
- Which files implement OpenCL setup, allocation, kernel launch and teardown.
- The calling convention: who allocates, who owns, which handles are passed.
- Which parts are OpenCL-specific and which are generic.
- Map existing math primitives.
- CPU reference implementations of GEMM, softmax, layer norm, etc.
- Any tensor / op-graph abstraction.
- Existing numerical test harnesses or golden files.
- Memory layout convention for matrices (row-major vs column-major).
- Map existing training code.
- Gradient code, optimizers, loss functions, backprop scaffolding.
- Any LoRA or adapter code, including its naming convention for the A/B matrices.
- Map the build system. How backends are selected at
build time, existing
USE_*flags, how libraries are linked. - Find leaks. List every place where an OpenCL-specific type or call appears in code that should be backend-agnostic.
- Measure the baseline. The OpenCL product kernel's
time on the shapes of a training step (from
OKLM_CL_PROFILE=1), to set against R1. - Decide reuse for R2–R4. For each of backend abstraction, device memory API, GEMM wrapper and numerical test harness: reuse, extend or create new, with the reason.
Output:
the "ROCm" section of docs/gap-analysis.md, and
docs/rocm-design.md
The gap-analysis section holds the inventory with evidence; the design note holds these sections (as the other design notes):
## Repository layout— tree plus one-line descriptions.## Existing abstractions— table: name | file | backend-specific? | extendable? | notes.## OpenCL path breakdown— what is OpenCL-specific, what is generic.## Existing math primitives— list and location.## Existing training code— list and location.## Build system— how backends are selected today.## Reuse / extend / create decision table— for R2–R4 components, and the five decisions of "Before starting (OKLM)".## Risks— things likely to bite during the HIP port.## Recommended deviations from this plan— if the audit reveals a better path, say so explicitly.
Acceptance criteria
🛑 Decision point — stop and wait for approval
Present the audit summary and recommended deviations. The user decides:
- which R2–R4 parts to reuse as-is;
- which parts to extend, and the scope of the extension;
- which parts to create new;
- whether the phase order should change.
Do not start R1 until the user explicitly approves.
Known pitfalls
- Do not assume the framework lacks an abstraction. Small C projects often have ad-hoc but working dispatch tables.
- Do not assume the OpenCL path is monolithic. It may already be split into device / buffer / kernel / queue layers that HIP can implement.
- Watch for code that looks generic but has OpenCL types in its headers. Extending those may be simpler than duplicating.
R1 — Environment verification and minimal GEMM
Goal: Prove that the HIP runtime and hipBLASLt can compile and run a GEMM on this machine, using whatever reusable primitives R0 identified, and measure it against the OpenCL product kernel on the same shapes.
Tasks
Run
hipcc --versionandrocminfo. Confirmgfx1100is listed and record total VRAM. Ifgfx1100is missing, stop and report.Create the GEMM test under
tests/train/(built by the Makefile, passing with a message where no HIP device is found). If R0 found an existing GEMM wrapper or test harness, extend it instead.- Allocate input matrices A, B and output C with
hipMalloc. - Run one GEMM with
hipblasLtMatmul. - Copy the result back with
hipMemcpy. - Compare against a CPU reference.
- Allocate input matrices A, B and output C with
Build it with
make tests, as R0's decision 1 says (libraries loaded at run time, or a build option). For a first manual check:ROCM=$(hipconfig --rocm-path) hipcc -o test_hip_gemm test_hip_gemm.c \ -I"$ROCM/include" \ -L"$ROCM/lib" \ -lhipblaslt -lamdhip64Time the GEMM on the shapes of a training step (R0's baseline) against the OpenCL product kernel.
Acceptance criteria
🛑 Decision point — go on or stop
If hipBLASLt is not clearly faster than the OpenCL product kernel on these shapes, and LoRA does not need HIP, stop and report: LoRA may then be added to the existing trainer (R7) without R2 to R6.
Known pitfalls
- hipBLASLt needs an algorithm from
hipblasLtMatmulAlgoGetHeuristicbeforehipblasLtMatmul. Cache it per shape; do not query on every matmul. - hipBLASLt is column-major. If the framework is
row-major, either compute
Cᵀ = Bᵀ·Aᵀby swapping operands, or set layouts explicitly. Wronglda/ldb/ldcgives silent garbage. - If the heuristic returns zero algorithms for
gfx1100in FP32, stop and report. Fallback options (hipBLAS / rocBLAS) need user approval. - When compiling with a plain C compiler instead of
hipcc, define__HIP_PLATFORM_AMD__before including HIP headers.
R2 — Backend abstraction layer
Goal: Let HIP coexist with OpenCL without changing existing OpenCL code. Scope is set by the R0 decision.
Tasks
If R0 found an existing backend abstraction (it did:
oklm_backend_iface, and the trainer's device launchers): implement the HIP side of that interface. Do not invent a parallel one.If R0 found no abstraction: create the minimal interface below, after the user approves the refactor scope.
int hip_backend_init(int device_id); int hip_backend_alloc(void **ptr, size_t bytes); int hip_backend_free(void *ptr); int hip_backend_memcpy_h2d(void *dst, const void *src, size_t bytes); int hip_backend_memcpy_d2h(void *dst, const void *src, size_t bytes); int hip_backend_gemm_fp32(const float *A, const float *B, float *C, int M, int N, int K, int transA, int transB, float alpha, float beta);Implement in
backend_hip.cusing the HIP runtime and hipBLASLt (or as the HIP side of the existing abstraction).Wire backend selection at run time, matching the convention found in R0 (
--backend hip, the trainer'sbackendoption,autopreferring it where it is faster).
🛑 Decision point — stop and ask before editing
- Confirm the exact list of files that will change.
- Confirm the OpenCL path will not be modified.
Acceptance criteria
R3 — LoRA forward (GEMM composition)
Goal: Implement the LoRA forward pass with two hipBLASLt GEMMs, reusing the R2 GEMM wrapper.
Math
Naming in this plan: B is the down-projection
(in_dim × rank), A is the up-projection
(rank × out_dim). This is the reverse of the LoRA paper's
convention — match whatever the framework already uses (from R0)
and keep it consistent. OKLM has no LoRA yet: R0's decision 4
takes the paper's naming, and the formulas below are rewritten in it
when R3 starts.
h = input @ B # (batch × in_dim) @ (in_dim × rank) -> (batch × rank)
output = (h @ A) * scaling # (batch × rank) @ (rank × out_dim) -> (batch × out_dim)
Tasks
- Implement both GEMMs through the backend abstraction.
- Apply
scalingvia the second GEMM'salpha(no extra kernel). - Omit bias and activation in this phase; validate GEMM numerics only.
- Keep
hon the device; R4 needs it.
Acceptance criteria
R4 — LoRA backward (highest risk)
Goal: Implement LoRA gradients by GEMM composition. Correctness first.
Math
g = grad_output @ Aᵀ # (batch × rank), reuse below
grad_A = hᵀ @ grad_output * scaling # (rank × out_dim)
grad_B = inputᵀ @ g * scaling # (in_dim × rank)
grad_input = g @ Bᵀ * scaling # (batch × in_dim)
scaling appears exactly once in each gradient. Apply it
via GEMM alpha.
Tasks
- Implement each gradient using the backend GEMM. Multiple launches and intermediate buffers are fine.
- Compute
gonce and reuse it forgrad_Bandgrad_input. Do not foldscalingintogitself, or it will be applied twice. - Validate each gradient independently against a CPU reference.
Common causes of wrong numerics
- Scaling applied twice, or to the wrong operand.
- Transpose flags swapped (
HIPBLAS_OP_TvsHIPBLAS_OP_N), especially combined with the column-major layout. - An intermediate buffer reused before a later GEMM finished with it.
Do not
- Do not port ThemisDB's fused LoRA HIP kernels. They are C++ and tightly fused. Compose with GEMMs first.
Acceptance criteria
R5 — Numerical validation harness
Goal: A repeatable test guarding the HIP path
against regressions. Reuse the R0 test harness if one exists
(oklm_test.h, the trainer's finite-difference and device
checks).
Tasks
- Create or extend the LoRA test under
tests/train/. - Fixed small problem:
batch=4, in_dim=64, out_dim=32, rank=8, fixed seed. - Compute forward output and all three gradients with a CPU FP32 reference.
- Run the same through the HIP path.
- Report max absolute error per tensor.
Acceptance criteria
For larger, realistic shapes, absolute tolerance becomes unreliable. If the harness is later extended to bigger sizes, add a relative-error check alongside it.
R6 — Performance baseline
Goal: Establish a baseline; only then consider optimization.
Tasks
- Use
hipblaslt-benchto find the best algorithm for the target GEMM shapes. Cache or hardcode the result. - Time one forward + backward on a realistic LoRA problem size.
- Compare against the existing OpenCL training path on the same step (the 4.95M Llama and SmolLM2-135M of training.md).
- Record all numbers in training.md and benchmarks.md.
Rules
- Do not optimize before R5 passes.
- Do not write custom fused kernels unless profiling shows a specific hotspot.
🛑 Decision point — custom kernels
Any device kernel (__global__) must be compiled as HIP
C++, which conflicts with the pure-C constraint. If profiling justifies
a custom kernel, stop and ask whether a separate .hip
translation unit with an extern "C" interface is
acceptable.
R7 — LoRA end to end (OKLM)
Goal: LoRA usable from oklm train and
oklm run, not only as validated GEMMs.
Tasks
- The CPU reference in the trainer. Adapters on the
chosen projections (R0's decision 4) in
src/train/: frozen base weights, trainable A and B, their gradients checked against finite differences like every other gradient. This is the reference the HIP and OpenCL paths are held to. - The devices. The HIP path of R3 to R5 wired into the trainer; LoRA on the OpenCL path too if R1 kept it as the default (the same kernels' products).
- Files. Adapters saved and loaded in the format R0
chose;
oklm run --lora FILE [--lora-scale S]applies them at inference, and a merge writes a model with them folded in. oklm train --lora RANK [--lora-alpha A] [--lora-targets ...], checkpoints of the adapters only, resumable.- The 27B path, as far as R0's decision 5 says it goes: a frozen quantized base, the memory measured.
Acceptance criteria
Hard constraints
- R0 is read-only apart from its documents. No source edits before approval.
- Do not port or link ggml's HIP backend.
- Do not write a custom GEMM. Use hipBLASLt or the existing GEMM wrapper.
- Do not touch the Vulkan path.
- Do not use HIPRTC.
- Do not modify files outside R0's documents and the files approved after R0 without asking first.
- No C++ constructs.
.cfiles only. If a HIP header forces C++, stop and report. - OKLM's own rules apply to every file: C17, Doxygen on every function
and structure (
make docwithout warnings),make checkclean,make testper commit, no ggml or llama.cpp code in the library (llama.cpp and llm.c are test-only references).
Reference: key HIP / hipBLASLt APIs
| Purpose | API |
|---|---|
| Initialize HIP | hipInit(0) |
| Select device | hipSetDevice(id) |
| Allocate device memory | hipMalloc(&ptr, bytes) |
| Free device memory | hipFree(ptr) |
| Host → device | hipMemcpy(dst, src, bytes, hipMemcpyHostToDevice) |
| Device → host | hipMemcpy(dst, src, bytes, hipMemcpyDeviceToHost) |
| Create BLAS handle | hipblasLtCreate(&handle) |
| Create matrix layout | hipblasLtMatrixLayoutCreate(&desc, HIP_R_32F, rows, cols, ld) |
| Create matmul descriptor | hipblasLtMatmulDescCreate(&desc, HIPBLAS_COMPUTE_32F, HIP_R_32F) |
| Set transpose | hipblasLtMatmulDescSetAttribute(desc, HIPBLASLT_MATMUL_DESC_TRANSA, ...) |
| Query algorithm | hipblasLtMatmulAlgoGetHeuristic(...) |
| Execute GEMM | hipblasLtMatmul(...) |
| Create stream | hipStreamCreate(&stream) |
| Sync stream | hipStreamSynchronize(stream) |
Link flags:
-lhipblaslt -lamdhip64Reporting format
After each phase, output only the following. Do not restate the plan.
RN status: DONE | BLOCKED | NEEDS DECISION
Files changed: <path> — <one-line description> (one per line)
Validation: R0: the gap-analysis section and docs/rocm-design.md
R1–R2: PASSED / FAILED (R1: and its time against OpenCL)
R3–R5: max abs error per tensor
R7: the loss curves, CPU against GPU
Reuse decisions (R2+): which R0 findings were acted on
Blockers / questions: anything needing user input
Next action: the single next concrete step