OKLMdocumentation v1.0.3
On this page

ROCm/HIP Training Backend — Implementation Plan

How to work through this plan

This file describes a multi-phase plan to add ROCm/HIP support to OKLM, a pure C framework.

  • Treat each phase as a discrete task. Do not skip phases.
  • Do not start the next phase until the current phase meets its acceptance criteria.
  • At every 🛑 Decision point, stop and wait for explicit user approval.
  • R0 (the audit) is mandatory and blocking. Do not write any new code until R0 is complete and the user has approved its findings.
  • After each phase, report using the format in Reporting format.

Status: R0 done (2026-11-04: rocm-design.md, the gap analysis's ROCm section); R1 done with rocBLAS in hipBLASLt's place (hipBLASLt measured no faster than OKLM's OpenCL product kernel; rocBLAS approved as the plan's fallback); R2 done (--backend rocm: the trainer's large products through rocBLAS on shared virtual memory, 1.26 to 1.43 times a step); R3 to R5 and R7 done (LoRA in the trainer on the CPU, opencl and rocm, adapter files in llama.cpp's format, oklm train --lora, oklm lora-merge, --lora on run / chat / ppl / serve: ADR-0020, rocm-design.md); R6 done (the baseline: LoRA 1.3 to 1.6 times a full step, Llama-3.2-1B trained with LoRA in 344 ms a step on rocm; rocBLAS's default solutions kept; no custom kernel needed: rocm-design.md). Every phase done; the 27B path (a quantized frozen model) split out.


Before starting (OKLM)

What this plan needs settled before its R0 can start, how its names map onto OKLM's, and what already exists. The plan was written for a generic pure-C framework with an OpenCL path; OKLM is that framework, and much of what the plan asks R0 to look for exists already.

Phases. The plan's phases are numbered R0 to R6 here, as the other specifications number theirs (S, P, G, C): R0 is the plan's Phase 0 (the audit), R1 to R6 its Phases 1 to 6. R7, added for OKLM, takes LoRA from validated GEMMs to the trainer, files and models end to end.

Names.

The plan says OKLM
AUDIT.md at the repository root the "ROCm" section of gap-analysis.md and a design note, docs/rocm-design.md, as for the other specifications; an ADR when R0 decides how HIP is loaded and where the backend lives
PERF.md the "Speed" section of training.md and benchmarks.md
test_hip_gemm.c, test_hip_lora.c tests under tests/train/ built by the Makefile with the other tests (oklm_test.h), which pass, saying so, on a machine without a HIP device
backend_hip.c where R0 decides: next to src/train/train_cl.c (a training-only device path, as the plan intends) or under src/core/hip/ beside the OpenCL backend
USE_HIP / USE_OPENCL OKLM has no USE_* switches: OpenCL is loaded at run time (src/core/opencl/cl_loader.c), so one binary runs with or without it, and the backend is chosen at run time (`--backend cpu
"the existing GEMM wrapper" src/train/gemm_train.c on the CPU, the tiled product kernel of src/train/train_kernels.cl on OpenCL
"CI or a pre-commit hook" make test (per commit), make asan-test / make tsan-test (daily)

Hardware, as measured on 2026-11-02 (rocminfo, hipcc --version): an AMD Radeon RX 7900 XT, gfx1100, with 20 GB of memory (20 955 136 KB), not the 7900 XTX's 24 GB the plan assumed. ROCm 7.2.4, HIP 6.4, hipBLASLt installed (/opt/rocm/include/hipblaslt/hipblaslt.h, /opt/rocm/lib/libhipblaslt.so). Host: i7-11700K, 128 GB.

What OKLM already does. R0 measures each part against the plan rather than assuming:

The plan asks for OKLM today Where
A backend abstraction oklm_backend_iface: buffers, upload / download, supports / compute of an op descriptor, synchronize, streams, memory query; CPU and OpenCL instances (ADR-0017) include/oklm/core/backend.h, src/core/backend.c, src/core/opencl/backend_cl.c
Device memory API the backend's buffers; the trainer's own device memory handed out as fake addresses, so its pointer arithmetic is the same on the CPU and the device src/train/train_cl.c, training.md
GPU training the whole trainer on OpenCL: parameters, gradients, AdamW moments and activations on the device, one launcher per kernel, deterministic (no atomics), within 1e-4 relative of the CPU src/train/train_cl.c, train_kernels.cl
GEMM on the device one tiled product kernel (64 x 64 outputs per work group, operands through strides, batched); on this gfx1100 the products are three quarters of a training step and bound by their loads training.md
CPU reference GEMM gemm_train.c: F32, scalar, AVX2 and AVX-512 giving the same bits src/train/gemm_train.c
Layout row-major throughout (hipBLASLt is column-major: the plan's operand swap applies) include/oklm/core/tensor.h
Training code forward and backward of GPT-2 and Llama-family blocks (Qwen's included), losses, AdamW, gradient clipping, checkpoints (ADR-0008), fine-tuning published models, reinforcement learning src/train/, training.md
LoRA / adapters none: oklm train fine-tunes every parameter
Numerical test harness oklm_test.h; every gradient against finite differences; device against CPU at TOL_TRAIN_DEVICE; GPT-2 fine-tuning against llm.c (a test-only oracle) tests/train/, adr/0015-test-references.md
Run-time loading of a GPU library cl_loader.c opens the OpenCL ICD loader at run time src/core/opencl/cl_loader.c
Speed to beat on this GPU through OpenCL: the 4.95M-parameter Llama at 94 000 tokens/s (16 x 256), SmolLM2-135M fine-tuning at 4 240 tokens/s, 17 to 20 times the CPU training.md

So the plan's premise changes: OKLM already trains on this GPU through OpenCL, with its own product kernel. What HIP brings is hipBLASLt's products (three quarters of a step today) and LoRA, which OKLM lacks on every backend. R0 states how much of each is worth it: R1's GEMM against the OpenCL kernel on the same shapes is the first number, and the plan goes on to R2 only if hipBLASLt is clearly faster or LoRA needs it.

Decisions R0 takes (with the design note):

  1. How HIP is linked. Loaded at run time like OpenCL (libamdhip64, libhipblaslt opened when present, so one oklm binary runs on machines without ROCm, and the MinGW build stays as it is), or linked behind a build option (make HIP=1). The pure-C constraint holds either way: the C HIP API and hipBLASLt's C header, the system compiler with __HIP_PLATFORM_AMD__.
  2. Where it sits. A third device of the trainer next to train_cl.c, using hipBLASLt for the products and the existing OpenCL kernels (or CPU) for the rest; or a HIP instance of oklm_backend_iface, which inference could use later. The plan says training only; the backend interface is the reusable seam either way.
  3. The kernels that are not products (norms, softmax, RoPE, AdamW, losses: about a quarter of a step). The plan forbids HIP device kernels in C, so either the data goes through the OpenCL kernels on the same GPU (two runtimes on one device: R0 checks that buffers can be shared or must be copied), or those steps stay on OpenCL and only the products move, or the plan's last decision point (a .hip unit with an extern "C" interface) is taken.
  4. LoRA's conventions. The plan's own A / B naming is the reverse of the paper's; OKLM has none yet, so it takes the paper's (A down to the rank, B up, B initialized to zero, scaling = alpha / rank) and the plan's formulas are rewritten in it at R3. Which projections get adapters (Q, K, V, O and the MLP), and how adapters are saved: llama.cpp's GGUF LoRA format (so llama.cpp reads them, and oklm run --lora can apply or merge them) or OKLM's checkpoint format.
  5. The 27B target (since 2026-11-04 Qwen3-8B, the 27B being too large for this GPU: roadmap.md, item 18 for the path). The default models first made an abliterated Qwen3.8-27B, made by OKLM from the official BF16 safetensors, the base for security training. In F16 it is 54 GB; in 20 GB it fits only frozen and quantized (Q4_K about 16 GB, QLoRA-style), with the products of the frozen weights dequantized by tiles before hipBLASLt (or done by OKLM's quantized OpenCL matrix kernels), activations checkpointed, and possibly layers kept in host memory (128 GB). R0 says whether this plan's F32 path reaches it or where it stops, and what R7 must add.

Owned elsewhere, not built twice. The trainer, its checkpoints and oklm train are training.md's; the OpenCL backend is ADR-0017's and is not modified (the plan's first rule); the default models and the security training runs are the roadmap's "Default models".


Project context

Item Value
Framework language Pure C (no C++, no PyTorch, no Python bindings)
Existing compute path OpenCL. Working, with benchmarks close to llama.cpp. Do not modify or replace without explicit approval.
New compute path ROCm/HIP, used only for training / LoRA fine-tuning, unless R0 shows existing code can be reused
Target GPU AMD Radeon RX 7900 XT (Navi 31, gfx1100), ROCm 7.2.4, HIP 6.4
Host i7-11700K, 128 GB DDR4-3200, 2 TB NVMe
GPU memory 20 GB (measured with rocminfo on 2026-11-02; the plan first assumed the 7900 XTX's 24 GB)
API constraint Host code uses the C-compatible HIP API (hip/hip_runtime.h, hipblaslt/hipblaslt.h). Only .c files.

Core principles

  1. Audit before adding. The default outcome of R0 may be "reuse existing code" rather than "write new HIP code."
  2. The HIP backend is additive. Add new code only where existing code cannot be extended.
  3. Correctness before performance. Every phase has a numerical validation gate.
  4. Use hipBLASLt for GEMM. Do not write custom GEMM kernels.
  5. Do not port ggml's HIP backend. It is inference-oriented and does not fit training.
  6. Do not touch Vulkan. It does not help training.
  7. Do not use HIPRTC in the first iteration.
  8. Prefer GEMM composition over fused kernels until correctness is proven. Fold scalar factors into GEMM alpha where possible so no custom kernels are needed.

Execution order

Phase Name Notes
R0 Audit existing framework Blocking. Read-only except its documents.
R1 Environment + minimal GEMM Its speed against the OpenCL product kernel decides whether R2 follows
R2 Backend abstraction layer Decision point before editing
R3 LoRA forward
R4 LoRA backward Highest risk
R5 Numerical validation harness
R6 Performance baseline
R7 LoRA end to end (OKLM) The trainer, adapter files, oklm train and oklm run

R0 — Audit existing framework (blocking, read-only)

Goal: Produce a precise map of what already exists, so later phases reuse, extend or replace rather than duplicate. The output is a written report, not code.

Absolute rule: Do not edit, create or delete any file except the audit's documents: the "ROCm" section of docs/gap-analysis.md and docs/rocm-design.md.

Tasks

  1. Inventory the repository layout. List top-level directories with a one-line role for each.
  2. Identify existing abstractions. Search for anything resembling:
    • a compute backend interface (function pointer tables, struct of ops, #ifdef USE_* switches, runtime dispatch);
    • memory management wrappers (device alloc / free / memcpy);
    • GEMM or matrix-op wrappers, even if OpenCL-only today;
    • kernel loading / compilation paths;
    • error handling and logging conventions.
  3. Map the OpenCL path in detail.
    • Which files implement OpenCL setup, allocation, kernel launch and teardown.
    • The calling convention: who allocates, who owns, which handles are passed.
    • Which parts are OpenCL-specific and which are generic.
  4. Map existing math primitives.
    • CPU reference implementations of GEMM, softmax, layer norm, etc.
    • Any tensor / op-graph abstraction.
    • Existing numerical test harnesses or golden files.
    • Memory layout convention for matrices (row-major vs column-major).
  5. Map existing training code.
    • Gradient code, optimizers, loss functions, backprop scaffolding.
    • Any LoRA or adapter code, including its naming convention for the A/B matrices.
  6. Map the build system. How backends are selected at build time, existing USE_* flags, how libraries are linked.
  7. Find leaks. List every place where an OpenCL-specific type or call appears in code that should be backend-agnostic.
  8. Measure the baseline. The OpenCL product kernel's time on the shapes of a training step (from OKLM_CL_PROFILE=1), to set against R1.
  9. Decide reuse for R2–R4. For each of backend abstraction, device memory API, GEMM wrapper and numerical test harness: reuse, extend or create new, with the reason.

Output: the "ROCm" section of docs/gap-analysis.md, and docs/rocm-design.md

The gap-analysis section holds the inventory with evidence; the design note holds these sections (as the other design notes):

  1. ## Repository layout — tree plus one-line descriptions.
  2. ## Existing abstractions — table: name | file | backend-specific? | extendable? | notes.
  3. ## OpenCL path breakdown — what is OpenCL-specific, what is generic.
  4. ## Existing math primitives — list and location.
  5. ## Existing training code — list and location.
  6. ## Build system — how backends are selected today.
  7. ## Reuse / extend / create decision table — for R2–R4 components, and the five decisions of "Before starting (OKLM)".
  8. ## Risks — things likely to bite during the HIP port.
  9. ## Recommended deviations from this plan — if the audit reveals a better path, say so explicitly.

Acceptance criteria

🛑 Decision point — stop and wait for approval

Present the audit summary and recommended deviations. The user decides:

  • which R2–R4 parts to reuse as-is;
  • which parts to extend, and the scope of the extension;
  • which parts to create new;
  • whether the phase order should change.

Do not start R1 until the user explicitly approves.

Known pitfalls

  • Do not assume the framework lacks an abstraction. Small C projects often have ad-hoc but working dispatch tables.
  • Do not assume the OpenCL path is monolithic. It may already be split into device / buffer / kernel / queue layers that HIP can implement.
  • Watch for code that looks generic but has OpenCL types in its headers. Extending those may be simpler than duplicating.

R1 — Environment verification and minimal GEMM

Goal: Prove that the HIP runtime and hipBLASLt can compile and run a GEMM on this machine, using whatever reusable primitives R0 identified, and measure it against the OpenCL product kernel on the same shapes.

Tasks

  1. Run hipcc --version and rocminfo. Confirm gfx1100 is listed and record total VRAM. If gfx1100 is missing, stop and report.

  2. Create the GEMM test under tests/train/ (built by the Makefile, passing with a message where no HIP device is found). If R0 found an existing GEMM wrapper or test harness, extend it instead.

    • Allocate input matrices A, B and output C with hipMalloc.
    • Run one GEMM with hipblasLtMatmul.
    • Copy the result back with hipMemcpy.
    • Compare against a CPU reference.
  3. Build it with make tests, as R0's decision 1 says (libraries loaded at run time, or a build option). For a first manual check:

    ROCM=$(hipconfig --rocm-path)
    hipcc -o test_hip_gemm test_hip_gemm.c \
      -I"$ROCM/include" \
      -L"$ROCM/lib" \
      -lhipblaslt -lamdhip64
  4. Time the GEMM on the shapes of a training step (R0's baseline) against the OpenCL product kernel.

Acceptance criteria

🛑 Decision point — go on or stop

If hipBLASLt is not clearly faster than the OpenCL product kernel on these shapes, and LoRA does not need HIP, stop and report: LoRA may then be added to the existing trainer (R7) without R2 to R6.

Known pitfalls

  • hipBLASLt needs an algorithm from hipblasLtMatmulAlgoGetHeuristic before hipblasLtMatmul. Cache it per shape; do not query on every matmul.
  • hipBLASLt is column-major. If the framework is row-major, either compute Cᵀ = Bᵀ·Aᵀ by swapping operands, or set layouts explicitly. Wrong lda / ldb / ldc gives silent garbage.
  • If the heuristic returns zero algorithms for gfx1100 in FP32, stop and report. Fallback options (hipBLAS / rocBLAS) need user approval.
  • When compiling with a plain C compiler instead of hipcc, define __HIP_PLATFORM_AMD__ before including HIP headers.

R2 — Backend abstraction layer

Goal: Let HIP coexist with OpenCL without changing existing OpenCL code. Scope is set by the R0 decision.

Tasks

  1. If R0 found an existing backend abstraction (it did: oklm_backend_iface, and the trainer's device launchers): implement the HIP side of that interface. Do not invent a parallel one.

  2. If R0 found no abstraction: create the minimal interface below, after the user approves the refactor scope.

    int hip_backend_init(int device_id);
    int hip_backend_alloc(void **ptr, size_t bytes);
    int hip_backend_free(void *ptr);
    int hip_backend_memcpy_h2d(void *dst, const void *src, size_t bytes);
    int hip_backend_memcpy_d2h(void *dst, const void *src, size_t bytes);
    int hip_backend_gemm_fp32(const float *A, const float *B, float *C,
                              int M, int N, int K,
                              int transA, int transB,
                              float alpha, float beta);
  3. Implement in backend_hip.c using the HIP runtime and hipBLASLt (or as the HIP side of the existing abstraction).

  4. Wire backend selection at run time, matching the convention found in R0 (--backend hip, the trainer's backend option, auto preferring it where it is faster).

🛑 Decision point — stop and ask before editing

  • Confirm the exact list of files that will change.
  • Confirm the OpenCL path will not be modified.

Acceptance criteria


R3 — LoRA forward (GEMM composition)

Goal: Implement the LoRA forward pass with two hipBLASLt GEMMs, reusing the R2 GEMM wrapper.

Math

Naming in this plan: B is the down-projection (in_dim × rank), A is the up-projection (rank × out_dim). This is the reverse of the LoRA paper's convention — match whatever the framework already uses (from R0) and keep it consistent. OKLM has no LoRA yet: R0's decision 4 takes the paper's naming, and the formulas below are rewritten in it when R3 starts.

h      = input @ B             # (batch × in_dim) @ (in_dim × rank)   -> (batch × rank)
output = (h @ A) * scaling     # (batch × rank)   @ (rank × out_dim)  -> (batch × out_dim)

Tasks

  1. Implement both GEMMs through the backend abstraction.
  2. Apply scaling via the second GEMM's alpha (no extra kernel).
  3. Omit bias and activation in this phase; validate GEMM numerics only.
  4. Keep h on the device; R4 needs it.

Acceptance criteria


R4 — LoRA backward (highest risk)

Goal: Implement LoRA gradients by GEMM composition. Correctness first.

Math

g          = grad_output @ Aᵀ                  # (batch × rank), reuse below
grad_A     = hᵀ @ grad_output   * scaling      # (rank × out_dim)
grad_B     = inputᵀ @ g         * scaling      # (in_dim × rank)
grad_input = g @ Bᵀ             * scaling      # (batch × in_dim)

scaling appears exactly once in each gradient. Apply it via GEMM alpha.

Tasks

  1. Implement each gradient using the backend GEMM. Multiple launches and intermediate buffers are fine.
  2. Compute g once and reuse it for grad_B and grad_input. Do not fold scaling into g itself, or it will be applied twice.
  3. Validate each gradient independently against a CPU reference.

Common causes of wrong numerics

  • Scaling applied twice, or to the wrong operand.
  • Transpose flags swapped (HIPBLAS_OP_T vs HIPBLAS_OP_N), especially combined with the column-major layout.
  • An intermediate buffer reused before a later GEMM finished with it.

Do not

  • Do not port ThemisDB's fused LoRA HIP kernels. They are C++ and tightly fused. Compose with GEMMs first.

Acceptance criteria


R5 — Numerical validation harness

Goal: A repeatable test guarding the HIP path against regressions. Reuse the R0 test harness if one exists (oklm_test.h, the trainer's finite-difference and device checks).

Tasks

  1. Create or extend the LoRA test under tests/train/.
  2. Fixed small problem: batch=4, in_dim=64, out_dim=32, rank=8, fixed seed.
  3. Compute forward output and all three gradients with a CPU FP32 reference.
  4. Run the same through the HIP path.
  5. Report max absolute error per tensor.

Acceptance criteria

For larger, realistic shapes, absolute tolerance becomes unreliable. If the harness is later extended to bigger sizes, add a relative-error check alongside it.


R6 — Performance baseline

Goal: Establish a baseline; only then consider optimization.

Tasks

  1. Use hipblaslt-bench to find the best algorithm for the target GEMM shapes. Cache or hardcode the result.
  2. Time one forward + backward on a realistic LoRA problem size.
  3. Compare against the existing OpenCL training path on the same step (the 4.95M Llama and SmolLM2-135M of training.md).
  4. Record all numbers in training.md and benchmarks.md.

Rules

  • Do not optimize before R5 passes.
  • Do not write custom fused kernels unless profiling shows a specific hotspot.

🛑 Decision point — custom kernels

Any device kernel (__global__) must be compiled as HIP C++, which conflicts with the pure-C constraint. If profiling justifies a custom kernel, stop and ask whether a separate .hip translation unit with an extern "C" interface is acceptable.


R7 — LoRA end to end (OKLM)

Goal: LoRA usable from oklm train and oklm run, not only as validated GEMMs.

Tasks

  1. The CPU reference in the trainer. Adapters on the chosen projections (R0's decision 4) in src/train/: frozen base weights, trainable A and B, their gradients checked against finite differences like every other gradient. This is the reference the HIP and OpenCL paths are held to.
  2. The devices. The HIP path of R3 to R5 wired into the trainer; LoRA on the OpenCL path too if R1 kept it as the default (the same kernels' products).
  3. Files. Adapters saved and loaded in the format R0 chose; oklm run --lora FILE [--lora-scale S] applies them at inference, and a merge writes a model with them folded in.
  4. oklm train --lora RANK [--lora-alpha A] [--lora-targets ...], checkpoints of the adapters only, resumable.
  5. The 27B path, as far as R0's decision 5 says it goes: a frozen quantized base, the memory measured.

Acceptance criteria


Hard constraints

  1. R0 is read-only apart from its documents. No source edits before approval.
  2. Do not port or link ggml's HIP backend.
  3. Do not write a custom GEMM. Use hipBLASLt or the existing GEMM wrapper.
  4. Do not touch the Vulkan path.
  5. Do not use HIPRTC.
  6. Do not modify files outside R0's documents and the files approved after R0 without asking first.
  7. No C++ constructs. .c files only. If a HIP header forces C++, stop and report.
  8. OKLM's own rules apply to every file: C17, Doxygen on every function and structure (make doc without warnings), make check clean, make test per commit, no ggml or llama.cpp code in the library (llama.cpp and llm.c are test-only references).

Reference: key HIP / hipBLASLt APIs

Purpose API
Initialize HIP hipInit(0)
Select device hipSetDevice(id)
Allocate device memory hipMalloc(&ptr, bytes)
Free device memory hipFree(ptr)
Host → device hipMemcpy(dst, src, bytes, hipMemcpyHostToDevice)
Device → host hipMemcpy(dst, src, bytes, hipMemcpyDeviceToHost)
Create BLAS handle hipblasLtCreate(&handle)
Create matrix layout hipblasLtMatrixLayoutCreate(&desc, HIP_R_32F, rows, cols, ld)
Create matmul descriptor hipblasLtMatmulDescCreate(&desc, HIPBLAS_COMPUTE_32F, HIP_R_32F)
Set transpose hipblasLtMatmulDescSetAttribute(desc, HIPBLASLT_MATMUL_DESC_TRANSA, ...)
Query algorithm hipblasLtMatmulAlgoGetHeuristic(...)
Execute GEMM hipblasLtMatmul(...)
Create stream hipStreamCreate(&stream)
Sync stream hipStreamSynchronize(stream)

Link flags:

-lhipblaslt -lamdhip64

Reporting format

After each phase, output only the following. Do not restate the plan.

RN status:      DONE | BLOCKED | NEEDS DECISION
Files changed:  <path> — <one-line description>   (one per line)
Validation:     R0: the gap-analysis section and docs/rocm-design.md
                R1–R2: PASSED / FAILED (R1: and its time against OpenCL)
                R3–R5: max abs error per tensor
                R7: the loss curves, CPU against GPU
Reuse decisions (R2+): which R0 findings were acted on
Blockers / questions: anything needing user input
Next action:    the single next concrete step