ROCm design
The audit (R0) of rocm_spec.md: what OKLM has for a ROCm/HIP training path and LoRA, what each later phase reuses, extends or creates, and the five decisions the specification leaves to R0. The inventory with its evidence is the "ROCm" section of gap-analysis.md.
Status: audit approved (2026-11-04) with its recommendations; R1 done
with rocBLAS in hipBLASLt's place (approved 2026-11-04); R2 done
(--backend rocm); LoRA next.
Repository layout
| Directory | Role |
|---|---|
apps/ |
the oklm command (oklm.c) and the GUI |
include/oklm/ |
public headers, one directory per module (core, io, tok, model, mm, harness, server, client, train) |
src/core/ |
tensors, arenas, dtypes and quantization, the op dispatcher
(ops.c, backend.c), the CPU backend
(cpu/), the OpenCL backend (opencl/: run-time
loader, backend, inference kernels) |
src/io/ |
GGUF and safetensors reading and writing, conversion, quantization |
src/tok/ |
tokenizers (BPE, Unigram, WordPiece) and their trainer |
src/model/ |
inference: architectures, contexts, the paged cache, sampling, grammars, speculative decoding |
src/mm/ |
images, sounds, video and documents |
src/train/ |
training: CPU kernels (layers_train.c,
gemm_train.c), the OpenCL device (train_cl.c,
train_kernels.cl), the trainer (train.c),
checkpoints, reinforcement learning |
src/harness/, src/server/,
src/client/ |
the agent, the HTTP server, the client |
tests/ |
one program per module (oklm_test.h), tolerances in
tests/tolerances.h, fixtures; benches |
tools/ |
reference generators (llama.cpp, llm.c, Hugging Face), benches |
docs/ |
documentation, ADRs, specifications, design notes |
Existing abstractions
| Name | File | Backend-specific? | Extendable? | Notes |
|---|---|---|---|---|
oklm_backend_iface |
include/oklm/core/backend.h:51-84 | no | yes | buffers, upload / download, supports /
compute of an op descriptor, synchronize, streams, memory
and free memory; used by inference (src/core/ops.c),
not by the trainer |
| The trainer's device seam | src/train/train_internal.h:560-759 | OpenCL by name (tr_cl_*) |
yes | one launcher per CPU kernel tr_*, same arguments;
train.c picks one or the other per call
(k_matmul_forward, train.c:1191-1196) |
| Device memory as fake addresses | src/train/train_cl.c:95-112 (tr_cl_buf),
tr_cl_alloc |
OpenCL | yes | pointers never dereferenced on the host, so the trainer's arithmetic is the same on both sides; a launcher maps a pointer to (buffer, offset) |
| The device product | src/train/train_cl.c:903 (gemm), train_kernels.cl:167
(tr_gemm) |
OpenCL | yes | strides on both operands (sai, sak,
sbk, sbj), accumulate flag, batched
(gemm_batch, train_cl.c:623-630), the attention's batches
mapping query heads to KV heads (pairs,
train_cl.c:1105) |
| The CPU product | src/train/gemm_train.c:233 (tr_gemm) |
no | reference | F32, multiply then add, scalar / AVX2 / AVX-512 giving the same bits |
| Run-time library loading | src/core/opencl/cl_loader.c:31-35, 68 (oklm_cl_load),
cl_loader.h |
OpenCL | the model to copy | dlopen / LoadLibrary, its own declarations
of the API's types and functions (no system header) |
| Shared OpenCL handles | src/core/opencl/backend_cl.h:19-43 (oklm_cl_handles,
oklm_opencl_handles) |
OpenCL | as is | the trainer borrows the inference backend's device, context and queue |
| Errors and logging | OKLM_ERR_* (include/oklm/oklm.h), n_log
(nilorea) |
no | as is | the device seam keeps its first error and returns it from
tr_cl_finish() |
| Test harness | tests/oklm_test.h, tests/tolerances.h:87
(TOL_TRAIN_DEVICE 1e-4) |
no | as is | finite-difference gradients, device against CPU, llm.c as a test-only oracle |
OpenCL path breakdown
- Generic: the trainer's forward and backward
(
train.c), its parameter layout and fake-address arithmetic, the CPU kernels, the checkpoints, the test harness. - OpenCL-specific, isolated:
src/core/opencl/(inference) andsrc/train/train_cl.cwithtrain_kernels.cl(training). No OpenCL type or call appears anywhere else (a search forcl_mem,cl_kernel,cl_int,clEnqueue,cl_context,cl_command_queueoutside those two places finds nothing). The trainer chooses its device through one pointer,tr->cl(train.c:654-662). - Ownership: the trainer owns its device side
(
tr_cl_new/tr_cl_free), which owns its buffers and kernels; the OpenCL context and queue belong to the inference backend instance and are borrowed. - The products: every product of a training step, the
attention's included, goes through
gemm()in train_cl.c: one function to route elsewhere.
Existing math primitives
- Products:
tr_gemm(CPU, gemm_train.c:233; device, train_kernels.cl:167). - Norms (LayerNorm, RMSNorm, per-head RMSNorm), softmax with cross-entropy, attention (causal, grouped KV), GELU, SwiGLU, RoPE (with frequency factors), dropout, AdamW, gradient norm: CPU in layers_train.c, device in train_kernels.cl, declared side by side in train_internal.h.
- Inference ops (
include/oklm/core/ops.h) with CPU and OpenCL kernels, quantized matrix kernels for every GGUF type on both. - Layout: row-major throughout (include/oklm/core/tensor.h); a weight
is OC rows of C inputs (train_internal.h:16-17). hipBLASLt is
column-major: a row-major
C = A Bis itsC^T = B^T A^T, the operands swapped.
Existing training code
- The trainer: GPT-2 and Llama-family blocks (Qwen2 biases, Qwen3 head norms, grouped KV), losses, AdamW, gradient clipping and accumulation, learning-rate schedules, dropout, evaluation (src/train/train.c, layers_train.c; training.md).
- Checkpoints with optimizer state and resume (src/train/checkpoint.c, ADR-0008); fine-tuning from Hugging Face checkpoints; reinforcement learning (src/train/rl.c).
- On OpenCL: the whole step on the device, deterministic (no atomics), within 1e-4 relative of the CPU (tests/train/test_train_cl.c).
- LoRA or adapters: none. No A / B naming to match.
(Added since by R3 to R5:
oklm train --lora, training.md.) - Activation recompute: none: every activation of the
step is kept (train.c:538-574 sizes them). (Added since, on 2026-11-04:
oklm train --recompute, training.md.)
Build system
- No
USE_*switches. OpenCL is loaded at run time bycl_loader.c(Makefile:239), so one binary runs with or without it; the backend is chosen at run time (--backend, the trainer'sbackendoption, include/oklm/train/train.h:71). - Optional dependencies found by
pkg-config(Allegro, OpenSSL, Ogg);-ldllinked (Makefile:159); kernels embedded as generated includes (Makefile:525-530). - MinGW builds the same sources for Windows; ROCm has no part there.
Baseline (task 8)
OKLM_CL_PROFILE=1 (each launch waited for and timed),
gfx1100, 2026-11-04:
| Run | Step | Products (tr_gemm) |
Their share | Products' rate |
|---|---|---|---|---|
| Llama Nano, 4.95M, 16 x 256 | 47 ms (profiled) | 3052 launches in 20 steps, 38.6 ms a step | 69 % | about 3.6 TFLOP/s |
| SmolLM2-135M fine-tune, 4 x 4 x 128 | 528 ms | 18808 launches in 6 steps, 429 ms a step | 69.5 % | about 3.8 TFLOP/s (6 x 2048 x 134.5M FLOP a step) |
About 150 to 800 products a micro-batch, many small (the attention's per-head batches). R1 measures hipBLASLt on the same shapes; a product must then win more than what handing it over costs (below, decision 3).
Reuse / extend / create decision table
| Component | Decision | Reason |
|---|---|---|
| Backend abstraction | reuse the trainer's device seam, not
oklm_backend_iface |
the trainer never uses the backend interface; its seam already pairs
every CPU kernel with a device launcher, and the products have one
entry, gemm() |
| Device memory API | reuse the fake-address allocator; extend it to shared virtual memory only if R1 shows HIP can read it | the trainer's arithmetic depends on it; a second allocator would split parameters and activations between two runtimes |
| GEMM wrapper | extend gemm() in train_cl.c: hipBLASLt
for the products it wins, the OpenCL kernel for the rest (the
attention's head-mapped batches, small products) |
one function, strides already explicit; hipBLASLt's strided batches cannot express the query-to-KV head mapping |
| Numerical test harness | reuse oklm_test.h,
TOL_TRAIN_DEVICE, the device-against-CPU pattern of
test_train_cl.c |
it already holds devices to the CPU |
| LoRA (R3, R4, R7) | create, in the trainer first (CPU reference, then the OpenCL device), independent of HIP | OKLM has none; the plan's GEMM composition is plain products that both existing devices run |
The five decisions of "Before starting (OKLM)":
- How HIP is linked: loaded at run time.
hipblaslt.hcannot be compiled as C: it includes<memory>,<regex>and<vector>unconditionally (/opt/rocm/include/hipblaslt/hipblaslt.h:54-56), which the plan's constraint 7 says to report. Its functions are exported as C symbols (nm -D libhipblaslt.so:hipblasLtCreate,hipblasLtMatmul,hipblasLtMatmulAlgoGetHeuristic, ... unmangled), so a loader likecl_loader.cwith OKLM's own declarations of the few functions, opaque handles and enum values needed reaches them without the header.hip_runtime_api.hcompiles as C with__HIP_PLATFORM_AMD__, but the loader declares its handful of HIP functions too, so no ROCm header is needed to build and one binary still runs without ROCm (and MinGW is untouched). The enum values are checked numerically by R1's test.rocblas.hdoes compile as C, a fallback needing your approval (the plan's R1 pitfall). - Where it sits: inside the trainer's OpenCL device,
as the engine of its products, not a new
oklm_backend_ifaceinstance. A HIP backend for inference would need every inference op, which HIP cannot provide in C (no device kernels without HIP C++, no HIPRTC). - The other kernels stay on OpenCL, and only products
move. That holds only if the two runtimes share memory: ROCm's OpenCL
and HIP sit on the same HSA runtime, so a buffer allocated as OpenCL
shared virtual memory (
clSVMAlloc) may be a pointer hipBLASLt can read on the same device; R1 tests it first. If it is not, each product would copy its operands both ways, and the plan stops at R1 (LoRA through R7 on the existing devices). Each hand-over also needs the OpenCL queue drained before the HIP stream starts and the reverse (no shared events): only products large enough to pay for two waits move. - LoRA's conventions: the paper's naming (A down to
the rank, B up, B zero at start,
scaling = alpha / rank). Targets chosen by name (q, k, v, o, gate, up, down; default q, k, v, o). Files in llama.cpp's GGUF adapter format, so llama.cpp reads them too:general.type = adapter,adapter.type = lora,adapter.lora.alpha, tensors<base name>.lora_aand.lora_b(build/llama.cpp/src/llama-adapter.cpp:205-207, 276-284, 374 for the checks it makes); the optimizer state for resuming in OKLM's checkpoint sidecar. - The 27B target is out of this plan's reach. The
trainer keeps every weight, gradient, moment and activation in F32: 27B
is 108 GB of weights alone. A frozen base would have to stay quantized
(Q4_K about 16 GB of the 20), its products dequantized by tiles, the
backward pass through it a transposed quantized product OKLM does not
have (
dX = dY Wwith W quantized), activations recomputed (none today) and possibly layers kept in host memory. R7 should stop at a frozen F32 or F16 base (SmolLM2-135M, Qwen3-0.6B, up to about 4B); the 27B path is a milestone of its own.
Risks
- Hand-over cost. Products are many and small (150 to 800 a micro-batch); waiting on the OpenCL queue and the HIP stream for each (tens of microseconds each) can cost more than hipBLASLt saves on small shapes. Mitigation: route only large products, measured in R1.
- Memory sharing between the runtimes is not
documented by AMD; if
clSVMAllocmemory is not a valid HIP device pointer, decision 3 fails and R2 to R6 do not follow. - Our own declarations of hipBLASLt's enums drift
with ROCm releases: the loader checks
hipblasLtGetVersionagainst the version it was written for and refuses others; R1's test catches a wrong value as wrong numbers. - Determinism. The trainer gives the same result on every run; hipBLASLt's algorithms may split K with atomic adds. The heuristic's choice must be restricted (or measured) to keep runs reproducible.
- FP32 algorithms on gfx1100: the heuristic may return none for some shapes or transposes (the plan's R1 pitfall); the OpenCL kernel stays the fallback for those.
- Column-major operand swaps and leading dimensions:
silent garbage when wrong; R1's test compares every transpose
combination the trainer uses against
tr_gemm. - Numerics: hipBLASLt fuses multiply and add where
OKLM's kernels do not; results differ from the CPU in the last bits,
within
TOL_TRAIN_DEVICE.
Recommended deviations from this plan
- R1 first tests memory sharing, then times hipBLASLt
against the OpenCL kernel on the baseline's shapes, the hand-over cost
included. Only if both pass (sharing works, products clearly faster
after the waits) do R2 to R6 follow, as the routing inside
gemm()above rather than a new backend. - LoRA does not wait for HIP. R3 and R4's GEMM
compositions are products both existing devices run; R7 (the CPU
reference, then the OpenCL device, files,
oklm train --lora,oklm run --lora) can start right after R1 whatever R1 decides, and R3 to R5's checks become the trainer's own gradient tests. - The 27B path is split out of R7 into a milestone of its own (frozen quantized base, transposed quantized products, activation recompute, host offload), after LoRA works on small models.
- No HIP header in the build (decision 1): a run-time loader, so the C-only rule holds without a C++ header and one binary serves machines with and without ROCm.
R1: measurements
Run outside the build (throwaway programs, 2026-11-04), as R1's gate before any code: hipBLASLt reached through OKLM's own declarations (the header being C++), the trainer's OpenCL launchers called through its internal header, rocBLAS for comparison only.
Declarations and numerics: hipBLASLt 1.2.2 (
hipblasLtGetVersion100202) through our own C declarations gives every transpose the trainer uses within 7e-6 of a CPU product (77 x 53 x 129).Memory sharing works: a buffer from
clSVMAllocis a device pointer to HIP (hipPointerGetAttributes: type device, device 0), and a hipBLASLt product on such pointers is correct (1e-5); an idle hand-over (clFinishthenhipStreamSynchronize) costs 2.2 microseconds.hipBLASLt is not faster than the OpenCL kernel. Its heuristic offers 4 algorithms for FP32 on gfx1100; the best of them reaches 4.4 to 7.4 TFLOP/s on the trainer's shapes, about the OpenCL kernel's speed and sometimes below it (the Nano's output projection, 4096 x 2048 x 256: 790 microseconds against 419).
rocBLAS is 1.5 to 3.8 times faster (
rocblas_sgemm, its header compiles as C), the trainer's products as it issues them, microseconds:Layer (rows x inputs x outputs) Forward, OpenCL / rocBLAS Backward (input and weight gradients), OpenCL / rocBLAS SmolLM2 Q/K/V, 512 x 576 x 960 179 / 95 343 / 212 SmolLM2 MLP up, 512 x 576 x 3072 393 / 103 745 / 223 SmolLM2 MLP down, 512 x 1536 x 576 178 / 117 264 / 120 SmolLM2 output, 512 x 576 x 49152 2971 / 1142 10393 / 6215 Nano Q/K/V, 4096 x 256 x 512 123 / 58 647 / 283 Nano MLP up, 4096 x 256 x 1408 310 / 130 890 / 379 Nano output, 4096 x 256 x 2048 419 / 180 1189 / 473 For SmolLM2-135M this is about 305 ms of products a step against 135 ms (the attention's per-head products staying on OpenCL), so a step of 528 ms near 360 (about 1.45 times), before the hand-overs.
By R1's rule hipBLASLt stopped there; rocBLAS, the plan's fallback, was approved in its place (2026-11-04) and takes it everywhere in the design above: run-time loading with our own declarations, products routed from the trainer's OpenCL device on shared memory, the rest unchanged.
In the tree (R1's deliverables):
src/train/hip_loader.c: the HIP runtime and rocBLAS opened at run time (OKLM_HIP_LIB,OKLM_ROCBLAS_LIBto name them, empty to disable; nothing on Windows, nor in an ASan/TSan build unless named), the entry points and values declared there.src/train/gemm_rocblas.c:tr_rb_gemm()takestr_gemm()'s operands (row-major, either operand read transposed through its strides) and callsrocblas_sgemmwith the operands swapped; HIP deviceOKLM_HIP_DEVICE(0).tests/train/test_rocblas.c: every operand layout, accumulation, padded rows and sizes of 1 againsttr_gemm()within 7e-6 (PASSED); skipped without ROCm.tests/train/bench_rocblas.c: the table above through the loader (1.7 to 3.5 times: SmolLM2's Q/K/V 1.7, MLP up 3.5, MLP down 1.8, output 1.7; the Nano's 2.2 to 2.5).
R2: the trainer's products through rocBLAS
--backend rocm (the trainer's backend
option): the first OpenCL device with its large single products through
rocBLAS. The files R2 changed, as approved:
src/core/opencl/cl_loader.c (clSVMAlloc,
clSVMFree resolved when present; the inference backend
untouched), src/train/train_cl.c,
src/train/train.c, include/oklm/train/train.h,
apps/oklm.c, tests/train/test_train_cl.c.
- Memory: each trainer buffer is shared virtual
memory wrapped in a buffer (
CL_MEM_USE_HOST_PTR), so every OpenCL launcher is unchanged andgemm()hands rocBLAS the shared address of the same bytes (checked first outside the tree: OpenCL writes, rocBLAS computes, OpenCL reads the product back within 2e-6). Coarse-grained SVM is device memory committed when allocated, where theopenclbackend's plain buffers may overflow into host memory; once the device is full (or pastOKLM_ROCM_DEVICE_MB), a buffer is allocated as fine-grained SVM, which this driver places in host memory the device and rocBLAS read over the bus (2026-11-08: Qwen3-8B LoRA at 6144 tokens, the activations 17.1 GB then the 5.0 GB packed base, which went to the host; see training.md). - Routing: a single product of at least 2^22
multiply-adds (
OKLM_ROCBLAS_MIN) goes to rocBLAS; the attention's batched products, whose batches map query heads to KV heads, stay on the OpenCL kernel. - Hand-overs: switching from OpenCL to rocBLAS drains the OpenCL queue, and back the HIP stream, only when the side left has work queued; consecutive products do not wait for each other.
- Checks: every test of
tests/train/test_train_cl.cruns again onrocmwith every single product through rocBLAS: the forward and backward pass against the CPU (logits and gradients within 1e-6), five AdamW steps (within 2.3e-5), the same bits on two runs, dropout, weighted rows, resuming on the CPU. - Speed: the Llama Nano 38 to 26.5 ms a step, SmolLM2-135M 390 to 310 ms (training.md); the threshold makes no difference on these models (every single product is large).
16-bit product inputs (2026-11-08)
--matmul-precision bf16|f16 (matmul_type):
tr_to16 rounds both operands of a product rocBLAS computes
(at least 128 in every size) into one scratch buffer, kept in their
layouts so the transpose flags and leading dimensions are those of the
F32 call (the kernel walks a transposed operand along its contiguous
dimension); rocblas_gemm_ex then writes a 16-bit product
with F32 accumulation, which tr_from16 widens into, or adds
to, the F32 output. rocBLAS 5 on gfx1100 has tuned kernels only for
16-bit outputs: bf16_r inputs with an f32_r
output measured 1.5 to 2 times slower than sgemm and 12 to
20 times with the second operand transposed, a bf16_r
output 2.5 to 4 times faster than sgemm (shapes of
SmolLM2-135M's layers and head and 4096 x 4096). A BF16 base under
--quantized-base is read in place:
tr_cl_dequantize() records the BF16 rows it unpacked as
they lie, and an operand within them points rocBLAS at the rows
themselves. Measured in training.md:
BF16 follows F32's validation curve to the third decimal, F16 alone does
not, Llama-3.2-1B's LoRA step 350 to 227 ms. With
--loss-scale dynamic (loss_scale: the
gradients of the loss times a scale, halved on an overflow with the
update skipped, doubled after 200 finite updates) F16 follows F32 too
(2026-11-08).
The weights' 16-bit copy (2026-11-08):
tr_cl_set_weights16() names the parameter buffer, whose
rounding train_cl.c keeps in a second buffer of 2 bytes a
parameter, element for element, with a flag per 65 536 floats. An
operand of gemm16() within the parameters points rocBLAS at
the copy, after tr_to16 has rounded the stale chunks it
spans; tr_adamw and tr_adamw16 write the
rounding of each parameter they update (w16_store, the
rounding tr_to16 does) and mark the chunks they cover whole
current; an upload marks its chunks stale. Each weight is thus rounded
once per update instead of at each of its products (forward, input
gradient, recomputed forward, each accumulated batch, evaluations), and
a LoRA run's frozen base once for the run. The copy is allocated on the
first 16-bit product (after the activations), in coarse-grained memory
only and when 512 MiB stay free after it
(TR_CL_W16_RESERVE), else not at all;
OKLM_ROCM_W16_CACHE=0 turns it off. Under
--quantized-base the frozen matrices are not in the
parameter buffer: each is unpacked and rounded before its product as
before (a BF16 base still read in place), and no copy is made. The
scratch also remembers the left operand it holds rounded (address and
view): the next product reading the same elements, as a weight's
gradient reads the output gradient its input gradient's product has just
rounded, skips the rounding; any other kernel, an upload, an F32 product
or a 16-bit output over those elements forgets it. The bits are those of
rounding every time (the same rounding of the same values), checked by
test_train_cl and by the checkpoints of 20-step runs (training.md).
R3 to R5 and R7: LoRA
LoRA in the trainer, on the CPU reference, the OpenCL device and
rocm alike, as recommended (deviation 2); the decisions in
ADR-0020, the use in training.md.
- R3, forward: h = s (x A^T), then y += h B^T after
the Q/K/V (or packed QKV), output, gate/up and down products, one
adapter per weight of the model file on its columns of the packed
product (
src/train/train.c:lora_forward). - R4, backward: dB += dy^T h, dh = s (dy B), dA +=
dh^T x, dx += dh A (
lora_backward); the model's weight gradients skipped (the products'dw, the norms' and the head norms' parameter passes, the embeddings' backward), the input gradients kept. - R5, validation: every adapter of the five tiny
architectures against finite differences, every target on; the model's
parameters and gradients untouched after steps; a fresh adapter giving
the model's loss to the bit (
tests/train/test_train.c,test_lora); one pass and five steps onopencland onrocm(every single product through rocBLAS) against the CPU (tests/train/test_train_cl.c); the device's split products against a double-precision reference (test_skinny). - R7, end to end: adapter files in llama.cpp's format
with the checkpoint's sidecar (
src/train/checkpoint.c,oklm_trainer_load_lora(); resuming bit-identical, another rank refused);oklm train --lora R --lora-alpha A --lora-targets LIST,-m <model> --resume <adapter>;oklm_lora_merge()(src/io/lora.c),oklm lora-merge,--loraonrun,chat,pplandserve(test_lora_merge: the merged model's loss is the adapted trainer's). llama.cpp reads the adapters: its perplexity with--loramatches it on OKLM's merged model (27.3985 and 27.3989, SmolLM2-135M). - Device products: LoRA's rank-wide products have one
to sixteen output tiles and an inner size of every row of the batch;
split along the inner size into a batch of parts (about 64 work groups
in all) summed in order, and the scale taken as the product's factor, a
step for every projection of SmolLM2-135M went from 197 to 140 ms on
opencland 125 to 102 ms onrocm(a full fine-tune: 156 and 110 ms). Adjacent tensors of the same weight decay are updated in one AdamW launch. - Not done: the 27B path (a quantized frozen model,
activations recomputed) stays split out, as decided at R0. Since then
(2026-11-04): activation recompute, the head in chunks, and the blocks'
matrices kept in their file's types (
--quantized-base, unpacked by tiles of one matrix before each product rather than a transposed quantized kernel; training.md).
R6: performance baseline
Measured 2026-11-04 (training.md, benchmarks.md).
- Solutions (task 1): hipBLASLt was replaced by
rocBLAS at R1, so rocBLAS's choice was checked instead:
bench_rocblas --solutionstimes its default against every solutionrocblas_gemm_ex_get_solutionslists for each product (both loaded at run time, optional). The default is within 5 % on most products and gives up at most 3 % of a step on the others (SmolLM2's MLP down forward, the Nano's Q/K/V weight gradient); nothing is hardcoded, as solution indices belong to one rocBLAS build. rocBLAS 4.4 aborts on some listed solutions of a 49 152-column output head, which the bench skips. - Steps (tasks 2, 3): full fine-tune against LoRA
(rank 16, Q, K, V, O), 4 x 256 tokens, on
openclandrocm: SmolLM2-135M, Qwen2.5-0.5B, Qwen3-0.6B and Llama-3.2-1B. LoRA is 1.3 to 1.6 times faster than a full step,rocm1.3 to 2 times faster thanopencl; Llama-3.2-1B only trains with LoRA (344 ms a step onrocm): a full fine-tune needs 16 bytes a parameter. - Memory: the gradients and AdamW moments now hold
only the trainable floats (
train_off): with LoRA the model has neither, its gradient pointers NULL. Before, every LoRA run allocated three model-sized buffers it never used. - Profile: the Llama-3.2-1B LoRA step on
rocmis 79 % rocBLAS products, 10 % norms, nothing else above 2 %: no hotspot for a custom kernel, so the plan's decision point (a.hipunit) is not reached.