ADR-0020: LoRA adapters
Records how LoRA adapters are named, trained, saved and applied.
Status: accepted (2026-11-04); its "Applying" decision amended by ADR-0021 (adapters also served beside the weights, several in one batch).
Context
rocm_spec.md (R3, R4, R7) asks for
LoRA in the trainer, adapter files, oklm train --lora and
oklm run --lora. Its decision 4 left open the naming (the
plan's A and B are the reverse of the paper's), the projections that get
adapters, and the file: llama.cpp's GGUF adapter format or OKLM's
checkpoint format. The design note (rocm-design.md) recommended doing LoRA in
the trainer on the existing devices, independent of HIP, and the user
approved it.
Decision
- Naming and initialization: the paper's. A (rank x inputs) maps down to the rank and is drawn uniformly in ±1/sqrt(inputs); B (outputs x rank) maps up and starts at zero; the update is W + (alpha / rank) B A, alpha the rank by default. A fresh adapter changes nothing, to the bit.
- Targets: Q, K, V, O, gate, up and down for Llama-family models; QKV, O, up and down for GPT-2; the attention's by default. A packed product (OKLM's Q/K/V and gate/up are one product each) gets one adapter per weight of the model file, on that weight's columns, so adapters map one to one onto the file's tensors.
- Training: the adapters follow the model in the parameter buffer, so the model's range is frozen by not touching it: its weight gradients are never computed, and the gradient norm, clipping and AdamW cover the adapters' range only.
- Files: llama.cpp's adapter format, so llama.cpp
reads OKLM's adapters and OKLM reads llama.cpp-format ones:
general.typeadapter,adapter.typelora,adapter.lora.alpha,general.architecturethe model's, the tensors<weight>.lora_a(shape {inputs, rank}) and<weight>.lora_b(shape {rank, outputs}). A LoRA checkpoint is this file plus ADR-0008's sidecar (the adapters' moments, the step, the options andtrain.lora_rank,train.lora_alpha,train.lora_targets, the hash over the adapters). - Applying: merged into the weights, not evaluated
beside them:
oklm_lora_merge()writes W + scale x (alpha / rank) x B A as a new model (F32, F16 and BF16 weights keep their type, quantized ones become F16), and--loraonrun,chat,pplandservemerges into a temporary copy as the model loads.
Consequences
An adapter is a small file (10 MB for every projection of SmolLM2-135M at rank 8) that llama.cpp and OKLM both use. Merging keeps inference unchanged (no extra products, every backend and quantized cache as before) at the cost of a copy of the model per adapter and a merge at load time; several adapters served in one batch (serving's S8.5) would need the products beside the weights instead, and are left for when they are asked for. A quantized weight an adapter touches is written as F16, which is larger than the original; requantizing would lose most of a small update.