OKLMdocumentation v1.0.3
On this page

ADR-0020: LoRA adapters

Records how LoRA adapters are named, trained, saved and applied.

Status: accepted (2026-11-04); its "Applying" decision amended by ADR-0021 (adapters also served beside the weights, several in one batch).

Context

rocm_spec.md (R3, R4, R7) asks for LoRA in the trainer, adapter files, oklm train --lora and oklm run --lora. Its decision 4 left open the naming (the plan's A and B are the reverse of the paper's), the projections that get adapters, and the file: llama.cpp's GGUF adapter format or OKLM's checkpoint format. The design note (rocm-design.md) recommended doing LoRA in the trainer on the existing devices, independent of HIP, and the user approved it.

Decision

  • Naming and initialization: the paper's. A (rank x inputs) maps down to the rank and is drawn uniformly in ±1/sqrt(inputs); B (outputs x rank) maps up and starts at zero; the update is W + (alpha / rank) B A, alpha the rank by default. A fresh adapter changes nothing, to the bit.
  • Targets: Q, K, V, O, gate, up and down for Llama-family models; QKV, O, up and down for GPT-2; the attention's by default. A packed product (OKLM's Q/K/V and gate/up are one product each) gets one adapter per weight of the model file, on that weight's columns, so adapters map one to one onto the file's tensors.
  • Training: the adapters follow the model in the parameter buffer, so the model's range is frozen by not touching it: its weight gradients are never computed, and the gradient norm, clipping and AdamW cover the adapters' range only.
  • Files: llama.cpp's adapter format, so llama.cpp reads OKLM's adapters and OKLM reads llama.cpp-format ones: general.type adapter, adapter.type lora, adapter.lora.alpha, general.architecture the model's, the tensors <weight>.lora_a (shape {inputs, rank}) and <weight>.lora_b (shape {rank, outputs}). A LoRA checkpoint is this file plus ADR-0008's sidecar (the adapters' moments, the step, the options and train.lora_rank, train.lora_alpha, train.lora_targets, the hash over the adapters).
  • Applying: merged into the weights, not evaluated beside them: oklm_lora_merge() writes W + scale x (alpha / rank) x B A as a new model (F32, F16 and BF16 weights keep their type, quantized ones become F16), and --lora on run, chat, ppl and serve merges into a temporary copy as the model loads.

Consequences

An adapter is a small file (10 MB for every projection of SmolLM2-135M at rank 8) that llama.cpp and OKLM both use. Merging keeps inference unchanged (no extra products, every backend and quantized cache as before) at the cost of a copy of the model per adapter and a merge at load time; several adapters served in one batch (serving's S8.5) would need the products beside the weights instead, and are left for when they are asked for. A quantized weight an adapter touches is written as F16, which is larger than the original; requantizing would lose most of a small update.