Training
Training a transformer from scratch, fine-tuning a loaded model and
resuming from checkpoints: the trainer's API, its kernels, checkpoint
files, the oklm train command, and how the results are
checked against llm.c.
Status: implemented for the GPT-2 architecture and the Llama family (Llama, Mistral, Qwen2, Qwen3), on the CPU and on OpenCL devices.
Trainer
include/oklm/train/train.h, sources in
src/train/. A trainer holds every parameter, its gradient
and the two AdamW moments in F32 flat buffers (four floats per
parameter), and the activations of one batch of at most
batch sequences of seq_len tokens, allocated
once (oklm_trainer_activation_bytes();
oklm train prints it).
Activation recompute (
recompute,oklm train --recompute): only each block's input is kept from the forward pass, with one set of the other activations shared by every block. Just before a block's backward pass its forward pass runs again from that input (the last block's activations are still in place). Dropout masks are keyed by the step, the batch and the site, so a recomputed block draws the same ones, and the blocks' gradients are those of keeping everything, to the bit, on the CPU,openclandrocm(test_train.c and test_train_cl.ctest_recompute).The head in chunks (with recompute): the logits and probabilities are held for a chunk of rows at a time, about 2^26 floats of each (multiples of 64 rows;
OKLM_TRAIN_LOGIT_ROWSsets it). The forward pass keeps the losses and each sequence's last logits (oklm_trainer_last_logits();oklm_trainer_logits()andoklm_trainer_probs()give NULL after a chunked pass). A full fine-tune's backward pass computes each chunk again for the head's gradient. With LoRA the head is frozen, so the forward pass also takes each row's (p - onehot) W while the chunk is there, and the backward pass only weights those rows. The gradients are those of the whole head within float rounding (the head's sums go chunk by chunk;test_chunked_head). Together the activations shrink 3 to 8 times (the table under Speed).A quantized base (
quantized_base,oklm train --lora R --quantized-base): with LoRA, the blocks' projection matrices (Q, K, V, O, gate, up, down; rows of whole groups of 32) stay in the model file's types, on the trainer's device, instead of F32. Before each product, forward or backward (dX = dY W included), the matrix is unpacked into one F32 scratch, the size of the largest matrix: on a device by a kernel built from the inference engine's dequantizers (every type its products read), on the CPU byoklm_dequantize_row(). The device's unpacking gives the CPU's values to the bit for all 28 types (test_train_cl.ctest_dequantize), so the products, losses and adapters are those of the F32 trainer to the bit, with or without recompute, on the CPU,openclandrocm(test_quantized_base, on Q8_0 and Q4_K_M files that OKLM quantized). The token embeddings are packed too (not GPT-2's, whose lookup adds the positions): a token's row is unpacked as it is looked up. So is the output head, unpacked whole when it fits the scratch (the same bits), else a tile of vocabulary rows at a time (the scratch's size;OKLM_TRAIN_HEAD_ROWSsets it), each tile giving its columns of the logits, the gradient through the head summed tile by tile within float rounding (test_quantized_head).oklm_trainer_weight_bytes()gives what the model takes;oklm trainprints it (the table under Speed).BF16 storage (
act_type,opt_type,grad_type,save_type;oklm train --act-type bf16 --opt-type bf16 --grad-type bf16 --save-type bf16), each opt-in, every computation still in F32. See BF16 storage for what each one stores, the measurements and the caveats.oklm_trainer_memory()gives the memory by kind (weights, gradients, optimizer, activations, and of those the activations kept for the backward pass);oklm trainprints it.oklm_trainer_new(): random weights, GPT-2's initialization for both architectures (normal with standard deviation 0.02, the attention and MLP output projections scaled by 1 / sqrt(2 n_layer), biases 0, norm scales 1).oklm_trainer_from_model(): the weights of a loaded model (GGUF of any type, or a Hugging Face checkpoint), converted to F32; the output head is tied to the token embeddings unless the model has its own. A Hugging Face Llama checkpoint's query and key rows (each head split in two halves) are reordered into the interleaved pairs of GGUF Llama files, so its checkpoints are ordinary Llama files; its trainer gives the same logits, bit for bit, as the trainer of its own checkpoint.oklm_trainer_forward()with targets gives the mean cross-entropy loss and keeps what the backward pass needs; without targets it gives logits (sampling during training uses it).oklm_trainer_backward(scale)adds the gradient of the loss timesscaleto the gradients, so several batches can be accumulated (accumbatches each scaled by 1 /accumgive the gradient of their mean loss;oklm train --accum).oklm_trainer_update(lr)clips the global gradient norm (whengrad_clipis set) and applies one AdamW step, with decoupled weight decay on the matrices only, or on every parameter (decay_all, as llm.c does).oklm_trainer_lr(step): linear warm-up, then cosine decay fromlrtolr_minuntilsteps.dropoutin the options: residual dropout, as nanoGPT's, on the embeddings and on the attention and MLP outputs before their residual adds, in training mode only (oklm_trainer_set_training(), on by default;oklm trainevaluates in evaluation mode). The attention weights are not dropped. Each mask is a hash of the seed, the step, the batch within the update, the site and the element, recomputed by the backward pass: it does not depend on the thread count, a device draws the CPU's masks, and a resumed checkpoint draws those of the uninterrupted run (the probability is kept in the sidecar).oklm_trainer_step()chains the four.backendin the options:cpu(default),auto(the first OpenCL device, else the CPU) or an OpenCL instance (opencl,opencl1);oklm_trainer_backend()tells which one runs. See On an OpenCL device.
Architectures, with llama.cpp's tensor names so checkpoints are GGUF files of the same architecture:
GPT-2: learned positions, LayerNorm with biases, GELU MLP with the tanh approximation, biases on every projection.
Llama: rotary positions, RMSNorm, SwiGLU MLP, grouped-query attention (fewer key/value heads than query heads), no biases. Rotation of adjacent pairs (GGUF) or half-split pairs, of all or the first
n_rotdimensions of each head, with Llama 3's frequency factors when the model has them (rope_freqs.weight, kept in the checkpoints). SmolLM2, TinyLlama and Llama 3.2 have this block. The head size is its own (head_dimxn_headneed not ben_embd: Qwen3's 16 heads of 128 over 1024 channels).Qwen2 (
qwen2): the Llama block with biases on the Q, K and V projections, half-split pairs (Qwen2, Qwen2.5).Qwen3 (
qwen3): the Llama block with an RMSNorm of each query and key head (attn_q_norm,attn_k_norm, one scale ofhead_dimshared by the heads) between the Q/K/V product and the rotation, as the inference engine computes it.Sliding windows, experts and other scalings are refused (
OKLM_ERR_UNSUPPORTED). Each block's Q, K and V projections are stored one after the other, as are their biases and its gate and up projections, so each group is one matrix product (and one bias).
Kernels
src/train/train_internal.h: explicit forward and
backward functions over the preallocated buffers, in the manner of
llm.c, run on the CPU worker team.
- Matrix product (
gemm_train.c): one form, C (+)= A B, with A read directly or transposed. The forward pass multiplies by the transposed weight (transposed once per call into a scratch buffer), the backward pass gives the input gradient (dout W) and adds the weight gradient (dout^T x). Tiles of 4 x 24 (AVX2) or 8 x 32 (AVX-512) outputs, blocks of 256 along the inner dimension. Each output is summed in one lane from k = 0 up, by a multiplication then an addition (no fused multiply-add): the scalar, AVX2 and AVX-512 variants give the same bits. - Attention: causal, softmax of scaled dot products, the weights kept for the backward pass (batch x heads x T x T floats per layer). The packed Q/K/V layout takes fewer key/value heads than query heads, for grouped-query attention.
- Rotary positions: the query and key heads of the packed rows rotated in place after the Q/K/V product, and rotated back (the transposed rotation) on their gradients after the attention's backward pass. The cosines and sines are tabulated once, with the inference engine's arithmetic (the angle multiplied by base^(-2 / n_rot) from one pair to the next).
- Per-head Q/K norms (Qwen3): each query and key head
of the packed rows normalized over its
head_dimvalues, the value heads copied; the Q/K/V product before it is kept for the backward pass, which replaces the heads' gradients in place after the rotation's. The two scale gradients are summed per channel over the rows then the heads in order (on a device by chunks of 64 rows, then the chunks in order). - Norms (LayerNorm, and RMSNorm without a bias), GELU, SwiGLU (silu(gate) x up over the gate and up halves of one product), softmax with cross-entropy (the logit gradients replace the probabilities in place; each row's normalizer is summed in double, which brings the loss of GPT-2 124M to PyTorch's within 1e-6).
Determinism: rows are split between threads, sums over rows (weight
and bias gradients, embedding gradients) are split by blocks of channels
with the rows walked in order, the gradient norm is summed in fixed
chunks. A run gives the same parameters, bit for bit, with any thread
count and at any SIMD level (test_train), for both
architectures.
On an OpenCL device
src/train/train_cl.c and
src/train/train_kernels.cl: the same kernels as OpenCL C,
built on the device, context and queue of an OpenCL backend instance
(ADR-0017), with the CPU kernels' arguments. Device memory is handed out
as fake addresses, so the trainer's layout and pointer arithmetic are
unchanged and each kernel call goes to the CPU or the device.
- The parameters, gradients, AdamW moments and activations stay on the device. Per step only the token ids go up and the row losses come back; the gradient norm is summed on the device in chunks of 4096 and the partial sums added on the host in double.
- The host keeps copies of the parameters, gradients and moments,
fetched when they are read (checkpoints,
oklm_trainer_tensor(), the logits for sampling) and sent back when the host changed them (initialization, loading a state, writes throughoklm_trainer_tensor()), so checkpoints are the same files on both sides and a device run resumes on the CPU. - Matrix products: one tiled kernel (64 x 64 outputs per work group of 16 x 16, blocks of 16 along k), operands read through strides (no transposed copies) and batched along a third dimension. Attention is batched products (scores = Q K^T, weights x V, and for the gradients dout V^T, dpre K, dpre^T Q, weights^T dout, the query heads of a group added in order) with the causal softmax and its backward as row kernels. A work group's batch offsets are computed with 32-bit divisions: GPUs have no 64-bit integer divide, and its software routine, five times per item, cost a fifth of the products' time (an 8-layer, 512-channel Llama step of 16 x 512 tokens: 284 to 238 ms, the same bits). Larger register tiles (128 x 128 outputs per work group, 8 x 4 or 8 x 8 per item, loads following each operand's contiguous dimension) measured no faster on this gfx1100 once the divisions were gone: the products are bound by their loads, not their arithmetic.
- Determinism: no atomics; every output is summed in a fixed order
(the sums over rows of the norm and bias gradients by chunks of 64 rows,
then the chunks in order), so a device gives the same bits on every run.
The device and the CPU sum in different orders and use different exp and
tanh, so they agree to
TOL_TRAIN_DEVICE(1e-4 relative), not bit for bit. - BF16 storage (
--act-type,--opt-type,--grad-type) keeps its 16-bit copies on the device too:tr_bf16_st/tr_bf16_ldround and widen a block's activations,tr_adamw16updates BF16 moments andtr_grad16_addadds a block's gradients into BF16 ones, both with the CPU's stochastic-rounding bits (BF16 storage). - A hybrid stack's Gated DeltaNet blocks run on the device too
(
tr_gd_*kernels): the input projections as products, the short conv, the recurrence (one work-group per sequence and value head, heads of up to 128 keys and values), the gated output norm. Their backward computes the forward again, keeping the recurrence's state every 16 tokens, and walks each 16-token chunk backward from its states computed again; the q and k gradients of a key group's heads are summed before the L2 norms' backward, and every weight gradient is a fixed-order sum, as above. Larger heads, orOKLM_TRAIN_GD_HOST=1, run the block on the host as before. A step of a 4-block, 512-channel hybrid stack (two linear blocks of eight 64-value heads, 4 x 128 tokens): 2 626 ms with the block on the host, 43.9 ms on the device, the same loss (2026-11-11);test_train_clchecks the gradients against the CPU over 7 and 37 tokens (three chunks). OKLM_CL_PROFILE=1waits for and times every launch and prints a table per kernel when the trainer is freed.
Checkpoints
adr/0008-training-checkpoints.md:
<name>.gguf: the weights as an ordinary F32 GGUF model, with hyper-parameters and tokenizer.oklm run,oklm chat,oklm quantizeand llama.cpp load it as it is. With--save-type bf16its matrices are BF16 (half the file; norms and biases stay F32), and the sidecar's hash is that of the rounded weights, so resuming it works, from the rounded weights (BF16 storage).<name>.gguf.train: the AdamW moments (F32, or BF16 with--opt-type bf16), the step, the options (the storage types among them), and a hash of the weights' data; a sidecar of other weights is refused, and so is one whose layout (train.version) is newer than the build reads.oklm_trainer_save(),oklm_train_read_params(),oklm_trainer_load_state(). Training continued from a checkpoint gives the same parameters, bit for bit, as the same steps run without stopping; weights without a sidecar start a fresh optimizer state (a fine-tune).- With LoRA (LoRA)
<name>.ggufis the adapters only, in llama.cpp's adapter format, and the sidecar holds their moments and the LoRA options;oklm_trainer_load_lora()reads them back into a trainer of the same rank and targets.
oklm train
--out is a prefix: checkpoints are
<prefix>-<step>.gguf, and the directories it
names (build/nano/ below) are created when missing, before
any work starts.
# from scratch: a 2048-token BPE tokenizer trained on the text, then a GPT-2
oklm train -f data/tinyshakespeare.txt --out build/nano/nano --vocab 2048 \
--layers 6 --embd 256 --heads 8 --ctx 256 --batch 16 \
--steps 1200 --lr 1e-3 --lr-min 1e-4 --warmup 100 --save-every 200 --sample-every 400
# fine-tune a model
oklm train -m models/gpt2-hf -f data/tinyshakespeare.txt --out build/ft/gpt2 --steps 200 --lr 1e-4
# a Llama from scratch: 8 query heads over 4 key / value heads, MLP of about 8/3 x 256
oklm train -f data/tinyshakespeare.txt --out build/nano-llama/nano --vocab 2048 --arch llama \
--layers 6 --embd 256 --heads 8 --kv-heads 4 --ctx 256 --batch 16 \
--steps 600 --lr 1e-3 --lr-min 1e-4 --warmup 100
# the same on the GPU
oklm train -f data/tinyshakespeare.txt --out build/nano-llama/nano --vocab 2048 --arch llama \
--layers 6 --embd 256 --heads 8 --kv-heads 4 --ctx 256 --batch 16 \
--steps 600 --lr 1e-3 --lr-min 1e-4 --warmup 100 --backend opencl
# fine-tune a Hugging Face Llama with updates over 4 batches of 4 x 128
oklm train -m models/smollm2-135m-hf -f data/tinyshakespeare.txt --out build/ft/smollm2 \
--batch 4 --seq 128 --accum 4 --steps 50 --lr 1e-4
# continue a checkpoint (its options come from the sidecar; --steps extends the run)
oklm train --resume build/nano/nano-000600.gguf -f data/tinyshakespeare.txt --out build/nano/nano
oklm run -m build/nano/nano-001200.gguf -p "ROMEO:" -n 100
# LoRA: rank-8 adapters on every projection of a frozen model, then run with them, or merge them
oklm train -m models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf -f data/tinyshakespeare.txt --out build/lora/smol \
--lora 8 --lora-alpha 16 --lora-targets all --batch 8 --seq 128 --steps 200 --lr 1e-3 --warmup 20 --backend rocm
oklm train -m models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf --resume build/lora/smol-000200.gguf \
-f data/tinyshakespeare.txt --out build/lora/smol --steps 400
oklm run -m models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf --lora build/lora/smol-000400.gguf -p "ROMEO:" -n 100
oklm lora-merge models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf build/lora/smol-000400.gguf build/lora/smol-merged.gguf
# BF16 storage: activations and AdamW moments in BF16, checkpoints and the merged model in BF16
oklm train -f data/tinyshakespeare.txt --out build/nano/nano --vocab 2048 --layers 6 --embd 256 --heads 8 --ctx 256 --batch 16 \
--steps 1200 --lr 1e-3 --lr-min 1e-4 --warmup 100 --backend cpu --act-type bf16 --opt-type bf16 --save-type bf16
oklm lora-merge models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf build/lora/smol-000400.gguf build/lora/smol-merged.gguf --save-type bf16
# F16 products on the matrix units with dynamic loss scaling
oklm train -m models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf -f data/tinyshakespeare.txt --out build/ft/smol16 \
--batch 4 --seq 256 --steps 300 --warmup 20 --lr 1e-4 --backend rocm --matmul-precision f16 --loss-scale dynamic--backend is auto by default (the first
OpenCL device, else the CPU), as for oklm run;
cpu gives the same bits on every machine. A new model is
GPT-2 by default; --arch llama builds Llama blocks
(--kv-heads, --head-dim, --ff,
--rope-theta), --arch qwen2 adds Q/K/V biases
and --arch qwen3 per-head Q/K norms to them.
--dropout P sets the residual dropout (0 by default).
--accum N sums the gradients of N batches per update, for a
larger effective batch in the same memory; the run's options,
--accum included, are kept in the sidecar.
The text is tokenized once; its last --val-frac (10 %)
is kept for the validation loss, printed every --eval-every
steps over --eval-batches consecutive windows. Each batch
is drawn from the seed, its index (the step times --accum,
plus its place in the step) and the row, so a resumed run sees the
batches the uninterrupted one would have. Checkpoints are written as
<prefix>-<step>.gguf every
--save-every steps and at the end.
--sample-every N prints a sample every N steps and once
more at the end (none without it): a text the model being trained writes
from its end-of-text token (a newline for a tokenizer without one),
between two --- lines. --sample-len L is the
sample's length in tokens (200 by default); each token is drawn at
temperature 0.8 among the 40 likeliest, from a seed made of the run's
seed and the step, with the last --seq tokens as the
context, run again for each new token, so a long sample of a large model
takes a while. --progress-json FILE writes the run as JSON
lines a program can follow (the section "Machine-readable progress"
below).
LoRA
--lora R (the trainer's lora_rank) freezes
the model and trains adapters of rank R on its projections: a weight W
(outputs x inputs) becomes W + (alpha / R) B A, A (R x inputs) drawn
uniformly in ±1/sqrt(inputs), B (outputs x R) zero, so a fresh adapter
changes nothing (the loss of the model alone, to the bit).
--lora-alpha sets alpha (the rank by default: a scale of
1). --lora-targets lists the projections: q,
k, v, o, gate,
up, down for Llama-family models,
qkv, o, up, down for
GPT-2, or all; the default is the attention's
(q,k,v,o, or qkv,o). Q, K and V (and gate and
up) have adapters of their own on their columns of the packed product,
as separate weights have in the model file.
- Forward: after the product, h = s (x A^T), kept for the backward pass, then y += h B^T.
- Backward: dB += dy^T h, dh = s (dy B), dA += dh^T x, dx += dh A. The model's own weight gradients are not computed (the products that would give them are skipped, and the norms' and embeddings' passes with them); the gradients flowing through the model to earlier adapters are.
- State: the adapters follow the model in the parameter buffer; gradients, AdamW moments, the gradient norm and clipping cover only them (2.4 M parameters for every projection of SmolLM2-135M at rank 8, against 135 M).
- Files: a LoRA checkpoint's weights file is an
adapter in llama.cpp's format:
general.typeadapter,adapter.typelora,adapter.lora.alpha,general.architecturethe model's, and for a weight<name>the tensors<name>.lora_a(shape {inputs, rank}) and<name>.lora_b(shape {rank, outputs}), in F32 (10 MB for the example above, against 540 MB for a full checkpoint). llama.cpp's--lorareads it. - Resuming:
-m <model> --resume <adapter.gguf>: the rank, alpha and targets come from the sidecar; the adapters are loaded, then the optimizer state. The same parameters, bit for bit, as without stopping. - Using them:
oklm run,chat,pplandservetake--lora <adapter.gguf> [--lora-scale S]: the adapter is merged into a temporary copy of the model as it loads.oklm lora-merge <model> <adapter> <out> [--scale S]writes the merged model (oklm_lora_merge(),oklm/io/lora.h): W + S x (alpha / rank) x B A, F32, F16 and BF16 weights keeping their type and quantized ones becoming F16; every other tensor and key copied.oklm serve --lora-module NAME=<adapter.gguf>serves adapters beside the model instead, several in the same batches, each request choosing one by itsmodel(server.md, ADR-0021); in the library,oklm_adapter_load()andoklm_seq_set_adapter()(oklm/model/model.h). - On a device: the rank-wide products have few output tiles and a long inner size (dA is rank x inputs summed over every row of the batch), so the device product splits such products along the inner size, one batch per part, and sums the parts in order; the scale is the product's own factor (rocBLAS's alpha, or the kernel's) rather than a pass of its own.
Stopping a run. oklm train and
oklm rl asked to end (Ctrl-C, SIGTERM, a tool's
Stop, which on Windows is the event of the registry of
processes) finish the step they are in, write the checkpoint of that
step and its training state, print
stopped on request at step N of TOTAL: a checkpoint is written
and end with the status 0; the final validation and the final sample are
left out. --resume on that checkpoint goes on from step N +
1. A second Ctrl-C ends the program at once, without a checkpoint.
Chat-data fine-tuning (loss on the assistant turns)
oklm train --chat-data conv.jsonl (in place of
-f text.txt) trains on conversations with the loss on the
assistant turns only — supervised fine-tuning for tool use and
instruction following, not plain next-token over the whole text. Each
JSONL line is {"text": ...} as
oklm agent --save-training writes it: the conversation
rendered through the model's chat template. It needs a model with that
template (-m or --resume), not a new one.
- The mask. Each conversation is tokenized and its
assistant tokens marked (the content a model generated and the
<|im_end|>that closes it, found by reading the role after each<|im_start|>); the system, user and tool turns are not targets. A conversation longer than--seqkeeps its last--seqtokens, where the assistant turns usually are. - The loss. The forward pass's per-token losses
(
oklm_trainer_row_losses()) are averaged over the masked tokens; the gradient isoklm_trainer_backward_weighted()with weights of mask / (masked count) / accum, so only the assistant tokens pull the weights. The step and the validation loss (a held-out--val-fracof the conversations) are this masked mean. A token the model is merely given (the prompt) never contributes, so the run does not learn to generate the inputs. - Use.
oklm train --chat-data data.jsonl -m model.gguf --lora 16 --out adapter, then run or serve with--lora. The MCP tool-use and the defensive security fine-tunes (mcp-training-design.md, security-training-design.md) both train this way on data captured with--save-training.
Verification
tests/train/test_train.c, on tiny GPT-2 and Llama models (adjacent and half-split rotations, partial rotation, 4 query heads over 2 key/value heads): the matrix product against a plain loop at every SIMD level (the same bits); every parameter tensor's gradient against a central difference of the loss (TOL_TRAIN_FD_REL); the same parameters after training with 1 or 3 threads and at every SIMD level; two half batches accumulated against the whole batch; a tiny model learning a batch (loss 3.6 to 0.0007 in 150 steps); the schedule; the forward pass of loaded models against their fixtures' float64 references (GPT-2 1.7e-5, Llama with Llama 3 frequency factors 1.3e-6); a Hugging Face Llama checkpoint's trainer against its own checkpoint's (the same bits) and against inference on it (1e-6); checkpoints loaded as models, resumed bit for bit, a sidecar of other weights refused.tests/train/test_train_cl.c(skipped without an OpenCL device): the tiny GPT-2 and Llama models on the device against the CPU trainer: loss, logits and every gradient of a pass (within 6e-7 relative), the losses and parameters of five AdamW steps, the same bits from two device runs, and a device checkpoint resumed on the CPU.tests/train/test_train_gpt2.c(skipped withoutsh tools/fetch_llmc.sh, 1.1 GB): GPT-2 124M from the Hugging Face checkpoint against llm.c, on the CPU and on an OpenCL device.- llm.c's reference state, written with PyTorch by its
train_gpt2.py, with the criteria of itstest_gpt2.c: logits of one batch of 4 x 64 tokens, the loss, every parameter's gradient, then the losses of ten AdamW steps. - llm.c's CPU training run of its README (
train_gpt2.c: 40 steps of 4 x 64 tokens of Tiny Shakespeare in order, AdamW at 1e-4): the validation loss before and after, and the train losses it prints.
- llm.c's reference state, written with PyTorch by its
| Check | llm.c / PyTorch | OKLM, CPU | OKLM, OpenCL | Tolerance |
|---|---|---|---|---|
| logits of one batch | PyTorch | max diff 0.0020 | max diff 0.0015 | 1e-2 |
| loss | 5.270009 | 5.270009 | 5.270011 | 1e-2 |
| gradients, every parameter | PyTorch | max diff 0.00032 | max diff 0.00031 | 2e-2 |
| loss after 10 steps (step 9) | 0.376511 | 0.376524 | 0.376529 | 1e-2 |
| Tiny Shakespeare, validation loss before training | 5.252026 | 5.252031 | 5.252031 | 1e-3 |
| train loss, step 0 / 1 / 2 / 3 / 39 | 5.356189 / 4.301069 / 4.623322 / 4.600470 / 3.970751 | 5.356194 / 4.301096 / 4.623336 / 4.600519 / 3.970691 | 5.356193 / 4.301094 / 4.623335 / 4.600514 / 3.970695 | 1e-2 |
| validation loss after 40 steps | 4.107781 | 4.107731 | 4.107734 | 1e-2 |
The validation loss of llm.c's published run is the mean over the
first ten batches of 4 x 64 validation tokens (its code has since gone
to five). The tests use llm.c's published numbers and reference files,
so they run without llm.c. llm.c itself (commit f1e2ace6, built in
build/llm.c as a test-only reference, never linked into
OKLM) was also run directly:
test_gpt2passes on the same reference state (step 9 loss 0.376691).train_gpt2with its training batches taken in order and ten validation batches (the conditions of its README run; its current code shuffles the batches and averages five): every one of the 41 train losses and 5 validation losses agrees with OKLM's CPU trainer within 0.0008 (largest at step 34, 4.516505 against 4.515689; llm.c sums the softmax normalizer in float, OKLM in double); validation after 40 steps is 4.107841 against 4.107731.
Speed
Intel Core i7-11700K, 8 threads, AVX-512:
| Model | Batch | Time per step | Tokens/s |
|---|---|---|---|
| GPT-2 124M (fine-tune) | 4 x 64 | 0.6 s | 430 |
| GPT-2, 5.3M parameters (6 x 256, vocabulary 2048) | 16 x 256 | 0.8 to 0.9 s | 4 400 to 5 000 |
| Llama, 4.95M parameters (6 x 256, 8 / 4 heads, vocabulary 2048) | 16 x 256 | 0.75 s | 5 450 |
| SmolLM2-135M (fine-tune, Llama) | 4 x 4 x 128 | 9 s | 210 |
On the GPU (AMD Radeon gfx1100, 42 compute units,
--backend opencl):
| Model | Batch | Time per step | Tokens/s | Against the CPU |
|---|---|---|---|---|
| Llama, 4.95M parameters (the Nano above) | 16 x 256 | 42 ms | 94 000 | 17 x |
| SmolLM2-135M (fine-tune) | 4 x 4 x 128 | 0.44 s | 4 240 | 20 x |
The 600-step Llama Nano takes 27 seconds instead of 7.5 minutes, with the same validation curve (3.8318 against 3.8319 after 600 steps); the SmolLM2 fine-tune 16 seconds instead of 4.5 minutes (3.4302 against 3.4298). Matrix products are three quarters of the device's time.
With ROCm, --backend rocm is the same OpenCL device with
its large single products through rocBLAS (loaded at run time, on shared
virtual memory; the attention's batched products and every other kernel
stay on OpenCL), the same losses (2026-11-04):
| Model | Batch | opencl |
rocm |
|---|---|---|---|
| Llama Nano | 16 x 256 | 38 ms a step | 26.5 ms (1.43 x) |
| SmolLM2-135M (fine-tune) | 4 x 4 x 128 | 390 ms | 310 ms (1.26 x) |
The products alone are 1.7 to 3.5 times faster
(tests/train/bench_rocblas.c); each switch between the two
runtimes drains the queue it leaves. OKLM_HIP_DEVICE picks
the HIP device (0), which must be the OpenCL device's GPU;
OKLM_ROCBLAS_MIN the multiply-adds from which a product
goes to rocBLAS (2^22).
rocm holds its buffers in shared virtual memory, which
the driver commits in device memory when allocated (the
opencl backend's plain buffers can overflow into host
memory instead). When the device is full, a buffer goes to fine-grained
shared virtual memory, in host memory the device reads over the bus,
with a notice (the device is full, N MiB in host memory);
OKLM_ROCM_DEVICE_MB caps the device memory used the same
way. Qwen3-8B Q4_K_M, LoRA rank 16 on Q, K, V, O,
--quantized-base --recompute, 1 x 6144 tokens with 2
accumulated (2026-11-08): the activations take 17.1 GB, so the 5.0 GB
packed base lands in host memory; a step takes 56.4 s on
rocm against 91.2 s on opencl, with the same
losses (5.0787, 5.0576). Before, rocm stopped there with
cannot allocate 5020594176 bytes of shared virtual memory:
the device's per-allocation limit
(CL_DEVICE_MAX_MEM_ALLOC_SIZE, 17 GiB on the gfx1100) was
not the cause, the device's 20 GB were.
--matmul-precision bf16|f16
(oklm_train_params::matmul_type, rocm only,
refused on other backends) runs rocBLAS's large products on the matrix
units of RDNA3: for a product of at least 128 rows, columns and inner
size (OKLM_ROCM_MM16_MIN), a kernel rounds both operands to
BF16 / F16 (to nearest even, in their own layouts),
rocblas_gemm_ex multiplies them with F32 accumulation into
a product of the same 16-bit type, and a kernel widens it into the F32
output (or adds it). The product goes through 16 bits because rocBLAS's
tuned gfx1100 kernels write 16-bit outputs: with F32 outputs its 16-bit
products ran 1.5 to 2 times slower than its F32 sgemm (20
times when the second operand is transposed), with 16-bit outputs 2.5 to
4 times faster (2026-11-08, SmolLM2 and 4096 x 4096 shapes). Under
--quantized-base a BF16 base is read as it lies (the same
bits as its rounded unpacked copy). The norms, attention, softmax and
optimizer stay F32, as do the thinner products (LoRA's rank-wide ones),
whose conversions cost more than they gain.
SmolLM2-135M on Tiny Shakespeare,
--batch 4 --seq 256 --steps 300 --warmup 20, full
(--lr 1e-4) and LoRA (rank 16, alpha 16, every target,
--lr 1e-3), validation loss every 50 steps and the mean
step (2026-11-08):
| Run | Inputs | Validation 0 / 50 / 100 / 150 / 200 / 250 / 300 | Step |
|---|---|---|---|
| full | F32 | 3.5508 / 3.2509 / 3.1891 / 3.1605 / 3.1462 / 3.1377 / 3.1329 | 125 ms |
| full | BF16 | 3.5516 / 3.2579 / 3.1906 / 3.1627 / 3.1477 / 3.1378 / 3.1342 | 114 ms |
| full | F16 | 3.5509 / 3.4086 / 3.3298 / 3.2860 / 3.2618 / 3.2430 / 3.2365 | 114 ms |
| LoRA | F32 | 3.5508 / 3.2887 / 3.2213 / 3.1804 / 3.1774 / 3.1619 / 3.1573 | 117 ms |
| LoRA | BF16 | 3.5516 / 3.2892 / 3.2215 / 3.1805 / 3.1770 / 3.1625 / 3.1576 | 110 ms |
| LoRA | F16 | 3.5509 / 3.5266 / 3.4266 / 3.3863 / 3.3845 / 3.3648 / 3.3573 | |
| full | F16, --loss-scale dynamic |
3.5509 / 3.2520 / 3.1891 / 3.1607 / 3.1475 / 3.1377 / 3.1347 | 102 ms |
| LoRA | F16, --loss-scale dynamic |
3.5509 / 3.2886 / 3.2199 / 3.1789 / 3.1774 / 3.1618 / 3.1570 | 107 ms |
BF16 follows F32 to the third decimal; F16 alone trains worse: small
gradients underflow its range (F16's 11 bits help the forward pass
only). With --loss-scale dynamic F16 follows F32 as BF16
does (3.1347 against 3.1329 full, 3.1570 against 3.1573 LoRA; the last
two rows 2026-11-08, the step times of a busier machine than the rows
above).
--loss-scale dynamic|N
(oklm_train_params::loss_scale, any backend) is dynamic
loss scaling: the backward pass computes the gradients of the loss times
a scale S (65536 for dynamic, or N), which lifts SmolLM2's
logit gradients (about 1e-8 per element at 1024 rows, under F16's
smallest denormal) into F16's range; the update divides them back before
the norm is clipped. An update whose gradients are not all finite (an
F16 product overflowed) is skipped, the parameters and moments
unchanged, the step counted, and S halves; S doubles after 200 finite
updates in a row (OKLM_TRAIN_LOSS_SCALE_WINDOW). Both runs
above settled at 16384 after two skipped updates out of 300. The step
line prints the scale and the skipped updates; the sidecar keeps the
scale as it stands, so a resumed run continues with it. In F32 a
power-of-two scale changes no bit (the 200-step test of
test_train), unless it overflows or reaches the
denormals.
What stays F32 under --matmul-precision was measured not
to pay (2026-11-08, SmolLM2-135M full, 4 x 256,
OKLM_CL_PROFILE=1, BF16 products): every OpenCL tiled
product, the attention's batched ones included, is 9.9 % of the device's
time, rocBLAS 41.7 %, and the roundings into and out of 16 bits
(tr_to16, tr_from16) already 20.9 %; the
attention's operands (Q, K, V and the B x heads x T x T weights) would
each need the same rounding for at most a few percent. The thin LoRA
products on 16 bits made a LoRA step slower (110 to 164 ms). The LoRA
losses come from a run with every rocBLAS product on 16 bits, the thin
ones included (164 ms a step with BF16); the 110 ms is a 30-step run
once those stay F32. SmolLM2's products are small; on Llama-3.2-1B
Q4_K_M (LoRA rank 16, Q, K, V, O, 4 x 256) a step takes 227 ms with BF16
against 350 ms (1.54 x), the losses within 1e-3.
The weights' 16-bit copies are kept between updates (2026-11-08):
each weight is rounded once after AdamW changes it (the update kernel
writes the rounding of what it updates), not at each of its products
(forward, input gradient, a recomputed forward, every accumulated batch,
every evaluation); a LoRA run's frozen base is rounded once for the
whole run. The output gradient of a weight's two backward products
(input gradient and weight gradient) is also rounded once for both. The
copy costs 2 bytes a parameter of device memory (257 MiB for
SmolLM2-135M, 2.3 GiB for Llama-3.2-1B); it is allocated on the first
16-bit product, in device memory only and when 512 MiB stay free after
it (otherwise the weights are rounded at each product as before, with a
notice), and OKLM_ROCM_W16_CACHE=0 turns it off. Under
--quantized-base the frozen matrices are unpacked and
rounded before each product as before (a BF16 base is still read in
place) and no copy is made: that option is about memory. The results are
the same bits: the same rounding of the same values, made less often. 20
steps of 4 x 256 (--warmup 5), the median step of steps 6
to 20, the most device memory the trainer allocated ("before" with
OKLM_ROCM_W16_CACHE=0, the allocations of before)
(OKLM_LOG_LEVEL=INFO prints it at exit), two runs each
where two are given (another job shared the GPU); in every row the 20
losses and the checkpoint written after step 20 (sha256) are identical
with and without the copy:
| Run | Step before | Step after | Device memory before | After |
|---|---|---|---|---|
| SmolLM2-135M full, BF16 | 102 / 103 ms | 98 / 99 ms | 3930 MiB | 4187 MiB |
SmolLM2-135M full, F16, --loss-scale dynamic |
102 / 114 ms | 97 / 98 ms | 3930 MiB | 4187 MiB |
SmolLM2-135M full, BF16, --accum 2 --recompute |
236 ms | 221 ms | 2777 MiB | 3034 MiB |
| SmolLM2-135M LoRA (every target), BF16 | 108 / 109 ms | 104 / 106 ms | 2483 MiB | 2748 MiB |
SmolLM2-135M LoRA (every target), F16,
--loss-scale dynamic |
118 / 107 ms | 105 / 105 ms | 2483 MiB | 2748 MiB |
| Llama-3.2-1B Q4_K_M LoRA, BF16 | 219 / 228 ms | 207 / 203 ms | 9785 MiB | 12149 MiB |
Llama-3.2-1B Q4_K_M LoRA, F16,
--loss-scale dynamic |
217 / 219 ms | 201 / 200 ms | 9785 MiB | 12149 MiB |
In the profile (OKLM_CL_PROFILE=1, 5 steps and a
validation pass) the roundings into and out of 16 bits went from 20.1 to
16.3 % of the device's time on SmolLM2-135M full (tr_to16
202 to 109 ms, the device's total 1547 to 1330 ms) and from 14.9 to 8.2
% on Llama-3.2-1B LoRA (tr_to16 318 to 97 ms, the total
2814 to 2410 ms). What is left are the activations: each product's input
and output gradient rounded in, and its output widened out
(tr_from16), new at every product. Keeping the forward
pass's rounded inputs for the weight-gradient products would cost 2
bytes a kept activation for at most the remaining input roundings; it
was not done.
LoRA against a full fine-tune (2026-11-04, rocm_spec.md
R6): a step of 4 x 256 tokens, rank 16, alpha 16, the default targets
(Q, K, V, O), the mean of steps 4 to 6
(oklm train ... --steps 6 --batch 4 --seq 256 [--lora 16 --lora-alpha 16] --backend opencl|rocm):
| Model (file) | Full, opencl |
Full, rocm |
LoRA, opencl |
LoRA, rocm |
|---|---|---|---|---|
| SmolLM2-135M (F16) | 163 ms | 118 ms | 126 ms | 98 ms |
| Qwen2.5-0.5B (Q4_K_M) | 500 ms | 315 ms | 322 ms | 199 ms |
| Qwen3-0.6B (Q8_0) | 635 ms | 410 ms | 443 ms | 291 ms |
| Llama-3.2-1B (Q4_K_M) | 9 s (beyond the device's memory) | out of memory | 679 ms | 344 ms |
The model is held in F32 whatever its file (4 bytes a parameter on
the device and as many on the host); a full fine-tune adds its gradients
and two AdamW moments (16 bytes a parameter in all: 19.8 GB for
Llama-3.2-1B, more than the 20 GB GPU holds once activations are
counted), LoRA only the adapters' (11 M floats for Llama-3.2-1B at rank
16). A LoRA step skips the model's weight gradients, a third of the
large products. In the Llama-3.2-1B LoRA step on rocm
(OKLM_CL_PROFILE=1) rocBLAS's products are 79 % of the
device's time, the norms 10 %, nothing else above 2 %.
Activation recompute (2026-11-04, rocm, the mean of
steps 4 to 6, LoRA rank 16 on Q, K, V, O). "Activations" is what
oklm train prints:
| Model (file), mode | Batch | Activations | Blocks recomputed | And the head in chunks | Step | Blocks recomputed | And the head in chunks |
|---|---|---|---|---|---|---|---|
| SmolLM2-135M (F16), full | 4 x 256 | 1689 MiB | 536 MiB | 536 MiB (one chunk) | 119 ms | 142 ms | 144 ms |
| SmolLM2-135M (F16), full | 4 x 1024 | 10103 MiB | 2358 MiB | 1326 MiB | 405 ms | 531 ms | 550 ms |
| Qwen2.5-0.5B (Q4_K_M), LoRA | 4 x 256 | 3516 MiB | 1457 MiB | 719 MiB | 195 ms | 257 ms | 294 ms |
| Qwen2.5-0.5B (Q4_K_M), LoRA | 4 x 1024 | 19 GB: out of memory | 6164 MiB | 1875 MiB | 962 ms | 1091 ms | |
| Llama-3.2-1B (Q4_K_M), LoRA | 4 x 256 | 4070 MiB | 1490 MiB | 997 MiB | 344 ms | 469 ms | 463 ms |
| Llama-3.2-1B (Q4_K_M), LoRA | 4 x 1024 | 24 GB: out of memory | 6728 MiB | 3253 MiB | 1865 ms | 1881 ms | |
| Llama-3.2-1B (Q4_K_M), LoRA | 2 x 2048 | 33 GB: out of memory | 7752 MiB | 4277 MiB | 2204 ms | 2211 ms |
Each block keeps its attention weights (batch x heads x T x T floats), so without recompute the activations grow with the square of the sequence. With the blocks recomputed, most of what is left is the head's logits and probabilities (batch x T x vocabulary floats each: 3.9 GiB of Llama-3.2-1B's 6.6 at 4 x 1024), which the chunks bring down to about 512 MiB. What remains is each block's input, one block's activations, the attention weights' gradients, and the chunk. LoRA pays relatively more for the extra forward pass, because its backward pass skips the model's weight gradients. Qwen2.5-0.5B (vocabulary 151,936) also pays for its head in three chunks a batch.
A quantized base (2026-11-04, rocm, LoRA rank 16 on Q,
K, V, O, 4 x 256 tokens, --recompute, the mean of steps 4
to 6). "Weights" is what oklm train prints; the losses are
the same to the last digit:
| Model (file, size) | Weights, F32 | The blocks packed | And the embeddings and head | Step, F32 | Step, all packed |
|---|---|---|---|---|---|
| Qwen2.5-0.5B (Q4_K_M, 469 MiB) | 2412 MiB | 1316 MiB | 504 MiB | 298 ms | 313 ms |
| Llama-3.2-1B (Q4_K_M, 770 MiB) | 4727 MiB | 1700 MiB | 904 MiB | 479 ms | 503 ms |
| Qwen2.5-1.5B (Q4_K_M, 1066 MiB) | 6796 MiB | 2654 MiB | 1182 MiB | 716 ms | 757 ms |
| Qwen3-4B (Q4_K_M, 2382 MiB) | 15389 MiB | 3790 MiB | 2611 MiB | 1675 ms | 1762 ms |
Qwen3-8B, the default model (2026-11-04, the same settings, the steps after the first; its own head, unpacked in 7 tiles of 24,576 rows):
| File (size) | Batch | Weights | Activations | Step | Tokens/s |
|---|---|---|---|---|---|
| Q4_K_M (4795 MiB) | 4 x 256 | 5232 MiB | 1621 MiB | 3.06 s | 335 |
| Q8_0 (8306 MiB) | 4 x 256 | 8743 MiB | 1621 MiB | 3.07 s | 334 |
| Q4_K_M | 4 x 1024 | 5232 MiB | 5918 MiB | 11.4 s | 359 |
| Q4_K_M | 2 x 2048 | 5232 MiB | 6942 MiB | 12.7 s | 323 |
| Q8_0 | 2 x 2048 | 8743 MiB | 6942 MiB | 12.7 s | 323 |
| Q4_K_M | 1 x 4096 | 5232 MiB | 8990 MiB | 15.3 s | 267 |
| Q4_K_M | 4 x 2048 | 5232 MiB | 13439 MiB | 25.2 s | 326 |
Held in F32 the model would need 32.8 GB and is refused. The Q8_0 base costs no time over Q4_K_M; its unpacking is as cheap. The largest batch here, 8192 tokens, takes about 18.7 GiB of the card's 20.
With everything packed the model takes its file's size plus the scratch (the largest block matrix in F32). Unpacking each matrix before its products (about three times a step with recompute) costs 5 to 6 % of a step. These four models tie their head to the embeddings: the head is unpacked in tiles of the scratch's rows (Qwen2.5-0.5B: 16 tiles).
BF16 storage
The trainer computes in F32 everywhere; four options store some of
what it holds in BF16 (8 bits of mantissa, F32's exponent range), for
memory, not speed (OKLM's kernels widen BF16 to F32; the products on the
gfx1100's matrix units are --matmul-precision, under Speed), on the CPU and on a device alike. Each is
opt-in; without them the trainer's bits are those of before (the
existing tests, and a 20-step checkpoint of the same run byte for byte
against the previous binary).
--act-type bf16(act_type): what each block keeps from the forward pass for its backward pass (its input, norms' outputs and statistics, Q/K/V, attention weights, attention output, MLP activations; with--recompute, its input only) is stored in BF16, rounded to nearest even as the block's forward pass ends, and widened into one F32 set just before its backward pass (or its recompute). The residual stream itself flows in F32 from block to block; the head's logits and probabilities, the gradients' scratch and the final norm stay F32. The kept activations halve; one block's F32 set is added, so the total shrinks by about (L / 2 - 1) blocks' worth. The gradients are within 2 % (norm of the difference) of F32's on the tiny models ofmake test, every architecture, whole or LoRA, with or without recompute (test_bf16_gradients).--opt-type bf16(opt_type): AdamW's two moments are stored in BF16, widened for the update, and stored back with stochastic rounding: up in magnitude with the probability of the dropped low 16 bits (so the stored value is the moment in expectation), the bits a hash of the seed, the step, the moment (m or v) and the element's index alone, so runs are reproducible and do not depend on the thread count (test_bf16_trainingcompares 1 and 3 threads to the bit). Rounding to nearest would freeze a moment whose update is under half a BF16 step (2^-9 relative), as with a beta2 of 0.999 (updates of 0.1 % of v): stochastic rounding keeps those updates in expectation. The parameter update uses the F32 moments before they are rounded; the master weights stay F32. The optimizer's memory halves (4 instead of 8 bytes a trainable parameter), and the sidecar holds the moments in BF16, so resuming them is exact.--grad-type bf16(grad_type, full training): the gradients are held in BF16 between the backward passes and the update (2 bytes a trainable parameter instead of 4). They are still computed in F32: the backward pass works through the model in regions (each block's parameters, which follow each other in the layout; the final norm; the output head and the token embeddings a tile of vocabulary rows at a time; GPT-2's positions), each region's gradients computed into one F32 window and then added into the BF16 ones with stochastic rounding (tr_grad16_add(): the bits a hash of the seed, the updates done, the micro-batch and the element, so accumulated micro-batches stay unbiased and the bits do not depend on the thread count). The window is the size of the largest block (the vocabulary tiles are as many rows as fit it,OKLM_TRAIN_GRAD_TILE_ROWSsets them), so the vocabulary matrices never need an F32 copy. The update's norm and AdamW read the BF16 values widened (tr_sqsumandtr_adamw/tr_adamw16take them on a device). A tied head's rows get two additions a backward pass (the head and the embeddings), and a chunked head (--recompute) one a chunk, each a rounding. The gradientsoklm_trainer_tensor()gives are then a widened copy made when asked for, read-only. Refused with LoRA: the adapters' gradients are small. Within 2 % of F32's gradients on the tiny models (every architecture and the hybrid one, with and without recompute, one micro-batch or two, whole tables or tiles of 3 rows:test_bf16_grad_storage), the same bits with 1 or 3 threads (test_bf16_training), and on the device within 1e-4 of the CPU (test_bf16_storage).--save-type bf16(save_type, any backend): the checkpoint's matrices (the adapters with LoRA) are written in BF16, rounded to nearest even; norms, biases and other 1-D tensors stay F32;general.file_typeis BF16's. The sidecar keeps the moments as the trainer holds them (F32 unless--opt-type bf16).oklm lora-merge --save-type bf16writes the merged weights and the model's other F32 and F16 matrices in BF16 (oklm_lora_merge_as()).
The options are kept in the sidecar and come back on
--resume (a flag given again overrides).
--act-type and --opt-type run on every
backend. On a device (opencl, rocm) the BF16
copies stay in device memory: tr_bf16_st rounds a block's
activations as its forward pass ends and tr_bf16_ld widens
them before its backward pass (the CPU's bits), and
tr_adamw16 updates BF16 moments with the CPU's stochastic
rounding (the same hash of seed, step, moment and element, in 64-bit
integers); the host's copies of the moments, the sidecar and resuming
are those of the CPU. With --grad-type bf16 the gradients
and their window are in device memory, tr_grad16_add
drawing the CPU's bits. The parameters stay F32.
Caveats:
- Resuming from a BF16 checkpoint is lossy for full
training: the trainer starts from the matrices rounded to BF16
(the moments, the step and the schedule are restored), so it trains on
as the saved trainer would from its rounded matrices, to the bit
(
test_bf16_checkpoint), not as the uninterrupted run. For LoRA the same holds for the adapters. Keep F32 checkpoints to resume a run exactly; BF16 ones to ship. - BF16 activations change the gradients by the rounding of what is kept (2^-9 relative per value); BF16 moments add the stochastic rounding's noise to the moments. Neither changes a step's arithmetic otherwise.
- The F32 moments of an older sidecar load into a trainer of BF16 moments (rounded to nearest), and BF16 ones into an F32 trainer (widened).
Measured (2026-11-07, --backend cpu, 8 threads, the
machine shared with other jobs so the step times are not comparable).
OKLM-Nano as below (GPT-2,
6 layers, 256 channels, 16 x 256 tokens, the 2048-token tokenizer), 300
steps at 1e-3 with 100 warm-up steps and a cosine decay to 1e-4;
validation over 8 batches; memory as oklm train prints it,
and the process's peak RSS:
| Nano, 300 steps | Val 100 | Val 200 | Final val | Train, step 300 | Optimizer | Kept activations | All activations | Peak RSS | Checkpoint |
|---|---|---|---|---|---|---|---|---|---|
| F32 | 5.0133 | 4.6725 | 4.5278 | 4.3863 | 40.7 MiB | 576.4 MiB | 695 MiB | 813 MiB | 20.4 MiB |
--act-type bf16 |
5.0133 | 4.6714 | 4.5280 | 4.3864 | 40.7 MiB | 288.2 MiB | 502 MiB | 624 MiB | 20.4 MiB |
--opt-type bf16 |
5.0130 | 4.6703 | 4.5269 | 4.3855 | 20.3 MiB | 576.4 MiB | 695 MiB | 792 MiB | 20.4 MiB |
| all three | 5.0133 | 4.6713 | 4.5281 | 4.3864 | 20.3 MiB | 288.2 MiB | 502 MiB | 601 MiB | 10.3 MiB |
The BF16 checkpoint of the last run, evaluated as resumed (its rounded weights): 4.5281, the same as its F32 master weights. Moments rounded to nearest instead of stochastically (a build made for this comparison, not shipped) gave 4.5272: at the default beta2 of 0.95 each update of v is 5 % of it, ten times half a BF16 step, so nearest rounding does not stall it here; stochastic rounding is kept because it stays unbiased at a beta2 of 0.999 and beyond, where it would.
SmolLM2-135M (the F16 file, full fine-tune on the text, 4 x 128 tokens, 40 steps at 1e-4, 5 warm-up steps):
| SmolLM2-135M, 40 steps | Val 20 | Final val | Train, step 40 | Optimizer | Kept activations | All activations | Peak RSS | Checkpoint |
|---|---|---|---|---|---|---|---|---|
| F32 | 3.5355 | 3.5236 | 3.5516 | 1026 MiB | 562.7 MiB | 880 MiB | 3233 MiB | 515 MiB |
| all three | 3.5355 | 3.5235 | 3.5516 | 513 MiB | 281.4 MiB | 618 MiB | 2510 MiB | 258 MiB |
--recompute |
3.5355 | 3.5236 | 3.5516 | 1026 MiB | 33.8 MiB | 369 MiB | 2724 MiB | 515 MiB |
--recompute, activations and moments BF16 |
3.5355 | 3.5236 | 3.5516 | 513 MiB | 16.9 MiB | 353 MiB | 2193 MiB | 515 MiB |
The BF16 checkpoint resumed scores 3.5187 (its rounded weights; a fine-tune of 40 small updates moves the weights by less than a BF16 step, so rounding them is a change of the same order as the training). The losses are those of F32 to three or four digits; what changes is the memory: the optimizer and the kept activations halve, the checkpoint halves. With recompute the kept activations are already small (each block's input), so BF16 activations save little there; BF16 moments save 4 bytes a trainable parameter whatever the mode, which matters most for a full fine-tune (513 MiB of SmolLM2's 3.2 GB here; for Llama-3.2-1B, 4.9 GB). Gradients stay F32 (4 bytes a trainable parameter).
On the GPU (2026-11-08, --backend rocm, gfx1100):
SmolLM2-135M (the F16 file) fully fine-tuned on the text,
--batch 4 --seq 256 --steps 300 --warmup 20 --lr 1e-4,
validation every 50 steps; memory as oklm train prints it
(device memory), the mean step:
SmolLM2-135M, 300 steps, rocm |
Validation 0 / 50 / 100 / 150 / 200 / 250 / 300 | Optimizer | Kept activations | All activations | Step |
|---|---|---|---|---|---|
| F32 | 3.5508 / 3.2509 / 3.1891 / 3.1605 / 3.1462 / 3.1377 / 3.1329 | 1026 MiB | 1261 MiB | 1689 MiB | 127 ms |
--act-type bf16 --opt-type bf16 |
3.5508 / 3.2509 / 3.1895 / 3.1612 / 3.1464 / 3.1379 / 3.1335 | 513 MiB | 630 MiB | 1101 MiB | 129 ms |
and --matmul-precision bf16 |
3.5516 / 3.2578 / 3.1912 / 3.1646 / 3.1471 / 3.1377 / 3.1356 | 513 MiB | 630 MiB | 1101 MiB | 111 ms |
BF16 gradients (2026-11-11, the same run, oklm train
printing the gradients' memory too; step times are the mean over the 300
steps of this session's runs, F32 again for the baseline):
SmolLM2-135M, 300 steps, rocm |
Validation 0 / 50 / 100 / 150 / 200 / 250 / 300 | Gradients | Optimizer | Kept activations | Step |
|---|---|---|---|---|---|
| F32 | 3.5508 / 3.2509 / 3.1891 / 3.1605 / 3.1462 / 3.1377 / 3.1329 | 513.1 MiB | 1026 MiB | 1261 MiB | 127 ms |
--grad-type bf16 |
3.5508 / 3.2509 / 3.1891 / 3.1605 / 3.1462 / 3.1376 / 3.1329 | 270.1 MiB | 1026 MiB | 1261 MiB | 132 ms |
--act-type bf16 --opt-type bf16 --grad-type bf16 |
3.5508 / 3.2509 / 3.1892 / 3.1611 / 3.1464 / 3.1379 / 3.1334 | 270.1 MiB | 513 MiB | 630 MiB | 135 ms |
The gradients' 270.1 MiB are 256.6 MiB of BF16 values and a 13.5 MiB window (one block of SmolLM2's 30, 6146 x 576 floats: the vocabulary tiles of 6146 rows fit it). BF16 gradients leave the curve on F32's to the fourth decimal; with all three the gradients, moments and kept activations take 1413 MiB instead of 2799, for 6 % more time a step (the roundings in and out).
The curve follows F32's to the third decimal, as on the CPU; the
device memory drops by 1.1 GB (of 3.2 GB of gradients, moments and
activations), for two more launches a block and step time within the
noise. With the products on 16 bits too, every large tensor but the
master weights and the gradients is 16-bit and the step 13 % faster. The
gradients can be BF16 too (--grad-type bf16, below the
table): they are accumulated (the weight products add into them, as do
micro-batches and the norm and bias sums), and BF16 sums of many small
terms would lose them, so each region of them is accumulated in an F32
window and added into the BF16 gradients once, with stochastic
rounding.
OKLM-Nano on Tiny Shakespeare
The command of oklm train above: a
2048-token BPE tokenizer trained on the text (1.1 MB, 388 533 tokens,
the last 10 % held out), a GPT-2 of 6 layers, 256 channels, 8 heads and
a 256-token context (5.3 million parameters), 1200 steps of 16 x 256
tokens (about 18 epochs), AdamW at 1e-3 with 100 warm-up steps and a
cosine decay to 1e-4, weight decay 0.1, clipping at 1.0, no dropout. 18
minutes on 8 threads.
| Step | 0 | 100 | 200 | 400 | 600 | 800 | 1000 | 1200 |
|---|---|---|---|---|---|---|---|---|
| validation loss | 7.646 | 5.013 | 4.671 | 4.362 | 4.166 | 4.075 | 4.025 | 3.988 |
3.99 nats per token is 1.39 per character (0.348 tokens per
character), next to nanoGPT's character-level model of this text (10.7
million parameters with dropout, 1.47).
oklm run -m build/nano/nano-001200.gguf -p "ROMEO:" -n 80 --temp 0.8 --seed 7:
ROMEO:
O, with his grave, and with a din
Must, as I have a painted to do.
ROMEO:
O, nurse; for I pray you; if you shall
Toof, if you will I have: but the
mory, I will take you have bird,
doot it not be gone to my side
With dropout, past the point of overfitting
The same model and settings for 3000 steps (about 45 epochs) on the
GPU (57 ms a step), without and with --dropout 0.2:
| Step | 1000 | 1500 | 2000 | 2500 | 3000 |
|---|---|---|---|---|---|
| no dropout | 4.135 | 4.280 | 4.755 | 5.227 | 5.546 |
| dropout 0.2 | 4.167 | 4.046 | 4.018 | 4.019 | 4.036 |
Without dropout the model learns the training text by heart after about 15 epochs and its validation loss climbs; with it, the loss settles near 4.02.
With Llama blocks
The same tokenizer, data and optimizer settings, with
--arch llama --kv-heads 4: 6 layers, 256 channels, 8 query
heads over 4 key/value heads, SwiGLU MLP of 704, rotary positions (4.95
million parameters). With the 1200-step schedule it reaches 3.865 at
step 600 and then overfits (training loss 2.04, validation 4.22 at step
1200: no dropout), so the run is 600 steps with the schedule ending
there. 7.5 minutes on 8 threads.
| Step | 0 | 100 | 200 | 300 | 400 | 500 | 600 |
|---|---|---|---|---|---|---|---|
| validation loss | 7.621 | 4.747 | 4.267 | 4.076 | 3.954 | 3.868 | 3.832 |
3.83 nats per token, 1.33 per character, in half the steps of the
GPT-2 block. llama.cpp loads the checkpoint as it is, and its greedy
continuation of "ROMEO:" is the same as oklm run's.
oklm run -m build/nano-llama/nano-000600.gguf -p "ROMEO:" -n 80 --temp 0.8 --seed 7:
ROMEO:
Out to them!
Nurse:
A noble boy!
Nurse:
I know it is, sir.
MERCUTIO:
Fell me, sir.
The v0.1.0 models
The models of release-notes-0.1.0.md, trained on
one AMD gfx1100 GPU through the OpenCL backend. The data
(tools/prepare_release_data.py) and the shared 16 384-token
tokenizer
(oklm train-tokenizer --vocab 16384 --pre llama-bpe, with
<|endoftext|>, <|im_start|>,
<|im_end|> and the ChatML template of
tools/oklm-chatml.jinja) live in
models/datasets/oklm-release/ ($R below). They
are not kept in the tree: tools/fetch_datasets.sh --release
downloads the sources and
tools/prepare_release_data.py <datasets dir> models/datasets/oklm-release
writes the texts again; the tokenizer is released
(oklm-tok-16k-0.1.0.gguf), so a later release copies it
rather than training another (tools/make_release.sh reads
it from $R/oklm-tok-16k.gguf).
OKLM-Micro, 33.6 M parameters, on TinyStories (98 M tokens seen, about 1 hour), then tuned on SmolTalk (24.6 M tokens, 35 minutes). The checkpoint before the tune is released too, as OKLM-Micro-Stories:
oklm train -f $R/tinystories.txt --tokenizer $R/oklm-tok-16k.gguf --out micro/oklm-micro \
--arch llama --layers 8 --embd 512 --heads 8 --kv-heads 4 --ff 1536 --ctx 512 \
--seq 512 --batch 16 --steps 12000 --lr 6e-4 --lr-min 6e-5 --warmup 500 --wd 0.1 --clip 1 \
--val-frac 0.005 --eval-every 1000 --save-every 2000 --backend opencl
oklm train -m micro/oklm-micro-012000.gguf -f $R/smoltalk.txt --out micro-chat/oklm-micro-chat \
--seq 512 --batch 16 --steps 3000 --lr 3e-4 --lr-min 3e-5 --warmup 100 --wd 0.1 --clip 1 --backend opencl
OKLM-Small, 88.1 M parameters, on FineWeb-Edu (229 M tokens seen, 6.7 hours at 0.85 s per step), then tuned on SmolTalk (20.5 M tokens, 35 minutes):
oklm train -f $R/fineweb.txt --tokenizer $R/oklm-tok-16k.gguf --out small/oklm-small \
--arch llama --layers 12 --embd 768 --heads 12 --kv-heads 4 --ff 2048 --ctx 1024 \
--seq 1024 --batch 8 --steps 28000 --lr 6e-4 --lr-min 6e-5 --warmup 1000 --wd 0.1 --clip 1 \
--val-frac 0.002 --eval-every 2000 --save-every 2000 --backend opencl
oklm train -m small/oklm-small-028000.gguf -f $R/smoltalk.txt --out small-chat/oklm-small-chat \
--seq 1024 --batch 8 --steps 2500 --lr 2e-4 --lr-min 2e-5 --warmup 100 --wd 0.1 --clip 1 --backend opencl
| Validation loss | start | end |
|---|---|---|
| Micro on TinyStories | 9.83 | 1.39 |
| Micro tuned on SmolTalk | 7.68 | 2.93 |
| Small on FineWeb-Edu | 9.86 | 3.51 |
| Small tuned on SmolTalk | 3.64 | 2.37 |
Longer conversation tunes
The released Micro and Small continued on the same SmolTalk text for
6 000 more steps each (2026-11-07), from their F16 files (a fresh
optimizer), at 1e-4 with 100 warm-up steps and cosine decay to 1e-5, the
release's batches, the same held-out tenth. The text is made again with
sh tools/fetch_datasets.sh --release smoltalk (the first
smol-smoltalk shard at a pinned revision, its sha256 checked) and
tools/prepare_release_data.py models/datasets/sources models/datasets/oklm-release
(a source not fetched is left out):
oklm train -m OKLM-Small-0.1.0-f16.gguf -f $R/smoltalk.txt --out small-chat2 --seq 1024 --batch 8 --steps 6000 \
--lr 1e-4 --lr-min 1e-5 --warmup 100 --wd 0.1 --clip 1 --eval-every 1000 --recompute --backend opencl
oklm train -m OKLM-Micro-0.1.0-f16.gguf -f $R/smoltalk.txt --out micro-chat2 --seq 512 --batch 16 --steps 6000 \
--lr 1e-4 --lr-min 1e-5 --warmup 100 --wd 0.1 --clip 1 --eval-every 1000 --backend opencl
| Held-out loss, step | 0 | 1 000 | 2 000 | 3 000 | 4 000 | 5 000 | 6 000 | time |
|---|---|---|---|---|---|---|---|---|
| Small (0.92 s a step) | 2.262 | 2.304 | 2.296 | 2.262 | 2.235 | 2.222 | 2.217 | 92 min |
| Micro (0.27 s a step) | 2.993 | 3.010 | 2.956 | 2.915 | 2.877 | 2.857 | 2.844 | 27 min |
(The step-0 figures are the released models on this split, measured over 8 batches; the release's table measured fewer.) The loss rises for the first thousand steps, the fresh optimizer's moments settling at the higher rate, then falls below the start: 0.045 for Small, 0.149 for Micro, which had seen 25 M conversation tokens against Small's 20 M and has more to gain per token. Small's greedy answers read the same before and after (fluent lists that lose the thread: "Take regular breaks... make sure your body is getting enough sleep"); the tune does not buy facts the pretraining lacks. The checkpoints are not released.
tools/make_release.sh 0.1.0 NANO MICRO SMALL STORIES
writes each model in F16 and Q8_0 with its name, version, licence and
data in general.*
(oklm quantize --override-kv), checks that each loads and
generates, and adds the tokenizer, the template, the notes, the licence
and SHA256SUMS in build/release-0.1.0/.
llama.cpp loads and runs the files as well.
Fine-tuning SmolLM2-135M
The Hugging Face checkpoint loaded directly (30 layers, 576 channels,
9 query heads over 3 key/value heads, tied output head), the
oklm train command above: 30 updates of 4 batches of 4 x
128 tokens of Tiny Shakespeare (its own tokenizer: 341 094 tokens),
AdamW at 1e-4, 5 warm-up steps, cosine decay to 1e-5. 4.5 minutes.
| Step | 0 | 10 | 20 | 30 |
|---|---|---|---|---|
| validation loss | 3.810 | 3.524 | 3.446 | 3.430 |
The checkpoint is a Llama GGUF in F32 (540 MB, optimizer sidecar 1.08
GB). llama.cpp's greedy continuation of "KING HENRY:" on it is the same
as oklm run's, token for token, which checks the reordering
of the query and key rows end to end.
LoRA on SmolLM2-135M
The published F16 GGUF loaded directly, rank 8, alpha 16, every
projection (210 adapters, 2 442 240 parameters), batches of 8 x 128
tokens of Tiny Shakespeare, AdamW at 1e-3, 20 warm-up steps, cosine
decay, --backend rocm: 200 steps in 30 s.
| Step | 0 | 50 | 100 | 150 | 200 |
|---|---|---|---|---|---|
| validation loss | 3.834 | 3.418 | 3.337 | 3.304 | 3.297 |
The CPU, opencl and rocm give the same
losses, to the fourth decimal, over the first steps. llama.cpp's
perplexity on the first 60 000 bytes of the text, 256-token windows:
34.96 for the model, 27.399 with the adapter
(llama-perplexity --lora), 27.399 on the model
oklm lora-merge wrote (F16 weights);
oklm ppl --lora 35.07 to 27.48 (its windows are scored
differently).
A step of 8 x 128 tokens:
| Backend | Every parameter | LoRA, every projection | LoRA, q, k, v, o |
|---|---|---|---|
opencl |
156 ms | 140 ms | 119 ms |
rocm |
110 ms | 102 ms | 91 ms |
cpu |
3.3 s |
The model's weight gradients skipped save a third of the large products; the adapters add six small products each (1260 a step for every projection), which the split products keep to a few microseconds each.
Fine-tuning Qwen2.5-0.5B and Qwen3-0.6B
The published GGUF files loaded directly (Qwen2.5-0.5B-Instruct in Q4_K_M, Qwen3-0.6B in Q8_0, both converted to F32), on the GPU: 30 updates of 4 batches of 4 x 128 tokens of Tiny Shakespeare (the models' own tokenizer: 301 829 tokens), AdamW at 1e-4, 5 warm-up steps, cosine decay to 1e-5.
oklm train -f data/tinyshakespeare.txt -m Qwen3-0.6B-Q8_0.gguf --out build/ft-qwen3/ft \
--steps 30 --batch 4 --accum 4 --seq 128 --lr 1e-4 --lr-min 1e-5 --warmup 5 --eval-every 10| Model | Step 0 | 10 | 20 | 30 | Time |
|---|---|---|---|---|---|
| Qwen3-0.6B (16 heads of 128 over 1024 channels, Q/K norms) | 4.566 | 4.078 | 3.989 | 3.911 | 57 s |
| Qwen2.5-0.5B (Q/K/V biases) | 4.340 | 4.199 | 4.082 | 3.945 | 47 s |
The checkpoints are qwen3 and qwen2 GGUF
files in F32 (2.4 and 2.5 GB, optimizer sidecars twice that).
llama.cpp's greedy continuation of "KING HENRY:" on each is the same as
oklm run's, token for token, as is that of a 4-layer Qwen3
trained from scratch with --head-dim 48 over 128
channels.
Reinforcement learning
src/train/rl.c, oklm rl: fine-tuning from
rewards on tasks with known answers, with group-relative advantages
(GRPO), several updates per batch of samples with PPO's clipped ratio,
and a KL penalty towards a reference model.
oklm_trainer_generate()samples completions from the model being trained (temperature, or the most likely token at 0), evaluating the whole sequence again for each new token: meant for the short completions of such tasks.oklm_trainer_last_logits()fetches only the last position's logits, so a device does not send back the whole logits.oklm_rl_step(): for each task, a group of completions is sampled and scored by the caller's reward; each completion's advantage is its reward minus the group's mean, over the group's standard deviation. The loss is minus the advantage times the log-probability of each sampled token, averaged over the completion's tokens, the group and the tasks: the cross-entropy gradient of the sampled tokens weighted by their advantage,oklm_trainer_backward_weighted()(per-row weights, negative ones included; weights of 1 / N giveoklm_trainer_backward()bit for bit). One AdamW update follows by default. Groups whose rewards are all equal carry no advantage; a step with no term at all makes no update.- Several updates per batch
(
oklm_rl_params.updates,--updates): the samples serve that many AdamW updates. The first records each sampled token's log-probability under the sampling policy (from the same forward pass,oklm_trainer_row_losses()); the later ones weigh a token by its advantage times its probability ratio to that policy, and by nothing once the ratio left 1 +/-clip(0.2,--clip) on the advantage's side: the gradient of PPO's clipped surrogate. With one update the ratio is 1 and the weights are the advantages alone, as before. In the test, four updates a batch learn the task in 26 sampling steps against 81 with one (lr 5e-3; at 2e-2 four overshoot and collapse onto wrong answers, which one update does not). - KL penalty (
oklm_rl_params.klandref,--kl,--ref): each sampled token addsklx (ref/pi - log(ref/pi) - 1), the estimate whose expectation under the policy is KL(pi || ref), positive and zero where the two agree; its gradient in the token's log-probability is kl x (1 - ref/pi). The reference is another trainer, only evaluated (oklm rlbuilds it from--ref, or the starting model, behind rank-1 LoRA adapters whose B starts at zero: it computes the model itself without a full copy's optimizer state). Groups with equal rewards still carry the KL term.oklm_rl_stats.klreports the mean estimate. In the test, 40 steps without the penalty end 3.34 nats from the start (exact KL of the next token), withkl1 at 1.12. oklm rl -m model -f tasks.jsonl --out prefix: one{"prompt": ..., "answer": ...}a line, a share held out (--val-frac) whose greedy accuracy is printed every--eval-everysteps. A completion earns 1 when its last number is a numeric answer ("13 + 6 = 19." answers 19), or when it starts with any other answer. Checkpoints are those ofoklm train.
oklm rl -m SmolLM2-135M-Instruct-f16.gguf -f additions.jsonl --out build/rl/add \
--steps 150 --tasks 16 --group 8 --max-new 12 --lr 2e-5 --eval-every 25SmolLM2-135M-Instruct on the 400 sums of two numbers from 0 to 19, asked through its chat template ("What is 13+6?"), 40 held out, on the GPU (150 steps of 16 tasks and groups of 8, 12 new tokens, 13.5 minutes):
| Step | 0 | 25 | 50 | 100 | 125 | 150 |
|---|---|---|---|---|---|---|
| held-out greedy accuracy | 0.025 | 0.825 | 0.800 | 0.875 | 0.900 | 0.925 |
| training reward (sampled) | 0.195 | 0.734 | 0.805 | 0.930 | 1.000 | 0.945 |
With two updates a batch and the KL penalty
(--updates 2 --kl 0.04, the same tasks, 30 steps, 225 s on
the GPU), the held-out accuracy reaches 0.875 at step 10 and 0.900 at
step 30; the KL estimate starts at 0 (the reference is the starting
model) and stays near 0.3, and 1 to 9 % of the second update's tokens
are clipped, fewer as the policy settles.
The base model starts with an explanation it rarely finishes ("To
find 17 + 15, we need to add..."); the checkpoint, through
oklm run, answers "17 + 15 = 32". A run gives the same
steps on the same device. The test (test_train) has a
freshly initialized model learn, from rewards alone, which of 37 tokens
answers each of 8 prompts (4.7 % reward at the first step, all 8
answered greedily after about 30).
Rewards teach only what the model sometimes does already: on additions of two-digit numbers, where SmolLM2-135M's samples are right less than once in 600, no group ever differs and nothing is learned. Not done: a reward computed by a command.
Abliteration
src/model/abliterate.c, oklm abliterate: a
chat model's refusal direction found and projected out of its weights
(Arditi et al., 2024), pure C on the CPU, the same loaders and writers
as oklm convert. It makes the trainable base the
fine-tuning runs of "Default models" start from, without Python or
PyTorch.
- The direction. Two sets of instructions, one the
model declines and one it answers, are run through it in its chat
template. At each prompt's last token the residual stream entering every
layer is read: a matrix product observer (the importance-matrix
mechanism of
oklm_imatrix_compute()) catches the input of each block's first attention product, RMSNorm(h) * g, and divides it back by the norm's weights g to recover h / rms(h), so a prompt's scale does not sway it. Per layer, the candidate direction is the difference of the two sets' mean residuals, normalized.oklm_refusal_directions()returns one per layer with two figures: the separation (the gap of the two sets' projections on the direction over their pooled deviation) and the gap (the mean difference's norm relative to the residual's). The layer that separates best within a band of the depth (0.3 to 0.7 by default) is chosen,oklm_refusal_pick_layer(). - The weights.
oklm_abliterate_write()makes every matrix that writes into the residual stream orthogonal to the unit direction r: W' = W - s r (r^T W) for each block'sattn_outputandffn_down(the experts'ffn_down_expsandffn_down_shexpincluded), and the token embeddings e' = e - s r (r . e) when the model has anoutput.weightof its own (a tied head is left, since it also reads the residual). Each is produced as the file is written, dequantized, changed and converted back, rows over the CPU threads; a matrix kept in F32, F16 or BF16 stays its type, else it becomes F16, and every other tensor is copied. The scale s (1 by default, 0 to 2) weakens the removal below 1. The result is trainable weights (or an F16/BF16 GGUF) whose making OKLM controls. - Measuring.
oklm refusals -m model --prompts refused.txtanswers each instruction greedily and counts those that read as a refusal (oklm_is_refusal(), the usual markers);--ref base --harmless answered.txtadds the mean KL divergence of the model's first-token distribution from the reference's on instructions it should answer, which measures how much the projection moved its behaviour elsewhere.-n Nis the length of each answer in tokens (48),--limit Nreads the first N lines of each file only, and--showprints every answer as it is judged:REFUSEDoranswered, the prompt, then the answer on one line, before the total.
tools/fetch_abliterate_data.sh fetches two prompt sets
at pinned commits and splits each into a part for the direction and a
held-out part for the measures: AdvBench (Zou et al.) for the declined
set, Alpaca instructions without an input for the answered one.
sh tools/fetch_abliterate_data.sh
oklm abliterate -m models/qwen3-8b/Qwen3-8B-BF16.gguf \
--harmful models/abliterate/harmful_train.txt \
--harmless models/abliterate/harmless_train.txt --out Qwen3-8B-abliterated.gguf
oklm refusals -m Qwen3-8B-abliterated.gguf --prompts models/abliterate/harmful_test.txt
oklm refusals -m Qwen3-8B-abliterated.gguf --ref models/qwen3-8b/Qwen3-8B-BF16.gguf \
--harmless models/abliterate/harmless_test.txtOn Qwen3-8B (BF16, 36 layers), the direction found from the 416 declined and 416 answered training prompts separates the two sets ever more deeply: the picker's band (0.3 to 0.7 of the depth) takes layer 24 (separation 11.2, the mean difference 0.71 of the residual there). With its direction removed at scale 1, the held-out refusal rate (104 declined instructions, greedy) falls from 97.5 % to 7.5 %, near the published figures; but the first-token KL divergence from the base on held-out answered instructions is 1.29, so the projection also moves answered replies. The removal is a trade-off along the depth, measured here with 24 prompts of each held-out set:
| layer | separation | refusals | first-token KL (answered) |
|---|---|---|---|
| base | — | 97.5 % | 0 |
| 14 | 3.3 | 79 % | 1.07 |
| 16 | 4.4 | 62 % | 0.18 |
| 19 | 7.6 | 38 % | 0.81 |
| 24 (picked) | 11.2 | 7.5 % | 1.29 |
The strongest refusal removal costs the most on answered prompts; a
shallower layer, or --scale below 1, keeps more of the
base's behaviour for less refusal reduction. The operating point is the
security training's to choose. On Qwen3-0.6B (Q8_0, 28 layers) the
middle layers separate best; with the chosen layer's direction removed
the held-out refusal rate goes from a near-half to none while the KL on
answered instructions stays small. The per-layer table and the 0.6B
figures are checked in test_infer_public's
test_abliterate, which also confirms that the written
model's own mean differences keep almost nothing along the removed
direction. Computing the 8B directions on the CPU over 832 prompts takes
about half an hour; --save-dirs writes them so
--load-dirs --layer N writes a model at any layer or scale
in seconds without recomputing.
Machine-readable progress
oklm train, oklm rl,
oklm abliterate and oklm refusals take
--progress-json FILE, and so do oklm convert,
oklm quantize and oklm ppl (their lines are
the last two tables): the run written as JSON lines, one object a line,
each line flushed as it is written, so a program can follow a run from
the file (or a pipe) while it goes. With --progress-json -
the lines go to standard output and everything the command otherwise
prints there (the report, the samples, the tables) goes to standard
error, so standard output holds JSON lines only. Without the option
nothing changes. The training window follows a run through such a file
(oklm-gui --tool train, gui.md). The file is truncated
when the command starts, after --out's directories are
made; a file that cannot be opened stops the command (status 1). The
writer is oklm/io/progress.h.
$ oklm train -f data/tinyshakespeare.txt --out nano/nano --vocab 512 --layers 2 --embd 64 --heads 2 --ctx 64 \
--steps 4 --eval-every 2 --save-every 2 --backend cpu --progress-json - 2>/dev/null
{"type":"start","elapsed":0.432,"command":"train","mode":"scratch","model":null,"resume":null,"out":"nano/nano","arch":"gpt2","parameters":136960,"trainable":136960,"layers":2,"embd":64,"heads":2,"vocab":512,"context":64,"device":"cpu","total":4,"first_step":0,"batch":4,"seq":64,"accum":1,"lr":0.0003,"lr_min":3e-05,"warmup":50,"weight_decay":0.1,"grad_clip":1,"dropout":0,"seed":1337,"lora_rank":0,"eval_every":2,"save_every":2,"sample_every":0,"data":"data/tinyshakespeare.txt","data_kind":"text","tokens":575809,"train_tokens":518229,"val_tokens":57580}
{"type":"step","elapsed":0.499,"step":1,"total":4,"loss":6.232001,"val_loss":6.244404,"lr":6e-06,"grad_norm":1.616281,"step_seconds":0.06725377,"tokens_per_second":3806.447,"eta_seconds":0.201763}
{"type":"step","elapsed":0.514,"step":2,"total":4,"loss":6.24755,"lr":1.2e-05,"grad_norm":1.521801,"step_seconds":0.01515267,"tokens_per_second":6211.878,"eta_seconds":0.08242274}
{"type":"checkpoint","elapsed":0.515,"path":"nano/nano-000002.gguf","step":2}
{"type":"step","elapsed":0.554,"step":3,"total":4,"loss":6.221951,"val_loss":6.243486,"lr":1.8e-05,"grad_norm":1.663938,"step_seconds":0.03843528,"tokens_per_second":6308.998,"eta_seconds":0.04057697}
{"type":"step","elapsed":0.561,"step":4,"total":4,"loss":6.223746,"lr":2.4e-05,"grad_norm":1.538278,"step_seconds":0.007306504,"tokens_per_second":7934.69,"eta_seconds":0}
{"type":"checkpoint","elapsed":0.582,"path":"nano/nano-000004.gguf","step":4}
{"type":"end","elapsed":0.582,"ok":true,"step":4,"total":4,"loss":6.223746,"val_loss":6.241247,"files":["nano/nano-000002.gguf","nano/nano-000004.gguf"]}
Every line is a JSON object on one line, in UTF-8,
holding first type (a string, the tables below) then
elapsed (seconds since the command opened the stream, three
decimals), then the fields of its type in the tables' order. Numbers
have 7 significant digits; a number that is not finite (a loss that
diverged) is null. A string field with no value (no
--resume, no --system) is null.
Bytes of a text that are not UTF-8 (a sample cut inside a character) are
written as U+FFFD. A reader should ignore the fields and the types it
does not know: later versions may add some.
The order. A run that gets as far as its model
writes one start line first (abliterate writes
it before loading anything). The last line is always end,
once, whether the run succeeded or not: a run that fails before
start (a file that cannot be read, a model that cannot be
loaded) writes end alone. A stream without an
end line is a command that was killed. An error in the
options (the usage, status 2) is found before the stream is opened, and
writes nothing.
Lines of every command
| Type | Field | Value |
|---|---|---|
end |
ok |
true when the command succeeded (exit status 0) |
error |
the error's text, only when ok is false
(the detailed message is on standard error) |
|
| figures | the command's final figures, in its table below | |
files |
an array of the paths of the files the command wrote, in the order written, each once; empty when none |
oklm train
| Type | Field | Value |
|---|---|---|
start |
command |
"train" |
mode |
"scratch" (a new model), "finetune"
(-m), "lora" (-m with
--lora) or "resume"
(--resume) |
|
model, resume, out |
the options' values (null when not given) |
|
arch |
the architecture (gpt2, llama,
qwen2, qwen3, ...) |
|
parameters, trainable |
the model's parameter count, and how many are trained (the adapters' with LoRA) | |
layers, embd, heads,
vocab, context |
the model's shape | |
device |
where it trains: cpu, opencl,
rocm |
|
total |
the step the run ends at | |
first_step |
updates already made (0, or a resumed checkpoint's step) | |
batch, seq, accum |
sequences a batch, tokens a sequence, batches an update | |
lr, lr_min, warmup,
weight_decay, grad_clip, dropout,
seed |
the options in effect (a resumed run's come from its sidecar) | |
lora_rank, lora_alpha |
the adapters' rank (0 without LoRA) and, with LoRA, their alpha | |
eval_every, save_every,
sample_every |
the three periods, in steps (0: never) | |
data, data_kind |
the data file, and "text" (-f) or
"chat" (--chat-data) |
|
tokens, train_tokens,
val_tokens |
with "text": the text's tokens, and the two shares |
|
conversations, train_conversations,
val_conversations, assistant_tokens |
with "chat": the conversations, the two shares, the
tokens the loss is taken on |
|
step |
step, total |
updates made, this one included, and the step the run ends at |
loss |
the training loss of this update's batches (on the assistant tokens
with "chat") |
|
val_loss |
the validation loss, only on the steps it was measured at (every
eval_every, before the update: loss and
val_loss of a line are of the same weights) |
|
lr, grad_norm |
the learning rate of the update, the gradient's norm before clipping | |
step_seconds |
the step's time, its evaluation and sample included | |
tokens_per_second |
tokens trained on a second, since the run's first step | |
eta_seconds |
an estimate of the seconds left: the mean time of the steps made (evaluations and checkpoints included) times the steps left | |
loss_scale, skipped |
with --loss-scale: the scale and the updates skipped so
far |
|
sample |
step, text |
with --sample-every: the updates made when the sample
was taken, and its text |
checkpoint |
path, step |
a checkpoint written (its .train sidecar beside it),
and its step |
end |
step, total |
updates made and planned, when the trainer was set up |
loss |
the last step's training loss, when a step was made | |
val_loss |
the final validation loss, when there is validation data |
oklm rl
| Type | Field | Value |
|---|---|---|
start |
command |
"rl" |
model, resume, out |
the options' values (null when not given) |
|
arch, parameters, layers |
the model | |
device |
where it trains | |
data, tasks, train_tasks,
val_tasks |
the task file, its tasks, those trained on and those held out | |
total |
the steps of this run (--steps) |
|
first_update |
updates the model had (0, or a resumed checkpoint's) | |
tasks_per_step, group,
max_new, temperature |
--tasks, --group, --max-new,
--temp |
|
lr, weight_decay, updates,
clip, kl, seed |
the options in effect | |
ref |
the model the KL penalty is taken against (null without
--kl) |
|
eval_every, save_every |
the two periods, in steps | |
step |
step, total |
steps of this run made, this one included, and planned |
reward, solved, length |
the mean reward, the share of completions rewarded, their mean length in tokens | |
learned |
tasks whose completions differed in reward, so gave a gradient | |
grad_norm, lr |
the gradient's norm (0 when no update was made), the learning rate | |
accuracy |
the held-out accuracy (greedy), only on the steps it was measured at, before the update | |
kl |
with a reference: the mean KL divergence from it a sampled token | |
updates, clipped |
with --updates above 1: the updates made, the share of
tokens whose ratio was clipped |
|
step_seconds, samples_per_second |
the step's time (its evaluation included), the completions sampled a second in it | |
eta_seconds |
an estimate of the seconds left, as for train |
|
checkpoint |
path, step |
a checkpoint written; step is the model's update count,
as in the file's name (a step without signal makes no update) |
end |
step, total |
steps made and planned, when the trainer was set up |
reward |
the last step's mean reward, when a step was made | |
accuracy |
the final held-out accuracy, when tasks are held out |
oklm abliterate
| Type | Field | Value |
|---|---|---|
start |
command |
"abliterate" |
model, harmful, harmless,
load_dirs, save_dirs, out,
system |
the options' values (null when not given) |
|
layer |
--layer as given: "auto" or a number, as a
string |
|
band_low, band_high, scale,
limit |
--band, --scale, -n (0: every
prompt) |
|
device |
"cpu" |
|
stage |
stage |
a stage begins: "directions" (the prompts run through
the model, the long one) or "write" (the model
written) |
harmful, harmless,
layers |
with "directions": the prompts of each set, the model's
layers |
|
directions |
harmful, harmless, layers,
embd, seconds |
the directions are measured: the prompts used, the layers, the width, the time taken |
layer |
layer, layers |
one line a layer, in order: its index and the layer count |
separation, gap |
the two figures of the table: how far apart the two sets' projections are in standard deviations, and the gap of their means relative to the residual's norm | |
chosen |
layer, auto |
the layer whose direction is used, and whether
--layer auto chose it |
separation, gap |
its figures (not with --load-dirs) |
|
file |
path, kind |
a file written: "directions" (--save-dirs)
or "model" (--out) |
seconds |
with "model": the time the writing took |
|
end |
layer |
the layer used, when the run succeeded |
With --load-dirs the stream is start,
chosen, stage ("write"),
file, end.
oklm refusals
| Type | Field | Value |
|---|---|---|
start |
command |
"refusals" |
model, arch, layers |
the model judged | |
prompts, total |
the prompt file (null without --prompts)
and its prompts |
|
ref, harmless,
harmless_total |
the reference model, the answered prompts and their count
(null, null, 0 without them) |
|
max_tokens, limit,
system |
-n, --limit (0: every prompt),
--system |
|
device |
the backend asked for ("auto" when
--backend is not given) |
|
prompt |
index, total |
one line a prompt judged: its number from 1, and their count |
refused |
whether the answer is a refusal | |
refused_so_far |
refusals among the prompts judged so far | |
prompt, answer |
the prompt and the answer's first max_tokens
tokens |
|
stage |
stage, total |
"kl": the first-token distributions of the two models
begin, over total answered prompts |
summary |
refused, total, rate |
with --prompts: the refusals, the prompts, and their
ratio (0 to 1) |
kl, kl_prompts |
with --ref: the mean first-token KL divergence from the
reference, and the prompts it is a mean of |
|
end |
refused, total, rate,
kl, kl_prompts |
the summary's figures again, when the run succeeded |
tests/harness/test_cli_progress.c checks the writer
(escapes, null, the files' list), then runs the four
commands: each line parses as JSON and holds its type's fields above,
the stream on standard output holds nothing else, and a run without the
option prints what it printed before.
oklm convert,
oklm quantize
start is written once the file's header is ready, before
the first tensor; a run that fails before (a source that cannot be read,
a tensor no type fits) writes end alone, and so does
--dry-run (convert.md).
| Type | Field | Value |
|---|---|---|
start |
command |
"convert" or "quantize" |
in, out |
the source and the file written | |
ftype |
the target type as given ("f16",
"Q4_K_M") |
|
tensors |
tensors of the file | |
bytes |
the size the file will have | |
tensor |
index, total |
the tensor just written, from 1, and their count |
name |
its name | |
bytes_done, bytes_total |
bytes of the file written so far, and its size | |
end |
dry_run |
true for --dry-run: nothing was
written |
tensors, bytes |
the file's tensor count and size, when ok |
oklm ppl
| Type | Field | Value |
|---|---|---|
start |
command |
"ppl" |
model, file |
the model and the text | |
ctx |
tokens a window | |
tokens |
tokens of the text | |
windows |
whole windows in it: the ones scored | |
window |
index, total |
the window just scored, from 1, and their count |
perplexity |
the perplexity over the windows so far | |
end |
perplexity |
the perplexity, when ok |
tokens, windows |
tokens scored (the second half of each window) and windows, when
ok |
Next
Longer runs of the released models (Small saw about 230 M tokens, several billion would suit its size).