OKLMdocumentation v1.0.3
On this page

Training

Training a transformer from scratch, fine-tuning a loaded model and resuming from checkpoints: the trainer's API, its kernels, checkpoint files, the oklm train command, and how the results are checked against llm.c.

Status: implemented for the GPT-2 architecture and the Llama family (Llama, Mistral, Qwen2, Qwen3), on the CPU and on OpenCL devices.

Trainer

include/oklm/train/train.h, sources in src/train/. A trainer holds every parameter, its gradient and the two AdamW moments in F32 flat buffers (four floats per parameter), and the activations of one batch of at most batch sequences of seq_len tokens, allocated once (oklm_trainer_activation_bytes(); oklm train prints it).

  • Activation recompute (recompute, oklm train --recompute): only each block's input is kept from the forward pass, with one set of the other activations shared by every block. Just before a block's backward pass its forward pass runs again from that input (the last block's activations are still in place). Dropout masks are keyed by the step, the batch and the site, so a recomputed block draws the same ones, and the blocks' gradients are those of keeping everything, to the bit, on the CPU, opencl and rocm (test_train.c and test_train_cl.c test_recompute).

  • The head in chunks (with recompute): the logits and probabilities are held for a chunk of rows at a time, about 2^26 floats of each (multiples of 64 rows; OKLM_TRAIN_LOGIT_ROWS sets it). The forward pass keeps the losses and each sequence's last logits (oklm_trainer_last_logits(); oklm_trainer_logits() and oklm_trainer_probs() give NULL after a chunked pass). A full fine-tune's backward pass computes each chunk again for the head's gradient. With LoRA the head is frozen, so the forward pass also takes each row's (p - onehot) W while the chunk is there, and the backward pass only weights those rows. The gradients are those of the whole head within float rounding (the head's sums go chunk by chunk; test_chunked_head). Together the activations shrink 3 to 8 times (the table under Speed).

  • A quantized base (quantized_base, oklm train --lora R --quantized-base): with LoRA, the blocks' projection matrices (Q, K, V, O, gate, up, down; rows of whole groups of 32) stay in the model file's types, on the trainer's device, instead of F32. Before each product, forward or backward (dX = dY W included), the matrix is unpacked into one F32 scratch, the size of the largest matrix: on a device by a kernel built from the inference engine's dequantizers (every type its products read), on the CPU by oklm_dequantize_row(). The device's unpacking gives the CPU's values to the bit for all 28 types (test_train_cl.c test_dequantize), so the products, losses and adapters are those of the F32 trainer to the bit, with or without recompute, on the CPU, opencl and rocm (test_quantized_base, on Q8_0 and Q4_K_M files that OKLM quantized). The token embeddings are packed too (not GPT-2's, whose lookup adds the positions): a token's row is unpacked as it is looked up. So is the output head, unpacked whole when it fits the scratch (the same bits), else a tile of vocabulary rows at a time (the scratch's size; OKLM_TRAIN_HEAD_ROWS sets it), each tile giving its columns of the logits, the gradient through the head summed tile by tile within float rounding (test_quantized_head). oklm_trainer_weight_bytes() gives what the model takes; oklm train prints it (the table under Speed).

  • BF16 storage (act_type, opt_type, grad_type, save_type; oklm train --act-type bf16 --opt-type bf16 --grad-type bf16 --save-type bf16), each opt-in, every computation still in F32. See BF16 storage for what each one stores, the measurements and the caveats. oklm_trainer_memory() gives the memory by kind (weights, gradients, optimizer, activations, and of those the activations kept for the backward pass); oklm train prints it.

  • oklm_trainer_new(): random weights, GPT-2's initialization for both architectures (normal with standard deviation 0.02, the attention and MLP output projections scaled by 1 / sqrt(2 n_layer), biases 0, norm scales 1).

  • oklm_trainer_from_model(): the weights of a loaded model (GGUF of any type, or a Hugging Face checkpoint), converted to F32; the output head is tied to the token embeddings unless the model has its own. A Hugging Face Llama checkpoint's query and key rows (each head split in two halves) are reordered into the interleaved pairs of GGUF Llama files, so its checkpoints are ordinary Llama files; its trainer gives the same logits, bit for bit, as the trainer of its own checkpoint.

  • oklm_trainer_forward() with targets gives the mean cross-entropy loss and keeps what the backward pass needs; without targets it gives logits (sampling during training uses it).

  • oklm_trainer_backward(scale) adds the gradient of the loss times scale to the gradients, so several batches can be accumulated (accum batches each scaled by 1 / accum give the gradient of their mean loss; oklm train --accum).

  • oklm_trainer_update(lr) clips the global gradient norm (when grad_clip is set) and applies one AdamW step, with decoupled weight decay on the matrices only, or on every parameter (decay_all, as llm.c does).

  • oklm_trainer_lr(step): linear warm-up, then cosine decay from lr to lr_min until steps.

  • dropout in the options: residual dropout, as nanoGPT's, on the embeddings and on the attention and MLP outputs before their residual adds, in training mode only (oklm_trainer_set_training(), on by default; oklm train evaluates in evaluation mode). The attention weights are not dropped. Each mask is a hash of the seed, the step, the batch within the update, the site and the element, recomputed by the backward pass: it does not depend on the thread count, a device draws the CPU's masks, and a resumed checkpoint draws those of the uninterrupted run (the probability is kept in the sidecar).

  • oklm_trainer_step() chains the four.

  • backend in the options: cpu (default), auto (the first OpenCL device, else the CPU) or an OpenCL instance (opencl, opencl1); oklm_trainer_backend() tells which one runs. See On an OpenCL device.

Architectures, with llama.cpp's tensor names so checkpoints are GGUF files of the same architecture:

  • GPT-2: learned positions, LayerNorm with biases, GELU MLP with the tanh approximation, biases on every projection.

  • Llama: rotary positions, RMSNorm, SwiGLU MLP, grouped-query attention (fewer key/value heads than query heads), no biases. Rotation of adjacent pairs (GGUF) or half-split pairs, of all or the first n_rot dimensions of each head, with Llama 3's frequency factors when the model has them (rope_freqs.weight, kept in the checkpoints). SmolLM2, TinyLlama and Llama 3.2 have this block. The head size is its own (head_dim x n_head need not be n_embd: Qwen3's 16 heads of 128 over 1024 channels).

  • Qwen2 (qwen2): the Llama block with biases on the Q, K and V projections, half-split pairs (Qwen2, Qwen2.5).

  • Qwen3 (qwen3): the Llama block with an RMSNorm of each query and key head (attn_q_norm, attn_k_norm, one scale of head_dim shared by the heads) between the Q/K/V product and the rotation, as the inference engine computes it.

    Sliding windows, experts and other scalings are refused (OKLM_ERR_UNSUPPORTED). Each block's Q, K and V projections are stored one after the other, as are their biases and its gate and up projections, so each group is one matrix product (and one bias).

Kernels

src/train/train_internal.h: explicit forward and backward functions over the preallocated buffers, in the manner of llm.c, run on the CPU worker team.

  • Matrix product (gemm_train.c): one form, C (+)= A B, with A read directly or transposed. The forward pass multiplies by the transposed weight (transposed once per call into a scratch buffer), the backward pass gives the input gradient (dout W) and adds the weight gradient (dout^T x). Tiles of 4 x 24 (AVX2) or 8 x 32 (AVX-512) outputs, blocks of 256 along the inner dimension. Each output is summed in one lane from k = 0 up, by a multiplication then an addition (no fused multiply-add): the scalar, AVX2 and AVX-512 variants give the same bits.
  • Attention: causal, softmax of scaled dot products, the weights kept for the backward pass (batch x heads x T x T floats per layer). The packed Q/K/V layout takes fewer key/value heads than query heads, for grouped-query attention.
  • Rotary positions: the query and key heads of the packed rows rotated in place after the Q/K/V product, and rotated back (the transposed rotation) on their gradients after the attention's backward pass. The cosines and sines are tabulated once, with the inference engine's arithmetic (the angle multiplied by base^(-2 / n_rot) from one pair to the next).
  • Per-head Q/K norms (Qwen3): each query and key head of the packed rows normalized over its head_dim values, the value heads copied; the Q/K/V product before it is kept for the backward pass, which replaces the heads' gradients in place after the rotation's. The two scale gradients are summed per channel over the rows then the heads in order (on a device by chunks of 64 rows, then the chunks in order).
  • Norms (LayerNorm, and RMSNorm without a bias), GELU, SwiGLU (silu(gate) x up over the gate and up halves of one product), softmax with cross-entropy (the logit gradients replace the probabilities in place; each row's normalizer is summed in double, which brings the loss of GPT-2 124M to PyTorch's within 1e-6).

Determinism: rows are split between threads, sums over rows (weight and bias gradients, embedding gradients) are split by blocks of channels with the rows walked in order, the gradient norm is summed in fixed chunks. A run gives the same parameters, bit for bit, with any thread count and at any SIMD level (test_train), for both architectures.

On an OpenCL device

src/train/train_cl.c and src/train/train_kernels.cl: the same kernels as OpenCL C, built on the device, context and queue of an OpenCL backend instance (ADR-0017), with the CPU kernels' arguments. Device memory is handed out as fake addresses, so the trainer's layout and pointer arithmetic are unchanged and each kernel call goes to the CPU or the device.

  • The parameters, gradients, AdamW moments and activations stay on the device. Per step only the token ids go up and the row losses come back; the gradient norm is summed on the device in chunks of 4096 and the partial sums added on the host in double.
  • The host keeps copies of the parameters, gradients and moments, fetched when they are read (checkpoints, oklm_trainer_tensor(), the logits for sampling) and sent back when the host changed them (initialization, loading a state, writes through oklm_trainer_tensor()), so checkpoints are the same files on both sides and a device run resumes on the CPU.
  • Matrix products: one tiled kernel (64 x 64 outputs per work group of 16 x 16, blocks of 16 along k), operands read through strides (no transposed copies) and batched along a third dimension. Attention is batched products (scores = Q K^T, weights x V, and for the gradients dout V^T, dpre K, dpre^T Q, weights^T dout, the query heads of a group added in order) with the causal softmax and its backward as row kernels. A work group's batch offsets are computed with 32-bit divisions: GPUs have no 64-bit integer divide, and its software routine, five times per item, cost a fifth of the products' time (an 8-layer, 512-channel Llama step of 16 x 512 tokens: 284 to 238 ms, the same bits). Larger register tiles (128 x 128 outputs per work group, 8 x 4 or 8 x 8 per item, loads following each operand's contiguous dimension) measured no faster on this gfx1100 once the divisions were gone: the products are bound by their loads, not their arithmetic.
  • Determinism: no atomics; every output is summed in a fixed order (the sums over rows of the norm and bias gradients by chunks of 64 rows, then the chunks in order), so a device gives the same bits on every run. The device and the CPU sum in different orders and use different exp and tanh, so they agree to TOL_TRAIN_DEVICE (1e-4 relative), not bit for bit.
  • BF16 storage (--act-type, --opt-type, --grad-type) keeps its 16-bit copies on the device too: tr_bf16_st / tr_bf16_ld round and widen a block's activations, tr_adamw16 updates BF16 moments and tr_grad16_add adds a block's gradients into BF16 ones, both with the CPU's stochastic-rounding bits (BF16 storage).
  • A hybrid stack's Gated DeltaNet blocks run on the device too (tr_gd_* kernels): the input projections as products, the short conv, the recurrence (one work-group per sequence and value head, heads of up to 128 keys and values), the gated output norm. Their backward computes the forward again, keeping the recurrence's state every 16 tokens, and walks each 16-token chunk backward from its states computed again; the q and k gradients of a key group's heads are summed before the L2 norms' backward, and every weight gradient is a fixed-order sum, as above. Larger heads, or OKLM_TRAIN_GD_HOST=1, run the block on the host as before. A step of a 4-block, 512-channel hybrid stack (two linear blocks of eight 64-value heads, 4 x 128 tokens): 2 626 ms with the block on the host, 43.9 ms on the device, the same loss (2026-11-11); test_train_cl checks the gradients against the CPU over 7 and 37 tokens (three chunks).
  • OKLM_CL_PROFILE=1 waits for and times every launch and prints a table per kernel when the trainer is freed.

Checkpoints

adr/0008-training-checkpoints.md:

  • <name>.gguf: the weights as an ordinary F32 GGUF model, with hyper-parameters and tokenizer. oklm run, oklm chat, oklm quantize and llama.cpp load it as it is. With --save-type bf16 its matrices are BF16 (half the file; norms and biases stay F32), and the sidecar's hash is that of the rounded weights, so resuming it works, from the rounded weights (BF16 storage).
  • <name>.gguf.train: the AdamW moments (F32, or BF16 with --opt-type bf16), the step, the options (the storage types among them), and a hash of the weights' data; a sidecar of other weights is refused, and so is one whose layout (train.version) is newer than the build reads.
  • oklm_trainer_save(), oklm_train_read_params(), oklm_trainer_load_state(). Training continued from a checkpoint gives the same parameters, bit for bit, as the same steps run without stopping; weights without a sidecar start a fresh optimizer state (a fine-tune).
  • With LoRA (LoRA) <name>.gguf is the adapters only, in llama.cpp's adapter format, and the sidecar holds their moments and the LoRA options; oklm_trainer_load_lora() reads them back into a trainer of the same rank and targets.

oklm train

--out is a prefix: checkpoints are <prefix>-<step>.gguf, and the directories it names (build/nano/ below) are created when missing, before any work starts.

# from scratch: a 2048-token BPE tokenizer trained on the text, then a GPT-2
oklm train -f data/tinyshakespeare.txt --out build/nano/nano --vocab 2048 \
    --layers 6 --embd 256 --heads 8 --ctx 256 --batch 16 \
    --steps 1200 --lr 1e-3 --lr-min 1e-4 --warmup 100 --save-every 200 --sample-every 400
# fine-tune a model
oklm train -m models/gpt2-hf -f data/tinyshakespeare.txt --out build/ft/gpt2 --steps 200 --lr 1e-4
# a Llama from scratch: 8 query heads over 4 key / value heads, MLP of about 8/3 x 256
oklm train -f data/tinyshakespeare.txt --out build/nano-llama/nano --vocab 2048 --arch llama \
    --layers 6 --embd 256 --heads 8 --kv-heads 4 --ctx 256 --batch 16 \
    --steps 600 --lr 1e-3 --lr-min 1e-4 --warmup 100
# the same on the GPU
oklm train -f data/tinyshakespeare.txt --out build/nano-llama/nano --vocab 2048 --arch llama \
    --layers 6 --embd 256 --heads 8 --kv-heads 4 --ctx 256 --batch 16 \
    --steps 600 --lr 1e-3 --lr-min 1e-4 --warmup 100 --backend opencl
# fine-tune a Hugging Face Llama with updates over 4 batches of 4 x 128
oklm train -m models/smollm2-135m-hf -f data/tinyshakespeare.txt --out build/ft/smollm2 \
    --batch 4 --seq 128 --accum 4 --steps 50 --lr 1e-4
# continue a checkpoint (its options come from the sidecar; --steps extends the run)
oklm train --resume build/nano/nano-000600.gguf -f data/tinyshakespeare.txt --out build/nano/nano
oklm run -m build/nano/nano-001200.gguf -p "ROMEO:" -n 100
# LoRA: rank-8 adapters on every projection of a frozen model, then run with them, or merge them
oklm train -m models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf -f data/tinyshakespeare.txt --out build/lora/smol \
    --lora 8 --lora-alpha 16 --lora-targets all --batch 8 --seq 128 --steps 200 --lr 1e-3 --warmup 20 --backend rocm
oklm train -m models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf --resume build/lora/smol-000200.gguf \
    -f data/tinyshakespeare.txt --out build/lora/smol --steps 400
oklm run -m models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf --lora build/lora/smol-000400.gguf -p "ROMEO:" -n 100
oklm lora-merge models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf build/lora/smol-000400.gguf build/lora/smol-merged.gguf
# BF16 storage: activations and AdamW moments in BF16, checkpoints and the merged model in BF16
oklm train -f data/tinyshakespeare.txt --out build/nano/nano --vocab 2048 --layers 6 --embd 256 --heads 8 --ctx 256 --batch 16 \
    --steps 1200 --lr 1e-3 --lr-min 1e-4 --warmup 100 --backend cpu --act-type bf16 --opt-type bf16 --save-type bf16
oklm lora-merge models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf build/lora/smol-000400.gguf build/lora/smol-merged.gguf --save-type bf16
# F16 products on the matrix units with dynamic loss scaling
oklm train -m models/smollm2-135m/SmolLM2-135M-Instruct-f16.gguf -f data/tinyshakespeare.txt --out build/ft/smol16 \
    --batch 4 --seq 256 --steps 300 --warmup 20 --lr 1e-4 --backend rocm --matmul-precision f16 --loss-scale dynamic

--backend is auto by default (the first OpenCL device, else the CPU), as for oklm run; cpu gives the same bits on every machine. A new model is GPT-2 by default; --arch llama builds Llama blocks (--kv-heads, --head-dim, --ff, --rope-theta), --arch qwen2 adds Q/K/V biases and --arch qwen3 per-head Q/K norms to them. --dropout P sets the residual dropout (0 by default). --accum N sums the gradients of N batches per update, for a larger effective batch in the same memory; the run's options, --accum included, are kept in the sidecar.

The text is tokenized once; its last --val-frac (10 %) is kept for the validation loss, printed every --eval-every steps over --eval-batches consecutive windows. Each batch is drawn from the seed, its index (the step times --accum, plus its place in the step) and the row, so a resumed run sees the batches the uninterrupted one would have. Checkpoints are written as <prefix>-<step>.gguf every --save-every steps and at the end. --sample-every N prints a sample every N steps and once more at the end (none without it): a text the model being trained writes from its end-of-text token (a newline for a tokenizer without one), between two --- lines. --sample-len L is the sample's length in tokens (200 by default); each token is drawn at temperature 0.8 among the 40 likeliest, from a seed made of the run's seed and the step, with the last --seq tokens as the context, run again for each new token, so a long sample of a large model takes a while. --progress-json FILE writes the run as JSON lines a program can follow (the section "Machine-readable progress" below).

LoRA

--lora R (the trainer's lora_rank) freezes the model and trains adapters of rank R on its projections: a weight W (outputs x inputs) becomes W + (alpha / R) B A, A (R x inputs) drawn uniformly in ±1/sqrt(inputs), B (outputs x R) zero, so a fresh adapter changes nothing (the loss of the model alone, to the bit). --lora-alpha sets alpha (the rank by default: a scale of 1). --lora-targets lists the projections: q, k, v, o, gate, up, down for Llama-family models, qkv, o, up, down for GPT-2, or all; the default is the attention's (q,k,v,o, or qkv,o). Q, K and V (and gate and up) have adapters of their own on their columns of the packed product, as separate weights have in the model file.

  • Forward: after the product, h = s (x A^T), kept for the backward pass, then y += h B^T.
  • Backward: dB += dy^T h, dh = s (dy B), dA += dh^T x, dx += dh A. The model's own weight gradients are not computed (the products that would give them are skipped, and the norms' and embeddings' passes with them); the gradients flowing through the model to earlier adapters are.
  • State: the adapters follow the model in the parameter buffer; gradients, AdamW moments, the gradient norm and clipping cover only them (2.4 M parameters for every projection of SmolLM2-135M at rank 8, against 135 M).
  • Files: a LoRA checkpoint's weights file is an adapter in llama.cpp's format: general.type adapter, adapter.type lora, adapter.lora.alpha, general.architecture the model's, and for a weight <name> the tensors <name>.lora_a (shape {inputs, rank}) and <name>.lora_b (shape {rank, outputs}), in F32 (10 MB for the example above, against 540 MB for a full checkpoint). llama.cpp's --lora reads it.
  • Resuming: -m <model> --resume <adapter.gguf>: the rank, alpha and targets come from the sidecar; the adapters are loaded, then the optimizer state. The same parameters, bit for bit, as without stopping.
  • Using them: oklm run, chat, ppl and serve take --lora <adapter.gguf> [--lora-scale S]: the adapter is merged into a temporary copy of the model as it loads. oklm lora-merge <model> <adapter> <out> [--scale S] writes the merged model (oklm_lora_merge(), oklm/io/lora.h): W + S x (alpha / rank) x B A, F32, F16 and BF16 weights keeping their type and quantized ones becoming F16; every other tensor and key copied. oklm serve --lora-module NAME=<adapter.gguf> serves adapters beside the model instead, several in the same batches, each request choosing one by its model (server.md, ADR-0021); in the library, oklm_adapter_load() and oklm_seq_set_adapter() (oklm/model/model.h).
  • On a device: the rank-wide products have few output tiles and a long inner size (dA is rank x inputs summed over every row of the batch), so the device product splits such products along the inner size, one batch per part, and sums the parts in order; the scale is the product's own factor (rocBLAS's alpha, or the kernel's) rather than a pass of its own.

Stopping a run. oklm train and oklm rl asked to end (Ctrl-C, SIGTERM, a tool's Stop, which on Windows is the event of the registry of processes) finish the step they are in, write the checkpoint of that step and its training state, print stopped on request at step N of TOTAL: a checkpoint is written and end with the status 0; the final validation and the final sample are left out. --resume on that checkpoint goes on from step N + 1. A second Ctrl-C ends the program at once, without a checkpoint.

Chat-data fine-tuning (loss on the assistant turns)

oklm train --chat-data conv.jsonl (in place of -f text.txt) trains on conversations with the loss on the assistant turns only — supervised fine-tuning for tool use and instruction following, not plain next-token over the whole text. Each JSONL line is {"text": ...} as oklm agent --save-training writes it: the conversation rendered through the model's chat template. It needs a model with that template (-m or --resume), not a new one.

  • The mask. Each conversation is tokenized and its assistant tokens marked (the content a model generated and the <|im_end|> that closes it, found by reading the role after each <|im_start|>); the system, user and tool turns are not targets. A conversation longer than --seq keeps its last --seq tokens, where the assistant turns usually are.
  • The loss. The forward pass's per-token losses (oklm_trainer_row_losses()) are averaged over the masked tokens; the gradient is oklm_trainer_backward_weighted() with weights of mask / (masked count) / accum, so only the assistant tokens pull the weights. The step and the validation loss (a held-out --val-frac of the conversations) are this masked mean. A token the model is merely given (the prompt) never contributes, so the run does not learn to generate the inputs.
  • Use. oklm train --chat-data data.jsonl -m model.gguf --lora 16 --out adapter, then run or serve with --lora. The MCP tool-use and the defensive security fine-tunes (mcp-training-design.md, security-training-design.md) both train this way on data captured with --save-training.

Verification

  • tests/train/test_train.c, on tiny GPT-2 and Llama models (adjacent and half-split rotations, partial rotation, 4 query heads over 2 key/value heads): the matrix product against a plain loop at every SIMD level (the same bits); every parameter tensor's gradient against a central difference of the loss (TOL_TRAIN_FD_REL); the same parameters after training with 1 or 3 threads and at every SIMD level; two half batches accumulated against the whole batch; a tiny model learning a batch (loss 3.6 to 0.0007 in 150 steps); the schedule; the forward pass of loaded models against their fixtures' float64 references (GPT-2 1.7e-5, Llama with Llama 3 frequency factors 1.3e-6); a Hugging Face Llama checkpoint's trainer against its own checkpoint's (the same bits) and against inference on it (1e-6); checkpoints loaded as models, resumed bit for bit, a sidecar of other weights refused.
  • tests/train/test_train_cl.c (skipped without an OpenCL device): the tiny GPT-2 and Llama models on the device against the CPU trainer: loss, logits and every gradient of a pass (within 6e-7 relative), the losses and parameters of five AdamW steps, the same bits from two device runs, and a device checkpoint resumed on the CPU.
  • tests/train/test_train_gpt2.c (skipped without sh tools/fetch_llmc.sh, 1.1 GB): GPT-2 124M from the Hugging Face checkpoint against llm.c, on the CPU and on an OpenCL device.
    • llm.c's reference state, written with PyTorch by its train_gpt2.py, with the criteria of its test_gpt2.c: logits of one batch of 4 x 64 tokens, the loss, every parameter's gradient, then the losses of ten AdamW steps.
    • llm.c's CPU training run of its README (train_gpt2.c: 40 steps of 4 x 64 tokens of Tiny Shakespeare in order, AdamW at 1e-4): the validation loss before and after, and the train losses it prints.
Check llm.c / PyTorch OKLM, CPU OKLM, OpenCL Tolerance
logits of one batch PyTorch max diff 0.0020 max diff 0.0015 1e-2
loss 5.270009 5.270009 5.270011 1e-2
gradients, every parameter PyTorch max diff 0.00032 max diff 0.00031 2e-2
loss after 10 steps (step 9) 0.376511 0.376524 0.376529 1e-2
Tiny Shakespeare, validation loss before training 5.252026 5.252031 5.252031 1e-3
train loss, step 0 / 1 / 2 / 3 / 39 5.356189 / 4.301069 / 4.623322 / 4.600470 / 3.970751 5.356194 / 4.301096 / 4.623336 / 4.600519 / 3.970691 5.356193 / 4.301094 / 4.623335 / 4.600514 / 3.970695 1e-2
validation loss after 40 steps 4.107781 4.107731 4.107734 1e-2

The validation loss of llm.c's published run is the mean over the first ten batches of 4 x 64 validation tokens (its code has since gone to five). The tests use llm.c's published numbers and reference files, so they run without llm.c. llm.c itself (commit f1e2ace6, built in build/llm.c as a test-only reference, never linked into OKLM) was also run directly:

  • test_gpt2 passes on the same reference state (step 9 loss 0.376691).
  • train_gpt2 with its training batches taken in order and ten validation batches (the conditions of its README run; its current code shuffles the batches and averages five): every one of the 41 train losses and 5 validation losses agrees with OKLM's CPU trainer within 0.0008 (largest at step 34, 4.516505 against 4.515689; llm.c sums the softmax normalizer in float, OKLM in double); validation after 40 steps is 4.107841 against 4.107731.

Speed

Intel Core i7-11700K, 8 threads, AVX-512:

Model Batch Time per step Tokens/s
GPT-2 124M (fine-tune) 4 x 64 0.6 s 430
GPT-2, 5.3M parameters (6 x 256, vocabulary 2048) 16 x 256 0.8 to 0.9 s 4 400 to 5 000
Llama, 4.95M parameters (6 x 256, 8 / 4 heads, vocabulary 2048) 16 x 256 0.75 s 5 450
SmolLM2-135M (fine-tune, Llama) 4 x 4 x 128 9 s 210

On the GPU (AMD Radeon gfx1100, 42 compute units, --backend opencl):

Model Batch Time per step Tokens/s Against the CPU
Llama, 4.95M parameters (the Nano above) 16 x 256 42 ms 94 000 17 x
SmolLM2-135M (fine-tune) 4 x 4 x 128 0.44 s 4 240 20 x

The 600-step Llama Nano takes 27 seconds instead of 7.5 minutes, with the same validation curve (3.8318 against 3.8319 after 600 steps); the SmolLM2 fine-tune 16 seconds instead of 4.5 minutes (3.4302 against 3.4298). Matrix products are three quarters of the device's time.

With ROCm, --backend rocm is the same OpenCL device with its large single products through rocBLAS (loaded at run time, on shared virtual memory; the attention's batched products and every other kernel stay on OpenCL), the same losses (2026-11-04):

Model Batch opencl rocm
Llama Nano 16 x 256 38 ms a step 26.5 ms (1.43 x)
SmolLM2-135M (fine-tune) 4 x 4 x 128 390 ms 310 ms (1.26 x)

The products alone are 1.7 to 3.5 times faster (tests/train/bench_rocblas.c); each switch between the two runtimes drains the queue it leaves. OKLM_HIP_DEVICE picks the HIP device (0), which must be the OpenCL device's GPU; OKLM_ROCBLAS_MIN the multiply-adds from which a product goes to rocBLAS (2^22).

rocm holds its buffers in shared virtual memory, which the driver commits in device memory when allocated (the opencl backend's plain buffers can overflow into host memory instead). When the device is full, a buffer goes to fine-grained shared virtual memory, in host memory the device reads over the bus, with a notice (the device is full, N MiB in host memory); OKLM_ROCM_DEVICE_MB caps the device memory used the same way. Qwen3-8B Q4_K_M, LoRA rank 16 on Q, K, V, O, --quantized-base --recompute, 1 x 6144 tokens with 2 accumulated (2026-11-08): the activations take 17.1 GB, so the 5.0 GB packed base lands in host memory; a step takes 56.4 s on rocm against 91.2 s on opencl, with the same losses (5.0787, 5.0576). Before, rocm stopped there with cannot allocate 5020594176 bytes of shared virtual memory: the device's per-allocation limit (CL_DEVICE_MAX_MEM_ALLOC_SIZE, 17 GiB on the gfx1100) was not the cause, the device's 20 GB were.

--matmul-precision bf16|f16 (oklm_train_params::matmul_type, rocm only, refused on other backends) runs rocBLAS's large products on the matrix units of RDNA3: for a product of at least 128 rows, columns and inner size (OKLM_ROCM_MM16_MIN), a kernel rounds both operands to BF16 / F16 (to nearest even, in their own layouts), rocblas_gemm_ex multiplies them with F32 accumulation into a product of the same 16-bit type, and a kernel widens it into the F32 output (or adds it). The product goes through 16 bits because rocBLAS's tuned gfx1100 kernels write 16-bit outputs: with F32 outputs its 16-bit products ran 1.5 to 2 times slower than its F32 sgemm (20 times when the second operand is transposed), with 16-bit outputs 2.5 to 4 times faster (2026-11-08, SmolLM2 and 4096 x 4096 shapes). Under --quantized-base a BF16 base is read as it lies (the same bits as its rounded unpacked copy). The norms, attention, softmax and optimizer stay F32, as do the thinner products (LoRA's rank-wide ones), whose conversions cost more than they gain.

SmolLM2-135M on Tiny Shakespeare, --batch 4 --seq 256 --steps 300 --warmup 20, full (--lr 1e-4) and LoRA (rank 16, alpha 16, every target, --lr 1e-3), validation loss every 50 steps and the mean step (2026-11-08):

Run Inputs Validation 0 / 50 / 100 / 150 / 200 / 250 / 300 Step
full F32 3.5508 / 3.2509 / 3.1891 / 3.1605 / 3.1462 / 3.1377 / 3.1329 125 ms
full BF16 3.5516 / 3.2579 / 3.1906 / 3.1627 / 3.1477 / 3.1378 / 3.1342 114 ms
full F16 3.5509 / 3.4086 / 3.3298 / 3.2860 / 3.2618 / 3.2430 / 3.2365 114 ms
LoRA F32 3.5508 / 3.2887 / 3.2213 / 3.1804 / 3.1774 / 3.1619 / 3.1573 117 ms
LoRA BF16 3.5516 / 3.2892 / 3.2215 / 3.1805 / 3.1770 / 3.1625 / 3.1576 110 ms
LoRA F16 3.5509 / 3.5266 / 3.4266 / 3.3863 / 3.3845 / 3.3648 / 3.3573
full F16, --loss-scale dynamic 3.5509 / 3.2520 / 3.1891 / 3.1607 / 3.1475 / 3.1377 / 3.1347 102 ms
LoRA F16, --loss-scale dynamic 3.5509 / 3.2886 / 3.2199 / 3.1789 / 3.1774 / 3.1618 / 3.1570 107 ms

BF16 follows F32 to the third decimal; F16 alone trains worse: small gradients underflow its range (F16's 11 bits help the forward pass only). With --loss-scale dynamic F16 follows F32 as BF16 does (3.1347 against 3.1329 full, 3.1570 against 3.1573 LoRA; the last two rows 2026-11-08, the step times of a busier machine than the rows above).

--loss-scale dynamic|N (oklm_train_params::loss_scale, any backend) is dynamic loss scaling: the backward pass computes the gradients of the loss times a scale S (65536 for dynamic, or N), which lifts SmolLM2's logit gradients (about 1e-8 per element at 1024 rows, under F16's smallest denormal) into F16's range; the update divides them back before the norm is clipped. An update whose gradients are not all finite (an F16 product overflowed) is skipped, the parameters and moments unchanged, the step counted, and S halves; S doubles after 200 finite updates in a row (OKLM_TRAIN_LOSS_SCALE_WINDOW). Both runs above settled at 16384 after two skipped updates out of 300. The step line prints the scale and the skipped updates; the sidecar keeps the scale as it stands, so a resumed run continues with it. In F32 a power-of-two scale changes no bit (the 200-step test of test_train), unless it overflows or reaches the denormals.

What stays F32 under --matmul-precision was measured not to pay (2026-11-08, SmolLM2-135M full, 4 x 256, OKLM_CL_PROFILE=1, BF16 products): every OpenCL tiled product, the attention's batched ones included, is 9.9 % of the device's time, rocBLAS 41.7 %, and the roundings into and out of 16 bits (tr_to16, tr_from16) already 20.9 %; the attention's operands (Q, K, V and the B x heads x T x T weights) would each need the same rounding for at most a few percent. The thin LoRA products on 16 bits made a LoRA step slower (110 to 164 ms). The LoRA losses come from a run with every rocBLAS product on 16 bits, the thin ones included (164 ms a step with BF16); the 110 ms is a 30-step run once those stay F32. SmolLM2's products are small; on Llama-3.2-1B Q4_K_M (LoRA rank 16, Q, K, V, O, 4 x 256) a step takes 227 ms with BF16 against 350 ms (1.54 x), the losses within 1e-3.

The weights' 16-bit copies are kept between updates (2026-11-08): each weight is rounded once after AdamW changes it (the update kernel writes the rounding of what it updates), not at each of its products (forward, input gradient, a recomputed forward, every accumulated batch, every evaluation); a LoRA run's frozen base is rounded once for the whole run. The output gradient of a weight's two backward products (input gradient and weight gradient) is also rounded once for both. The copy costs 2 bytes a parameter of device memory (257 MiB for SmolLM2-135M, 2.3 GiB for Llama-3.2-1B); it is allocated on the first 16-bit product, in device memory only and when 512 MiB stay free after it (otherwise the weights are rounded at each product as before, with a notice), and OKLM_ROCM_W16_CACHE=0 turns it off. Under --quantized-base the frozen matrices are unpacked and rounded before each product as before (a BF16 base is still read in place) and no copy is made: that option is about memory. The results are the same bits: the same rounding of the same values, made less often. 20 steps of 4 x 256 (--warmup 5), the median step of steps 6 to 20, the most device memory the trainer allocated ("before" with OKLM_ROCM_W16_CACHE=0, the allocations of before) (OKLM_LOG_LEVEL=INFO prints it at exit), two runs each where two are given (another job shared the GPU); in every row the 20 losses and the checkpoint written after step 20 (sha256) are identical with and without the copy:

Run Step before Step after Device memory before After
SmolLM2-135M full, BF16 102 / 103 ms 98 / 99 ms 3930 MiB 4187 MiB
SmolLM2-135M full, F16, --loss-scale dynamic 102 / 114 ms 97 / 98 ms 3930 MiB 4187 MiB
SmolLM2-135M full, BF16, --accum 2 --recompute 236 ms 221 ms 2777 MiB 3034 MiB
SmolLM2-135M LoRA (every target), BF16 108 / 109 ms 104 / 106 ms 2483 MiB 2748 MiB
SmolLM2-135M LoRA (every target), F16, --loss-scale dynamic 118 / 107 ms 105 / 105 ms 2483 MiB 2748 MiB
Llama-3.2-1B Q4_K_M LoRA, BF16 219 / 228 ms 207 / 203 ms 9785 MiB 12149 MiB
Llama-3.2-1B Q4_K_M LoRA, F16, --loss-scale dynamic 217 / 219 ms 201 / 200 ms 9785 MiB 12149 MiB

In the profile (OKLM_CL_PROFILE=1, 5 steps and a validation pass) the roundings into and out of 16 bits went from 20.1 to 16.3 % of the device's time on SmolLM2-135M full (tr_to16 202 to 109 ms, the device's total 1547 to 1330 ms) and from 14.9 to 8.2 % on Llama-3.2-1B LoRA (tr_to16 318 to 97 ms, the total 2814 to 2410 ms). What is left are the activations: each product's input and output gradient rounded in, and its output widened out (tr_from16), new at every product. Keeping the forward pass's rounded inputs for the weight-gradient products would cost 2 bytes a kept activation for at most the remaining input roundings; it was not done.

LoRA against a full fine-tune (2026-11-04, rocm_spec.md R6): a step of 4 x 256 tokens, rank 16, alpha 16, the default targets (Q, K, V, O), the mean of steps 4 to 6 (oklm train ... --steps 6 --batch 4 --seq 256 [--lora 16 --lora-alpha 16] --backend opencl|rocm):

Model (file) Full, opencl Full, rocm LoRA, opencl LoRA, rocm
SmolLM2-135M (F16) 163 ms 118 ms 126 ms 98 ms
Qwen2.5-0.5B (Q4_K_M) 500 ms 315 ms 322 ms 199 ms
Qwen3-0.6B (Q8_0) 635 ms 410 ms 443 ms 291 ms
Llama-3.2-1B (Q4_K_M) 9 s (beyond the device's memory) out of memory 679 ms 344 ms

The model is held in F32 whatever its file (4 bytes a parameter on the device and as many on the host); a full fine-tune adds its gradients and two AdamW moments (16 bytes a parameter in all: 19.8 GB for Llama-3.2-1B, more than the 20 GB GPU holds once activations are counted), LoRA only the adapters' (11 M floats for Llama-3.2-1B at rank 16). A LoRA step skips the model's weight gradients, a third of the large products. In the Llama-3.2-1B LoRA step on rocm (OKLM_CL_PROFILE=1) rocBLAS's products are 79 % of the device's time, the norms 10 %, nothing else above 2 %.

Activation recompute (2026-11-04, rocm, the mean of steps 4 to 6, LoRA rank 16 on Q, K, V, O). "Activations" is what oklm train prints:

Model (file), mode Batch Activations Blocks recomputed And the head in chunks Step Blocks recomputed And the head in chunks
SmolLM2-135M (F16), full 4 x 256 1689 MiB 536 MiB 536 MiB (one chunk) 119 ms 142 ms 144 ms
SmolLM2-135M (F16), full 4 x 1024 10103 MiB 2358 MiB 1326 MiB 405 ms 531 ms 550 ms
Qwen2.5-0.5B (Q4_K_M), LoRA 4 x 256 3516 MiB 1457 MiB 719 MiB 195 ms 257 ms 294 ms
Qwen2.5-0.5B (Q4_K_M), LoRA 4 x 1024 19 GB: out of memory 6164 MiB 1875 MiB 962 ms 1091 ms
Llama-3.2-1B (Q4_K_M), LoRA 4 x 256 4070 MiB 1490 MiB 997 MiB 344 ms 469 ms 463 ms
Llama-3.2-1B (Q4_K_M), LoRA 4 x 1024 24 GB: out of memory 6728 MiB 3253 MiB 1865 ms 1881 ms
Llama-3.2-1B (Q4_K_M), LoRA 2 x 2048 33 GB: out of memory 7752 MiB 4277 MiB 2204 ms 2211 ms

Each block keeps its attention weights (batch x heads x T x T floats), so without recompute the activations grow with the square of the sequence. With the blocks recomputed, most of what is left is the head's logits and probabilities (batch x T x vocabulary floats each: 3.9 GiB of Llama-3.2-1B's 6.6 at 4 x 1024), which the chunks bring down to about 512 MiB. What remains is each block's input, one block's activations, the attention weights' gradients, and the chunk. LoRA pays relatively more for the extra forward pass, because its backward pass skips the model's weight gradients. Qwen2.5-0.5B (vocabulary 151,936) also pays for its head in three chunks a batch.

A quantized base (2026-11-04, rocm, LoRA rank 16 on Q, K, V, O, 4 x 256 tokens, --recompute, the mean of steps 4 to 6). "Weights" is what oklm train prints; the losses are the same to the last digit:

Model (file, size) Weights, F32 The blocks packed And the embeddings and head Step, F32 Step, all packed
Qwen2.5-0.5B (Q4_K_M, 469 MiB) 2412 MiB 1316 MiB 504 MiB 298 ms 313 ms
Llama-3.2-1B (Q4_K_M, 770 MiB) 4727 MiB 1700 MiB 904 MiB 479 ms 503 ms
Qwen2.5-1.5B (Q4_K_M, 1066 MiB) 6796 MiB 2654 MiB 1182 MiB 716 ms 757 ms
Qwen3-4B (Q4_K_M, 2382 MiB) 15389 MiB 3790 MiB 2611 MiB 1675 ms 1762 ms

Qwen3-8B, the default model (2026-11-04, the same settings, the steps after the first; its own head, unpacked in 7 tiles of 24,576 rows):

File (size) Batch Weights Activations Step Tokens/s
Q4_K_M (4795 MiB) 4 x 256 5232 MiB 1621 MiB 3.06 s 335
Q8_0 (8306 MiB) 4 x 256 8743 MiB 1621 MiB 3.07 s 334
Q4_K_M 4 x 1024 5232 MiB 5918 MiB 11.4 s 359
Q4_K_M 2 x 2048 5232 MiB 6942 MiB 12.7 s 323
Q8_0 2 x 2048 8743 MiB 6942 MiB 12.7 s 323
Q4_K_M 1 x 4096 5232 MiB 8990 MiB 15.3 s 267
Q4_K_M 4 x 2048 5232 MiB 13439 MiB 25.2 s 326

Held in F32 the model would need 32.8 GB and is refused. The Q8_0 base costs no time over Q4_K_M; its unpacking is as cheap. The largest batch here, 8192 tokens, takes about 18.7 GiB of the card's 20.

With everything packed the model takes its file's size plus the scratch (the largest block matrix in F32). Unpacking each matrix before its products (about three times a step with recompute) costs 5 to 6 % of a step. These four models tie their head to the embeddings: the head is unpacked in tiles of the scratch's rows (Qwen2.5-0.5B: 16 tiles).

BF16 storage

The trainer computes in F32 everywhere; four options store some of what it holds in BF16 (8 bits of mantissa, F32's exponent range), for memory, not speed (OKLM's kernels widen BF16 to F32; the products on the gfx1100's matrix units are --matmul-precision, under Speed), on the CPU and on a device alike. Each is opt-in; without them the trainer's bits are those of before (the existing tests, and a 20-step checkpoint of the same run byte for byte against the previous binary).

  • --act-type bf16 (act_type): what each block keeps from the forward pass for its backward pass (its input, norms' outputs and statistics, Q/K/V, attention weights, attention output, MLP activations; with --recompute, its input only) is stored in BF16, rounded to nearest even as the block's forward pass ends, and widened into one F32 set just before its backward pass (or its recompute). The residual stream itself flows in F32 from block to block; the head's logits and probabilities, the gradients' scratch and the final norm stay F32. The kept activations halve; one block's F32 set is added, so the total shrinks by about (L / 2 - 1) blocks' worth. The gradients are within 2 % (norm of the difference) of F32's on the tiny models of make test, every architecture, whole or LoRA, with or without recompute (test_bf16_gradients).
  • --opt-type bf16 (opt_type): AdamW's two moments are stored in BF16, widened for the update, and stored back with stochastic rounding: up in magnitude with the probability of the dropped low 16 bits (so the stored value is the moment in expectation), the bits a hash of the seed, the step, the moment (m or v) and the element's index alone, so runs are reproducible and do not depend on the thread count (test_bf16_training compares 1 and 3 threads to the bit). Rounding to nearest would freeze a moment whose update is under half a BF16 step (2^-9 relative), as with a beta2 of 0.999 (updates of 0.1 % of v): stochastic rounding keeps those updates in expectation. The parameter update uses the F32 moments before they are rounded; the master weights stay F32. The optimizer's memory halves (4 instead of 8 bytes a trainable parameter), and the sidecar holds the moments in BF16, so resuming them is exact.
  • --grad-type bf16 (grad_type, full training): the gradients are held in BF16 between the backward passes and the update (2 bytes a trainable parameter instead of 4). They are still computed in F32: the backward pass works through the model in regions (each block's parameters, which follow each other in the layout; the final norm; the output head and the token embeddings a tile of vocabulary rows at a time; GPT-2's positions), each region's gradients computed into one F32 window and then added into the BF16 ones with stochastic rounding (tr_grad16_add(): the bits a hash of the seed, the updates done, the micro-batch and the element, so accumulated micro-batches stay unbiased and the bits do not depend on the thread count). The window is the size of the largest block (the vocabulary tiles are as many rows as fit it, OKLM_TRAIN_GRAD_TILE_ROWS sets them), so the vocabulary matrices never need an F32 copy. The update's norm and AdamW read the BF16 values widened (tr_sqsum and tr_adamw / tr_adamw16 take them on a device). A tied head's rows get two additions a backward pass (the head and the embeddings), and a chunked head (--recompute) one a chunk, each a rounding. The gradients oklm_trainer_tensor() gives are then a widened copy made when asked for, read-only. Refused with LoRA: the adapters' gradients are small. Within 2 % of F32's gradients on the tiny models (every architecture and the hybrid one, with and without recompute, one micro-batch or two, whole tables or tiles of 3 rows: test_bf16_grad_storage), the same bits with 1 or 3 threads (test_bf16_training), and on the device within 1e-4 of the CPU (test_bf16_storage).
  • --save-type bf16 (save_type, any backend): the checkpoint's matrices (the adapters with LoRA) are written in BF16, rounded to nearest even; norms, biases and other 1-D tensors stay F32; general.file_type is BF16's. The sidecar keeps the moments as the trainer holds them (F32 unless --opt-type bf16). oklm lora-merge --save-type bf16 writes the merged weights and the model's other F32 and F16 matrices in BF16 (oklm_lora_merge_as()).

The options are kept in the sidecar and come back on --resume (a flag given again overrides). --act-type and --opt-type run on every backend. On a device (opencl, rocm) the BF16 copies stay in device memory: tr_bf16_st rounds a block's activations as its forward pass ends and tr_bf16_ld widens them before its backward pass (the CPU's bits), and tr_adamw16 updates BF16 moments with the CPU's stochastic rounding (the same hash of seed, step, moment and element, in 64-bit integers); the host's copies of the moments, the sidecar and resuming are those of the CPU. With --grad-type bf16 the gradients and their window are in device memory, tr_grad16_add drawing the CPU's bits. The parameters stay F32.

Caveats:

  • Resuming from a BF16 checkpoint is lossy for full training: the trainer starts from the matrices rounded to BF16 (the moments, the step and the schedule are restored), so it trains on as the saved trainer would from its rounded matrices, to the bit (test_bf16_checkpoint), not as the uninterrupted run. For LoRA the same holds for the adapters. Keep F32 checkpoints to resume a run exactly; BF16 ones to ship.
  • BF16 activations change the gradients by the rounding of what is kept (2^-9 relative per value); BF16 moments add the stochastic rounding's noise to the moments. Neither changes a step's arithmetic otherwise.
  • The F32 moments of an older sidecar load into a trainer of BF16 moments (rounded to nearest), and BF16 ones into an F32 trainer (widened).

Measured (2026-11-07, --backend cpu, 8 threads, the machine shared with other jobs so the step times are not comparable). OKLM-Nano as below (GPT-2, 6 layers, 256 channels, 16 x 256 tokens, the 2048-token tokenizer), 300 steps at 1e-3 with 100 warm-up steps and a cosine decay to 1e-4; validation over 8 batches; memory as oklm train prints it, and the process's peak RSS:

Nano, 300 steps Val 100 Val 200 Final val Train, step 300 Optimizer Kept activations All activations Peak RSS Checkpoint
F32 5.0133 4.6725 4.5278 4.3863 40.7 MiB 576.4 MiB 695 MiB 813 MiB 20.4 MiB
--act-type bf16 5.0133 4.6714 4.5280 4.3864 40.7 MiB 288.2 MiB 502 MiB 624 MiB 20.4 MiB
--opt-type bf16 5.0130 4.6703 4.5269 4.3855 20.3 MiB 576.4 MiB 695 MiB 792 MiB 20.4 MiB
all three 5.0133 4.6713 4.5281 4.3864 20.3 MiB 288.2 MiB 502 MiB 601 MiB 10.3 MiB

The BF16 checkpoint of the last run, evaluated as resumed (its rounded weights): 4.5281, the same as its F32 master weights. Moments rounded to nearest instead of stochastically (a build made for this comparison, not shipped) gave 4.5272: at the default beta2 of 0.95 each update of v is 5 % of it, ten times half a BF16 step, so nearest rounding does not stall it here; stochastic rounding is kept because it stays unbiased at a beta2 of 0.999 and beyond, where it would.

SmolLM2-135M (the F16 file, full fine-tune on the text, 4 x 128 tokens, 40 steps at 1e-4, 5 warm-up steps):

SmolLM2-135M, 40 steps Val 20 Final val Train, step 40 Optimizer Kept activations All activations Peak RSS Checkpoint
F32 3.5355 3.5236 3.5516 1026 MiB 562.7 MiB 880 MiB 3233 MiB 515 MiB
all three 3.5355 3.5235 3.5516 513 MiB 281.4 MiB 618 MiB 2510 MiB 258 MiB
--recompute 3.5355 3.5236 3.5516 1026 MiB 33.8 MiB 369 MiB 2724 MiB 515 MiB
--recompute, activations and moments BF16 3.5355 3.5236 3.5516 513 MiB 16.9 MiB 353 MiB 2193 MiB 515 MiB

The BF16 checkpoint resumed scores 3.5187 (its rounded weights; a fine-tune of 40 small updates moves the weights by less than a BF16 step, so rounding them is a change of the same order as the training). The losses are those of F32 to three or four digits; what changes is the memory: the optimizer and the kept activations halve, the checkpoint halves. With recompute the kept activations are already small (each block's input), so BF16 activations save little there; BF16 moments save 4 bytes a trainable parameter whatever the mode, which matters most for a full fine-tune (513 MiB of SmolLM2's 3.2 GB here; for Llama-3.2-1B, 4.9 GB). Gradients stay F32 (4 bytes a trainable parameter).

On the GPU (2026-11-08, --backend rocm, gfx1100): SmolLM2-135M (the F16 file) fully fine-tuned on the text, --batch 4 --seq 256 --steps 300 --warmup 20 --lr 1e-4, validation every 50 steps; memory as oklm train prints it (device memory), the mean step:

SmolLM2-135M, 300 steps, rocm Validation 0 / 50 / 100 / 150 / 200 / 250 / 300 Optimizer Kept activations All activations Step
F32 3.5508 / 3.2509 / 3.1891 / 3.1605 / 3.1462 / 3.1377 / 3.1329 1026 MiB 1261 MiB 1689 MiB 127 ms
--act-type bf16 --opt-type bf16 3.5508 / 3.2509 / 3.1895 / 3.1612 / 3.1464 / 3.1379 / 3.1335 513 MiB 630 MiB 1101 MiB 129 ms
and --matmul-precision bf16 3.5516 / 3.2578 / 3.1912 / 3.1646 / 3.1471 / 3.1377 / 3.1356 513 MiB 630 MiB 1101 MiB 111 ms

BF16 gradients (2026-11-11, the same run, oklm train printing the gradients' memory too; step times are the mean over the 300 steps of this session's runs, F32 again for the baseline):

SmolLM2-135M, 300 steps, rocm Validation 0 / 50 / 100 / 150 / 200 / 250 / 300 Gradients Optimizer Kept activations Step
F32 3.5508 / 3.2509 / 3.1891 / 3.1605 / 3.1462 / 3.1377 / 3.1329 513.1 MiB 1026 MiB 1261 MiB 127 ms
--grad-type bf16 3.5508 / 3.2509 / 3.1891 / 3.1605 / 3.1462 / 3.1376 / 3.1329 270.1 MiB 1026 MiB 1261 MiB 132 ms
--act-type bf16 --opt-type bf16 --grad-type bf16 3.5508 / 3.2509 / 3.1892 / 3.1611 / 3.1464 / 3.1379 / 3.1334 270.1 MiB 513 MiB 630 MiB 135 ms

The gradients' 270.1 MiB are 256.6 MiB of BF16 values and a 13.5 MiB window (one block of SmolLM2's 30, 6146 x 576 floats: the vocabulary tiles of 6146 rows fit it). BF16 gradients leave the curve on F32's to the fourth decimal; with all three the gradients, moments and kept activations take 1413 MiB instead of 2799, for 6 % more time a step (the roundings in and out).

The curve follows F32's to the third decimal, as on the CPU; the device memory drops by 1.1 GB (of 3.2 GB of gradients, moments and activations), for two more launches a block and step time within the noise. With the products on 16 bits too, every large tensor but the master weights and the gradients is 16-bit and the step 13 % faster. The gradients can be BF16 too (--grad-type bf16, below the table): they are accumulated (the weight products add into them, as do micro-batches and the norm and bias sums), and BF16 sums of many small terms would lose them, so each region of them is accumulated in an F32 window and added into the BF16 gradients once, with stochastic rounding.

OKLM-Nano on Tiny Shakespeare

The command of oklm train above: a 2048-token BPE tokenizer trained on the text (1.1 MB, 388 533 tokens, the last 10 % held out), a GPT-2 of 6 layers, 256 channels, 8 heads and a 256-token context (5.3 million parameters), 1200 steps of 16 x 256 tokens (about 18 epochs), AdamW at 1e-3 with 100 warm-up steps and a cosine decay to 1e-4, weight decay 0.1, clipping at 1.0, no dropout. 18 minutes on 8 threads.

Step 0 100 200 400 600 800 1000 1200
validation loss 7.646 5.013 4.671 4.362 4.166 4.075 4.025 3.988

3.99 nats per token is 1.39 per character (0.348 tokens per character), next to nanoGPT's character-level model of this text (10.7 million parameters with dropout, 1.47). oklm run -m build/nano/nano-001200.gguf -p "ROMEO:" -n 80 --temp 0.8 --seed 7:

ROMEO:
O, with his grave, and with a din
Must, as I have a painted to do.

ROMEO:
O, nurse; for I pray you; if you shall
Toof, if you will I have: but the
mory, I will take you have bird,
doot it not be gone to my side

With dropout, past the point of overfitting

The same model and settings for 3000 steps (about 45 epochs) on the GPU (57 ms a step), without and with --dropout 0.2:

Step 1000 1500 2000 2500 3000
no dropout 4.135 4.280 4.755 5.227 5.546
dropout 0.2 4.167 4.046 4.018 4.019 4.036

Without dropout the model learns the training text by heart after about 15 epochs and its validation loss climbs; with it, the loss settles near 4.02.

With Llama blocks

The same tokenizer, data and optimizer settings, with --arch llama --kv-heads 4: 6 layers, 256 channels, 8 query heads over 4 key/value heads, SwiGLU MLP of 704, rotary positions (4.95 million parameters). With the 1200-step schedule it reaches 3.865 at step 600 and then overfits (training loss 2.04, validation 4.22 at step 1200: no dropout), so the run is 600 steps with the schedule ending there. 7.5 minutes on 8 threads.

Step 0 100 200 300 400 500 600
validation loss 7.621 4.747 4.267 4.076 3.954 3.868 3.832

3.83 nats per token, 1.33 per character, in half the steps of the GPT-2 block. llama.cpp loads the checkpoint as it is, and its greedy continuation of "ROMEO:" is the same as oklm run's. oklm run -m build/nano-llama/nano-000600.gguf -p "ROMEO:" -n 80 --temp 0.8 --seed 7:

ROMEO:
Out to them!

Nurse:
A noble boy!

Nurse:
I know it is, sir.

MERCUTIO:
Fell me, sir.

The v0.1.0 models

The models of release-notes-0.1.0.md, trained on one AMD gfx1100 GPU through the OpenCL backend. The data (tools/prepare_release_data.py) and the shared 16 384-token tokenizer (oklm train-tokenizer --vocab 16384 --pre llama-bpe, with <|endoftext|>, <|im_start|>, <|im_end|> and the ChatML template of tools/oklm-chatml.jinja) live in models/datasets/oklm-release/ ($R below). They are not kept in the tree: tools/fetch_datasets.sh --release downloads the sources and tools/prepare_release_data.py <datasets dir> models/datasets/oklm-release writes the texts again; the tokenizer is released (oklm-tok-16k-0.1.0.gguf), so a later release copies it rather than training another (tools/make_release.sh reads it from $R/oklm-tok-16k.gguf).

OKLM-Micro, 33.6 M parameters, on TinyStories (98 M tokens seen, about 1 hour), then tuned on SmolTalk (24.6 M tokens, 35 minutes). The checkpoint before the tune is released too, as OKLM-Micro-Stories:

oklm train -f $R/tinystories.txt --tokenizer $R/oklm-tok-16k.gguf --out micro/oklm-micro \
    --arch llama --layers 8 --embd 512 --heads 8 --kv-heads 4 --ff 1536 --ctx 512 \
    --seq 512 --batch 16 --steps 12000 --lr 6e-4 --lr-min 6e-5 --warmup 500 --wd 0.1 --clip 1 \
    --val-frac 0.005 --eval-every 1000 --save-every 2000 --backend opencl
oklm train -m micro/oklm-micro-012000.gguf -f $R/smoltalk.txt --out micro-chat/oklm-micro-chat \
    --seq 512 --batch 16 --steps 3000 --lr 3e-4 --lr-min 3e-5 --warmup 100 --wd 0.1 --clip 1 --backend opencl

OKLM-Small, 88.1 M parameters, on FineWeb-Edu (229 M tokens seen, 6.7 hours at 0.85 s per step), then tuned on SmolTalk (20.5 M tokens, 35 minutes):

oklm train -f $R/fineweb.txt --tokenizer $R/oklm-tok-16k.gguf --out small/oklm-small \
    --arch llama --layers 12 --embd 768 --heads 12 --kv-heads 4 --ff 2048 --ctx 1024 \
    --seq 1024 --batch 8 --steps 28000 --lr 6e-4 --lr-min 6e-5 --warmup 1000 --wd 0.1 --clip 1 \
    --val-frac 0.002 --eval-every 2000 --save-every 2000 --backend opencl
oklm train -m small/oklm-small-028000.gguf -f $R/smoltalk.txt --out small-chat/oklm-small-chat \
    --seq 1024 --batch 8 --steps 2500 --lr 2e-4 --lr-min 2e-5 --warmup 100 --wd 0.1 --clip 1 --backend opencl
Validation loss start end
Micro on TinyStories 9.83 1.39
Micro tuned on SmolTalk 7.68 2.93
Small on FineWeb-Edu 9.86 3.51
Small tuned on SmolTalk 3.64 2.37

Longer conversation tunes

The released Micro and Small continued on the same SmolTalk text for 6 000 more steps each (2026-11-07), from their F16 files (a fresh optimizer), at 1e-4 with 100 warm-up steps and cosine decay to 1e-5, the release's batches, the same held-out tenth. The text is made again with sh tools/fetch_datasets.sh --release smoltalk (the first smol-smoltalk shard at a pinned revision, its sha256 checked) and tools/prepare_release_data.py models/datasets/sources models/datasets/oklm-release (a source not fetched is left out):

oklm train -m OKLM-Small-0.1.0-f16.gguf -f $R/smoltalk.txt --out small-chat2 --seq 1024 --batch 8 --steps 6000 \
    --lr 1e-4 --lr-min 1e-5 --warmup 100 --wd 0.1 --clip 1 --eval-every 1000 --recompute --backend opencl
oklm train -m OKLM-Micro-0.1.0-f16.gguf -f $R/smoltalk.txt --out micro-chat2 --seq 512 --batch 16 --steps 6000 \
    --lr 1e-4 --lr-min 1e-5 --warmup 100 --wd 0.1 --clip 1 --eval-every 1000 --backend opencl
Held-out loss, step 0 1 000 2 000 3 000 4 000 5 000 6 000 time
Small (0.92 s a step) 2.262 2.304 2.296 2.262 2.235 2.222 2.217 92 min
Micro (0.27 s a step) 2.993 3.010 2.956 2.915 2.877 2.857 2.844 27 min

(The step-0 figures are the released models on this split, measured over 8 batches; the release's table measured fewer.) The loss rises for the first thousand steps, the fresh optimizer's moments settling at the higher rate, then falls below the start: 0.045 for Small, 0.149 for Micro, which had seen 25 M conversation tokens against Small's 20 M and has more to gain per token. Small's greedy answers read the same before and after (fluent lists that lose the thread: "Take regular breaks... make sure your body is getting enough sleep"); the tune does not buy facts the pretraining lacks. The checkpoints are not released.

tools/make_release.sh 0.1.0 NANO MICRO SMALL STORIES writes each model in F16 and Q8_0 with its name, version, licence and data in general.* (oklm quantize --override-kv), checks that each loads and generates, and adds the tokenizer, the template, the notes, the licence and SHA256SUMS in build/release-0.1.0/. llama.cpp loads and runs the files as well.

Fine-tuning SmolLM2-135M

The Hugging Face checkpoint loaded directly (30 layers, 576 channels, 9 query heads over 3 key/value heads, tied output head), the oklm train command above: 30 updates of 4 batches of 4 x 128 tokens of Tiny Shakespeare (its own tokenizer: 341 094 tokens), AdamW at 1e-4, 5 warm-up steps, cosine decay to 1e-5. 4.5 minutes.

Step 0 10 20 30
validation loss 3.810 3.524 3.446 3.430

The checkpoint is a Llama GGUF in F32 (540 MB, optimizer sidecar 1.08 GB). llama.cpp's greedy continuation of "KING HENRY:" on it is the same as oklm run's, token for token, which checks the reordering of the query and key rows end to end.

LoRA on SmolLM2-135M

The published F16 GGUF loaded directly, rank 8, alpha 16, every projection (210 adapters, 2 442 240 parameters), batches of 8 x 128 tokens of Tiny Shakespeare, AdamW at 1e-3, 20 warm-up steps, cosine decay, --backend rocm: 200 steps in 30 s.

Step 0 50 100 150 200
validation loss 3.834 3.418 3.337 3.304 3.297

The CPU, opencl and rocm give the same losses, to the fourth decimal, over the first steps. llama.cpp's perplexity on the first 60 000 bytes of the text, 256-token windows: 34.96 for the model, 27.399 with the adapter (llama-perplexity --lora), 27.399 on the model oklm lora-merge wrote (F16 weights); oklm ppl --lora 35.07 to 27.48 (its windows are scored differently).

A step of 8 x 128 tokens:

Backend Every parameter LoRA, every projection LoRA, q, k, v, o
opencl 156 ms 140 ms 119 ms
rocm 110 ms 102 ms 91 ms
cpu 3.3 s

The model's weight gradients skipped save a third of the large products; the adapters add six small products each (1260 a step for every projection), which the split products keep to a few microseconds each.

Fine-tuning Qwen2.5-0.5B and Qwen3-0.6B

The published GGUF files loaded directly (Qwen2.5-0.5B-Instruct in Q4_K_M, Qwen3-0.6B in Q8_0, both converted to F32), on the GPU: 30 updates of 4 batches of 4 x 128 tokens of Tiny Shakespeare (the models' own tokenizer: 301 829 tokens), AdamW at 1e-4, 5 warm-up steps, cosine decay to 1e-5.

oklm train -f data/tinyshakespeare.txt -m Qwen3-0.6B-Q8_0.gguf --out build/ft-qwen3/ft \
    --steps 30 --batch 4 --accum 4 --seq 128 --lr 1e-4 --lr-min 1e-5 --warmup 5 --eval-every 10
Model Step 0 10 20 30 Time
Qwen3-0.6B (16 heads of 128 over 1024 channels, Q/K norms) 4.566 4.078 3.989 3.911 57 s
Qwen2.5-0.5B (Q/K/V biases) 4.340 4.199 4.082 3.945 47 s

The checkpoints are qwen3 and qwen2 GGUF files in F32 (2.4 and 2.5 GB, optimizer sidecars twice that). llama.cpp's greedy continuation of "KING HENRY:" on each is the same as oklm run's, token for token, as is that of a 4-layer Qwen3 trained from scratch with --head-dim 48 over 128 channels.

Reinforcement learning

src/train/rl.c, oklm rl: fine-tuning from rewards on tasks with known answers, with group-relative advantages (GRPO), several updates per batch of samples with PPO's clipped ratio, and a KL penalty towards a reference model.

  • oklm_trainer_generate() samples completions from the model being trained (temperature, or the most likely token at 0), evaluating the whole sequence again for each new token: meant for the short completions of such tasks. oklm_trainer_last_logits() fetches only the last position's logits, so a device does not send back the whole logits.
  • oklm_rl_step(): for each task, a group of completions is sampled and scored by the caller's reward; each completion's advantage is its reward minus the group's mean, over the group's standard deviation. The loss is minus the advantage times the log-probability of each sampled token, averaged over the completion's tokens, the group and the tasks: the cross-entropy gradient of the sampled tokens weighted by their advantage, oklm_trainer_backward_weighted() (per-row weights, negative ones included; weights of 1 / N give oklm_trainer_backward() bit for bit). One AdamW update follows by default. Groups whose rewards are all equal carry no advantage; a step with no term at all makes no update.
  • Several updates per batch (oklm_rl_params.updates, --updates): the samples serve that many AdamW updates. The first records each sampled token's log-probability under the sampling policy (from the same forward pass, oklm_trainer_row_losses()); the later ones weigh a token by its advantage times its probability ratio to that policy, and by nothing once the ratio left 1 +/- clip (0.2, --clip) on the advantage's side: the gradient of PPO's clipped surrogate. With one update the ratio is 1 and the weights are the advantages alone, as before. In the test, four updates a batch learn the task in 26 sampling steps against 81 with one (lr 5e-3; at 2e-2 four overshoot and collapse onto wrong answers, which one update does not).
  • KL penalty (oklm_rl_params.kl and ref, --kl, --ref): each sampled token adds kl x (ref/pi - log(ref/pi) - 1), the estimate whose expectation under the policy is KL(pi || ref), positive and zero where the two agree; its gradient in the token's log-probability is kl x (1 - ref/pi). The reference is another trainer, only evaluated (oklm rl builds it from --ref, or the starting model, behind rank-1 LoRA adapters whose B starts at zero: it computes the model itself without a full copy's optimizer state). Groups with equal rewards still carry the KL term. oklm_rl_stats.kl reports the mean estimate. In the test, 40 steps without the penalty end 3.34 nats from the start (exact KL of the next token), with kl 1 at 1.12.
  • oklm rl -m model -f tasks.jsonl --out prefix: one {"prompt": ..., "answer": ...} a line, a share held out (--val-frac) whose greedy accuracy is printed every --eval-every steps. A completion earns 1 when its last number is a numeric answer ("13 + 6 = 19." answers 19), or when it starts with any other answer. Checkpoints are those of oklm train.
oklm rl -m SmolLM2-135M-Instruct-f16.gguf -f additions.jsonl --out build/rl/add \
    --steps 150 --tasks 16 --group 8 --max-new 12 --lr 2e-5 --eval-every 25

SmolLM2-135M-Instruct on the 400 sums of two numbers from 0 to 19, asked through its chat template ("What is 13+6?"), 40 held out, on the GPU (150 steps of 16 tasks and groups of 8, 12 new tokens, 13.5 minutes):

Step 0 25 50 100 125 150
held-out greedy accuracy 0.025 0.825 0.800 0.875 0.900 0.925
training reward (sampled) 0.195 0.734 0.805 0.930 1.000 0.945

With two updates a batch and the KL penalty (--updates 2 --kl 0.04, the same tasks, 30 steps, 225 s on the GPU), the held-out accuracy reaches 0.875 at step 10 and 0.900 at step 30; the KL estimate starts at 0 (the reference is the starting model) and stays near 0.3, and 1 to 9 % of the second update's tokens are clipped, fewer as the policy settles.

The base model starts with an explanation it rarely finishes ("To find 17 + 15, we need to add..."); the checkpoint, through oklm run, answers "17 + 15 = 32". A run gives the same steps on the same device. The test (test_train) has a freshly initialized model learn, from rewards alone, which of 37 tokens answers each of 8 prompts (4.7 % reward at the first step, all 8 answered greedily after about 30).

Rewards teach only what the model sometimes does already: on additions of two-digit numbers, where SmolLM2-135M's samples are right less than once in 600, no group ever differs and nothing is learned. Not done: a reward computed by a command.

Abliteration

src/model/abliterate.c, oklm abliterate: a chat model's refusal direction found and projected out of its weights (Arditi et al., 2024), pure C on the CPU, the same loaders and writers as oklm convert. It makes the trainable base the fine-tuning runs of "Default models" start from, without Python or PyTorch.

  • The direction. Two sets of instructions, one the model declines and one it answers, are run through it in its chat template. At each prompt's last token the residual stream entering every layer is read: a matrix product observer (the importance-matrix mechanism of oklm_imatrix_compute()) catches the input of each block's first attention product, RMSNorm(h) * g, and divides it back by the norm's weights g to recover h / rms(h), so a prompt's scale does not sway it. Per layer, the candidate direction is the difference of the two sets' mean residuals, normalized. oklm_refusal_directions() returns one per layer with two figures: the separation (the gap of the two sets' projections on the direction over their pooled deviation) and the gap (the mean difference's norm relative to the residual's). The layer that separates best within a band of the depth (0.3 to 0.7 by default) is chosen, oklm_refusal_pick_layer().
  • The weights. oklm_abliterate_write() makes every matrix that writes into the residual stream orthogonal to the unit direction r: W' = W - s r (r^T W) for each block's attn_output and ffn_down (the experts' ffn_down_exps and ffn_down_shexp included), and the token embeddings e' = e - s r (r . e) when the model has an output.weight of its own (a tied head is left, since it also reads the residual). Each is produced as the file is written, dequantized, changed and converted back, rows over the CPU threads; a matrix kept in F32, F16 or BF16 stays its type, else it becomes F16, and every other tensor is copied. The scale s (1 by default, 0 to 2) weakens the removal below 1. The result is trainable weights (or an F16/BF16 GGUF) whose making OKLM controls.
  • Measuring. oklm refusals -m model --prompts refused.txt answers each instruction greedily and counts those that read as a refusal (oklm_is_refusal(), the usual markers); --ref base --harmless answered.txt adds the mean KL divergence of the model's first-token distribution from the reference's on instructions it should answer, which measures how much the projection moved its behaviour elsewhere. -n N is the length of each answer in tokens (48), --limit N reads the first N lines of each file only, and --show prints every answer as it is judged: REFUSED or answered, the prompt, then the answer on one line, before the total.

tools/fetch_abliterate_data.sh fetches two prompt sets at pinned commits and splits each into a part for the direction and a held-out part for the measures: AdvBench (Zou et al.) for the declined set, Alpaca instructions without an input for the answered one.

sh tools/fetch_abliterate_data.sh
oklm abliterate -m models/qwen3-8b/Qwen3-8B-BF16.gguf \
    --harmful models/abliterate/harmful_train.txt \
    --harmless models/abliterate/harmless_train.txt --out Qwen3-8B-abliterated.gguf
oklm refusals -m Qwen3-8B-abliterated.gguf --prompts models/abliterate/harmful_test.txt
oklm refusals -m Qwen3-8B-abliterated.gguf --ref models/qwen3-8b/Qwen3-8B-BF16.gguf \
    --harmless models/abliterate/harmless_test.txt

On Qwen3-8B (BF16, 36 layers), the direction found from the 416 declined and 416 answered training prompts separates the two sets ever more deeply: the picker's band (0.3 to 0.7 of the depth) takes layer 24 (separation 11.2, the mean difference 0.71 of the residual there). With its direction removed at scale 1, the held-out refusal rate (104 declined instructions, greedy) falls from 97.5 % to 7.5 %, near the published figures; but the first-token KL divergence from the base on held-out answered instructions is 1.29, so the projection also moves answered replies. The removal is a trade-off along the depth, measured here with 24 prompts of each held-out set:

layer separation refusals first-token KL (answered)
base — 97.5 % 0
14 3.3 79 % 1.07
16 4.4 62 % 0.18
19 7.6 38 % 0.81
24 (picked) 11.2 7.5 % 1.29

The strongest refusal removal costs the most on answered prompts; a shallower layer, or --scale below 1, keeps more of the base's behaviour for less refusal reduction. The operating point is the security training's to choose. On Qwen3-0.6B (Q8_0, 28 layers) the middle layers separate best; with the chosen layer's direction removed the held-out refusal rate goes from a near-half to none while the KL on answered instructions stays small. The per-layer table and the 0.6B figures are checked in test_infer_public's test_abliterate, which also confirms that the written model's own mean differences keep almost nothing along the removed direction. Computing the 8B directions on the CPU over 832 prompts takes about half an hour; --save-dirs writes them so --load-dirs --layer N writes a model at any layer or scale in seconds without recomputing.

Machine-readable progress

oklm train, oklm rl, oklm abliterate and oklm refusals take --progress-json FILE, and so do oklm convert, oklm quantize and oklm ppl (their lines are the last two tables): the run written as JSON lines, one object a line, each line flushed as it is written, so a program can follow a run from the file (or a pipe) while it goes. With --progress-json - the lines go to standard output and everything the command otherwise prints there (the report, the samples, the tables) goes to standard error, so standard output holds JSON lines only. Without the option nothing changes. The training window follows a run through such a file (oklm-gui --tool train, gui.md). The file is truncated when the command starts, after --out's directories are made; a file that cannot be opened stops the command (status 1). The writer is oklm/io/progress.h.

$ oklm train -f data/tinyshakespeare.txt --out nano/nano --vocab 512 --layers 2 --embd 64 --heads 2 --ctx 64 \
    --steps 4 --eval-every 2 --save-every 2 --backend cpu --progress-json - 2>/dev/null
{"type":"start","elapsed":0.432,"command":"train","mode":"scratch","model":null,"resume":null,"out":"nano/nano","arch":"gpt2","parameters":136960,"trainable":136960,"layers":2,"embd":64,"heads":2,"vocab":512,"context":64,"device":"cpu","total":4,"first_step":0,"batch":4,"seq":64,"accum":1,"lr":0.0003,"lr_min":3e-05,"warmup":50,"weight_decay":0.1,"grad_clip":1,"dropout":0,"seed":1337,"lora_rank":0,"eval_every":2,"save_every":2,"sample_every":0,"data":"data/tinyshakespeare.txt","data_kind":"text","tokens":575809,"train_tokens":518229,"val_tokens":57580}
{"type":"step","elapsed":0.499,"step":1,"total":4,"loss":6.232001,"val_loss":6.244404,"lr":6e-06,"grad_norm":1.616281,"step_seconds":0.06725377,"tokens_per_second":3806.447,"eta_seconds":0.201763}
{"type":"step","elapsed":0.514,"step":2,"total":4,"loss":6.24755,"lr":1.2e-05,"grad_norm":1.521801,"step_seconds":0.01515267,"tokens_per_second":6211.878,"eta_seconds":0.08242274}
{"type":"checkpoint","elapsed":0.515,"path":"nano/nano-000002.gguf","step":2}
{"type":"step","elapsed":0.554,"step":3,"total":4,"loss":6.221951,"val_loss":6.243486,"lr":1.8e-05,"grad_norm":1.663938,"step_seconds":0.03843528,"tokens_per_second":6308.998,"eta_seconds":0.04057697}
{"type":"step","elapsed":0.561,"step":4,"total":4,"loss":6.223746,"lr":2.4e-05,"grad_norm":1.538278,"step_seconds":0.007306504,"tokens_per_second":7934.69,"eta_seconds":0}
{"type":"checkpoint","elapsed":0.582,"path":"nano/nano-000004.gguf","step":4}
{"type":"end","elapsed":0.582,"ok":true,"step":4,"total":4,"loss":6.223746,"val_loss":6.241247,"files":["nano/nano-000002.gguf","nano/nano-000004.gguf"]}

Every line is a JSON object on one line, in UTF-8, holding first type (a string, the tables below) then elapsed (seconds since the command opened the stream, three decimals), then the fields of its type in the tables' order. Numbers have 7 significant digits; a number that is not finite (a loss that diverged) is null. A string field with no value (no --resume, no --system) is null. Bytes of a text that are not UTF-8 (a sample cut inside a character) are written as U+FFFD. A reader should ignore the fields and the types it does not know: later versions may add some.

The order. A run that gets as far as its model writes one start line first (abliterate writes it before loading anything). The last line is always end, once, whether the run succeeded or not: a run that fails before start (a file that cannot be read, a model that cannot be loaded) writes end alone. A stream without an end line is a command that was killed. An error in the options (the usage, status 2) is found before the stream is opened, and writes nothing.

Lines of every command

Type Field Value
end ok true when the command succeeded (exit status 0)
error the error's text, only when ok is false (the detailed message is on standard error)
figures the command's final figures, in its table below
files an array of the paths of the files the command wrote, in the order written, each once; empty when none

oklm train

Type Field Value
start command "train"
mode "scratch" (a new model), "finetune" (-m), "lora" (-m with --lora) or "resume" (--resume)
model, resume, out the options' values (null when not given)
arch the architecture (gpt2, llama, qwen2, qwen3, ...)
parameters, trainable the model's parameter count, and how many are trained (the adapters' with LoRA)
layers, embd, heads, vocab, context the model's shape
device where it trains: cpu, opencl, rocm
total the step the run ends at
first_step updates already made (0, or a resumed checkpoint's step)
batch, seq, accum sequences a batch, tokens a sequence, batches an update
lr, lr_min, warmup, weight_decay, grad_clip, dropout, seed the options in effect (a resumed run's come from its sidecar)
lora_rank, lora_alpha the adapters' rank (0 without LoRA) and, with LoRA, their alpha
eval_every, save_every, sample_every the three periods, in steps (0: never)
data, data_kind the data file, and "text" (-f) or "chat" (--chat-data)
tokens, train_tokens, val_tokens with "text": the text's tokens, and the two shares
conversations, train_conversations, val_conversations, assistant_tokens with "chat": the conversations, the two shares, the tokens the loss is taken on
step step, total updates made, this one included, and the step the run ends at
loss the training loss of this update's batches (on the assistant tokens with "chat")
val_loss the validation loss, only on the steps it was measured at (every eval_every, before the update: loss and val_loss of a line are of the same weights)
lr, grad_norm the learning rate of the update, the gradient's norm before clipping
step_seconds the step's time, its evaluation and sample included
tokens_per_second tokens trained on a second, since the run's first step
eta_seconds an estimate of the seconds left: the mean time of the steps made (evaluations and checkpoints included) times the steps left
loss_scale, skipped with --loss-scale: the scale and the updates skipped so far
sample step, text with --sample-every: the updates made when the sample was taken, and its text
checkpoint path, step a checkpoint written (its .train sidecar beside it), and its step
end step, total updates made and planned, when the trainer was set up
loss the last step's training loss, when a step was made
val_loss the final validation loss, when there is validation data

oklm rl

Type Field Value
start command "rl"
model, resume, out the options' values (null when not given)
arch, parameters, layers the model
device where it trains
data, tasks, train_tasks, val_tasks the task file, its tasks, those trained on and those held out
total the steps of this run (--steps)
first_update updates the model had (0, or a resumed checkpoint's)
tasks_per_step, group, max_new, temperature --tasks, --group, --max-new, --temp
lr, weight_decay, updates, clip, kl, seed the options in effect
ref the model the KL penalty is taken against (null without --kl)
eval_every, save_every the two periods, in steps
step step, total steps of this run made, this one included, and planned
reward, solved, length the mean reward, the share of completions rewarded, their mean length in tokens
learned tasks whose completions differed in reward, so gave a gradient
grad_norm, lr the gradient's norm (0 when no update was made), the learning rate
accuracy the held-out accuracy (greedy), only on the steps it was measured at, before the update
kl with a reference: the mean KL divergence from it a sampled token
updates, clipped with --updates above 1: the updates made, the share of tokens whose ratio was clipped
step_seconds, samples_per_second the step's time (its evaluation included), the completions sampled a second in it
eta_seconds an estimate of the seconds left, as for train
checkpoint path, step a checkpoint written; step is the model's update count, as in the file's name (a step without signal makes no update)
end step, total steps made and planned, when the trainer was set up
reward the last step's mean reward, when a step was made
accuracy the final held-out accuracy, when tasks are held out

oklm abliterate

Type Field Value
start command "abliterate"
model, harmful, harmless, load_dirs, save_dirs, out, system the options' values (null when not given)
layer --layer as given: "auto" or a number, as a string
band_low, band_high, scale, limit --band, --scale, -n (0: every prompt)
device "cpu"
stage stage a stage begins: "directions" (the prompts run through the model, the long one) or "write" (the model written)
harmful, harmless, layers with "directions": the prompts of each set, the model's layers
directions harmful, harmless, layers, embd, seconds the directions are measured: the prompts used, the layers, the width, the time taken
layer layer, layers one line a layer, in order: its index and the layer count
separation, gap the two figures of the table: how far apart the two sets' projections are in standard deviations, and the gap of their means relative to the residual's norm
chosen layer, auto the layer whose direction is used, and whether --layer auto chose it
separation, gap its figures (not with --load-dirs)
file path, kind a file written: "directions" (--save-dirs) or "model" (--out)
seconds with "model": the time the writing took
end layer the layer used, when the run succeeded

With --load-dirs the stream is start, chosen, stage ("write"), file, end.

oklm refusals

Type Field Value
start command "refusals"
model, arch, layers the model judged
prompts, total the prompt file (null without --prompts) and its prompts
ref, harmless, harmless_total the reference model, the answered prompts and their count (null, null, 0 without them)
max_tokens, limit, system -n, --limit (0: every prompt), --system
device the backend asked for ("auto" when --backend is not given)
prompt index, total one line a prompt judged: its number from 1, and their count
refused whether the answer is a refusal
refused_so_far refusals among the prompts judged so far
prompt, answer the prompt and the answer's first max_tokens tokens
stage stage, total "kl": the first-token distributions of the two models begin, over total answered prompts
summary refused, total, rate with --prompts: the refusals, the prompts, and their ratio (0 to 1)
kl, kl_prompts with --ref: the mean first-token KL divergence from the reference, and the prompts it is a mean of
end refused, total, rate, kl, kl_prompts the summary's figures again, when the run succeeded

tests/harness/test_cli_progress.c checks the writer (escapes, null, the files' list), then runs the four commands: each line parses as JSON and holds its type's fields above, the stream on standard output holds nothing else, and a run without the option prints what it printed before.

oklm convert, oklm quantize

start is written once the file's header is ready, before the first tensor; a run that fails before (a source that cannot be read, a tensor no type fits) writes end alone, and so does --dry-run (convert.md).

Type Field Value
start command "convert" or "quantize"
in, out the source and the file written
ftype the target type as given ("f16", "Q4_K_M")
tensors tensors of the file
bytes the size the file will have
tensor index, total the tensor just written, from 1, and their count
name its name
bytes_done, bytes_total bytes of the file written so far, and its size
end dry_run true for --dry-run: nothing was written
tensors, bytes the file's tensor count and size, when ok

oklm ppl

Type Field Value
start command "ppl"
model, file the model and the text
ctx tokens a window
tokens tokens of the text
windows whole windows in it: the ones scored
window index, total the window just scored, from 1, and their count
perplexity the perplexity over the windows so far
end perplexity the perplexity, when ok
tokens, windows tokens scored (the second half of each window) and windows, when ok

Next

Longer runs of the released models (Small saw about 230 M tokens, several billion would suit its size).