ADR-0008: Training checkpoints
Records how a training run is saved and resumed: a model file that every tool loads as it is, and a sidecar next to it with the rest of the state.
Status: accepted (2026-11-01).
Context
A checkpoint must let a run continue exactly where it stopped, and the trained model must be usable for inference. Resuming needs more than the weights: the optimizer moments (AdamW keeps two values per parameter, twice the size of the weights), the step count, the learning-rate schedule, the random state and the position in the training data. Inference needs none of that.
Two layouts were considered:
- One file holding the weights and the optimizer state (llama.cpp's
old
train-text-from-scratchdid this, with an export step to get a model). llama.cpp refuses a model file with tensors it does not expect ("wrong number of tensors"), so such a file is not a model until it is exported, and it is three times the size of one. - The weights as an ordinary model file, and the training state in a second file next to it (the default proposed when the question was opened).
Decision
- Weights:
<name>.gguf, a normal GGUF model: hyper-parameters, tokenizer and every parameter under its usual tensor name, in F32 (the values the optimizer updates, bit for bit).oklm run,oklm chat,oklm quantizeand llama.cpp load it unchanged; exporting a trained model is copying this file (or quantizing it). - Sidecar:
<name>.gguf.train, also in GGUF layout (one reader and writer for both, readable bygguf-dump) with the architecture keyoklm-train, so it is never mistaken for a model:- the AdamW moments, F32 tensors named
adam.m.<tensor>andadam.v.<tensor>; - the step, the optimizer and schedule settings, the batch shape, the
random state and the data position, as
train.*keys; - a 64-bit hash of the weights' tensor data. A sidecar whose hash does not match the weights next to it is refused, so a pair that does not belong together (one of the two files replaced or half written) is never resumed.
- the AdamW moments, F32 tensors named
- Writing: each file is written to a temporary name
and renamed over the final one, weights first, sidecar second. A crash
between the two renames leaves new weights with the old sidecar, which
the hash rejects; the CLI names checkpoints by step
(
<prefix>-<step>.gguf), so the previous pair is still there. - Resuming loads the weights through the ordinary model loader, then the sidecar. Resuming from a weights file without a sidecar starts a new optimizer state at step 0 (this is how a fine-tune starts from a published model). Training continued from a checkpoint gives the same parameters, bit for bit, as the same steps run without stopping.
Consequences
A checkpoint is a usable model at every save, with nothing to convert, and deleting the sidecar is all it takes to ship it. Two files must travel together to resume, and the hash makes a mismatch an error rather than a silent restart. F32 weights are four times the size of a Q8_0 file; quantizing is a separate step, as for any model.
Amended (2026-11-07): oklm train --save-type bf16 writes
the weights' matrices in BF16 (the 1-D tensors stay F32), and
--opt-type bf16 keeps the moments in BF16 in the trainer
and in the sidecar. The hash is that of the weights as written, so a
BF16 checkpoint resumes, from its rounded weights: bit for bit as the
saved trainer would train on from them, not as the uninterrupted run (training.md).