OKLM 0.1.0 release notes
The first models trained from scratch by OKLM, with its own trainer, on one machine, and the files to run them with OKLM or llama.cpp.
Status: released 2026-11-02 (tag v0.1.0); OKLM-Micro-Stories added to the release the same day.
The models
| Model | Architecture | Parameters | Context | Trained on | Tokens seen | Validation loss |
|---|---|---|---|---|---|---|
| OKLM-Nano | GPT-2, 6 layers, 256 channels, 8 heads | 5.3 M | 256 | Tiny Shakespeare | 4.9 M (1 200 steps of 16 × 256) | 3.99 per token (its own 2 048-token tokenizer) |
| OKLM-Micro | Llama, 8 layers, 512 channels, 8 heads (4 key / value), MLP 1 536 | 33.6 M | 512 | TinyStories, then SmolTalk conversations | 98.3 M stories (12 000 steps of 16 × 512), then 24.6 M conversation tokens (3 000 steps) | 1.39 on stories; 2.93 on conversations after the tune |
| OKLM-Micro-Stories | the same as Micro | 33.6 M | 512 | TinyStories | 98.3 M (12 000 steps of 16 × 512) | 1.39 |
| OKLM-Small | Llama, 12 layers, 768 channels, 12 heads (4 key / value), MLP 2 048 | 88.1 M | 1 024 | FineWeb-Edu, then SmolTalk conversations | 229 M web tokens (28 000 steps of 8 × 1 024), then 20.5 M conversation tokens (2 500 steps) | 3.51 on web text; 2.37 on conversations after the tune |
Micro, Micro-Stories and Small share one byte-level BPE tokenizer of
16 384 tokens, trained by oklm train-tokenizer on the
release data, with <|endoftext|>,
<|im_start|> and <|im_end|> as
control tokens and a ChatML template. Nano carries its own 2 048-token
tokenizer.
Training ran with oklm train on an AMD gfx1100 GPU (20
GiB) through OKLM's OpenCL backend: AdamW, cosine decay after warm-up,
weight decay 0.1, gradient clipping at 1, float32 weights and optimizer
state. Small took about 6.7 hours of pretraining (0.85 s per step) and
35 minutes of tuning; Micro about 1 hour 40 minutes in all; Nano 18
minutes on 8 CPU threads. The commands and settings are in training.md.
Files
Each model comes in F16 and Q8_0
(OKLM-<Model>-0.1.0-f16.gguf,
...-q8_0.gguf), with its name, version, licence and
training data in its metadata (general.*). With them:
oklm-tok-16k-0.1.0.gguf (the shared tokenizer on its own),
oklm-chatml.jinja (the chat template, also inside the
models), LICENSE, these notes, and SHA256SUMS.
Every file was loaded back and made to generate before it was
listed.
Using them
oklm chat -m OKLM-Small-0.1.0-q8_0.gguf --repeat-penalty 1.15
oklm run -m OKLM-Micro-0.1.0-q8_0.gguf -p "Once upon a time" --repeat-penalty 1.1
oklm run -m OKLM-Micro-Stories-0.1.0-q8_0.gguf -p "Once upon a time" --temp 0.7 --repeat-penalty 1.1
oklm run -m OKLM-Nano-0.1.0-f16.gguf -p "ROMEO:" --temp 0.8
oklm serve -m OKLM-Small-0.1.0-q8_0.gguf -c 1024
They are ordinary GGUF Llama and GPT-2 models: llama.cpp loads them as well.
What to expect
These are small models trained on a single GPU in a day; they show what the stack can do end to end, not an assistant to rely on.
Nano writes Shakespeare-shaped verse with speaker names, often with invented words:
ROMEO: The king is dead; and that the crown? FRIAR LAURENCE: It is the lably; and the rest of thy looks,Micro-Stories is Micro before its conversation tune: it writes coherent children's stories from an opening line:
Once upon a time, there was a little girl named Mia. She loved to play with her toys and make things. One day, she decided to make something special for her mom.Micro is the same model tuned on conversations: it follows the chat format but repeats itself quickly;
--repeat-penaltyhelps only somewhat. A demonstration of fine-tuning more than a chat model.Small answers in fluent, well-formed English, with Markdown lists, but loses the thread and makes facts up: about 230 M tokens is far below what a model of its size needs (several billion). Use
--repeat-penalty 1.1to1.15.
Licences
The weights are released under the Apache License 2.0
(LICENSE), like OKLM. The training data keeps its own
licences: Tiny Shakespeare (public domain), TinyStories
(CDLA-Sharing-1.0), FineWeb-Edu (ODC-By 1.0), SmolTalk (Apache-2.0).