OKLMdocumentation v1.0.3
On this page

OKLM 0.1.0 release notes

The first models trained from scratch by OKLM, with its own trainer, on one machine, and the files to run them with OKLM or llama.cpp.

Status: released 2026-11-02 (tag v0.1.0); OKLM-Micro-Stories added to the release the same day.

The models

Model Architecture Parameters Context Trained on Tokens seen Validation loss
OKLM-Nano GPT-2, 6 layers, 256 channels, 8 heads 5.3 M 256 Tiny Shakespeare 4.9 M (1 200 steps of 16 × 256) 3.99 per token (its own 2 048-token tokenizer)
OKLM-Micro Llama, 8 layers, 512 channels, 8 heads (4 key / value), MLP 1 536 33.6 M 512 TinyStories, then SmolTalk conversations 98.3 M stories (12 000 steps of 16 × 512), then 24.6 M conversation tokens (3 000 steps) 1.39 on stories; 2.93 on conversations after the tune
OKLM-Micro-Stories the same as Micro 33.6 M 512 TinyStories 98.3 M (12 000 steps of 16 × 512) 1.39
OKLM-Small Llama, 12 layers, 768 channels, 12 heads (4 key / value), MLP 2 048 88.1 M 1 024 FineWeb-Edu, then SmolTalk conversations 229 M web tokens (28 000 steps of 8 × 1 024), then 20.5 M conversation tokens (2 500 steps) 3.51 on web text; 2.37 on conversations after the tune

Micro, Micro-Stories and Small share one byte-level BPE tokenizer of 16 384 tokens, trained by oklm train-tokenizer on the release data, with <|endoftext|>, <|im_start|> and <|im_end|> as control tokens and a ChatML template. Nano carries its own 2 048-token tokenizer.

Training ran with oklm train on an AMD gfx1100 GPU (20 GiB) through OKLM's OpenCL backend: AdamW, cosine decay after warm-up, weight decay 0.1, gradient clipping at 1, float32 weights and optimizer state. Small took about 6.7 hours of pretraining (0.85 s per step) and 35 minutes of tuning; Micro about 1 hour 40 minutes in all; Nano 18 minutes on 8 CPU threads. The commands and settings are in training.md.

Files

Each model comes in F16 and Q8_0 (OKLM-<Model>-0.1.0-f16.gguf, ...-q8_0.gguf), with its name, version, licence and training data in its metadata (general.*). With them: oklm-tok-16k-0.1.0.gguf (the shared tokenizer on its own), oklm-chatml.jinja (the chat template, also inside the models), LICENSE, these notes, and SHA256SUMS. Every file was loaded back and made to generate before it was listed.

Using them

oklm chat -m OKLM-Small-0.1.0-q8_0.gguf --repeat-penalty 1.15
oklm run  -m OKLM-Micro-0.1.0-q8_0.gguf -p "Once upon a time" --repeat-penalty 1.1
oklm run  -m OKLM-Micro-Stories-0.1.0-q8_0.gguf -p "Once upon a time" --temp 0.7 --repeat-penalty 1.1
oklm run  -m OKLM-Nano-0.1.0-f16.gguf -p "ROMEO:" --temp 0.8
oklm serve -m OKLM-Small-0.1.0-q8_0.gguf -c 1024

They are ordinary GGUF Llama and GPT-2 models: llama.cpp loads them as well.

What to expect

These are small models trained on a single GPU in a day; they show what the stack can do end to end, not an assistant to rely on.

  • Nano writes Shakespeare-shaped verse with speaker names, often with invented words:

    ROMEO:
    The king is dead; and that the crown?
    
    FRIAR LAURENCE:
    It is the lably; and the rest of thy looks,
  • Micro-Stories is Micro before its conversation tune: it writes coherent children's stories from an opening line:

    Once upon a time, there was a little girl named Mia. She loved to play
    with her toys and make things. One day, she decided to make something
    special for her mom.
  • Micro is the same model tuned on conversations: it follows the chat format but repeats itself quickly; --repeat-penalty helps only somewhat. A demonstration of fine-tuning more than a chat model.

  • Small answers in fluent, well-formed English, with Markdown lists, but loses the thread and makes facts up: about 230 M tokens is far below what a model of its size needs (several billion). Use --repeat-penalty 1.1 to 1.15.

Licences

The weights are released under the Apache License 2.0 (LICENSE), like OKLM. The training data keeps its own licences: Tiny Shakespeare (public domain), TinyStories (CDLA-Sharing-1.0), FineWeb-Edu (ODC-By 1.0), SmolTalk (Apache-2.0).