OKLMdocumentation v1.0.3
On this page

Tokenizer

How OKLM turns text into token ids and back, which tokenizer families and file formats it reads and writes, how it is checked against Hugging Face, and how to train a tokenizer. API: include/oklm/tok/tokenizer.h, regex.h, unicode.h, bpe_train.h.

Status: implemented (byte-level BPE, SentencePiece-style BPE, training).

Families

Family GGUF tokenizer.ggml.model Examples Symbols Merging
Byte-level BPE gpt2 GPT-2, Llama 3, Qwen2.5, SmolLM2 bytes, through GPT-2's byte -> character table by merge rank
SentencePiece-style BPE llama, gemma4 Llama 2, TinyLlama, Mistral v1, Gemma 4 UTF-8 characters, <0xNN> byte fallback by merge rank when merges are known, else by piece score
WordPiece bert BERT, BGE embedding models words (lower case and accents per the vocabulary), word starts marked with ▁ greedy longest match; a word with an unmatched position is [UNK]
Unigram t5 T5 words split on whitespace, each behind ▁, after SentencePiece's precompiled normalization map best total piece score (Viterbi, double precision); uncovered characters are <unk>, runs fused

WordPiece follows llama.cpp: decomposition and the combining marks dropped when accents are stripped (tokenizer.ggml.normalizer.*, on with lower case by default), control characters dropped, whitespace ends a word, punctuation, ASCII symbols and CJK ideographs are words of their own. From a tokenizer.json (model WordPiece, BertNormalizer, BertPreTokenizer, BertProcessing) the vocabulary is stored as llama.cpp's converter stores it: ## pieces without the prefix, the other pieces with the ▁ word-start marker, [...] tokens as they are; such tokenizers follow Hugging Face's normalizer (only non-spacing marks stripped), which an OKLM key keeps through GGUF. With special tokens, a text is framed [CLS] ... [SEP] both ways: from a GGUF vocabulary as llama.cpp does (tokenizer.ggml.bos_token_id, else 101, and tokenizer.ggml.seperator_token_id, else 102; add_sep_token false turns [SEP] off), from a tokenizer.json by its BertProcessing.

Unigram follows Hugging Face's model: the Precompiled normalizer's map (SentencePiece's double-array trie of replacements, the longest key at each position) or GGUF tokenizer.ggml.precompiled_charsmap, words split on whitespace, then the segmentation maximizing the sum of piece scores; a character no single-character piece covers is <unk>, scored 10 below the lowest score, and adjacent <unk> fuse.

Encoding pipeline

This is Hugging Face tokenizers' pipeline, step for step:

  1. Added tokens. Special and added tokens are found in the raw text, leftmost-longest, honouring lstrip, rstrip and single_word; each becomes its id and splits the text into segments. Special (control and unknown) tokens are only matched with parse_special (the default).
  2. Normalization of each segment: NFC when the tokenizer asks for it (Qwen); for SentencePiece models, spaces become ▁ and a ▁ is prepended (to every segment, or only the first with Metaspace prepend_scheme: "first").
  3. Pre-tokenization. Each pattern of the tokenizer splits every piece: matches and the gaps between them become pieces ("Isolated" splitting).
  4. Model. Each piece becomes symbols and is merged with a binary heap of candidate pairs, lowest rank first (ties: leftmost). With ignore_merges (Llama 3), a piece that is a whole vocabulary entry is used directly. Score-based merging (GGUF llama vocabularies without merges) picks the highest-scoring pair whose concatenation is a token, then falls back to bytes for what is left, as llama.cpp does. A character missing from the vocabulary becomes its <0xNN> tokens when byte fallback is on and all of them exist, else the unknown token; with fuse_unk (tokenizer.json only, GGUF has no key for it) a run of such characters gives a single unknown token.
  5. Post-processing. With add_special, BOS and/or EOS are added as the tokenizer's settings say (TemplateProcessing, add_bos_token).

BPE dropout. With a probability p, each merge candidate popped from the heap is skipped and set aside, coming back after the next merge that is not skipped (Hugging Face's rule; ignore_merges is then not applied). p comes from oklm_tokenize_params.dropout, or from the dropout field of tokenizer.json when that is 0; a negative value turns it off. The draws come from a stream seeded by oklm_tokenize_params.seed, so a seed always gives the same ids. Dropout applies to merge-rank BPE only, for training data; p = 1 gives one token per symbol.

Decoding maps byte-level characters back to bytes, ▁ back to spaces and <0xNN> tokens to their byte; control tokens are output as their text unless skip_special is set; SentencePiece decoding drops the leading space it added.

Pre-tokenizers

Patterns run on OKLM's own regex engine (ADR-0004). Named sets, as GGUF tokenizer.ggml.pre calls them:

Name Steps Also implies
gpt-2 (gpt2, default) 's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+
llama-bpe (llama3) the Llama 3 pattern (\p{N}{1,3}, case-insensitive contractions) ignore_merges
qwen2 the Llama 3 pattern with single digits NFC
smollm \p{N}, then the GPT-2 pattern

Beyond these four, every name llama.cpp knows (88: deepseek-llm, deepseek-coder, deepseek-v3, falcon, starcoder, refact, command-r, qwen35, chatglm-bpe, glm4, gpt-4o, llama4, tekken, chameleon, viking, bloom, exaone, seed-coder, grok-2, ...) is read with llama.cpp's steps and ignore_merges flag, from src/tok/pre_presets.inc, generated by tools/gen_pre_tokenizers.py from llama.cpp's src/llama-vocab.cpp.

BPE over raw UTF-8 (GGUF model gemma4, or gpt2 with the pre-tokenizer gemma4, granite-embed-multi-311m, sarvam-moe or domyn-small) is loaded as merge-rank SentencePiece: spaces become ▁ with none prepended, the text is split at newlines only, characters missing from the vocabulary fall back to <0xNN> tokens, and for Gemma 4 a run of newlines that is a token is used whole and BOS is always added, as llama.cpp does. OKLM writes it back under the same GGUF names. In a Gemma 4 vocabulary (one with the <|tool_call> and <tool_call|> tokens) <|tool_response> and <eos> are read as control tokens whatever type the file gives them: a reply ends there (inference.md). tokenizer.ggml.suppress_tokens (the tokens a file says are never to be sampled) is read for every tokenizer model and handed to the samplers (inference.md).

The two that llama.cpp implements with custom code follow the models' own tokenizers, which llama.cpp's code only approximates:

  • kimi-k2: the pattern of Kimi-K2's tiktoken tokenizer, Han runs then words split by case without Han, Han being the Unicode 16 Script=Han ranges. llama.cpp's handler uses CJK block ranges and does not split by case ("HELLOworld" is one piece there, two here).
  • afmoe: the steps of AFMoE's tokenizer.json (Trinity-Mini): numeric runs isolated, decimal digits cut in threes from the right, one run per script group (CJK, Thai, Lao, Khmer, Myanmar, Hangul), then the main pattern. llama.cpp attaches the text before a digit group to it and merges the script groups into one class. The tokenizer.json writes its digit steps with atomic groups and \G, which OKLM's regex engine does not have; the loader recognizes them and uses the equivalent pattern.

Three more BPE vocabularies over characters follow their models' tokenizer.json, with the steps Hugging Face runs:

  • mmbert (mmBERT): spaces become ▁, a ▁ is prepended to each segment that does not start with one, and every ▁ starts a piece (Metaspace with split); characters missing from the vocabulary fall back to bytes. llama.cpp also splits at newlines, which the model's tokenizer does not.
  • soprano (Soprano TTS): lower case, each whitespace run one space, single digits, then runs of spaces, word characters (\w) or other characters; a missing character is the unknown token. llama.cpp's word class leaves out the combining marks that \w includes.
  • whitespace (Jina embeddings v2 zh, GGUF model whitespace): NFC and lower case when the tokenizer asks for them (tokenizer.ggml.normalizer.*), then words and punctuation runs with the whitespace dropped (Hugging Face's Whitespace pre-tokenizer). llama.cpp splits at whitespace only, so "hello," stays one piece there and is two here.

Lower case is Hugging Face's: the full mapping of each code point alone, so a capital sigma is always σ (Python's str.lower() gives ς at a word end).

A tokenizer.json is read with its own patterns (Split, Digits, ByteLevel steps), and its GGUF name is found by comparing the patterns with this table ("default" when none matches, with a warning). An unknown name in a GGUF file falls back to the GPT-2 patterns with a warning.

A BPE GGUF file that names no pre-tokenizer (files written before tokenizer.ggml.pre existed, in 2023 and early 2024) gets the one of its family when the file tells the family: starcoder for the starcoder2 architecture, deepseek-coder for DeepSeek-Coder's vocabulary (32 256 tokens and its BOS). Both were compared with Hugging Face's tokenizers on the models' own tokenizer.json: identical ids on 400 lines of C, Python and Markdown each (StarCoder2-3B, DeepSeek-Coder-1.3B-instruct). Any other such file gets default, which is GPT-2's pattern here. llama.cpp differs on all three: it warns that generation will be degraded and applies its own default steps (punctuation runs split off first), so its ids for such a file are not the model's wherever a space precedes punctuation ( {, =): two tokens there, one in the model's vocabulary.

Files

Hugging Face (oklm_tokenizer_from_hf()): tokenizer.json with model BPE (merges as strings or pairs, byte_fallback, ignore_merges, unk_token), normalizers NFC, Prepend, Replace and Sequence, pre-tokenizers ByteLevel, Split (Isolated), Digits, Metaspace and Sequence, post-processor TemplateProcessing; the sibling tokenizer_config.json supplies bos_token, eos_token, pad_token, add_bos_token, add_eos_token and chat_template.

GGUF (oklm_tokenizer_from_gguf(), oklm_tokenizer_to_gguf()): the llama.cpp keys tokenizer.ggml.model, .pre, .tokens, .token_type, .merges, .scores, .bos_token_id, .eos_token_id, .unknown_token_id, .padding_token_id, .add_bos_token, .add_eos_token, .add_space_prefix, and tokenizer.chat_template (a DeepSeek-Coder instruct file without one, recognized by its BOS and <|EOT|> as EOS, gets the template its tokenizer_config.json publishes). For SentencePiece-style vocabularies OKLM writes the merges (llama.cpp ignores them for llama models) and scores derived from the merge ranks (-rank of the merge that creates each token), so llama.cpp's score-ordered merging follows the same order; llama.cpp's own converter writes a placeholder score of -1000 for every token in that case. GGUF has no key for added-token options, so OKLM adds oklm.tokenizer.added_flags (int32 pairs: token id, bits lstrip=1, rstrip=2, single_word=4).

oklm convert writes the tokenizer keys when the checkpoint has a tokenizer.json: on SmolLM2 they are identical to the keys of llama.cpp's published GGUF (tokens, types, merges, pre, ids, flags, chat template).

Command line

oklm tokenize --model model.gguf "Hello world"          # ids
oklm tokenize --model tokenizer_dir --pieces "Hello"     # ids with their texts
oklm tokenize --model tokenizer.json --bos "Hello"       # with BOS/EOS
oklm tokenize --model model.gguf --decode 9906,1917      # text of ids
oklm train-tokenizer --vocab 4096 --special "<|endoftext|>" --out tok.gguf data/*.txt

oklm train-tokenizer takes the text files after its options (their contents joined, a newline added to a file that ends without one):

Option Meaning
--vocab N the vocabulary's size, special tokens included (required; 256 bytes plus the special tokens at least)
--out FILE the vocabulary-only GGUF written (required)
--min-freq F the smallest count a pair must have to be merged: the merges stop when the most frequent pair left occurs fewer than F times, and the vocabulary is then smaller than --vocab (default 2: a pair seen once is never merged)
--pre NAME the pre-tokenizer's pattern set: gpt-2 (the default), llama-bpe, qwen2, smollm
--special TEXT a special token, appended after the merges (repeatable, 16 at most)
--chat-template FILE a Jinja chat template kept in the file, which the models trained with this tokenizer carry
-v a log line every 500 merges, with the last pair's count

A larger --min-freq keeps rare pairs out of a vocabulary trained on a small text: with --vocab 4096 --min-freq 5 the training ends at the first merge whose pair occurs four times or fewer, whatever is left of the 4096.

Training

oklm_bpe_train() (and oklm train-tokenizer, which saves a vocabulary-only GGUF) trains byte-level BPE:

  1. pre-tokenize with a named pattern set and count identical pieces, in parallel over chunks cut after a newline followed by a non-space (no piece of the supported patterns spans such a point, so chunking cannot change the counts);
  2. start from the 256 bytes (id = byte value);
  3. repeatedly merge the most frequent adjacent pair (ties: smallest left id, then smallest right id), every occurrence left to right, until the vocabulary is full or the best count is below min_frequency (--min-freq, 2 when not given);
  4. append the special tokens.

Pair counts are updated incrementally, only for the pieces that contain the merged pair, and the best pair comes from a heap with lazy deletion (every changed count is pushed, including decreases).

Check: on tinyshakespeare with 4096 tokens, the 3840 merges are identical to tools/bpe_train_ref.py, an independent Python implementation of the same rules that splits text with Hugging Face's own ByteLevel pre-tokenizer; the trained tokenizer round-trips the text exactly. Hugging Face's BpeTrainer breaks ties differently and is not the reference.

Verification

Check Result
10 000-entry multilingual corpus (tests/fixtures/tok/corpus.jsonl), GPT-2, Llama 2, Llama 3.2, Qwen2.5, SmolLM2, from tokenizer.json ids and decoding identical to Hugging Face on every entry
the same five after writing and reading OKLM's GGUF keys identical on every entry
SmolLM2 from llama.cpp's published GGUF identical on every entry
four tiny tokenizers trained by Hugging Face (committed, offline), plus copies with fuse_unk (no byte fallback) and with dropout 1 identical, including lstrip/rstrip added tokens, byte fallback and fused unknown tokens
NFC on every decomposable code point and 3 500 mark / Hangul sequences identical to Python's unicodedata
llama.cpp's own tokenizer tests (models/ggml-vocab-*.gguf with .inp / .out): gpt-2, llama-bpe, llama-spm, qwen2, qwen35, phi-3, mpt, falcon, starcoder, refact, command-r, deepseek-llm, deepseek-coder, gemma-4 (tests/tok/test_llama_vocabs.c) 708 / 708 cases identical (bert-bge included), also after OKLM writes each vocabulary back to GGUF; gemma-4 decodes back to the input; bert-bge (WordPiece) also from BAAI/bge-small-en-v1.5's tokenizer.json
The 10 000-entry corpus with Hugging Face's tokenizers package: AFMoE (Trinity-Mini tokenizer.json), then through GGUF keys 10 000 identical both ways
The 10 000-entry corpus with Hugging Face's tokenizers package: mmBERT, Jina v2 zh and Soprano tokenizer.json, then through GGUF keys 10 000 identical for each, both ways
kimi-k2 and afmoe splitting on the corpus plus 5 000 generated mixed-script strings, against Kimi-K2's pattern in Python's regex module and AFMoE's pre-tokenizer in the tokenizers package 15 024 / 15 024 identical for each
The 10 000-entry corpus with Hugging Face's tokenizers package: T5 (Unigram) and BGE (WordPiece) tokenizer.json, then through GGUF keys (test_tokenizer_parity) T5 9 912 identical, the 88 others segmentations of exactly equal score that the package settles by the last-bit rounding of its JSON parser; BGE 9 998, the 2 others holding U+061D (punctuation since Unicode 14, unknown to the package's tables)
fuzzing (fuzz_regex, fuzz_tokenizer) no crash, leak or hang; byte-level decoding returns the input

Throughput is in benchmarks.md.

Limits

  • Case-insensitive groups use simple Unicode case folding ("K" matches "k" and KELVIN SIGN), not full folding ("SS" does not match "ß"). Possessive quantifiers are treated as greedy.
  • Added tokens with normalized: true are matched on the raw text.
  • BPE dropout draws from OKLM's own random stream, so with 0 < p < 1 the ids differ from Hugging Face's (whose stream is not seedable); both follow the same rule.
  • Unicode tables are version 16.0 (Python 3.14's unicodedata); Hugging Face builds may use an older Unicode, which only matters for characters assigned since.