Tokenizer
How OKLM turns text into token ids and back, which tokenizer families
and file formats it reads and writes, how it is checked against Hugging
Face, and how to train a tokenizer. API:
include/oklm/tok/tokenizer.h, regex.h,
unicode.h, bpe_train.h.
Status: implemented (byte-level BPE, SentencePiece-style BPE, training).
Families
| Family | GGUF tokenizer.ggml.model |
Examples | Symbols | Merging |
|---|---|---|---|---|
| Byte-level BPE | gpt2 |
GPT-2, Llama 3, Qwen2.5, SmolLM2 | bytes, through GPT-2's byte -> character table | by merge rank |
| SentencePiece-style BPE | llama, gemma4 |
Llama 2, TinyLlama, Mistral v1, Gemma 4 | UTF-8 characters, <0xNN> byte fallback |
by merge rank when merges are known, else by piece score |
| WordPiece | bert |
BERT, BGE embedding models | words (lower case and accents per the vocabulary), word starts
marked with ▁ |
greedy longest match; a word with an unmatched position is
[UNK] |
| Unigram | t5 |
T5 | words split on whitespace, each behind ▁, after
SentencePiece's precompiled normalization map |
best total piece score (Viterbi, double precision); uncovered
characters are <unk>, runs fused |
WordPiece follows llama.cpp: decomposition and the combining marks
dropped when accents are stripped
(tokenizer.ggml.normalizer.*, on with lower case by
default), control characters dropped, whitespace ends a word,
punctuation, ASCII symbols and CJK ideographs are words of their own.
From a tokenizer.json (model WordPiece,
BertNormalizer, BertPreTokenizer,
BertProcessing) the vocabulary is stored as llama.cpp's
converter stores it: ## pieces without the prefix, the
other pieces with the ▁ word-start marker,
[...] tokens as they are; such tokenizers follow Hugging
Face's normalizer (only non-spacing marks stripped), which an OKLM key
keeps through GGUF. With special tokens, a text is framed
[CLS] ... [SEP] both ways: from a GGUF vocabulary as
llama.cpp does (tokenizer.ggml.bos_token_id, else 101, and
tokenizer.ggml.seperator_token_id, else 102;
add_sep_token false turns [SEP] off), from a
tokenizer.json by its BertProcessing.
Unigram follows Hugging Face's model: the Precompiled
normalizer's map (SentencePiece's double-array trie of replacements, the
longest key at each position) or GGUF
tokenizer.ggml.precompiled_charsmap, words split on
whitespace, then the segmentation maximizing the sum of piece scores; a
character no single-character piece covers is <unk>,
scored 10 below the lowest score, and adjacent <unk>
fuse.
Encoding pipeline
This is Hugging Face tokenizers' pipeline, step for step:
- Added tokens. Special and added tokens are found in
the raw text, leftmost-longest, honouring
lstrip,rstripandsingle_word; each becomes its id and splits the text into segments. Special (control and unknown) tokens are only matched withparse_special(the default). - Normalization of each segment: NFC when the
tokenizer asks for it (Qwen); for SentencePiece models, spaces become
▁and a▁is prepended (to every segment, or only the first with Metaspaceprepend_scheme: "first"). - Pre-tokenization. Each pattern of the tokenizer splits every piece: matches and the gaps between them become pieces ("Isolated" splitting).
- Model. Each piece becomes symbols and is merged
with a binary heap of candidate pairs, lowest rank first (ties:
leftmost). With
ignore_merges(Llama 3), a piece that is a whole vocabulary entry is used directly. Score-based merging (GGUFllamavocabularies without merges) picks the highest-scoring pair whose concatenation is a token, then falls back to bytes for what is left, as llama.cpp does. A character missing from the vocabulary becomes its<0xNN>tokens when byte fallback is on and all of them exist, else the unknown token; withfuse_unk(tokenizer.json only, GGUF has no key for it) a run of such characters gives a single unknown token. - Post-processing. With
add_special, BOS and/or EOS are added as the tokenizer's settings say (TemplateProcessing,add_bos_token).
BPE dropout. With a probability p, each
merge candidate popped from the heap is skipped and set aside, coming
back after the next merge that is not skipped (Hugging Face's rule;
ignore_merges is then not applied). p comes
from oklm_tokenize_params.dropout, or from the
dropout field of tokenizer.json when that is 0; a negative
value turns it off. The draws come from a stream seeded by
oklm_tokenize_params.seed, so a seed always gives the same
ids. Dropout applies to merge-rank BPE only, for training data;
p = 1 gives one token per symbol.
Decoding maps byte-level characters back to bytes, ▁
back to spaces and <0xNN> tokens to their byte;
control tokens are output as their text unless skip_special
is set; SentencePiece decoding drops the leading space it added.
Pre-tokenizers
Patterns run on OKLM's own regex engine (ADR-0004). Named sets, as
GGUF tokenizer.ggml.pre calls them:
| Name | Steps | Also implies |
|---|---|---|
gpt-2 (gpt2, default) |
's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+ |
|
llama-bpe (llama3) |
the Llama 3 pattern (\p{N}{1,3}, case-insensitive
contractions) |
ignore_merges |
qwen2 |
the Llama 3 pattern with single digits | NFC |
smollm |
\p{N}, then the GPT-2 pattern |
Beyond these four, every name llama.cpp knows (88:
deepseek-llm, deepseek-coder,
deepseek-v3, falcon, starcoder,
refact, command-r, qwen35,
chatglm-bpe, glm4, gpt-4o,
llama4, tekken, chameleon,
viking, bloom, exaone,
seed-coder, grok-2, ...) is read with
llama.cpp's steps and ignore_merges flag, from
src/tok/pre_presets.inc, generated by
tools/gen_pre_tokenizers.py from llama.cpp's
src/llama-vocab.cpp.
BPE over raw UTF-8 (GGUF model gemma4, or
gpt2 with the pre-tokenizer gemma4,
granite-embed-multi-311m, sarvam-moe or
domyn-small) is loaded as merge-rank SentencePiece: spaces
become ▁ with none prepended, the text is split at newlines
only, characters missing from the vocabulary fall back to
<0xNN> tokens, and for Gemma 4 a run of newlines that
is a token is used whole and BOS is always added, as llama.cpp does.
OKLM writes it back under the same GGUF names. In a Gemma 4 vocabulary
(one with the <|tool_call> and
<tool_call|> tokens)
<|tool_response> and <eos> are
read as control tokens whatever type the file gives them: a reply ends
there (inference.md).
tokenizer.ggml.suppress_tokens (the tokens a file says are
never to be sampled) is read for every tokenizer model and handed to the
samplers (inference.md).
The two that llama.cpp implements with custom code follow the models' own tokenizers, which llama.cpp's code only approximates:
kimi-k2: the pattern of Kimi-K2's tiktoken tokenizer, Han runs then words split by case without Han, Han being the Unicode 16 Script=Han ranges. llama.cpp's handler uses CJK block ranges and does not split by case ("HELLOworld" is one piece there, two here).afmoe: the steps of AFMoE's tokenizer.json (Trinity-Mini): numeric runs isolated, decimal digits cut in threes from the right, one run per script group (CJK, Thai, Lao, Khmer, Myanmar, Hangul), then the main pattern. llama.cpp attaches the text before a digit group to it and merges the script groups into one class. The tokenizer.json writes its digit steps with atomic groups and\G, which OKLM's regex engine does not have; the loader recognizes them and uses the equivalent pattern.
Three more BPE vocabularies over characters follow their models' tokenizer.json, with the steps Hugging Face runs:
mmbert(mmBERT): spaces become▁, a▁is prepended to each segment that does not start with one, and every▁starts a piece (Metaspace with split); characters missing from the vocabulary fall back to bytes. llama.cpp also splits at newlines, which the model's tokenizer does not.soprano(Soprano TTS): lower case, each whitespace run one space, single digits, then runs of spaces, word characters (\w) or other characters; a missing character is the unknown token. llama.cpp's word class leaves out the combining marks that\wincludes.whitespace(Jina embeddings v2 zh, GGUF modelwhitespace): NFC and lower case when the tokenizer asks for them (tokenizer.ggml.normalizer.*), then words and punctuation runs with the whitespace dropped (Hugging Face's Whitespace pre-tokenizer). llama.cpp splits at whitespace only, so "hello," stays one piece there and is two here.
Lower case is Hugging Face's: the full mapping of each code point
alone, so a capital sigma is always σ (Python's
str.lower() gives ς at a word end).
A tokenizer.json is read with its own patterns (Split,
Digits, ByteLevel steps), and its GGUF name is found by comparing the
patterns with this table ("default" when none matches, with a warning).
An unknown name in a GGUF file falls back to the GPT-2 patterns with a
warning.
A BPE GGUF file that names no pre-tokenizer (files written before
tokenizer.ggml.pre existed, in 2023 and early 2024) gets
the one of its family when the file tells the family:
starcoder for the starcoder2 architecture,
deepseek-coder for DeepSeek-Coder's vocabulary (32 256
tokens and its BOS). Both were compared with Hugging Face's
tokenizers on the models' own tokenizer.json:
identical ids on 400 lines of C, Python and Markdown each
(StarCoder2-3B, DeepSeek-Coder-1.3B-instruct). Any other such file gets
default, which is GPT-2's pattern here. llama.cpp differs
on all three: it warns that generation will be degraded and applies its
own default steps (punctuation runs split off first), so its ids for
such a file are not the model's wherever a space precedes punctuation
( {, =): two tokens there, one in the model's
vocabulary.
Files
Hugging Face
(oklm_tokenizer_from_hf()): tokenizer.json
with model BPE (merges as strings or pairs,
byte_fallback, ignore_merges,
unk_token), normalizers NFC,
Prepend, Replace and Sequence,
pre-tokenizers ByteLevel, Split (Isolated),
Digits, Metaspace and Sequence,
post-processor TemplateProcessing; the sibling
tokenizer_config.json supplies bos_token,
eos_token, pad_token,
add_bos_token, add_eos_token and
chat_template.
GGUF (oklm_tokenizer_from_gguf(),
oklm_tokenizer_to_gguf()): the llama.cpp keys
tokenizer.ggml.model, .pre,
.tokens, .token_type, .merges,
.scores, .bos_token_id,
.eos_token_id, .unknown_token_id,
.padding_token_id, .add_bos_token,
.add_eos_token, .add_space_prefix, and
tokenizer.chat_template (a DeepSeek-Coder instruct file
without one, recognized by its BOS and <|EOT|> as
EOS, gets the template its tokenizer_config.json
publishes). For SentencePiece-style vocabularies OKLM writes the merges
(llama.cpp ignores them for llama models) and scores
derived from the merge ranks (-rank of the merge that
creates each token), so llama.cpp's score-ordered merging follows the
same order; llama.cpp's own converter writes a placeholder score of
-1000 for every token in that case. GGUF has no key for added-token
options, so OKLM adds oklm.tokenizer.added_flags (int32
pairs: token id, bits lstrip=1, rstrip=2, single_word=4).
oklm convert writes the tokenizer keys when the
checkpoint has a tokenizer.json: on SmolLM2 they are
identical to the keys of llama.cpp's published GGUF (tokens, types,
merges, pre, ids, flags, chat template).
Command line
oklm tokenize --model model.gguf "Hello world" # ids
oklm tokenize --model tokenizer_dir --pieces "Hello" # ids with their texts
oklm tokenize --model tokenizer.json --bos "Hello" # with BOS/EOS
oklm tokenize --model model.gguf --decode 9906,1917 # text of ids
oklm train-tokenizer --vocab 4096 --special "<|endoftext|>" --out tok.gguf data/*.txtoklm train-tokenizer takes the text files after its
options (their contents joined, a newline added to a file that ends
without one):
| Option | Meaning |
|---|---|
--vocab N |
the vocabulary's size, special tokens included (required; 256 bytes plus the special tokens at least) |
--out FILE |
the vocabulary-only GGUF written (required) |
--min-freq F |
the smallest count a pair must have to be merged: the merges stop
when the most frequent pair left occurs fewer than F times, and the
vocabulary is then smaller than --vocab (default 2: a pair
seen once is never merged) |
--pre NAME |
the pre-tokenizer's pattern set: gpt-2 (the default),
llama-bpe, qwen2, smollm |
--special TEXT |
a special token, appended after the merges (repeatable, 16 at most) |
--chat-template FILE |
a Jinja chat template kept in the file, which the models trained with this tokenizer carry |
-v |
a log line every 500 merges, with the last pair's count |
A larger --min-freq keeps rare pairs out of a vocabulary
trained on a small text: with --vocab 4096 --min-freq 5 the
training ends at the first merge whose pair occurs four times or fewer,
whatever is left of the 4096.
Training
oklm_bpe_train() (and oklm train-tokenizer,
which saves a vocabulary-only GGUF) trains byte-level BPE:
- pre-tokenize with a named pattern set and count identical pieces, in parallel over chunks cut after a newline followed by a non-space (no piece of the supported patterns spans such a point, so chunking cannot change the counts);
- start from the 256 bytes (id = byte value);
- repeatedly merge the most frequent adjacent pair (ties: smallest
left id, then smallest right id), every occurrence left to right, until
the vocabulary is full or the best count is below
min_frequency(--min-freq, 2 when not given); - append the special tokens.
Pair counts are updated incrementally, only for the pieces that contain the merged pair, and the best pair comes from a heap with lazy deletion (every changed count is pushed, including decreases).
Check: on tinyshakespeare with 4096 tokens, the 3840 merges are
identical to tools/bpe_train_ref.py, an independent Python
implementation of the same rules that splits text with Hugging Face's
own ByteLevel pre-tokenizer; the trained tokenizer round-trips the text
exactly. Hugging Face's BpeTrainer breaks ties differently and is not
the reference.
Verification
| Check | Result |
|---|---|
10 000-entry multilingual corpus
(tests/fixtures/tok/corpus.jsonl), GPT-2, Llama 2, Llama
3.2, Qwen2.5, SmolLM2, from tokenizer.json |
ids and decoding identical to Hugging Face on every entry |
| the same five after writing and reading OKLM's GGUF keys | identical on every entry |
| SmolLM2 from llama.cpp's published GGUF | identical on every entry |
four tiny tokenizers trained by Hugging Face (committed, offline),
plus copies with fuse_unk (no byte fallback) and with
dropout 1 |
identical, including lstrip/rstrip added tokens, byte fallback and fused unknown tokens |
| NFC on every decomposable code point and 3 500 mark / Hangul sequences | identical to Python's unicodedata |
llama.cpp's own tokenizer tests
(models/ggml-vocab-*.gguf with .inp /
.out): gpt-2, llama-bpe, llama-spm, qwen2, qwen35, phi-3,
mpt, falcon, starcoder, refact, command-r, deepseek-llm, deepseek-coder,
gemma-4 (tests/tok/test_llama_vocabs.c) |
708 / 708 cases identical (bert-bge included), also after OKLM
writes each vocabulary back to GGUF; gemma-4 decodes back to the input;
bert-bge (WordPiece) also from BAAI/bge-small-en-v1.5's
tokenizer.json |
The 10 000-entry corpus with Hugging Face's tokenizers package:
AFMoE (Trinity-Mini tokenizer.json), then through GGUF
keys |
10 000 identical both ways |
The 10 000-entry corpus with Hugging Face's tokenizers package:
mmBERT, Jina v2 zh and Soprano tokenizer.json, then through
GGUF keys |
10 000 identical for each, both ways |
kimi-k2 and afmoe splitting on the corpus
plus 5 000 generated mixed-script strings, against Kimi-K2's pattern in
Python's regex module and AFMoE's pre-tokenizer in the tokenizers
package |
15 024 / 15 024 identical for each |
The 10 000-entry corpus with Hugging Face's tokenizers package: T5
(Unigram) and BGE (WordPiece) tokenizer.json, then through
GGUF keys (test_tokenizer_parity) |
T5 9 912 identical, the 88 others segmentations of exactly equal score that the package settles by the last-bit rounding of its JSON parser; BGE 9 998, the 2 others holding U+061D (punctuation since Unicode 14, unknown to the package's tables) |
fuzzing (fuzz_regex, fuzz_tokenizer) |
no crash, leak or hang; byte-level decoding returns the input |
Throughput is in benchmarks.md.
Limits
- Case-insensitive groups use simple Unicode case folding ("K" matches "k" and KELVIN SIGN), not full folding ("SS" does not match "ß"). Possessive quantifiers are treated as greedy.
- Added tokens with
normalized: trueare matched on the raw text. - BPE dropout draws from OKLM's own random stream, so with
0 < p < 1the ids differ from Hugging Face's (whose stream is not seedable); both follow the same rule. - Unicode tables are version 16.0 (Python 3.14's unicodedata); Hugging Face builds may use an older Unicode, which only matters for characters assigned since.