How it works
A guided tour of OKLM for someone who knows a terminal but is new to the inside of language models. For each topic: the idea, how OKLM does it, and commands to try. The details live in the reference pages linked from each section; this page explains why things are the way they are and gives the commands that show it.
Status: maintained. Every command below was run with the CPU backend
on the small models of models/; the outputs are those runs,
cut short.
oklm stands for build/release/bin/oklm (see
build.md). oklm help lists every
command and option; each command also prints its own usage when its
arguments are wrong.
Contents
- Model files
- Quantization
- From a checkpoint to a quantized model, end to end
- Memory and the KV cache
- Backends and speed
- Generation
- Documents
- Embeddings
- The server
- The coding agent
- Training
- Images, sounds and speech
- Refusal analysis
- Developer tooling
- Files, caches and environment
- Limits worth knowing
Model files
A model is a list of named tensors (arrays of numbers: the weights) plus metadata: the architecture, its sizes, the tokenizer, the chat template. Two file families carry them.
- Hugging Face checkpoints: a directory with
config.json(the architecture and sizes), one or more.safetensorsfiles (the tensors, with an index when sharded),tokenizer.jsonandtokenizer_config.json(the vocabulary and the chat template). This is how models are published for training frameworks, usually in BF16. - GGUF: one file (or a few shards) holding the tensors and every piece of metadata, tokenizer included. It is llama.cpp's format, and the one block-quantized weights (Q4_K, Q8_0, ...) are stored in.
OKLM reads both and writes GGUF. The byte layouts are in formats/gguf.md and formats/safetensors.md.
oklm convert turns a checkpoint into a
GGUF file: it renames each tensor to the GGUF name, reverses the shape
order, permutes the Q and K rows of Llama-style models into the layout
GGUF expects, keeps norms in F32 and writes the hyperparameters and
tokenizer as metadata. Target types are f32,
f16 (the default), bf16, q8_0,
q4_0, q4_1, q5_0,
q5_1; the K-quants come from oklm quantize
afterwards. The supported architectures and what each needs
are in convert.md.
$ oklm convert models/smollm2-135m-hf --out smollm2-f16.gguf --type f16
convert: models/smollm2-135m-hf -> smollm2-f16.gguf (272 tensors, 0 fell back to F16)
Why run loads a checkpoint directly.
Safetensors files are raw arrays at known offsets, so OKLM maps them and
uses F32, F16 and BF16 matrices in place, with the hyperparameters from
config.json and the tokenizer from
tokenizer.json. FP8 matrices are widened to BF16 at load
time. Nothing is written to disk. Converting still pays off for a single
file, faster loading, and quantization (inference.md).
$ oklm run -m models/smollm2-135m-hf -p "The capital of France is" --temp 0 -n 12 --backend cpu
The capital of France is Paris. Paris is the largest city in France and the capital
Inspecting. oklm info model.gguf prints
the metadata, the tensor count per type and a verdict (loadable,
loadable with warnings, unsupported, with reasons); -t
lists every tensor. --json prints the same as one JSON
object for programs, with the model's layout: each block's tensors
grouped by role (attention, feed-forward, experts, norms) with their
types and bytes, the embeddings and the head (convert.md).
Shards. Large models are often published as several
GGUF files. oklm split writes such shards and
oklm merge joins them; the files are those of llama.cpp's
gguf-split, byte for byte (formats/gguf.md), so either
tool reads the other's. Every oklm command opens a split
model from any of its shards.
$ oklm split SmolLM2-135M-Instruct-Q8_0.gguf part --split-max-size 64M
3 shard(s): part-00001-of-00003.gguf ...
$ oklm info part-00002-of-00003.gguf | grep -i 'split\|verdict'
split.no = 0
split.tensors.count = 272
split.count = 3
verdict : loadable
$ oklm merge part-00001-of-00003.gguf merged.gguf
--split-max-tensors N splits by tensor count instead of
size.
Quantization
A weight stored in F32 takes 32 bits, in F16 or BF16 16. Generation reads every weight once per token, so a smaller file is both less memory and less to read: on a CPU, decoding speed follows memory bandwidth. Quantization stores weights in fewer bits at the cost of a small error.
Blocks. The ggml quants group weights into blocks
that share a scale: Q8_0 stores 32 weights as 32 signed bytes plus one
F16 scale (34 bytes, 8.5 bits per weight). The K-quants use super-blocks
of 256 with sub-block scales, themselves quantized.
oklm types lists every element type with its block size and
bytes per block; bits per weight are bytes x 8 / block:
| Type | Block | Bytes | Bits per weight | Kind |
|---|---|---|---|---|
| F32 | 1 | 4 | 32 | float |
| F16, BF16 | 1 | 2 | 16 | float (BF16 keeps F32's range, fewer digits) |
| Q8_0 | 32 | 34 | 8.5 | legacy block quant |
| Q6_K | 256 | 210 | 6.56 | K-quant |
| Q5_K | 256 | 176 | 5.5 | K-quant |
| Q5_0 | 32 | 22 | 5.5 | legacy |
| Q4_K | 256 | 144 | 4.5 | K-quant |
| Q4_0 | 32 | 18 | 4.5 | legacy |
| IQ4_XS | 256 | 136 | 4.25 | non-linear 4-bit |
| MXFP4 | 32 | 17 | 4.25 | 4-bit float |
| Q3_K | 256 | 110 | 3.44 | K-quant |
| Q2_K | 256 | 84 | 2.63 | K-quant |
| IQ2_XXS | 256 | 66 | 2.06 | codebook (grid) |
| IQ1_S | 256 | 50 | 1.56 | codebook (grid) |
The dequant and matmul columns of
oklm types say whether OKLM can read a type and multiply
with it directly (Q8_1 and Q8_K have no matrix product of their own).
The layouts are in formats/quant.md.
File types are mixes. oklm quantize
takes llama-quantize's file types (Q4_K_M,
Q5_K_M, Q8_0, IQ4_XS, ...), not
single element types. Q4_K_M keeps most matrices in Q4_K,
but the output matrix in Q6_K, attention V and some FFN down matrices a
step or two higher, and norms in F32. A matrix whose rows are not a
whole number of blocks falls back to a type with smaller blocks:
SmolLM2's rows are 576 wide, not a multiple of 256, so its "Q4_K_M" is
mostly Q5_0:
$ oklm info smollm2-q4_k_m.gguf
...
tensors : 272, 134.52 M parameters, 98.87 MiB
F32 61
Q5_0 166
Q8_0 15
Q4_K 16
Q6_K 14
--pure puts every matrix in the base type instead. The
output equals llama-quantize's byte for byte for every file type except
the grid IQ types, whose blocks come from OKLM's own codebook search (convert.md).
The trade-off. Fewer bits mean a smaller file and faster decoding, and more error. The table of the next section measures it on one model: Q8_0 is within 0.1 % of F16's perplexity, Q4_K_M within 3 %, at 54 % and 39 % of the F16 size. Below 4 bits the error grows quickly, which is what the importance matrix is for.
Importance matrices. Not every weight matters
equally. A weight multiplied by inputs that are large most of the time
changes the output more than one whose inputs are near zero.
oklm imatrix runs the model over a calibration text and
records, for each matrix of each block, the sum of the squares of every
input column (llama-imatrix's method and file format).
oklm quantize --imatrix divides those sums by their counts
and weights the quantizer's error by them, so the scales are chosen to
keep the important columns exact at the expense of the others. The
legacy quants, Q2_K to Q6_K, IQ4_NL, IQ4_XS and the grid IQ types use
it. oklm quantize warns when a type meant for one (Q2_K_S,
the IQ1 and IQ2 types, IQ3_XXS) gets none, where llama-quantize refuses
the 1- and 2-bit ones; the 1-bit types are unusable without one.
Choosing the calibration text. The importance matrix records what inputs the model sees on that text. Calibrate on text like the real use (code for a coding model, the target language for a translator) so the columns that matter in use are the ones protected. Measure on a different text: perplexity on the calibration text itself rewards the quantizer for fitting that one text and overstates the quality.
From a checkpoint to a quantized model, end to end
SmolLM2-135M-Instruct from its Hugging Face checkpoint to a Q4_K_M GGUF, checked at each stage. The calibration text is the first 400 kB of Tiny Shakespeare, the evaluation text its last 100 kB, so the two do not overlap.
head -c 400000 data/tinyshakespeare.txt > calib.txt
tail -c 100000 data/tinyshakespeare.txt > eval.txt1. Convert to F16. oklm plan reads GGUF
headers only, so the checkpoint is converted first (it refuses a
checkpoint directory with "convert a Hugging Face checkpoint
first").
$ oklm convert models/smollm2-135m-hf --out smollm2-f16.gguf --type f16
2. Check the memory it needs.
$ oklm plan -m smollm2-f16.gguf -c 2048
model : smollm2-f16.gguf (llama, 30 layers, 1 file)
context : 2048 tokens x 1 sequence, batch 512, KV F16 / F16, backend cpu
...
host memory :
weights resident 256.6 MiB
KV cache 45.0 MiB (30 layers x 1.5 MiB)
activations 114.0 MiB
scratch 1.6 MiB
total 417.2 MiB
RAM : 105.50 GiB available of 124.64 GiB, 3.78 GiB kept free
verdict : fits resident
3. Baseline perplexity. Perplexity is the
exponential of the mean negative log-probability the model gives each
actual next token: how many choices it is, on average, unsure between.
Lower is better. oklm ppl cuts the text into windows of
-c tokens (512 by default) and scores the second half of
each, so every scored token has at least half a window of context, as
llama.cpp does.
$ oklm ppl -m smollm2-f16.gguf -f eval.txt -c 512 --backend cpu
...
perplexity: 23.4932 over 16065 tokens (512-token windows)
4. Importance matrix on the calibration text (100 chunks of 512 tokens; about two minutes on 8 threads here):
$ oklm imatrix -m smollm2-f16.gguf -f calib.txt -o smollm2.imatrix.gguf -c 512 --chunks 100
imatrix: 210 matrices over 100 chunks of 512 tokens -> smollm2.imatrix.gguf
Without --chunks every chunk of the text is used.
--process-output also collects the output matrix.
5. Quantize.
$ oklm quantize --imatrix smollm2.imatrix.gguf smollm2-f16.gguf smollm2-q4_k_m.gguf Q4_K_M
quantize: 210 importance matrix entries from smollm2.imatrix.gguf
quantize: smollm2-f16.gguf -> smollm2-q4_k_m.gguf (Q4_K_M), 211 tensors converted, 180 fallbacks
The 180 fallbacks are the 576-wide rows above, each logged as a
warning. oklm quantize --dry-run with the same arguments
says the size the file will have without writing it (convert.md).
6. Perplexity again, with the same file and the same
-c; anything else changes the number for reasons other than
the quantization:
$ oklm ppl -m smollm2-q4_k_m.gguf -f eval.txt -c 512 --backend cpu
perplexity: 24.1685 over 16065 tokens (512-token windows)
7. A quick look at the output:
$ oklm run -m smollm2-q4_k_m.gguf -p "The capital of France is" --temp 0 -n 24 --backend cpu
The capital of France is Paris. Paris is a city that is known for its historical landmarks, art, fashion, and culture. It is the
prompt: 5 tokens in 0.012 s (404.5 t/s); generation: 24 tokens in 0.163 s (147.1 t/s)
The same steps with two other outputs, all on the same evaluation text:
| File | Size | Perplexity | Change from F16 |
|---|---|---|---|
| F16 | 258.3 MiB | 23.4932 | |
Q8_0 (quantize ... Q8_0) |
138.1 MiB | 23.5109 | +0.08 % |
| Q4_K_M with the importance matrix | 100.6 MiB | 24.1685 | +2.9 % |
| Q4_K_M without it | 100.6 MiB | 24.5347 | +4.4 % |
Reading the change. The ratio of the two perplexities is what matters, not the absolute value, which depends on the text and the context size. A change within a few tenths of a percent is noise-level; a few percent is a real but often acceptable loss; when it grows larger, or the use is sensitive to small errors, try the next larger type (Q5_K_M, Q6_K, Q8_0) and measure again. Perplexity is an average over text: it is the quick check, and a test on the real task (the prompts the model will get) is the final one.
The same in a window.
oklm-gui --tool convert does steps 1, 5, 3 and 6 from one
window (gui.md): it names the
source and says what it is, shows the size the result will have before
anything is written, starts oklm convert,
oklm quantize and oklm ppl in the background
with the command lines on screen, follows their log and progress, and
puts the two perplexities and sizes side by side.
Memory and the KV cache
Running a model takes four kinds of memory, which
oklm plan computes from the GGUF headers alone, before
anything is loaded, with the same functions the context allocates
with:
- Weights: the file's tensors. They are mapped, not
copied; on the CPU some matrices are repacked into a faster layout, and
plancounts the copy in place of the mapped bytes (a MoE model's routed experts, repacked when there is room, on top of them). - KV cache: see below. It grows with the context length.
- Activations: the buffers of one batch (512 tokens by default) flowing through the layers.
- Scratch: small temporary space.
It compares the total with the available RAM (less a margin of 3 %,
at least 256 MiB) and, with --backend opencl, the device
memory, and says "fits resident", "fits with N layers on the device" or
"does not fit" (exit status 3). run, chat and
serve make the same projection and warn, but do not refuse:
mapped weights can still page in from disk, slowly. Details in memory-plan.md.
No model comes with OKLM. oklm models lists the ones it
offers to download, by use (general chat, coding, images, sounds,
security, retrieval), with their size and licence;
oklm models pull qwen3-8b downloads one into
~/.local/share/oklm/models, resuming an interrupted
download and keeping the file only when its SHA-256 is the one compiled
into oklm; after that -m qwen3-8b works
wherever a model file is expected. Details in models.md.
The KV cache. Attention lets each new token look at every earlier one. For that it needs, per layer, a key and a value vector for each earlier token. Recomputing them for every new token would redo the whole prompt each time; the KV cache keeps them, so a token costs one pass through the model. Its size is
layers x 2 (key and value) x KV heads x head size x bytes per element x tokens
For SmolLM2-135M in F16, 30 x 2 x 3 x 64 x 2 bytes = 22.5 KiB a
token, so 2048 tokens take 45 MiB, as plan printed above.
(Gemma 4's blocks are of two widths, and its E2B and E4B models have
blocks that read another block's cache and keep none: the plan adds each
layer's own and prints (N layers, X for the widest). A
hybrid stack has blocks without attention, which keep no rows: 24 of
Qwen3.5-9B's 32 blocks, 22 of Cyber-Prime's 30; the plan prints
(8 of 32 layers x 32.0 MiB).) The context length multiplies
it directly, so it dominates at long contexts:
$ oklm plan -m qwen2.5-0.5b-instruct-q4_k_m.gguf -c 32768
...
weights resident 477.4 MiB (387.9 MiB repacked for the CPU, in place of 373.4 MiB mapped)
KV cache 384.0 MiB (24 layers x 16.0 MiB)
activations 337.3 MiB
total 1.17 GiB
$ oklm plan -m qwen2.5-0.5b-instruct-q4_k_m.gguf -c 32768 -ctk q8_0 -ctv q8_0
KV cache 204.0 MiB (24 layers x 8.5 MiB)
total 1020.1 MiB
-ctk / -ctv (on run,
chat, ppl, serve and
plan) store the cache in f32, f16
(the default), bf16 or the block quants q8_0,
q4_0, q4_1, q5_0,
q5_1, iq4_nl. A quantized cache holds rotated
heads (a Walsh-Hadamard transform, as llama.cpp does), which spreads
outliers before rounding: Q8_0 stays within 0.2 % of F16's perplexity on
Llama-3.2-1B (adr/0005-kv-cache-precision.md).
--parallel N multiplies the cache by the number of
sequences.
Listing what fits.
oklm plan --list [DIR] projects every GGUF model under
models/ (or DIR) at 8192 tokens (capped at each model's
trained context) and --recommend names the largest that
fits:
$ oklm plan --list models --recommend
size need verdict model
63.0 MiB 374.7 MiB fits resident on the CPU models/smollm2-135m-iq/SmolLM2-135M-Instruct-IQ1_S.gguf
...
782.1 MiB 1.32 GiB fits resident on the CPU models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf [MoE 8/32: resident 1.32 GiB, with the expert cache 656.4 MiB + 348.8 MiB]
...
recommended, resident: models/glm-4.7-flash/GLM-4.7-Flash-Q4_K_M.gguf (fits resident on the CPU)
recommended, with the expert cache: none (no MoE model here needs it to fit)
For a mixture-of-experts model the row also says what stays resident if its experts are read from disk instead (see MoE models below).
Saving the cache:
chat --kv-save --resume. With
--kv-save, oklm chat writes the conversation
to .oklm/chat/last.json and the cache's rows to
.oklm/chat/last.kv after each turn
(--chat-file FILE names another place).
--resume reloads both, so the first turn evaluates only the
new message instead of the whole history:
$ printf 'My name is Ada.\n/exit\n' | oklm chat -m SmolLM2-135M-Instruct-Q8_0.gguf --backend cpu --temp 0 --kv-save --stats
> (prompt 35 tokens: 0 cached, 35 evaluated)
...
$ printf 'What is my name?\n/exit\n' | oklm chat -m SmolLM2-135M-Instruct-Q8_0.gguf --backend cpu --temp 0 --kv-save --resume --stats
(continued: 2 messages from .oklm/chat/last.json; 95 tokens of the cache loaded from .oklm/chat/last.kv)
> (prompt 111 tokens: 95 cached, 16 evaluated)
My name is Ada, ...
The KV file (OKLMKVC, version 1; version 2 when the
conversation holds an image's or a sound's rows, which then also have
their media hashes and positions in it:
oklm describe --kv-save, and
oklm chat --mmproj --kv-save once a picture or a sound was
attached) starts with a header that identifies the model (a hash of its
architecture, sizes and tensors), the adapter, the context and the
per-layer cache types, then holds chunks: each one the token ids of a
range of positions and every layer's key and value rows for them, closed
by a trailer with a hash. A turn appends a chunk; a crash in the middle
leaves the previous turn's state. A file made by another model, cache
type or adapter is refused rather than misread. Here 95 tokens took 2.2
MB, the 22.5 KiB a token above (inference.md). The agent's
--kv-save keeps the same kind of file beside each session.
oklm serve --prefix-dir DIR does it for the server's prefix
cache: a finished request's full blocks go to a file in DIR, and a
restarted server loads the files before it takes requests, so a repeated
long prompt is found cached (server.md).
Backends and speed
CPU and OpenCL. The CPU backend is always there; the
OpenCL backend is compiled in but loads the system's OpenCL library at
run time, so a machine without one still runs the same binary.
--backend auto (the default of run,
chat, ppl, serve and
train; plan defaults to cpu)
takes the first OpenCL GPU when there is one, else the CPU;
--backend cpu gives the same numbers on every machine.
--gpu-layers N puts the last N layers on the device and the
rest on the CPU when the model does not fit. On an RDNA3 GPU the prompt
products of the common quantized types (Q4_K, Q5_K, Q6_K, Q8_0, Q4_0,
Q5_0) run on its 8-bit matrix units by default: the activations rounded
to 8 bits exactly as the CPU rounds them, and the CPU's products bit for
bit, so prompts follow the CPU's greedy choices
(OKLM_CL_Q8=0: the float kernels); the Q4_K and Q5_K tiles,
those of MoE prompts' indexed products included, have waves that load
the next step of weights and activations while the others multiply.
Decoding does the same for the larger K-quant products (Q4_K, Q5_K,
Q6_K, rows of 2048 values and more: each work-group rounds its
activation column in local memory, then 8-bit dot products;
OKLM_CL_Q8_MV=0: the float row kernels); the other
one-column products keep the float kernels, which measured faster. The
prompt products of the other quantized types (the grid IQ types of a
2-bit file, Q3_K, MXFP4, the ternary types: no 8-bit kernel) run on the
same device's float matrix units, their activations rounded to F16 as
Vulkan rounds them (OKLM_CL_WMMA=0: the F32 kernels, three
times slower on such a file). OKLM_CL_WMMA=1 runs every
prompt product on the F16 matrix units instead, and the prompt attention
of F16 caches too (not the CPU's bits, so it stays opt-in; decoding
keeps the 8-bit products; OKLM_CL_WMMA_ATT=0 keeps the
attention's half tiles). The backend also fuses a MoE layer's routing
and Qwen3's query and key norms with their rotation into one launch
each, and folds a decode step's RMS norm into the 8-bit product that
reads it (the same bits), and latent attention's prompts (GLM-4.7-Flash,
DeepSeek) have their own tiled kernel, as has the bidirectional
attention of an encoder's rows (the vision encoders, Whisper's), which
also takes a span of keys a row: the blocks of Qwen2.5-VL's encoder that
attend within windows are one launch each, on OpenCL and on Vulkan. An
image's rows that see each other (Gemma 3) take the prompt tiles with a
last key, for heads up to 128. See opencl.md.
Vulkan. The Vulkan backend
(--backend vulkan) is the main GPU backend on AMD, NVIDIA
and Intel (ADR-0023): it
loads the system's Vulkan library at run time like OpenCL, runs GLSL
shaders compiled into the binary at build time, and reaches tensors
through their buffer device addresses. Every op of the dense, MoE,
hybrid and latent-attention models runs on the GPU (norms, attention,
one-column, prompt (cooperative matrices where the device has them),
8-bit and MoE products); an op a shader does not take runs on the CPU
through the fallback. A model runs on it as on OpenCL
(--backend vulkan, --gpu-layers, its caches in
device memory, weights and caches larger than one allocation split over
several); a decode step is recorded once into command buffers and
replayed on the next tokens, its per-token values read from device
memory; the recording is cut into parts submitted as the host reaches
them, so the device does not wait for the host's pass over the step. Its
time on a discrete GPU is mostly the latency of its few hundred
dependent dispatches, so the decode shaders are written for latency (no
64-bit division, few loads per invocation and all of them before the
arithmetic, no long chain of arithmetic in one invocation: vulkan.md). As on
OpenCL, a MoE layer's routing (softmax, expert choice, weights) and its
gate and up products with their SwiGLU are held back op by op and run as
one dispatch each, and so is the end of a residual branch (its last
product or MoE sum, the scale of its output, the add into the stream),
with the bits of the ops one by one. The server, layers spread over
several Vulkan devices, weight streaming and the device expert tier (its
uploads on a transfer queue) work on it as on OpenCL; auto
picks it on every GPU it finds, OpenCL where Vulkan is not there
(OKLM_BACKEND_ORDER overrides). Decode products of Q4_K,
Q5_K, Q6_K, Q8_0, Q4_0 and Q5_0 weights can run on 8-bit integer dot
products with the activations rounded as the CPU rounds them, giving the
CPU's products bit for bit; they are taken where they were measured not
slower than the float products (OKLM_VK_Q8), the K-quants
in one of two layouts by vendor: eight lanes a super-block with the
rounding in the work-group where memory bounds the product (AMD), a
rounding dispatch then a weight row per invocation where the shader's
arithmetic does (Intel: Llama-3.2-1B decodes at 19.8 t/s on a UHD 750,
6.4 before, llama.cpp's Vulkan backend 15.6;
OKLM_VK_Q8_ROWS). See vulkan.md.
$ oklm devices
backends:
cpu priority 0 (selected)
cpu:
features : sse3 avx avx2 fma f16c avx512f avx512bw avx512vl avx512vnni
kernels : avx512 (best available: avx512)
threads : 8
opencl: not available
features are the instruction sets the CPU has;
kernels the SIMD level OKLM picked (scalar, AVX2 or
AVX-512; OKLM_SIMD forces one). The SIMD kernels give the
same bits as the scalar ones. A quantized product first rounds its input
to bytes, once, on the calling thread; that rounding runs on vector
instructions too (the scalar loop was 8 % of a SmolLM2-135M decoding
step, all of it with the memory idle), and a token's rotation computes
its cosines and sines once for all its heads. The scalar rounding is
OKLM's own as well, not the C library's roundf: the Windows
build's returns 1 for the float just below a half. The quantized
matrices of the CPU layers also get a repacked copy, eight rows
interleaved, which prompts read; decoding reads it too with one thread
(two when the copy is barely larger), where waiting on one row's sum at
a time would cost more than the memory, and at any thread count when the
copy is no larger than the file (Q8_0, Q5_0); with more threads a larger
copy loses, memory being the limit. Same bits either way. A MoE model's
routed experts get the same copy when it fits in memory beside the
mapped file (not with an expert cache), which makes its prompts about
1.45 times faster. On Linux the copies are cut one after the other from
chunks aligned on 2 MiB that the kernel backs with huge pages
(OKLM_CPU_PACK_HUGE=0: an allocation each): a decoding step
reads all of them once, and on 4 KiB pages every page of a small matrix
costs an address translation miss and ends the prefetcher's stream
(SmolLM2-135M decoding 3 to 5 % faster on the chunks).
The worker team. Decoding a token runs hundreds of
small parallel loops a few microseconds apart. Starting threads for each
would cost more than the work, so the CPU backend keeps a team of
threads alive (one per physical core by default,
OKLM_THREADS to change it) that claim chunks of each loop
from a shared counter. Each output element is computed by one thread in
a fixed order, so the result does not depend on the thread count (adr/0016-cpu-worker-team.md).
The products that read the same input (a layer's Q, K and V, its gate
and up) run as one loop, which saves the team's start and end of the
others. A decoding step's products are bound by memory, which half the
cores already read at full speed with the packed copies' light kernels:
more readers only contend, so those loops take at most that many threads
(OKLM_MATVEC_THREADS), as do the file layout's under 4 MiB
of weights, and the larger ones at most one per core (Qwen3-8B Q8_0
decoding from 8 threads 3.53 to 3.68 tokens/s). The workers a loop
leaves out keep waiting, and those no loop of a step takes go to sleep,
so extra threads cost a decoding step nearly nothing (SmolLM2-135M at
four, eight or sixteen threads 250 tokens/s, where sixteen gave 220).
With few keys, a decoding step's attention runs its tiles as they are,
one loop, instead of splitting them into three.
oklm tune. The CPU kernels cut their
work into tasks (rows of a matrix product per task, attention tile
sizes). The best split depends on the machine and the model.
oklm tune times a 512-token prompt (-n) with
other splits for up to 60 s (--budget), keeps a candidate
only if its logits equal the defaults' bit for bit and it is at least 3
% faster, and caches the result in ~/.cache/oklm/tuning/
(or $XDG_CACHE_HOME/oklm/tuning/, or
--cache-dir), keyed by the CPU, the backend, the model's
shapes and types, and the kernel sources. Contexts apply a cached result
(entries older than 90 days are ignored); OKLM_TUNE=0 turns
that off, --force tunes again, run --tune
tunes first when nothing is cached.
$ oklm tune -m SmolLM2-135M-Instruct-Q8_0.gguf --backend cpu --budget 10 --cache-dir tunecache
...
setting default kept
matmul rows 16 32
attn rows 32 16
attn keys 64 64
prompt of 512 tokens: 227.4 ms with the defaults, 202.6 ms kept (11 candidates, 0 rejected by the logit check)
Results across machines are in tuning/06-results.md.
MoE models
A mixture-of-experts (MoE) layer replaces one large feed-forward block with many small ones, the experts, and a router that picks a few of them per token (8 of 32 for Granite-3.1-1B-A400M). The model is large on disk but each token reads only a fraction of it.
- Grouped expert products. In a batch, different tokens pick different experts. OKLM's indexed matrix product groups the batch's (token, expert) pairs by expert, so each chosen expert's weights are read once for all the tokens that need it, on the CPU and on the GPU (where the grouping itself runs on the device) (inference.md).
- The expert cache
(
--expert-cache GB|auto, CPU, onrun,chat,pplandserve): the routed experts stay in the file and are read into a per-layer cache of that size, least recently used first out;autotakes the largest cache that fits beside the rest.--expert-loaders N(4 by default) read missing experts on threads while the experts already in memory compute;--expert-predict last|routerreads the next layer's likely experts ahead (its last routing, or its router applied to the current state);--expert-directbypasses the page cache;--pin-gb GBpins the most used experts of the usage file (--expert-usage FILE, by default~/.cache/oklm/usage/<model hash>.json, made on the first run and updated as the model runs);--expert-statsprints hit rates at exit;oklm expert-packwrites a copy of the experts with each one's three matrices together, which--expert-pack FILEreads in one read a miss instead of three, and--expert-mirror FILEsends every other read to a copy of that file on a second drive. The output is the same, bit for bit, as with every expert resident.
$ oklm run -m granite-3.1-1b-a400m-instruct-Q4_K_M.gguf -p "The capital of France is" -n 16 --temp 0 --backend cpu \
--expert-cache 0.2 --expert-loaders 2 --expert-predict router --expert-stats
The capital of France is Paris.
...
expert cache: 24 layers, 216 slots, 0 pinned, 194.6 MiB; hits 2948, misses 656 (81.8 % hits); 2186.9 MiB read in 0.639 s (3424 MiB/s)
expert cache: waited 0.064 s for reads; read ahead 1747, used 1583 (90.6 %); predicted 3152, right 2882 (83.5 % of 3452 routed)
--n-cpu-moe N|all(device backend): the routed experts of the first N layers (or all) stay in host memory and run on the CPU, while the rest of those layers (attention, router, shared experts) goes to the device. Experts are most of the weights but few are read per token, so this keeps attention on the GPU for a model larger than the GPU (moe.md).--expert-gpu GB|auto(device backend): a device tier holding the hottest experts of those CPU layers, chosen from the usage file's counts and refilled from the routing; in each step the device computes the experts it holds while the CPU computes the others, and a prompt batch copies the experts it lacks to the device instead (moe.md).
Generation
Tokens
A model reads and writes token ids, not characters. The tokenizer
cuts text into pieces from a learned vocabulary (byte-level BPE or
SentencePiece BPE); Ġ marks a leading space.
$ oklm tokenize --model models/smollm2-135m-hf --pieces "Hello world, quantization!"
19556 'Hello'
905 'Ġworld'
28 ','
3324 'Ġquant'
1237 'ization'
17 '!'
$ oklm tokenize --model models/smollm2-135m-hf --decode 19556,905
Hello world
--bos adds the start token, --no-special
reads special-token text as plain text.
oklm train-tokenizer learns a byte-level BPE vocabulary
from text files (--pre picks the pre-tokenizer:
gpt-2, llama-bpe, qwen2,
smollm; --special adds special tokens,
--chat-template embeds a template):
$ oklm train-tokenizer --vocab 1024 --out tok.gguf data/tinyshakespeare.txt
1024 tokens -> tok.gguf
See tokenizer.md.
Sampling
For each position the model outputs a score (logit) for every token
of the vocabulary. The sampler turns those into one choice, in this
order (src/model/sampling.c):
- Penalties over the last 64 accepted tokens:
--repeat-penalty Pdivides positive logits by P and multiplies negative ones (1 = off); the library and the server also have frequency and presence penalties (0 = off). - Temperature 0 is greedy: the highest score, lowest id on ties.
- Top-k keeps the k best tokens
(
--top-k, 0 = all). - Top-p keeps the smallest set of best tokens whose
probability sums to p (
--top-p, 1 = off), after the temperature. - Min-p drops tokens less than min-p times as likely
as the best (
--min-p, 0 = off). - Draw one of the rest, by probability, from a
xoshiro256** generator seeded with
--seed, so a seed reproduces a run on every platform.
Option (run, chat,
serve) |
Default |
|---|---|
--temp T |
0.8 |
--top-k K |
40 |
--top-p P |
0.95 |
--min-p P |
0.05 |
--repeat-penalty P |
1 (off) |
--seed S |
1234 |
-n N (tokens to generate) |
128 |
-c N (context) |
the model's trained context, at most 4096 |
describe and agent take the same options. A
server request may set temperature, top_p,
top_k, min_p, seed,
frequency_penalty, presence_penalty and
repeat_penalty; the command-line values are the defaults of
what it leaves out.
$ oklm run -m SmolLM2-135M-Instruct-Q8_0.gguf -p "List three fruits:" -n 30 --temp 0.7 --top-k 40 --top-p 0.9 \
--min-p 0.05 --repeat-penalty 1.1 --seed 7 --backend cpu
List three fruits: apple, banana and grape.
oklm run completes the raw prompt; it does not apply a
chat template. oklm chat does. With --url URL
in place of -m, oklm chat is the same chat on
the model of an OpenAI-compatible server (oklm serve,
llama.cpp's server, another, at http[s]://host[:port] or
under a base path the URL names, which then stands for
/v1), which applies its own template: the conversation goes
as chat messages in a streamed request, the model is chosen among those
the server lists, and the context's use is an estimate (inference.md).
Chat templates
Chat models are trained on conversations written in a particular
format, with markers for each role. The model file carries that format
as a Jinja template (tokenizer.chat_template). OKLM has its
own Jinja interpreter and renders the template as Hugging Face's
apply_chat_template() does, with the variables
messages, add_generation_prompt,
bos_token, eos_token, and tools
and documents set to none, plus the functions templates
call (raise_exception, strftime_now,
range, namespace, dict).
oklm chat --template-vars JSON adds or replaces variables,
such as Qwen3's enable_thinking; the server takes the same
as chat_template_kwargs. A DeepSeek-Coder instruct file
written before GGUF carried templates gets the published one, known by
its BOS and EOS tokens. A file without a template is given one with
--chat-template FILE|NAME (oklm chat,
run, serve, agent,
describe): a Jinja file, or the name of a built-in format
(vicuna, chatml, llama2,
llama3, mistral, mistral-v3,
gemma, phi3, alpaca,
deepseek-coder, foundation-sec), each a Jinja
template compiled in; a catalogue model whose file has none names its
format in the catalogue (LLaVA 1.5: vicuna) and needs no
option. Without either, the command says once that the file carries no
template and writes ChatML. A template given that way is rendered as it
is: oklm run --rag writes its turn with it, nothing
replaces it (not the template OKLM renders an LFM2 file's tools with),
and the agent's small model gets it when its own file carries none. A
template the interpreter rejects falls back to built-in ChatML, Llama 2,
Llama 3, Mistral, Gemma or Phi-3 renderings (inference.md, adr/0006-chat-templates.md).
$ printf 'What is 2+2?\n/exit\n' | oklm chat -m Qwen3-0.6B-Q8_0.gguf --backend cpu --temp 0 --template-vars '{"enable_thinking": false}'
> 2 + 2 = 4.
Each chat turn renders the whole conversation again but evaluates only the tokens after the part already in the cache.
A conversation longer than the context does not stop the chat. Past a
share of the context the older messages are summarized by the model into
a note that takes their place (/compact), and a reply that
the context or the token limit cut is gone on with: the conversation is
rendered again with the cut reply as the open start of the model's turn,
no end of turn after it, so the model writes on from the reply's last
token (/continue). oklm chat, the chat window,
the dashboard's chat and oklm agent do both by themselves,
and either can be left to the person; a limit the person gave
(-n, /max-tokens) still ends a reply for good.
When the context is what cut the reply, each of the four first asks for
a larger context where the memory plan (or the server's own, through the
dry run of its resize) holds one, and compacts only when none is made.
The agent asks for the larger context before a turn too, when the
conversation leaves less than a turn's room, makes no summary that would
be longer than what it replaces, and, when its own system prompt and
tools leave no room in the context, says so with the figures instead of
sending a request; oklm agent -p then ends with status 1,
as for any message the context refused (agent-sessions.md). A
model that writes its reasoning in a channel of its own has its turn
opened again in the channel it was cut in, and afresh when it was cut
inside the first header of its turn, whose markup is no text to go on
from. A reply cut inside a tool call is gone on with under the calls'
grammar, which first reads the call's text so far, so the call comes out
whole and is never run as the half that was written, through the chat
completions and through /v1/messages alike (inference.md).
Constrained output
A grammar restricts which tokens may come next: at each step the sampler checks its choice against the grammar and, if refused, masks every token the grammar does not allow and chooses again.
--grammar FILE.gbnf: a GBNF grammar, llama.cpp's syntax.--json-schema FILE.json: a JSON schema turned into a grammar.--json: any JSON value.
$ oklm run -m SmolLM2-135M-Instruct-Q8_0.gguf -p "A person as JSON:" --json-schema person.json -n 40 --temp 0 --backend cpu
A person as JSON:{"name": "John", "age": 30}
(person.json: an object with a required string
name and integer age.) Tool
calls use the same machinery: with
chat --tools tools.json (OpenAI-style definitions),
generation is free until the model writes the opening marker of a call,
then constrained to a valid call of a declared tool. The wire format is
the model's: hermes (<tool_call>),
llama3, mistral ([TOOL_CALLS]),
glm45 or glm4 (GLM-4-0414: the function's name
opens the turn, its arguments follow as JSON, the result comes back as
an observation turn), harmony (gpt-oss: a message addressed
to=functions.f), lfm2 (Liquid's LFM2: a Python
list of calls between <|tool_call_start|> and
<|tool_call_end|>, its literals turned into JSON
arguments) or gemma4 (Gemma 4:
call:f{key:value} between <|tool_call>
and <tool_call|>, strings between
<|"|> marks, turned into JSON arguments), recognized
from the template (for LFM2 from the vocabulary too: a file whose plain
ChatML template says nothing of tools is then rendered, in
oklm serve, oklm chat --tools and
oklm agent, with a template that writes them as Liquid's
does, unless --chat-template gave one) or set with
--tool-format. A reply's reasoning is told apart from its
answer by one splitter (src/harness/think.c): a
<think> block, the analysis channel of a harmony
reply, the message a Muse Glimmer reply addresses to itself
( to=self<|message|> ... <|eom|>),
or Gemma 4's thinking channel (<|channel>thought ...
<channel|>), which oklm chat,
/v1/messages (a thinking block) and
/v1/chat/completions (reasoning_content) show
without the channels' markup; a prompt that ends on an opening tag makes
the reply start as reasoning. --tool-choice required forces
a call. Results go back with /tool RESULT. See inference.md and adr/0009-tool-calls-and-grammars.md.
Speculative decoding
Checking k proposed tokens costs the large model about one batched pass, not k passes. Speculative decoding guesses a few tokens cheaply and lets the model verify them together; every emitted token is still drawn from the model's own logits with the same random draw, so the output is the one plain generation gives with the same seed. Only the speed changes.
| Option | Source of the guesses |
|---|---|
--draft MODEL (-md),
--draft-n 5 |
a smaller model with the same vocabulary |
--spec-ngram K |
prompt lookup: the K tokens that followed the last 1 to 3 tokens earlier in the text |
--mtp, --mtp-model FILE |
the model's own multi-token prediction blocks (DeepSeek V3 files that have them) |
--spec-grammar |
the text a grammar forces (a JSON key, a tool call's fixed text) |
--spec-guard RATE |
not a source: no guesses while the expert cache's recent hit rate is below RATE |
$ oklm run -m SmolLM2-135M-Instruct-Q8_0.gguf -p "Repeat: the cat sat on the mat, the cat sat on the mat, the cat" \
--spec-ngram 4 -n 24 --temp 0 --backend cpu
...
draft: 20 proposed, 20 accepted (100.0 %)
$ oklm run -m SmolLM2-135M-Instruct-Q8_0.gguf --draft SmolLM2-135M-Instruct-Q4_K_M.gguf -p "The capital of France is" -n 16 --temp 0 --backend cpu
...
draft: 20 proposed, 16 accepted (80.0 %)
A draft pays off when it is much smaller than the model; the second example only shows the mechanism (inference.md).
Decisions by scoring
Some questions have a fixed set of answers ("positive or negative?"). Instead of letting the model write an answer and parsing it, OKLM scores each allowed answer as the continuation of the prompt: it sums the log-probabilities of the answer's tokens (teacher-forced: the tokens are given, not generated) and turns the sums into probabilities over the options with a softmax. Nothing is generated, the answer is always one of the options, and the probabilities show how sure the model is relative to the alternatives (they are not calibrated frequencies). The sum, not the mean, is used so a long option is not favoured for its length.
oklm chat's lines that start with / are
commands, read from one table
(include/oklm/client/chat_commands.h) that the chat window
and the server's GET /api/commands share:
/help lists them, an unknown one is answered with the
nearest names and is not sent to the model, //text sends a
text starting with / (inference.md). The commands
change what the loop works with while the conversation stays:
/context N makes another context and /model
loads another model (chosen in a list of the catalogue by group,
downloaded only once the chat asked), and since each turn renders the
whole conversation again and evaluates what the cache does not hold, the
next turn simply evaluates everything with the new context or model (inference.md).
In oklm chat, /decide A | B | ... asks
about the conversation so far:
$ printf 'I love this phone.\n/decide positive | negative\n/exit\n' | oklm chat -m SmolLM2-135M-Instruct-Q8_0.gguf --backend cpu --temp 0
...
positive 64.86 %
negative 35.14 %
decision: positive (confidence 0.06)
Confidence is 1 minus the entropy of the distribution over that of a
flat one: 0 when every option is as likely, 1 when one is certain. The
server's /v1/decide (also /v1/systemone) does
the same for several named questions about a state, with yes/no, choice
and score questions, sharing the prompt's cache between options (server.md).
Looking inside a run
--observe FILE on oklm run and
oklm chat writes, for the prompt's last token and each
generated one, a JSON line of what the model did inside: the norm of
what each block's attention and feed-forward part added to the residual
stream, the experts each router picked with their weights and, with
--observe-attn, the attention of that token over the
positions before it, a row a head:
$ oklm run -m models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf --backend cpu --temp 0 -n 6 \
-p "The capital of France is" --observe run.jsonl --observe-attn 5
$ head -c 150 run.jsonl
{"seq":0,"pos":4,"token":438,"piece":" is","serial":1,"n_layer":24,"n_observed":24,"embd":7.30355,"resid":[3.72...
The blocks a device runs are recorded too: the rows are copied aside
on the device and read back once when the pass ends (what it costs: observe.md); off, it costs a pointer
test per block. --observe-lens K adds what the output head
would say after each block (the residual stream at that depth through
the final norm and the head: for "The capital of France is", Llama 3.2
1B's stream reads " Paris" from block 11 of 16 on).
The norms say how large a write is, not what it does.
--ablate runs the prompt again without a part (a block's
attention write, its feed-forward write, one head: zeroed, or with
--ablate-mean replaced by its mean over the prompt) and
says how far the next token's distribution moved;
--ablate-scan blocks does it for every block's two parts in
turn:
$ oklm run -m models/llama3.2-1b/Llama-3.2-1B-Instruct-Q4_K_M.gguf -p "The capital of France is" --temp 0 --ablate-scan blocks
part KL (nats) p(plain) likeliest after
0:attn 1.25425 0.9550 " Paris" p=0.9550
0:ffn 8.45301 0.0000 " out" p=0.0463
...
The stream is kept before the part and put back after it, through
host memory, so every kind of block and every backend takes it (observe.md).
A server started with --observe gives the same record for
the token it decoded last (GET /api/activations), and the
model inspector draws both (gui.md). The figures are exact,
and say less than they seem to: a large write is not an important one,
and attention is not an explanation (observe.md).
Documents
oklm ingest FILE... prints a document as the text a
model reads, and oklm run --doc FILE puts that text before
the prompt, inside <document> tags. The format is
chosen by extension, then by content (src/mm/ingest.c);
files up to 64 MiB:
| Format | How the text is extracted |
|---|---|
| Plain text, Markdown, CSV, TSV, code | as it is, without a byte order mark or carriage returns |
| JSON | parsed, then indented two spaces a level |
HTML (.html, .htm, .xhtml, or
a file starting with <html /
<!doctype html) |
rendered: scripts and styles dropped, entities decoded, blocks on their own lines |
DOCX (or any ZIP holding word/document.xml) |
the paragraphs; headings as # lines, list items as
- , tables as Markdown rows |
PDF (or a file starting with %PDF-) |
each page's text in order, spaces and line breaks placed from where the glyphs land; fonts decoded through their ToUnicode maps or encodings; documents encrypted with an empty user password are decrypted |
$ oklm ingest r.html
Report
Sales rose & costs fell.
$ oklm ingest tests/fixtures/pdf/simple.pdf
Hello, World!
Kerning stays one gap
...
$ oklm run -m model.gguf --doc r.html -p "What happened to sales?"
Document r.html:
<document>
Report
...
Details in multimodal.md. The agent's Read tool shows Word and PDF files the same way.
Embeddings
oklm embed -m MODEL TEXT... (or --file F,
one text a line) prints one vector a text: the first step of retrieval
(rag.md). Three kinds of model run:
- Text encoders,
bert(BGE, MiniLM, e5; RoBERTa and XLM-R as llama.cpp writes them) andnomic-bert, from GGUF or a Hugging Face checkpoint (src/model/arch_bert.c,src/io/names_bert.c): the token, type and position embeddings summed and normalized, then post-norm blocks whose attention sees every token of the text in both directions (nomic-bert rotates queries and keys and uses SwiGLU). No KV cache: each forward pass holds whole texts, several at once. - Decoder embedding models (Qwen3-Embedding): the usual forward pass without its output head; the last token's final normalized state is the embedding.
- The pooling (the first token for bert, the mean for nomic-bert, the last token for decoders, or what the file says) gives one vector a text, scaled to unit length.
$ oklm embed -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf "What is the boiling point of water?"
embed: 1 texts, 10 tokens in 0.005 s: 195.4 texts/s, 1954 tokens/s (cls pooling, 384 dimensions)
-0.01469593 -0.004681007 0.008391812 -0.03536071 0.05449276 ...
The library call is oklm_embed()
(include/oklm/rag/embed.h), on an embedding context
(oklm_context_params.embeddings); the server answers
POST /v1/embeddings with --embed-model. The
vectors match a numpy float64 encoder written from the model definitions
and llama.cpp's llama-embedding (rag.md).
Retrieval
Retrieval answers from documents the model was not trained on (rag.md is the user guide and the measures). It runs in four stages, each a C interface a program can replace: index, search, judge, answer.
The index.
oklm rag index PATH... --out INDEX reads each file as
oklm ingest reads it and cuts it into chunks
(src/rag/chunk.c, chunk_code.c: Markdown
sections, paragraphs and sentences; C-like and Python functions; fixed
windows; or semantic breaks). Each chunk is embedded through an encoder
interface (include/oklm/rag/encoder.h: an embedding model
through oklm_embed(), or a hashed bag of words that needs
no model), and everything goes into one GGUF file
(src/rag/index.c, ADR-0022): the chunks and
their sources, the vectors as one [dim, n] tensor, the BM25
postings (src/rag/bm25.c) and the embedding model's
identity, so a search with another model is refused. Run again, it
hashes the files and embeds only what changed. Optionally a chat model
extracts each chunk's entities and relations into the index
(--graph), and a vision model captions the documents'
figures, each caption a chunk (--caption-model, its
question --caption-prompt). A program that drives the
commands reads JSON instead of the text: the build's progress as lines
(--progress-json: each file added, updated, unchanged or
removed, the chunks embedded so far, the result), the index and whether
each source file changed since (rag info --sources --json),
and a search with its times and each hit's scores
(rag search --json-full) (rag.md).
The search. oklm rag search maps the
file. A dense search is one oklm_matmul() of the query
against the vectors where they lie in the mapping and an
oklm_topk(): exact, 0.04 ms over ten thousand chunks (384
dimensions, F16) and 27 ms over a million, where it reads the 768 MB of
vectors at the memory's bandwidth, so no approximate index is needed.
BM25 reads the postings of the query's terms; the default, hybrid, fuses
the two lists by their scores, min-max normalized and weighted 0.5 /
0.5, or by reciprocal rank (--fusion rrf;
src/rag/fuse.c). With a graph (--rag-graph on)
the search also walks from the question's entities, and those of its
three best hits, to the chunks that mention them or an entity related to
them, and fuses that list too (include/oklm/rag/graph.h). A
reranker can reorder the first candidates (--rerank,
oklm rerank, /v1/rerank;
src/rag/rerank.c): a cross-encoder (ms-marco-MiniLM-L-6-v2)
is a bert encoder with a classification head that reads
query and passage together as
[CLS] query [SEP] passage [SEP], the passage's tokens
adding the second row of the token type table
(oklm_eval_batch_types()), the head (cls, a
dense layer and tanh, then cls.output) turning the
[CLS] row into one logit
(oklm_model_classify()); without one, a chat model scores
each passage by a yes / no decision through decide.c.
oklm rag eval measures the search on a BEIR-format set
(src/rag/eval.c: the corpus indexed, with its graph when
--graph --gen-model is given, every judged query searched,
nDCG@10, MRR@10, recall@k and precision@k as BEIR scores them), and
make rag-eval runs it on SciFact and on 120 questions over
these docs (NFCorpus on request: 34.38 dense against the model card's
34.31, 36.67 hybrid; 112 questions over OKLM's own sources on request,
where cutting code at functions scores no better than cutting at
paragraphs and lines, 59.32 against 60.12 nDCG@10, and
Qwen3-Embedding-0.6B is the best encoder, 63.63 dense). With
bge-small-en-v1.5, SciFact's nDCG@10 is 71.27 dense (MTEB's published
figure), 67.41 with BM25, 71.29 fused by reciprocal rank and 73.04 by
weights (the default fusion for that); on the docs 60.48, 64.53, 69.78
and 70.61. The MiniLM reranker lowers SciFact's 71.64 to 68.95 over the
hybrid candidates, as the BEIR paper's figure for that model predicts,
so reranking is off by default; on the docs it puts the right section
first more often (recall@1 30.2 to 37.4 %) without changing the answers
beyond one question of 30. tools/rag_stats.py puts
bootstrap intervals on these rows: only the larger differences hold on
sets of 38 to 323 questions.
The judge. With --rag INDEX,
run, chat, serve (a request's
rag field) and the agent answer from an index
(include/oklm/rag/rag.h). oklm_rag_run()
chains the stages: the retriever (the index's search), an optional
reranker, the evaluator, the rewriter, an optional web search. The
evaluator is the generator itself: each passage becomes the state of a
yes / no decision by scoring ("Does the passage above contain
information that answers the question?", decide.h), one forward pass
from the shared question; the probability of yes makes it relevant (0.5
or more), irrelevant (under 0.2) or ambiguous. When none is relevant the
generator rewrites the question (a short greedy decode) and it is
searched again; then, for the agent only, the web through its WebSearch
tool; then the prompt asks for an answer saying that no reliable source
was found. A passage the graph found is kept beside a relevant one,
since the second fact of a chain does not answer the question alone.
oklm_rag_run_turn() adds three optional stages, the
generator again: a condenser rewrites a follow-up ("and how do I turn it
off?") as a standalone query from the earlier turns, and the previous
question is searched with it (on by default,
--rag-followup); a router decides by one three-way decision
whether to search not at all, once or several times
(--rag-route, off); and for several, a decomposer writes
the sub-questions, each retrieved and judged apart, their passages
merged rank by rank.
The answer. The relevant and ambiguous passages go
into one user message, numbered with their file, lines and heading,
within a token budget, after instructions to cite them as
[n] and to say when they do not answer. With
--rag-critique on each cited statement is then asked
against its sources and the answer against the question
(critique.h, decisions by scoring); an unsupported
statement gets the answer generated once more with the critique fed
back, the better of the two kept and flagged when still unsupported. An
answer already shown cannot be taken back, so a streamed one
(--rag-critique flag, a server stream) is checked once
complete and only flagged: the verdict follows it (a stream's last
chunk, a line after a chat reply), and a server stream sends a second
answer only when the client asks. The agent checks a final answer
against the passages its message retrieved and, with on,
sends the model back once with the critique as a reminder, through the
loop's answer_check hook. The server runs the judge, the
rewrites and the critique as scoring and generation requests of its
scheduler and returns the run in a rag member. Every run is
one record (query, rewrites, route, passages with scores and verdicts,
fallback, critique) written as a JSON line to --rag-log, to
the agent's debug log, and counted in /api/stats.
In the agent, Retrieve is a built-in
tool over an index of the project it keeps itself
(--rag-project, src/harness/rag_index.c): the
files Glob would list that the include globs select (the documentation
by default), stamped by name, size and modification time before each
search and updated by content hash when the stamp changed, so the
model's own edits are found by its next search. It reads only, so the
permission modes run it as they run Read. With --rag-memory
it also searches the project's earlier session transcripts and the
memory files, and with --rag-inject always|route each user
message gets its passages as a reminder through the agent's
rag_context hook. On a model served elsewhere
(--url) the judge, router and critic are requests to the
server's /v1/decide, or on another server chat completions
read by their log-probabilities or their text, and the rewrites chat
completions at temperature 0 (src/rag/rag_http.c). The
agent selects its tools by retrieval too (--tool-rag, The coding agent).
On 38 questions about OKLM's documentation, Qwen3-8B answers 16 % right without retrieval, 89 % with the top five passages and 92 % with corrective retrieval, which also declines all 8 questions the docs do not answer; self-critique raises the share of cited statements the same model finds supported from 87.5 % to 96.1 %, accuracy unchanged. On 30 questions that need two linked articles the graph brings both into the top five for 23 instead of 8; on 20 questions answered only by charts, SmolVLM2-2.2B's captions let the generator answer 18 instead of none (SmolVLM-256M's 9). On eight agent tasks that need the project's docs, the agent passes 2 without retrieval and 7 or 8 with it. Every figure, its set and its run are in rag.md.
The server
oklm serve puts a model behind an HTTP API on
127.0.0.1:7430 (--host, --port).
Concurrent requests share one context: each step gives every generating
request its next token in one batch, so serving many users costs much
less than running them one after another. Full cache blocks are kept as
a prefix cache, so requests that share a system prompt or a conversation
reuse its computed rows (server.md). The GPU's
products pick their kernel by a piece's rows, so a prompt found cached,
whose last rows go as a small piece, can answer otherwise at temperature
0 than the same prompt cold; --deterministic puts the
pieces on a fixed grid of positions and uses a cached prefix only up to
its last line, so a request alone answers the same either way (server.md).
| Endpoint | API |
|---|---|
POST /v1/chat/completions |
OpenAI chat (streaming, tools, response_format JSON and
JSON schema) |
POST /v1/completions |
OpenAI raw completion |
GET /v1/models |
the model and each --lora-module adapter |
POST /v1/messages,
POST /v1/messages/count_tokens |
Anthropic Messages (streaming, tools, thinking with its
budget_tokens cap) |
POST /v1/decide, POST /v1/systemone |
decisions by scoring |
GET /api/stats, /api/experts,
/api/profile |
the dashboard's data |
GET /api/activations |
with oklm serve --observe: what each block wrote, the
experts picked and the attention asked for, for the token decoded last,
or with serial=S for one of the last 512 records (observe.md) |
GET /api/commands |
the commands a chat on this server takes (/context,
/rag...), each with whether the server can do it |
GET /api/model |
the model's layout: its blocks by role, where each layer is (CPU or device), the context's memory by part |
GET /health |
{"status":"ok"} |
GET / |
the dashboard |
GET /activations |
with --observe: a page that draws what
/api/activations answers (each block's writes, the lens,
the attention of the blocks named, the experts), following the last
token or stepping through the kept records |
Embeddings and reranking are served when the server is given their
models (--embed-model: POST /v1/embeddings;
--rerank-model: POST /v1/rerank), as are
transcriptions, speech and documents (server.md).
The servers that run. A server writes a file for the
user alone at its start (pid, port, address, model, command line) and
removes it when it stops. oklm serve --list reads them
(--json for a program) and
oklm serve --stop PORT|PID stops one; the graphical tools
start their servers and jobs through the same registry (server.md). On Windows it is
the same with that system's means: a command started without a console,
an access list of one entry in place of the modes, and a stop asked
through a named event the server waits on, since a process without a
console cannot be sent Ctrl-C (run under Wine 11, not on Windows
itself).
Servers kept across logins.
oklm serve --keep NAME ... writes the same command as a
unit of systemd's user manager
(~/.config/systemd/user/oklm-serve-NAME.service, for the
user alone), enables it and starts it: the server then starts at each
login and again when it fails, its output appended to a log file the
registry names, and it registers itself like any server, marked with its
service's name. oklm serve --kept lists the kept ones with
what the manager says of them, oklm serve --unkeep NAME
removes one; the server manager does the same from its form
(Keep across logins) and its list, where a kept server
that is stopped or failed keeps its row (gui.md). The arguments are quoted
by the rules of ExecStart= (% and
$ doubled, quotes and backslashes escaped), a key given
inline is refused, and where the user manager does not answer (a
container, another init) the commands say so and do nothing.
--rekeep NAME ... changes the command kept under a name,
and so does the server manager's Re-keep with form for
the kept server selected, with the form's settings. On Windows the same
commands write a value of the user's Run key, which runs
oklm serve --start-kept NAME at login: the server is
started in the background through the registry, and is not started again
when it fails; on macOS a launchd agent, written and never built (server.md,
include/oklm/harness/kept.h).
Access. On a loopback address no key is needed
unless --api-keys FILE is given. Any other address needs
TLS (--tls-cert, --tls-key) and keys, or the
server refuses to start (--insecure overrides it). The keys
file holds one key per line, blank lines and # comments
ignored, and must not be readable by group or others
(chmod 600). Keys come as
Authorization: Bearer KEY or x-api-key: KEY; a
wrong or missing one gets 401.
A session against SmolLM2-135M on the CPU:
$ printf '# one key a line\nsecret-key-1\n' > keys.txt; chmod 600 keys.txt
$ oklm serve -m SmolLM2-135M-Instruct-Q8_0.gguf --backend cpu --port 18431 -c 2048 --api-keys keys.txt
serving SmolLM2-135M-Instruct-Q8_0 on http://127.0.0.1:18431/ (512 requests at once, 2048 tokens each, API keys required)
OpenAI style:
$ curl -s http://127.0.0.1:18431/v1/chat/completions \
-H 'Authorization: Bearer secret-key-1' -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":24,"temperature":0}'
{"id":"chatcmpl-1","object":"chat.completion","created":1794094858,"model":"SmolLM2-135M-Instruct-Q8_0",
"choices":[{"index":0,"message":{"role":"assistant","content":"The capital of France is Paris."},"finish_reason":"stop"}],
"usage":{"prompt_tokens":37,"completion_tokens":7,"total_tokens":44,"prompt_tokens_details":{"cached_tokens":0}}}
With "stream":true the answer comes as server-sent
events, one chat.completion.chunk per piece, then
data: [DONE].
Anthropic style:
$ curl -s http://127.0.0.1:18431/v1/messages \
-H 'x-api-key: secret-key-1' -H 'anthropic-version: 2023-06-01' -H 'Content-Type: application/json' \
-d '{"model":"any","max_tokens":24,"system":"Answer in one sentence.",
"messages":[{"role":"user","content":"What is the capital of France?"}]}'
{"id":"msg_0000000000000003","type":"message","role":"assistant","model":"SmolLM2-135M-Instruct-Q8_0",
"content":[{"type":"text","text":"The capital of France is Paris."}],"stop_reason":"end_turn","stop_sequence":null,
"usage":{"input_tokens":26,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":7}}
A decision:
$ curl -s http://127.0.0.1:18431/v1/decide -H 'Authorization: Bearer secret-key-1' -H 'Content-Type: application/json' \
-d '{"state":"The order arrived two weeks late and the box was crushed.",
"questions":{"complaint":{"type":"noul","instructions":"Is the customer complaining?"},
"topic":{"type":"choice","criteria":{"delivery":"shipping and delays","billing":null}}}}'
{"id":"decide-4","object":"systemone.result","model":"SmolLM2-135M-Instruct-Q8_0","answers":{
"complaint":{"type":"noul","label":"yes","probabilities":{"yes":0.669319449,"no":0.330680551},"confidence":0.0843798223,"p_yes":0.669319449},
"topic":{"type":"choice","label":"delivery","probabilities":{"delivery":0.949540009,"billing":0.0504599911},"confidence":0.711652238}},
"usage":{"prompt_tokens":215,"cached_tokens":80,"scored_tokens":5}}
Ctrl-C (or SIGTERM) stops the server.
Capacity. --parallel N is how many
requests generate at once. With the default paged cache a waiting
sequence costs only its block table, so by default it is the batch's 512
rows (or the cache's blocks, the fewer) and memory is the real bound;
--kv-mem MB sizes the shared cache, --queue N
(256) how many requests may wait, -c (4096) each request's
context.
Adapters. --lora FILE merges one LoRA
adapter into the model for every request.
--lora-module NAME=FILE (repeatable) serves an adapter
beside the model instead: a request whose model is NAME
runs with it, others without, in the same batches, and the base weights
stay as they are (a quantized model stays quantized):
$ oklm serve -m SmolLM2-135M-Instruct-f16.gguf --backend cpu --port 18432 --lora-module shakespeare=lora/a-000003.gguf
$ curl -s http://127.0.0.1:18432/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"shakespeare","messages":[{"role":"user","content":"Hello"}],"max_tokens":12,"temperature":0}'
{"id":"chatcmpl-1","object":"chat.completion",...,"model":"shakespeare",...}
The dashboard (http://127.0.0.1:7430/)
shows the figures of /api/stats (requests, tokens, cache
use, prefix hits), a chat that takes the chat's commands
(/help lists them) and files dropped, pasted or chosen
(images and sounds for a server with a projector, documents read by the
server, one too long for a message kept there for retrieval when the
server has an embedding model, a recording transcribed into the message
on a server started with --transcribe-model) and the
microphone (a Record button and /listen, on
localhost or over HTTPS: what is said is written in the
message box) and, on a server started with --speak-model, a
voice (a Speak button on each reply, /speak, a switch that
reads every reply: the page asks /v1/audio/speech for
samples and plays them with WebAudio while they are made; a reply read
by itself is spoken while it is written, each sentence asked for once no
later text can change it) and, with both, a spoken conversation (the
Talk switch, /talk: the microphone stays open with the
browser's echo cancellation, a detector on its level ends a turn at a
pause, the turn is transcribed and sent, the reply spoken, and a voice
at the microphone stops the voice and the reply and starts the next
turn), for a MoE model an expert map (one row per layer, one cell per
expert, lit by how often the router picks it), and a profile of the last
requests (queue, prompt, decode times and the distribution of step
times) (server.md). The chat
window does the same with the system's microphone
(apps/gui/chat_listen.h, gui.md): Record or
/listen keeps the fragments of Allegro's audio recorder in
memory, at 16 kHz mono or resampled to it, with a level meter; while it
grows, the recording so far is transcribed again every two seconds
(partial=true, which a busy recognizer refuses at once) and
the words two of those readings agree on are shown as said, in the
window's status line as under the dashboard's meter: Whisper reads a
whole sound, so the words while speaking are that, not a streaming
decoder; at its end the recording goes to
POST /v1/audio/transcriptions as a WAV file (the window's
own server is started with the smallest installed Whisper model, a
server at an address has to say it transcribes), or, attached to an
agent, through the library in the window itself, and the text lands in
the message box. It reads replies aloud too
(apps/gui/chat_speak.h, gui.md): Speak beside a reply,
/speak or the Read aloud box take what the dashboard reads
of a reply (the same rule, ported), cut it at sentence ends into the
pieces a server takes, and ask POST /v1/audio/speech for
raw samples, read while they are made
(oklm_client_post_data()), from the window's own server
when it was started with --speak-model or from a server at
an address that has a voice; elsewhere the window loads the installed
speech model itself at the first reading. The samples go to a queue that
a thread feeds to a stream of Allegro's audio, or, for the tests, to a
WAV file. With Read aloud on a reply is said while it is written: each
time the window draws, what was written since is read
(speak_stream), and the sentences that nothing written
later can change (no mark of Markdown left open before them, the same
whatever closing mark follows) join the queue of the thread making the
sound, the first piece in two passes so that its sound comes early. A
voice is the model's built-in one or a speaker file chosen in the
window, which loads its synthesizer again, or starts its own server
again, with it. And a spoken conversation (gui.md,
apps/gui/chat_talk.h): Talk leaves the microphone open; a
turn starts when its level held above a threshold for a quarter of a
second and ends at a pause (listen_turn); its words are
transcribed and sent without a press, the reply is said while it is
written, and the window listens again once the voice ended. The same
rule hears somebody speak over the voice: the voice and the reply stop,
and what is being said is the next turn. oklm-gui holds the
native graphical tools, one program with --tool NAME
choosing among them (gui.md):
launcher, a home window with a button for each tool and one
for the documentation that came with the program, models,
the model manager (the catalogue's models downloaded, listed and
deleted, and a chat, a server or the inspector opened on one),
servers, a manager that starts oklm serve in
the background from a form (with what the context takes of the machine's
memory), lists every server of the machine with its load and stops,
restarts, follows and resizes them (gui.md). The form and the chat
window share the controls of oklm serve's options: one
table (apps/gui/serve_opts.h) says for each option that has
a control what its value is, its range and when it applies, and the
command line, a kept profile, a command line read back and the memory
plan's parameters are made from it; an offload row (the layers on the
device, the cache quantized, the experts on the CPU, the cache's size)
feeds the memory line, which is the plan of a server: its cache is sized
by the rule oklm serve applies itself
(oklm_kv_cache_rule()), so that the line says
It fits, It fits with a smaller cache or
It does not fit as the server would turn out (a model
larger than its device, or one the device holds with less than one
context of cache beside it, has the cache of the layers that go there,
and the line says how many, the cache on each side and, for the second,
that it is slower than whole on the device; a model beside the model on
a device of its own is counted and judged on that device); one Advanced
window holds the rest, a tab a group, the last one a text field for the
options without a control: the server manager adds them to the command
line, the chat window gives its own server those it can take (gui.md). A chat window started
with nothing named loads no model: it lists the servers, the agents, the
installed models and a row for a GGUF file, and a model chosen there is
judged by the memory line before it is loaded in the window (gui.md). In the chat
window the same controls are the model settings: the sampling goes with
each request, and a change of where the window's own model computes
restarts it after asking, the conversation kept, while a server at an
address tells its own settings and its sampling defaults in
GET /api/context (gui.md). A server at an address
that is not OKLM's (llama.cpp's, any OpenAI-compatible one) is found out
once, when the window is linked to it, by asking each of OKLM's own
routes: the window then asks it for models and replies only, counts on a
context of its own setting, compacts by estimate, and says so where the
figures would be; such a server is kept with its key, its model's name
and that context in the window's settings file, which only its owner
reads (gui.md). The
memory line of a model the window already serves counts as free what its
server holds on its device and in the host's memory
(device_memory.held and host_memory.held of
GET /api/stats), since a restart frees it first; the line
counts what the server loads beside the model (a draft model, the
projector's encoder, the sound models, the embedding and the reranking
models) where the server places it, on the device or on the CPU with
what it works with there, sized from the files by the functions the
server uses, and in the chat window a speech model that is only
installed, which the window opens at the first reading (gui.md); a model loaded in place of
another runs with the window's backend, context, cache types and
sampling, not with the layers, the cache's size or the draft model
chosen for the model before, which are automatic again; and what a model
of the window's own runs with is kept for it, in the window's settings
file, for the next window on that model. convert, which
makes a GGUF file from a checkpoint or requantizes one and measures what
was lost, train, which builds an oklm train,
rl, refusals or abliterate
command from a form, starts it in the background and draws its loss from
the --progress-json file as it is written,
rag, the retrieval manager, inspect, the model
inspector, and chat, the default. The inspector draws a
model as a map, a bar a block sized by its bytes and split by role, from
the JSON the layout already gives: a file's tensor table and the memory
planner (no weight loaded), with the backend, the context and the layers
on the device chosen in the window, or a running server's
/api/model with its figures read every second; it also runs
a prompt through a file's model on the CPU (as it is, or through the
chat template), or follows a server started with --observe
and steps back through the records it keeps, and draws what each block
wrote, the attention of the block selected, the experts picked, the
tokens the output head would give after each block, and how far the
output moves when a block's write or a head is taken out (gui.md). The retrieval manager
works on one index: it starts oklm rag index as a
background process and follows the JSON lines of its progress, shows
each source of the index with the state of its file (the stale ones
marked, and an update that embeds only those), searches it in each mode
with or without the reranker, shows the table of
oklm rag eval, and starts a chat window whose answers come
from the index (oklm-gui -m MODEL --rag INDEX). The chat
window is on the same API: given a model it serves it itself, and
started alone it loads nothing and asks what to talk to, a server
already running on the machine or an installed model, since a second
copy of a model a server holds would take its memory twice. From its
side panel it downloads, uses and deletes the models of the catalogue,
changes the context's size while a conversation is open, and connects to
something else without closing: a server found on this machine or given
by its address, an agent started with oklm agent --listen,
or an agent session of the folder, read only. Files go with a message
(dropped on the window, chosen with Attach, or --attach):
an image, a sound or a video is sent in the request as the server takes
them (a video whole, as a video_url part the server
samples; the window decodes it once for its first frame and its length),
to a model whose projector the window's server opened (the catalogue's,
or --mmproj), and refused with the reason before sending
when the link has none; a PDF, DOCX, HTML, JSON or text file is read by
the window with the library's reader and its text sent under its name;
one longer than the context leaves room for is indexed by the window,
with an installed embedding model and the retrieval module, in a file of
the user's cache named by the text's hash and the embedding model, so
that the same document attached to another window is not embedded again;
each message is sent the passages found in it for that message, put in
order by an installed reranker when there is one, or passages spread
over the document when the message is about the whole, which the model
is told (its beginning only, when no embedding model is there and the
user chose so); an attached agent tells its context and is sent passages
too (gui.md). A message that
starts with / is a command: the window gives the commands'
parser the place it is in (its own server, a server at an address, an
agent) and what that link can do, runs what the parser finds right (the
system prompt, the model, the context, the sampling, saving and
exporting, retrying and rewinding), and writes the answer as a note in
the conversation that no request carries; a list above the message
completes the commands as they are typed (gui.md). sessions is the
browser of a project folder's agent sessions: listed with their date,
title, turns, model and size, searched by what their messages hold, read
with the tools' calls folded, written as Markdown or JSON, resumed in a
terminal, attached to when an agent runs them, or deleted (gui.md).
The context is the number of tokens a model works on at once. Each
model has the one it was trained with; -c chooses another
at start, and a running server takes POST /api/context to
change it without loading the model again: the cache is made again for
the new size, so the next message of a conversation is read whole once.
"dry_run": true asks what a size would take in memory first
(models.md, server.md). The model itself
is changed the same way, POST /api/model with a name: the
server opens the new file first, waits for the requests under way, frees
the old model and its cache, and makes the cache again for the new one
with the settings it was started with; a model it cannot serve leaves
the one before serving (server.md). The dashboard's
/model NAME and the chat window's Use this
model do that, and since a chat sends its whole conversation
with each message, the conversation goes on with the new model. The new
cache is filled whole when it is made, a second for 6 GiB and a minute
for tens of them, so the chat window asks for a resize on a thread of
its own and says when it ended (gui.md).
The coding agent
oklm agent reads, searches and edits the files of a
project and runs commands there, through a model that writes tool calls.
See user-guide.md for daily use and harness.md for the mechanics.
Local or remote model. With -m MODEL
the model runs in the same process. With --url URL the
agent talks to any OpenAI-compatible server (oklm serve or
another) through streamed chat completions;
--remote-model NAME names the model there and the key comes
from --api-key, else OKLM_API_KEY, else
OPENAI_API_KEY. With neither, the agent uses the
baseUrl setting (a server) or the model
setting (oklm agent config set model PATH), else looks for
Qwen3-8B-Q4_K_M.gguf in $OKLM_MODELS, the
user's models directory (~/.local/share/oklm/models),
./models and the source tree's models/. The
in-process context defaults to 32768 tokens within the model's trained
context.
Permission rules. Every tool call goes through rules
before it runs. Rules are Tool or
Tool(pattern) in allow, ask and
deny lists, in settings files read from the highest
priority down: managed (/etc/oklm/managed-settings.json),
the command line (--allow, --ask,
--deny), .oklm/settings.local.json (personal,
where "always" answers go), .oklm/settings.json (shared
with the project), then the user's
~/.config/oklm/settings.json. Deny beats ask beats allow,
whatever the file. A Bash line is split into its commands, and each must
match an allow rule; make:* matches make
followed by anything. Path rules resolve .. and symbolic
links first, so Read(./secrets/**) also catches
secrets/../secrets/key.pem. Calls no rule matches are
decided by the mode (default: reads inside the project run,
the rest asks; acceptEdits: edits inside run too;
plan: reads only, until the user accepts a plan). A list of
dangerous commands (rm -rf /,
git push --force, sudo, ...) always asks even
when allowed (agent-permissions.md).
oklm permissions shows what the rules decide for a call,
with exit status 0 (allow), 3 (ask) or 4 (deny):
$ cat proj/.oklm/settings.json
{"permissions":{"allow":["Bash(make:*)","Edit(./src/**)"],"ask":["Bash(git push:*)"],"deny":["Read(./secrets/**)","Bash(rm -rf:*)"]}}
$ oklm permissions --project proj Bash "make test && rm -rf build"
deny: Bash(rm -rf:*) (project settings), a command matches a deny rule
$ oklm permissions --project proj Read "secrets/../secrets/key.pem"
deny: Read(./secrets/**) (project settings), matches a deny rule
$ oklm permissions --project proj Bash "git push origin main"
ask: Bash(git push:*) (project settings), a command matches an ask rule
$ oklm permissions --project proj Bash "ls"
ask: default mode, not covered by a rule
$ oklm permissions --project proj --mode acceptEdits Edit "src/a.c"
allow: Edit(./src/**) (project settings), matches an allow rule
The audit log. Each decision is appended to
.oklm/audit/<session>.jsonl, one JSON object a line,
written before the tool runs: time, session,
tool, args, decision,
by (rule or mode),
source, rule, reason,
answer (the user's answer to an ask).
oklm audit lists them, filtered by --tool,
--decision and --session, from the project's
logs or the files given. After a scripted session (--mock
replays a fixture of tool calls instead of a model) in that project:
$ oklm audit
2026-11-07T23:44:50Z deny rule Bash {"command":"rm -rf build"} Bash(rm -rf:*)
2026-11-07T23:44:50Z allow rule Bash {"command":"make -v"} Bash(make:*)
$ oklm audit --decision deny
2026-11-07T23:44:50Z deny rule Bash {"command":"rm -rf build"} Bash(rm -rf:*)
Sessions. Each run is a session, its conversation
appended to .oklm/sessions/<id>.jsonl as it goes.
--resume resumes the latest, --resume ID a
given one (/resume lists them), --fork-session
copies it into a new one. oklm agent sessions lists them
without a model, sessions export ID writes one as Markdown
or JSON and sessions rm ID removes one. With
--kv-save and an in-process model, the model's cache is
kept beside the transcript (<id>.kv, the KV file
above), so a resumed session's first turn evaluates only what is new
instead of the whole conversation: a warm resume (agent-sessions.md).
With many tools (MCP servers above all), --tool-rag K
selects them by retrieval: the agent keeps the descriptions of its
tools, skills, plugin commands and sub-agent types in a retrieval index
(src/harness/tool_index.c over the RAG index of Retrieval, cached in
~/.local/state/oklm/rag/ and re-embedded only where a
description changed), grouped Type, Service, Tool. The prompt then
counts each MCP server's tools instead of naming them, and before each
message the K entries that fit it best (hybrid: bge-small-en-v1.5 on the
CPU and BM25, or BM25 alone without the model;
toolSearch.mode dense or bm25
ranks by one list) are looked up; the deferred tools among them come
with the message, as definitions in a reminder for the in-process model
or in the tools array for a server. ToolSearch's word queries use the
same index, and every decision is a line of the debug log (rag.md).
The agent has more: background jobs, sub-agents, skills, hooks, MCP
servers, plugins, a sandbox for commands. Sub-agents report to the agent
that started them (no channel between them): a commit or push asked
while background ones still run waits for their reports, so it is
decided with them read (agent-tools.md); each report ends
with the files the sub-agent wrote, from its edits; lines a sub-agent
wrote are kept from the others running beside it; and eval task 26
checks two run at once for undoing each other's work or a report
claiming what was not done. oklm agent config,
skills, mcp, plugin,
agents, attach, send and
stop manage them from the shell (extending.md, processes.md, commands.md).
Training
oklm train trains on the CPU, OpenCL or ROCm
(--backend auto|cpu|opencl|rocm); cpu gives
the same bits on every machine. Parameters, gradients and the AdamW
optimizer's two moments are held in F32 (four floats per parameter), and
oklm train prints what it will use before it starts.
Checkpoints are <prefix>-<step>.gguf models
(loadable by run) with a .train sidecar
holding the optimizer state and the run's options, for
--resume. The directories of --out are created
when missing (before any work, so a path that cannot be made fails at
once). See training.md.
train, rl, abliterate and
refusals take --progress-json FILE
(- for standard output, the report then on standard error):
the run as one JSON object a line, flushed as written — a
start line (the model, the data, the options in effect, the
device, the steps planned), a step line an update (loss,
validation loss when measured, learning rate, tokens a second, an
estimate of the time left), checkpoint and
sample lines, the layers and prompts of the two refusal
commands, and always a last end line (ok or
the error, the final figures, the files written). The fields of each
line are in training.md.
From scratch. With --vocab N a
tokenizer is first trained on the text; then a new model of the given
shape (--layers 4 --embd 128 --heads 4 --ctx 256 by
default) learns to predict the next token.
$ oklm train -f data/tinyshakespeare.txt --out nano/nano --vocab 512 --layers 2 --embd 64 --heads 2 --ctx 64 \
--batch 4 --steps 20 --eval-every 10 --backend cpu
tokenizer: 512 tokens trained on data/tinyshakespeare.txt
model: gpt2, 136960 parameters, 2 layers, 64 channels, 2 heads (2 key / value), MLP 256, context 64
data: 575809 tokens, 518229 for training, 57580 for validation; batches of 4 x 64, 1 per update; 20 steps; on cpu
...
final val loss 6.1651
saved nano/nano-000020.gguf
Twenty steps learn nothing useful; longer runs and their losses are in training.md. The architecture changes how a block is built:
| GPT-2 (default) | Llama (--arch llama) |
|
|---|---|---|
| Positions | learned position embeddings added to the tokens | rotary embeddings (RoPE) applied to queries and keys,
--rope-theta |
| Norm | LayerNorm | RMSNorm |
| MLP | 4 x width, GELU | gated (gate, up, down), about 8/3 x width (--ff) |
| Attention heads | as many key / value heads as query heads | fewer key / value heads allowed (--kv-heads,
grouped-query attention) |
--arch qwen2 adds biases to Q, K and V and
qwen3 norms on each head's Q and K, both on Llama
blocks.
Full fine-tuning (-m MODEL, a GGUF file
or a Hugging Face directory) updates every weight of an existing model,
starting from its tokenizer and shape. It needs the most memory: for
SmolLM2-135M, gradients 513 MiB and optimizer 1026 MiB beside the 513
MiB of weights.
LoRA. A low-rank adapter leaves the model frozen and learns a small correction to some of its matrices: a weight W (outputs x inputs) becomes
W' = W + (alpha / R) x B x A A: R x inputs, B: outputs x R
R, the rank, is much smaller than the matrix's sides, so B x A has the matrix's shape but only R x (inputs + outputs) free numbers. A starts random and B at zero, so a fresh adapter changes nothing. Only A and B get gradients and optimizer state:
$ oklm train -m SmolLM2-135M-Instruct-f16.gguf -f data/tinyshakespeare.txt --out lora/a --lora 8 --lora-targets all \
--batch 2 --seq 64 --steps 3 --eval-every 3 --lr 1e-3 --backend cpu
LoRA: rank 8, alpha 8, targets all; 2442240 trainable parameters, the model frozen
...
memory: gradients 9.3 MiB, optimizer 18.6 MiB (F32), activations kept for the backward pass 132.2 MiB (F32); checkpoints F32
saved lora/a-000003.gguf
2.4 M trainable parameters instead of 135 M; gradients and optimizer 28 MiB instead of 1.5 GiB; the adapter file 9.8 MB against 540 MB for a full F32 checkpoint (its sidecar 19.6 MB against 1.08 GB). The adapter is in llama.cpp's LoRA format.
- Targets:
--lora-targetslists the matrices: any ofq,k,v,o,gate,up,down(for GPT-2qkv,o,up,down), orall; the default is the attention's (q,k,v,o). - Rank: a larger rank can express a larger change and
costs proportionally more parameters, memory and file size. The
adapter's scale is alpha / R;
--lora-alphadefaults to the rank (scale 1). Start small (8 to 16) and raise it only when the training and validation losses stop improving while a larger change is needed; compare on the validation loss the run prints. - Use at runtime (
run,chat,ppl,serve --lora FILE [--lora-scale S]): the adapter is merged into a temporary copy of the model as it loads;serve --lora-moduleserves several beside one base. Merge (oklm lora-merge MODEL ADAPTER OUT [--scale S]) writes W + S x (alpha / R) x B x A into a new model file; F32, F16 and BF16 weights keep their type, quantized ones become F16 (quantize the merged file again if needed).
$ oklm run -m SmolLM2-135M-Instruct-f16.gguf --lora lora/a-000003.gguf -p "ROMEO:" -n 12 --temp 0 --backend cpu
lora: SmolLM2-135M-Instruct-f16.gguf + lora/a-000003.gguf (scale 1) -> /tmp/oklm-lora-2741031-0.gguf, 210 weights merged
...
$ oklm lora-merge SmolLM2-135M-Instruct-f16.gguf lora/a-000003.gguf merged.gguf
- Limits: a low-rank update changes how the model uses what it already knows (tone, format, following a kind of instruction, using a tool) well; it has few parameters to store new facts in, so teaching facts this way is unreliable. Give facts in the prompt (documents) instead.
Saving memory.
--recomputekeeps only each block's input and recomputes its activations in the backward pass: much less activation memory (longer sequences, larger batches) for about one more forward pass, the same gradients.--quantized-base(with--lora) keeps the frozen model's matrices in the file's types (Q8_0, Q4_K, ...) and unpacks each just before its product: SmolLM2-135M Q8_0 takes 147 MiB instead of 522 MiB of weights, with the same results.--act-type bf16stores the activations kept for the backward pass in BF16,--opt-type bf16the AdamW moments (with stochastic rounding) and--grad-type bf16the gradients (each block's computed in an F32 window, then added in with stochastic rounding), on the CPU or the GPU, every computation still in F32;--save-type bf16writes the checkpoints in BF16 (training.md).--matmul-precision bf16|f16(--backend rocm) runs the large products on the GPU's matrix units with 16-bit inputs; F16 needs--loss-scale dynamic, which computes the gradients of a scaled loss (so small ones do not vanish in F16) and divides them back in the update, skipping an update that overflowed (training.md). The weights' 16-bit copy is kept on the GPU between updates (2 bytes a parameter, when it fits): each weight is rounded once after the optimizer changes it, not at each of its products, with the same results.
Chat data. --chat-data conv.jsonl
(lines of {"text": ...}, conversations rendered through the
model's template, as oklm agent --save-training FILE writes
them) trains on the assistant turns only.
Reinforcement learning. oklm rl
fine-tunes from rewards instead of example answers: for each task it
samples a group of completions (--group 8,
--max-new 32 tokens at --temp 1), scores each
1 or 0, and pushes up the completions that scored above their group's
mean (group-relative advantages, GRPO). --updates N reuses
each batch for several clipped updates and --kl W (with
--ref MODEL, default the starting model) keeps the model
near the reference.
tasks.jsonl holds one
{"prompt": ..., "answer": ...} per line. The prompt is
tokenized as written, special tokens included, so it may carry the chat
template's markers. A completion earns 1 when the answer is a number and
the completion's last number equals it ("13 + 6 = 19." gives 19; 7 and
7.0 agree), or, for any other answer, when the completion starts with it
(leading spaces dropped, not followed by a letter or digit).
--val-frac (0.1) of the tasks are held out and their greedy
accuracy printed every --eval-every steps.
$ cat tasks.jsonl
{"prompt": "2+3=", "answer": 5}
{"prompt": "4+4=", "answer": 8}
...
{"prompt": "Capital of France:", "answer": "Paris"}
$ oklm rl -m SmolLM2-135M-Instruct-f16.gguf -f tasks.jsonl --out rl/rl --steps 2 --tasks 2 --group 4 --max-new 8 \
--eval-every 1 --val-frac 0.2 --backend cpu
model: llama, 134515008 parameters; 4 tasks for training, 1 held out; 2 tasks a step, groups of 4, 8 new tokens at temperature 1; on cpu
step 1/2: reward 0.000, solved 0.000, length 7.5, 0 tasks learned from, grad norm 0, 5.2 s
step 2/2: reward 0.125, solved 0.125, length 8.0, 1 tasks learned from, grad norm 7.77, 6.2 s
...
saved rl/rl-000001.gguf
A group whose completions all score the same teaches nothing (the first step above made no update, so the checkpoint counts one). Rewards only reinforce what the model already does sometimes (training.md).
Images, sounds and speech
oklm describe: a language model reads
text embeddings; an image or a sound has to become vectors of the same
kind. The projector file (--mmproj, llama.cpp's mmproj
GGUF) holds an encoder and a projector that do this: for SmolVLM, a
SigLIP vision encoder whose patch embeddings are pixel-shuffled and
projected (idefics3); for Gemma 3, the same encoder at 896 pixels, its
patches averaged in groups of 16 and projected to 256 embeddings, which
the model reads attending to one another in both directions; for LLaVA
1.5, a CLIP encoder whose 576 patch rows go through two linear layers;
for LLaVA 1.6, the same encoder on an overview and on two to four tiles
of the image, the tiles' rows put back side by side as one grid, the
part that covers the padding around the image cut away and each row of
the grid ended by an embedding the model learned for that; for Qwen2-VL
and Qwen2.5-VL, an encoder that reads the image at its own size (one
embedding per square of 28 pixels, up to 4 096), whose rows the language
model places by their row and column in the image: its rotary positions
have three parts (time, height, width), and the text after an image
continues from the largest one used; for Ultravox, a Whisper audio
encoder and a projector. Their output rows take the place of token
embeddings in the prompt, where the marker
<__media__> is (or before the prompt: the question
right after the picture, as the templates of SmolVLM, Gemma 3, Qwen2-VL
and Qwen2.5-VL write a picture and a text, or after a new line for
LLaVA, whose own conversations have one; a sound for Ultravox after the
question and a space, the order its own pipeline writes and the one it
answers about the sound in). A chat request to
oklm serve --mmproj holding a picture and a question is
written the same way, by the same function: the model reads the same
tokens. A video in a request (video_url, base64) is
decoded, sampled and laid out by the code oklm describe
uses, the frames it may decode bounded. The server keeps the embeddings
of recent media by a hash of their pixels or samples, and remembers
which bytes decoded to which of them: a chat window sends a
conversation's pictures and videos again with every message, and the
same bytes are then neither decoded nor encoded (server.md).
oklm chat --mmproj does the same for one conversation:
/attach FILE gives a picture, a sound or a video to the
next message, each turn's prompt is rows (tokens, and each item's
embeddings named by its content's hash) compared with the rows the
context holds, and an item is encoded once and its embeddings kept, so a
question after a picture costs its own tokens (SmolVLM-256M on the CPU:
14 rows evaluated of 375, the 320 of the picture among the 361 cached;
inference.md).
The projector file decides whether images or sounds are expected.
--tiles SIDE cuts large images into tiles plus an overview;
videos give frames (--fps 1, --frames 8) or,
for an audio model, their sound: Ogg Theora natively, other containers
through ffmpeg only with --ffmpeg or
OKLM_FFMPEG=1. Qwen2-VL and Qwen2.5-VL read the frames as
their own code reads a video: two by two through the two kernels of the
patch embedding, one grid of embeddings a pair, and in the language
model a time position that advances from a pair to the next (by one for
Qwen2-VL, by the seconds a pair covers times two for Qwen2.5-VL); the
other models get each frame as a still image with its time written
before it. With the language model on a device (--backend),
the vision encoder runs there too, from the image's patches to the
projector's rows (a prompt with a picture: 8.6 s to 0.35 s for Qwen2-VL
2B on Vulkan), and the audio encoder, from its first convolution to its
projector (a prompt with 11 seconds of speech: 5.3 s to 0.26 s for
Ultravox on Vulkan); the image is cut into patches and the sound's
spectrogram computed on the CPU.
$ oklm describe -m SmolVLM-256M-Instruct-Q8_0.gguf --mmproj mmproj-SmolVLM-256M-Instruct-f16.gguf \
--frames 2 -n 30 -p "What is shown?" tests/fixtures/mm/testsrc.ogv
[tests/fixtures/mm/testsrc.ogv: 2 frames, 0.0 s of sound]
A colorful image consisting of multiple vertical lines with varying shades of color.[640 media embeddings, prompt 713 tokens in 11.65 s; 14 tokens in 0.15 s]
oklm transcribe takes a Hugging Face
Whisper directory: config.json (a Whisper configuration;
the mel bin count is read from it), a single
model.safetensors and tokenizer.json. It
decodes greedily, a window of 30 seconds at a time, with Whisper's
timestamp tokens: the tokens between two timestamps are a segment with
its times (--timestamps,
--format srt|vtt|json), and the next window starts where
the last closed segment ended, so the cuts fall in the pauses and not
every 30 seconds. The language is the most probable language token after
the start token, found from the first window
(--language CODE gives it instead), and
--translate asks for English with the other task token (multimodal.md). The layer
counts and the special tokens' ids are read from the files too, and the
weights may be F32 or F16, so the large-v3 checkpoints load like the
small ones. Checked with Whisper base, small, large-v3-turbo and
large-v3 against transformers on the same files.
--backend opencl|vulkan runs the encoder (its two
convolutions and its blocks) and the decoder on a device (a 30-second
window of large-v3: 11.4 s on the CPU, 0.71 s on Vulkan; multimodal.md);
oklm speak --backend does the same for the language model
of a speech model and for its vocoder's network, whose convolutions are
products over views of the rows each position of a kernel reads and
whose group normalizations are a layer normalization between two copies
through a view that lays each group out as a row (speech.md). A server takes the same
devices for its image encoder (with its model), its recognizer
(--transcribe-backend) and its synthesizer
(--speak-backend): each computes in a thread of its own,
one pass on the device at a time, the passes in the order they asked for
the device. A decode step is a pass of a few milliseconds and an
encoder's is tenths of a second, so an encoder lets the passes that wait
in between two of its blocks, where its stream is whole in its own
buffer: it first waits for the blocks it queued, then hands the device
over. It does so on a device another thread uses: one whose pass waits,
or that said it keeps taking passes there, as the scheduler does for as
long as it has requests to decode, however slow a step. A model whose
layers are spread over two devices holds both for its forward pass,
always taken in the same order, so two such models cannot each hold the
device the other waits for, and lets the waiting passes in where its
stream goes from one device to the other. The server's cache, which
takes what the device has free, is sized after what the encoders and the
sound models will take there has been counted, each as it allocates,
with what the backend's own buffers are left. A request that is decoding
waits for one block of an image's or a window's encoder (0.013 to 0.08 s
on Vulkan where it waited 0.20 to 0.70 s; server.md).
$ oklm transcribe -m models/whisper-base models/samples/jfk.wav
[11.0 s of audio in 0.92 s]
And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.
A server does the same for its clients:
oklm serve --transcribe-model whisper-base answers
POST /v1/audio/transcriptions and
POST /v1/audio/translations (OpenAI's routes: a multipart
file, or the sound as the body; verbose_json
holds the language and the timed segments), and every server reads a
document sent to POST /api/ingest into text, as
oklm ingest does (server.md,
src/server/transcribe.c, src/server/upload.c).
A document too long for its client is cut, unless the server holds an
embedding model (--embed-model, or the one of its
--rag index) and the client names a conversation: the text
is then cut into passages, each embedded and kept in memory under that
name, and POST /api/ingest/search gives the passages
nearest a question (or, for a question about the whole document such as
"summarize this", passages at equal steps over it:
oklm_ingest_whole_question(), which the chat window uses
the same way), which the dashboard's chat puts before each message
(src/server/docs.c: bounded in conversations, documents and
bytes, dropped after an hour without a question or when the client says
so; server.md).
The server checks the sound models' files when it starts and loads their
weights at the first request of each route (--preload: at
start; /api/stats says loaded).
The other way too:
oklm serve --speak-model outetts-0.2-500m answers
POST /v1/audio/speech (OpenAI's route) with the synthesis
described next. Each request's synthesis runs in a thread of its own,
one at a time, and hands the sound over pass by pass; the HTTP thread
sends each pass as a chunk (pcm, a WAV file whose header
leaves its lengths open, or the FLAC frames the pass completes) or the
whole file at the end, and stops the synthesis when its client is gone.
A request names its voice among the speaker files the server offers
(GET /v1/audio/voices) and its speed (server.md,
src/server/speak.c).
oklm speak goes the other way, with two
models (speech.md): OuteTTS 0.2 or 0.3, a
qwen2 language model that continues a prompt of words and
audio codes (a voice's, then the text's words) with the codes of the
text, 75 a second, and WavTokenizer's decoder, a convolutional network
with one attention block that turns the codes into a spectrum, which an
inverse short-time Fourier transform makes a waveform at 24 kHz. A text
is spoken in passes, a sentence each (a long one in several), each
handed over when its last word is written, with a pause after a
sentence's end. The voice's codes come after the text in the prompt, so
a language model's cache cannot keep them from one text to the next:
what is saved is prompts, the passes that hold together being written
under one, the vocoder reading each pass with a second of the codes
before it; a first pass cut short gives an earlier first sound for no
prompt more. A speed is a change made on the samples that keeps the
pitch (overlap-add of frames placed where the waveforms agree), and a
FLAC file is written by OKLM's own encoder (fixed predictors and Rice
codes). On the CPU, or with the language model and the vocoder's network
on a device (--backend: a sentence of 7.0 s in 1.48 s on
Vulkan, the sound within one step of the CPU vocoder's 16-bit samples);
on the CPU the vocoder's output is within 131 dB of a double-precision
implementation's, and the prompt's tokens are llama.cpp's.
Soprano (speech.md, the catalogue's
soprano-1.1-80m) is the model that speaks faster than it is
heard on a CPU. It has no voice in its prompt and no audio codes: the
text is cleaned as the model's own package cleans it (numbers, dates,
amounts and abbreviations in words, src/mm/soprano_text.c),
each sentence is a prompt of three tokens more than its text, an 80
M-parameter qwen3 model writes tokens until
[STOP], and what goes to the decoder is the final hidden
state that predicted each token
(oklm_context_params.hidden), 16 a second. The decoder is
the ConvNeXt half of the vocoder above without its attention: the states
interpolated to four frames each, eight blocks, a spectrum, and the
inverse transform at 32 kHz. Since nothing in it reaches further than
four states, the first sound of a sentence is handed over while the rest
is written, and the pieces are the samples of one call. The tokens
written are llama.cpp's, the decoder's output within 107 dB of a
double-precision implementation's.
$ oklm speak -m outetts-0.2-500m -o cat.wav "The catalogue name finds both files."
[2.6 s of speech after 6.0 s]
cat.wav: 2.60 s of speech in 6.00 s (0.43 s of sound per second)
Audio formats: PCM WAV (8, 16, 24, 32-bit integer and 32-bit float) always; FLAC, Ogg Vorbis, Opus and MP3 when OKLM is built with Allegro 5. Images: PNM always; PNG, JPEG, BMP, TGA, PCX with Allegro 5. Sounds are mixed to mono and resampled to 16 kHz. See multimodal.md.
Refusal analysis
Chat models are tuned to decline some requests. Research on these models (Arditi et al., 2024) found that this behaviour is largely carried by one direction in the model's internal state.
This weakens the model's safety behaviour.
oklm abliterate exists for research and interpretability:
measuring where refusal lives in a model, and preparing a base whose
behaviour is then retrained deliberately. Do not deploy an abliterated
model in place of the original.
oklm refusalsanswers each prompt of a file greedily (-n 48tokens) through the chat template and counts the answers that read as refusals (the usual markers). With--ref MODEL --harmless FILEit also measures the mean KL divergence of the first token's distribution from a reference model on prompts the model should answer: how much a change moved the model's ordinary behaviour.oklm abliterateruns two prompt sets through the model, one it declines (--harmful) and one it answers (--harmless), and reads the residual stream (the running state between blocks) at each prompt's last token, entering every layer. Per layer, the difference of the two sets' mean states, normalized, is the candidate refusal direction; the layer that separates the two sets best within a band of the depth (0.3 to 0.7,--band) is chosen, or--layer N. With--out, every matrix that writes into the residual stream (each block's attention output and FFN down projection, and the token embeddings when the head is not tied to them) is made orthogonal to that direction, W' = W - s r (r^T W), so the model can no longer write along it.--scale s(1) weakens the removal below 1;--save-dirs/--load-dirskeep the directions to write other layers or scales without recomputing.
$ oklm abliterate -m SmolLM2-135M-Instruct-f16.gguf --harmful h32.txt --harmless l32.txt --out abl.gguf
abliterate: 32 + 32 prompts read in 2 s
abliterate: layer 11's direction removed (scale 1), abl.gguf written in 0 s
layer separation gap
...
11 4.368 0.093 <- chosen
...
$ oklm refusals -m abl.gguf --ref SmolLM2-135M-Instruct-f16.gguf --harmless answered8.txt --backend cpu
first-token KL divergence from the reference: 0.1127 (mean of 8 answered prompts)
(The prompt files are 32 lines of models/abliterate/'s
training sets, fetched by tools/fetch_abliterate_data.sh.)
SmolLM2-135M refused none of the 8 held-out harmful prompts tried here
even before; the measured effect on a model that does refuse (Qwen3-8B:
97.5 % to 7.5 %) and its cost are in training.md.
Developer tooling
| Command | What it does | When |
|---|---|---|
make |
library, apps, tests | always |
make test |
every test binary; skips what needs files not downloaded | every change |
make asan-test |
the tests under AddressSanitizer and UBSan (GPU drivers left out by the build itself) | before merging; memory errors and undefined behaviour |
make tsan-test |
the tests under ThreadSanitizer (GPU drivers left out) | before merging; data races |
make check |
cppcheck over src, apps,
tests, failing on findings |
before merging |
make CC=clang BUILD=build/clang test |
the tests built with clang | before merging |
make fuzz, make fuzz-run FUZZ_TIME=60 |
libFuzzer targets (GGUF, safetensors, regex, tokenizer, Jinja,
grammar, PDF, tool calls, shell splitting), each run
FUZZ_TIME seconds |
after touching a parser |
make test-public |
downloads a pinned corpus of public models, builds a Python venv, and checks OKLM's files, dequantization, tokenization, templates and outputs against gguf-py, Hugging Face, jinja2 and llama.cpp's files | after touching formats, kernels or the forward pass |
make bench |
matrix product micro-benchmark | kernel work |
sh tools/bench_vs_llama.sh |
prompt and decode t/s against llama.cpp's llama-bench
on the same files, the CPU at 1 to 16 threads and the GPU (benchmarks.md) |
speed work |
sh tools/gpu/report.sh |
on a GPU OKLM was never tested on: prerequisites, system and
drivers, build, the OpenCL and Vulkan device tests on each device, a
speed table with --models; one report file (testing.md) |
a new GPU or driver |
make doc |
Doxygen HTML in build/doc/html, warnings in
build/doc/doxygen_warnings.log |
after changing headers |
make docs-html |
README.md and docs/ converted by pandoc to
an HTML set in build/doc/docs-html (open
index.html): page list, table of contents, offline search;
fails on a dead link or a missing anchor |
to read or hand over the docs outside a Markdown viewer |
make deb, make appimage,
make installer-windows, make docker-build |
the packages in dist/: a .deb, an
AppImage, a Windows installer, and the first two built in a Debian
container (build.md) |
a release, or a machine without the build tools |
make docs-lint |
trailing spaces, tabs, a Status: line on every page,
relative links and their anchors, the README's quick start, every
environment variable documented; and, read from the sources without
running anything, every option of the usage texts, every route of the
server and of a listening agent, every slash command and every
oklm-gui tool documented, the synopses of the command
reference (cli.md) those of the usage text |
after changing docs, an option, a route or a command |
make eval |
the agent's evaluation tasks with EVAL_MODEL |
agent changes |
make rag-eval |
retrieval on SciFact and the docs set, dense / sparse / hybrid with bge-small-en-v1.5: the published retrieval table (rag.md) | embedding, chunking, index or search changes |
make tool-eval |
tool selection by retrieval: accuracy and prompt tokens over
tests/rag/tool_queries.jsonl |
tool index or ToolSearch changes |
make gui-test, make wine-test,
make asan-opencl |
the GUI under Xvfb, the Windows build under Wine, the OpenCL tests under ASan | the matching area |
sh tools/check_gui_wine.sh |
the Windows build's graphical tools under Wine and Xvfb, the flows of each, driven by the tools' own presses, and a server kept across logins (58 checks with the canary models; testing.md) | a tool's Windows paths, the registry, the listener |
A change is mergeable when the tests, both sanitizers, cppcheck and the clang build are clean (testing.md, build.md).
Files, caches and environment
What OKLM writes and where:
| Path | What |
|---|---|
~/.cache/oklm/tuning/
($XDG_CACHE_HOME) |
oklm tune results; safe to delete |
~/.cache/oklm/usage/<model hash>.json |
MoE routing counts of the expert cache and the device tier |
.oklm/chat/last.json, last.kv |
oklm chat --resume / --kv-save |
.oklm/sessions/<id>.jsonl,
<id>.kv |
agent sessions; removed after cleanupPeriodDays
(30) |
.oklm/audit/<session>.jsonl |
the agent's permission decisions |
.oklm/jobs/ |
the agent's background jobs |
.oklm/settings.json,
.oklm/settings.local.json,
~/.config/oklm/settings.json,
/etc/oklm/managed-settings.json |
agent settings and permission rules |
~/.local/state/oklm/logs/ |
the agent's debug log (--debug) |
~/.local/state/oklm/rag/tools-<hash>.gguf |
the agent's tool index (--tool-rag), one a project;
safe to delete |
~/.local/state/oklm/rag/project-<hash>.gguf,
memory-<hash>.gguf |
the agent's index of the project's documentation
(--rag-project) and of its earlier sessions
(--rag-memory), one a project; rebuilt when deleted |
~/.local/share/oklm/models/ |
one place the agent looks for its default model |
The environment variables met most often (every one, with its default and where it is read, is in build.md):
| Variable | Effect |
|---|---|
OKLM_THREADS |
CPU threads (default: physical cores) |
OKLM_BACKEND |
auto, cpu, opencl or
vulkan when --backend is not given |
OKLM_BACKEND_ORDER |
the backends auto tries first, in order
(vulkan,opencl,cpu) |
OKLM_VK_DEVICES |
which Vulkan device(s) the vulkan backend uses (by
default every discrete GPU, integrated ones only without a discrete
one); all: every GPU; none: no Vulkan
device |
OKLM_CL_DEVICES |
which OpenCL device(s) to use (by default every discrete GPU,
integrated ones only without a discrete one); none: the CPU
alone |
OKLM_GPU_MEM_MB |
the device memory to assume when fitting layers |
OKLM_STATE_KEEP |
states a context of a model with recurrent blocks keeps for cut sequences (default 8; 0: none) |
OKLM_GPU_MAX_ALLOC_MB |
the largest device allocation to assume (weights and caches split into allocations of that size) |
OKLM_LOG_LEVEL, OKLM_LOG_FILE |
log verbosity and destination |
OKLM_MODELS |
directories where the agent looks for models |
OKLM_CL_WMMA |
1, f16 or bf16: every OpenCL
prompt product on the RDNA3 float matrix units; unset: those of the
quantized types without an 8-bit kernel; 0: none |
OKLM_CL_Q8, OKLM_CL_Q8_MV |
0: quantized prompt products, or the K-quants'
one-column decode products, on the float kernels instead of the 8-bit
ones (default on) |
OKLM_CL_FUSE, OKLM_CL_MVG,
OKLM_CL_RES, OKLM_CL_ROPE_KV,
OKLM_CL_ROUTE_SIGMOID, OKLM_CL_ATT_MLA |
0: no fused OpenCL launches (routing, query / key norms
with rotation, a decode norm folded into its product;
OKLM_CL_MVG=0 only a MoE layer's gate, up and SwiGLU,
OKLM_CL_RES=0 only the end of a residual branch with the
norm after it, OKLM_CL_ROPE_KV=0 only the rotation with the
cache writes, OKLM_CL_ROUTE_SIGMOID=0 only a sigmoid
router's routing), or latent attention's prompts on the per-token kernel
(default on) |
OKLM_CPU_PACK_EXPERTS |
0 / 1: never / always keep repacked copies
of a MoE model's routed experts (default: when they fit) |
OKLM_GELU_F16 |
1: the tanh GELU rounded as ggml's half-precision table
rounds it (CPU), for comparisons with llama.cpp on the Gemma family
(default off: single precision) |
OKLM_ATTN_F16 |
1: attention over an F16 cache with the queries and the
probabilities rounded to half precision, as ggml's products round them
(CPU), for comparisons with llama.cpp (default off) |
OKLM_Q8_ROUND_EVEN |
1: Q8_0 and Q8_1 activations rounded as ggml's x86 code
rounds an exact half (to the even step), for comparisons with llama.cpp
(default off: away from zero, as ggml's portable code) |
OKLM_DUMP, OKLM_DUMP_FEED,
OKLM_DUMP_FEED_OPS |
the steps of a forward pass written to a file, and another program's taken in their place, for a comparison step by step with llama.cpp (testing.md) |
OKLM_MATVEC_THREADS |
threads at most of a one-row (decode) product; fewer readers get more from the memory |
OKLM_CPU_PACK_HUGE |
0: the repacked copies as one allocation each instead
of chunks on huge pages (Linux, default on) |
Architectures. oklm info gives the
verdict for a file. Inference covers llama, mistral, mistral3, qwen2,
qwen3, gemma, gemma2, gemma3, gemma4, phi2, phi3, granite, granitemoe,
qwen2moe, qwen3moe, deepseek, deepseek2 (V2 / V3), glm4, glm4moe, olmoe,
gpt2, starcoder2, lfm2 and lfm2moe (Liquid's LFM2: short convolution
blocks between attention blocks, whose per-sequence state, the last two
gated inputs of each block, is rebuilt from the last tokens when a
sequence is truncated, copied or read back), gpt-oss (attention sinks,
experts with biases; on the CPU, OpenCL and Vulkan,
src/model/arch_gptoss.c), spark2_5 and muse-glimmer (the
attention output times a sigmoid gate computed from the block's input, a
sliding window on three blocks out of four, the rotation set per kind of
block; on the CPU, OpenCL and Vulkan,
src/model/arch_gate.c), gemma4 (Gemma 4 12B and 26B-A4B:
window blocks and blocks without window with heads of two sizes, each
layer's cache in its block's shape, values taken from the key projection
where a block has no attn_v, a scale on each block's
output, routed experts beside the dense feed-forward; in the E2B and E4B
an embedding per layer added at each block's end and last blocks that
keep no cache and attend over an earlier block's; on the CPU and on both
GPU backends, a block and the cache it reads in one memory,
src/model/arch_gemma4.c), mimo2 (MiMo-V2: values with
narrower heads than the queries and keys, kept in cells of the keys'
size with zeros after them so the attention op and the cache stay as
they are; the window and the count of key heads set block by block; on
the CPU, checked on a tiny model only,
src/model/arch_mimo2.c), and the hybrid qwen3next stack
(checked on a synthetic model only), whose Gated DeltaNet blocks carry a
fixed-size state per sequence instead of a KV cache: a short conv with
its ring and the gated delta recurrence, two ops that run on the CPU or
the GPU like the others, the states of GPU layers kept in GPU memory;
training covers GPT-2, Llama, Qwen2 and Qwen3 blocks and that hybrid
stack, its Gated DeltaNet blocks on the GPU as well. qwen35
(Qwen3.5-9B) is the published form of such a stack, with a decay per
token and gated queries; it runs on the CPU, on OpenCL and on Vulkan
(the recurrence's kernel takes both forms) and is checked against
llama.cpp on the published file on each, as is qwen35moe
(Qwen3.6-35B-A3B), the same stack with routed experts and a shared one
in place of the dense feed-forward (hybrid-layers-design.md).
On a few steps of a sentence it repeats, the 9B file's logits move by 10
and more when only the last places of a run change, in llama.cpp against
itself as in OKLM against it: the model's own sensitivity, measured
block by block in hybrid-layers-design.md.
A sequence's recurrent state only moves forward, so the context keeps a
few states (eight tokens before the end of each prompt, every 32 tokens
of a growing sequence, before each draft) and a journal of what the
recurrences read at the last positions: cutting a hybrid sequence back
(a rejected draft, a turn asked again, a request sharing a system
prompt) starts from the nearest kept state and costs the tokens after
it, not the prefix (hybrid-layers-design.md);
with the blocks on a device the states are kept in host memory as well
and the journal is not written, a cut being a forward pass from the kept
state. The Ternary Bonsai 2 files are that layout with ternary weights
stored rotated: their prism.hadamard.* metadata lists the
matrices that were multiplied by signs and a block Walsh-Hadamard matrix
before being rounded, and each product with one of them reads its input
through the same signs and transform (src/model/fold.c; on
OpenCL and Vulkan one device op, oklm_hadamard_fold(), with
the signs copied to the device); two of their three tensor types, PQ2_0
and PTQ1_0, are a fork's and are read like ggml's, on the CPU and in the
devices' products (hybrid-layers-design.md).
Tokenizers, templates and tensor types per family are in compatibility.md.
Limits worth knowing
oklm planreads GGUF files only; convert a checkpoint first. A checkpoint's headers give every shape, but loading it builds tensors the file does not hold as such (per-expert matrices stacked, FP8 matrices widened, GPT-2's transposed), which reads their data; the plan would need a sizing path of its own for those, so a directory is refused instead.- A Hugging Face checkpoint loads directly only with F32, F16, BF16 or
FP8 matrices; other stored types go through
oklm convert. oklm runcompletes the raw prompt; usechat(or the server) for a chat model's template. With--docon a chat model, a raw prompt may get an immediate end of text.- The expert cache works on the CPU;
--n-cpu-moeand--expert-gpuneed a device backend. - The server serves one model (and its adapters), no embeddings, and
nmust be 1. The Windows build has no TLS. oklm transcribereads a singlemodel.safetensors, not sharded Whisper checkpoints.- Against llama.cpp on the same machine the CPU is ahead on prompts and level or ahead on decoding from four threads; on the GPU some rows stay behind (benchmarks.md).