OKLM
A self-contained LLMOps framework: a dependency-free LLM runtime and training stack in C17.
One source tree covers the whole life of a model: loading and converting its weights, tokenizing, quantizing, running it on the CPU, Vulkan or OpenCL, training and fine-tuning it, serving it over an OpenAI-compatible API, and driving it as a coding agent. No external library touches the model; the optional ones only serve the edges (TLS, the GUI window, video decoding).
Written from scratch: tensor engine, weight files, tokenizer, transformer inference and training, quantization, Vulkan and OpenCL acceleration, multimodal encoders, an agent harness, an HTTPS API with a web dashboard, and a native Allegro 5 client.
No external library touches the model. There is no ggml, llama.cpp, BLAS, sentencepiece, tiktoken, Hugging Face tokenizers or ONNX in OKLM, not even as a linked reference. Where an international format exists (GGUF, safetensors, the ggml block quants, byte-level BPE, SentencePiece-BPE), OKLM implements it itself, bit for bit, so it loads the open-weight models people already have.
The base library is nilorea-library, used as a source tree: OKLM compiles the nilorea modules it needs (logging, lists, timing, the worker team) straight from it.
Status
| Area | State |
|---|---|
| Build, tests, sanitizers, static analysis, Doxygen | working |
| Element types (ggml numbering), f16/bf16 conversion | working, bit-exact with F16C |
| Legacy quants Q4_0 Q4_1 Q5_0 Q5_1 Q8_0 Q8_1 | quantize + dequantize + dot, bit-identical to ggml's reference |
| K-quants Q2_K Q3_K Q4_K Q5_K Q6_K, Q8_K | quantize + dequantize + dot, quantization bit-identical to ggml's reference |
| IQ4_NL IQ4_XS, MXFP4 NVFP4, TQ1_0 TQ2_0, Q1_0 Q2_0 | quantize + dequantize + dot (CPU and OpenCL), bit-identical to ggml's |
| PQ2_0 PTQ1_0 (PrismML's llama.cpp fork: Ternary Bonsai 2) | quantize + dequantize + dot (CPU, OpenCL and Vulkan), bit-identical to the fork's |
| IQ1_S IQ1_M, IQ2_XXS IQ2_XS IQ2_S, IQ3_XXS IQ3_S | dequantize + dot (CPU and OpenCL), bit-identical to ggml's (its codebook tables, see NOTICE); written by an own codebook search (lower perplexity than llama-quantize's) |
| Tensors, views, arena allocator | working |
| CPU ops (matmul, norms, softmax, RoPE/YaRN, attention/GQA, ...) | working, scalar + AVX2 + AVX-512 (SIMD bit-identical to scalar); prompts ahead of llama.cpp's CPU build (1.02 to 1.94 times its rate), decoding level or ahead from four threads (1.00 to 1.09) |
| GGUF read/write, sharded models, safetensors read | working; published GGUFs rewrite byte for byte, every tensor dequantizes bit for bit like gguf-py |
oklm convert / oklm quantize /
oklm imatrix (Llama, Mistral, Qwen2/3, GPT-2) |
working; oklm quantize takes llama-quantize's types
(Q4_K_M, IQ4_XS, Q8_0, ...) and importance matrices and writes the same
files byte for byte; oklm imatrix computes importance
matrices |
| Tokenizer: byte-level BPE, SentencePiece BPE, training | working; ids identical to Hugging Face on 10 000 entries for GPT-2, Llama 2, Llama 3.2, Qwen2.5, SmolLM2 |
Inference: Llama family (llama, mistral, qwen2, qwen3), Gemma 1 / 2
/ 3, Phi-2 / Phi-3, Granite (dense and MoE) and GPT-2, sampling,
oklm run / chat / ppl, from GGUF
or straight from Hugging Face safetensors |
working on SmolLM2, Qwen2.5, Qwen3, Llama 3.2, TinyLlama, GPT-2; same greedy output as llama.cpp (identical with F16/BF16 weights, up to near ties with quantized ones) |
| Chat templates: Jinja interpreter | renders the published templates of 11 families byte for byte like jinja2 |
| OpenCL backend (runtime-loaded, no build dependency) | working: every op on the GPU, kernels checked against the CPU,
layers split with --gpu-layers; generation 4-13x and
prompts 14-51x faster than the CPU on the corpus |
Agent harness: oklm agent, permission rules and modes,
audit log, file and command tools, task lists, skills, background jobs,
sub-agents, Linux sandbox |
working; Qwen3-4B edits, builds and runs code within the rules, commands confined to the project |
Training: GPT-2, Llama and Qwen2 / Qwen3 blocks from scratch or
fine-tuned, AdamW, gradient accumulation, dropout, checkpoints,
oklm train; reinforcement learning from rewards (GRPO,
several clipped updates per batch, a KL penalty),
oklm rl |
working on the CPU and OpenCL (17 times faster on a gfx1100); GPT-2 124M logits, gradients and losses match llm.c's PyTorch reference and its published Tiny Shakespeare run; SmolLM2-135M, Qwen2.5-0.5B and Qwen3-0.6B fine-tune into files llama.cpp continues identically; RL takes SmolLM2-135M from 2.5 % to 92.5 % on held-out sums; same bits for any thread count and SIMD level |
HTTP API: oklm serve, OpenAI-compatible chat and
completions, an Anthropic-compatible /v1/messages, streams,
tool calls, JSON schemas, a dashboard, API keys and TLS |
working; OpenAI's Python client runs against it unchanged; concurrent requests batch on one context with the same answers |
Native window: oklm-gui (Allegro 5), one program for
the graphical tools (--tool NAME): a launcher, a model
manager, a server manager, a conversion and quantization tool, a model
inspector, a training window (a form for each kind of run, the run
started in the background and followed: loss curve, checkpoints,
samples), a browser of the agent's sessions, a retrieval manager, and
the chat window over the HTTP API, on a model of its own or a server at
a URL |
working on Linux and Windows: streamed Markdown replies, server
figures, settings, a model manager (download, use, delete), the context
resized while running, a Connect window (servers on this machine or by
address, agents started with --listen, sessions read); the
retrieval manager (an index built or updated in the background with its
progress, its stale sources, search with the modes and the reranker, the
evaluation's table, a chat window on the index) run on Linux, compiled
for Windows |
Multimodal: oklm describe (images, sounds, videos),
oklm transcribe (Whisper), oklm ingest /
--doc (text, JSON, HTML, DOCX, PDF) |
working: SmolVLM's vision embeddings within 1e-3 of llama.cpp's, Gemma 3 4B's, LLaVA 1.5's and 1.6's, Qwen2-VL's and Qwen2.5-VL's within 2e-5 of a float64 pass of their encoders, Ultravox hears speech, Whisper base transcribes exactly; Ogg Theora natively, other videos through ffmpeg on request (docs/multimodal.md) |
Speech synthesis: oklm speak (OuteTTS 0.2 or 0.3 with
WavTokenizer's decoder, or Soprano), a 24 kHz WAV or FLAC file from a
text (32 kHz with Soprano) |
working on the CPU, or with the language model and the vocoder's
network on OpenCL or Vulkan (--backend), in English with
one built-in voice and speaker files: the prompt's tokens are
llama.cpp's, the vocoder within 131 dB of a double-precision
implementation; a speed that keeps the pitch, a shorter first pass for
an earlier first sound, FLAC written by OKLM's own encoder. Soprano
(soprano-1.1-80m, Apache-2.0, one fixed voice) speaks
faster than it is heard on a CPU: the tokens it writes are llama.cpp's
(docs/speech.md) |
Retrieval: oklm embed,
oklm rag index / search / info / eval,
oklm rerank, --rag on run / chat / serve /
agent, /v1/embeddings, /v1/rerank, the agent's
Retrieve and --tool-rag |
working: embedding models (bge, nomic, Qwen3-Embedding) within 1e-5
of llama.cpp; chunkers over ingest, one GGUF index updated
by file hash, exact dense search, BM25 and fusion (SciFact nDCG@10 71.27
dense, MTEB's figure, 73.04 hybrid; make rag-eval
reproduces the table); cross-encoder reranking (off: it lowers SciFact);
answers citing their sources with corrective retrieval (Qwen3-8B: 92 %
of 38 doc questions right, 16 % without); follow-ups rewritten, routing,
graph retrieval (two-article questions 23 of 30 against 8),
self-critique, figures captioned; the agent's Retrieve over the project
and its sessions (doc tasks 8 of 8 against 2), tools picked by retrieval
(top-1 61.8 % against 27.0 %) (docs/rag.md) |
Model variants
| Name | Parameters | Purpose |
|---|---|---|
| OKLM-Nano | 1-15 M | development, unit tests, CI; trains in minutes on a CPU |
| OKLM-Micro | 30-60 M | TinyStories-class quality; hours on CPU, minutes on OpenCL |
| OKLM-Small | about 125 M | GPT-2-small shaped, the first "real" text model |
| OKLM-Base | 350 M - 1 B | imported from GGUF/safetensors first, fine-tuned later |
Modality suffixes: -V (vision projector),
-A (audio), -AV (both), for example
OKLM-Micro-V.
Quick start
# build and check (nilorea-library next to the OKLM checkout's parent, or NILOREA_DIR=...)
make # NILOREA_DIR=../../nilorea-library by default
make test # every test binary
make asan-test # under AddressSanitizer + UBSan
make tsan-test # under ThreadSanitizer
make bench # matmul micro-benchmark
make rag-eval # retrieval on SciFact and the docs set: the published table (docs/rag.md)
make check # cppcheck
make doc # Doxygen HTML in build/doc/html
make docs-html # README and docs/ as a browsable HTML set in build/doc/docs-html (pandoc)
make deb appimage # dist/oklm_VERSION_amd64.deb and dist/oklm-VERSION-x86_64.AppImage
make docker-build # both built in a Debian container (the release build)
make windows-deps installer-windows # dist/oklm-VERSION-setup.exe, cross-built with MinGW-w64 and NSIS
make test-public # download the corpus; check against gguf-py, Hugging Face, jinja2, llama.cpp
make fuzz-run FUZZ_TIME=60 # fuzz the GGUF and safetensors parsers
sh tools/gpu/report.sh --models # on another GPU: device tests and speeds, one report file (docs/testing.md)
# the machine and the models
build/release/bin/oklm version # the library version
build/release/bin/oklm help # every command and option
build/release/bin/oklm devices # backends, CPU features, threads
build/release/bin/oklm types # element types and what OKLM can do with them
build/release/bin/oklm info model.gguf # metadata, tensors, compatibility verdict (-t: every tensor)
build/release/bin/oklm info --json model.gguf # the same as one JSON object, with the layout: each block's tensors by role
build/release/bin/oklm plan -m model.gguf -c 32768 # memory it would take, from the headers, against the RAM
build/release/bin/oklm plan -m model.gguf --json # the same as JSON, with each layer's placement (CPU or device) and bytes
build/release/bin/oklm models # the models offered for download (none is shipped), by category
build/release/bin/oklm models pull qwen3-8b # download one (resumed, SHA-256 checked); then: oklm chat -m qwen3-8b
build/release/bin/oklm-gui # the chat window, nothing loaded: it lists the servers running on this machine and the installed models, and talks to the one chosen
build/release/bin/oklm-gui -m smollm2-135m # the chat window on an installed model: Models..., the Context slider, Connect...; /help in the message lists its commands
build/release/bin/oklm-gui --url http://127.0.0.1:7430 # the same window on a server (oklm serve)
build/release/bin/oklm-gui --tool servers # the server manager: start a server in the background, list, stop, restart, follow, resize
build/release/bin/oklm-gui --tool servers -m smollm2-135m --backend cpu -c 2048 # the same, its form filled; --profile NAME loads a kept one
build/release/bin/oklm-gui --tool servers -m qwen3-8b --options "--gpu-layers 20 -ctk q8_0 -ctv q8_0 --temp 0.6" --advanced sampling # options of oklm serve read into the form's offload row and its Advanced window (--save-profile NAME keeps the form)
build/release/bin/oklm-gui -m qwen3-8b --model-settings # the chat window on its model settings: backend, layers on the device, context, cache, sampling (--settings 'OPTIONS' applies some unattended)
build/release/bin/oklm-gui --tool chat --agent 7481 # the chat window attached to the agent listening on that port
build/release/bin/oklm-gui --tool models # the model manager: download, list and delete the catalogue's models; Chat, Serve..., Inspect on one
build/release/bin/oklm-gui --tool launcher # the home window: a button for each graphical tool, and the documentation
build/release/bin/oklm-gui --tool list # the graphical tools' names (chat, the default, models, servers, convert, launcher, inspect, train, sessions, rag)
build/release/bin/oklm-gui --tool inspect -m model.gguf # the model inspector: the blocks by role and size, the placement and its memory
build/release/bin/oklm-gui --tool inspect --url http://127.0.0.1:7430 # the same on a running server, with its live figures
build/release/bin/oklm-gui --tool inspect -m model.gguf --backend cpu --prompt "The capital of France is" --layer 5 # a prompt run on the CPU: what each block wrote, block 5's attention, the experts picked
build/release/bin/oklm-gui --tool inspect -m model.gguf --prompt "The capital of France is" --step 0 --view lens # what the output head would say after each block (--view effects: each block's write taken out in turn; --chat: the prompt through the chat template)
build/release/bin/oklm-gui --tool convert -m model.gguf --type Q4_K_M # convert or requantize in a window: the size before, the log, the perplexity after
build/release/bin/oklm-gui --tool train # the training window: from scratch, fine-tune, LoRA, chat data, RL; the loss as a curve
build/release/bin/oklm-gui --tool sessions --project . # the agent sessions of a folder: list, search, read, export, resume, delete
build/release/bin/oklm-gui --tool rag # the retrieval manager: build or update an index, see what is stale, search, evaluate, attach
build/release/bin/oklm-gui -m qwen3-8b --rag docs.gguf # the chat window answering each message from a retrieval index
build/release/bin/oklm-gui -m smolvlm-256m --attach photo.png # a picture with the first message (also dropped on the window, or Attach)
build/release/bin/oklm-gui -m qwen2-vl-2b --attach clip.ogv # a video with the first message, sampled by the window's server; --ffmpeg for an MP4 or another container
build/release/bin/oklm-gui -m qwen3-8b --attach report.pdf # a PDF, DOCX, HTML, JSON or text file read by the window, its text sent under its name
build/release/bin/oklm-gui -m qwen3-8b --attach book.pdf --rerank-model ms-marco-minilm-l6-v2 --no-doc-cache # its passages put in order by a reranker (an installed one is used by itself; --no-rerank: none); --no-doc-cache: its index a temporary file, not kept in ~/.cache/oklm/docs for the next window
build/release/bin/oklm-gui -m qwen3-8b --attach book.pdf --embed-model bge-small-en-v1.5 # a document longer than the context is indexed by the window and searched for each message, not cut
build/release/bin/oklm-gui -m model.gguf --mmproj mmproj.gguf # the projector of a model that is not in the catalogue
build/release/bin/oklm-gui -m qwen3-8b --transcribe-model whisper-base # Record and /listen: the microphone transcribed into the message (the smallest installed Whisper without the option)
build/release/bin/oklm-gui -m qwen3-8b --transcribe-language fr --listen-silence 2 --listen-send # spoken French; a recording ends after 2 s without a word and is sent
build/release/bin/oklm-gui -m qwen3-8b --transcribe-language fr --transcribe-translate # spoken French written in English (Whisper's translation task; the Speech window has the box)
build/release/bin/oklm-gui -m qwen3-8b --transcribe-attached --attach talk.ogg # an attached recording the model cannot hear, transcribed without a press on Transcribe
build/release/bin/oklm-gui -m qwen3-8b --mic alsa_input.usb-headset.mono-fallback --transcribe-language auto # another microphone, the language found by Whisper; the Speech window (Speech...) has both, the pause and the level
build/release/bin/oklm-gui -m qwen3-8b --no-transcribe # no Whisper model beside the chat model (253 MiB less with whisper-base)
build/release/bin/oklm-gui -m qwen3-8b --no-partial # no words shown while a recording is still coming (they take the recognizer's time while you speak)
build/release/bin/oklm-gui -m qwen3-8b --mic-file speech.wav --send "/listen" --screenshot out.png # a test aid: a sound file recorded as if it were the microphone
python3 -I tools/bench_partial.py --url http://127.0.0.1:7430 speech.wav # what the words while speaking cost a server's recognizer: partial=true requests of a recording played at its own pace
build/release/bin/oklm-gui -m qwen3-8b --read-aloud # every reply read aloud while it is written (Speak beside a reply, /speak): the installed speech model
build/release/bin/oklm-gui -m qwen3-8b --speak-model outetts-0.2-500m # the window's server speaks: its speech model is loaded at the first reading (1.6 GiB more from then)
build/release/bin/oklm-gui -m qwen3-8b --speak-model tts.gguf --speak-vocoder vocoder.gguf --speak-voice speaker.json # a speech model that is not the catalogue's, another voice
build/release/bin/oklm-gui -m qwen3-8b --no-speak # nothing is read aloud
build/release/bin/oklm-gui -m qwen3-8b --speak-file said.wav --send "/speak Hello there." --screenshot out.png # a test aid: what would be played, written to a WAV file
build/release/bin/oklm-gui -m qwen3-8b --chat-file chat.json # a saved conversation opened (what /save wrote; also dropped on the window)
build/release/bin/oklm-gui --url http://host:7430 --speak-voice anna --speak-speed 1.25 # replies read with one of that server's voices, by its name, a little faster; Pause holds the voice, and a reply read once is heard again without a new synthesis
build/release/bin/oklm-gui -m qwen3-8b --talk # a spoken conversation (the Talk box, /talk): speak, pause, the reply is spoken while it is written; speaking over it interrupts
build/release/bin/oklm-gui -m qwen3-8b --talk --listen-threshold -30 --no-interrupt # a noisier room; loudspeakers, which would interrupt the voice themselves
build/release/bin/oklm-gui -m qwen3-8b --speak-voice speaker.json # the voice chosen at start (the Voice chooser lists ~/.config/oklm/voices/*.json)
build/release/bin/oklm-gui -m qwen3-8b --talk --mic-file question.wav --mic-file other.wav@voice+3 --speak-file said.wav --screenshot out.png # a test aid: a scripted spoken conversation
# convert, quantize, split
build/release/bin/oklm convert <hf_dir> --out model.gguf --type f16
build/release/bin/oklm quantize model.gguf model-q8_0.gguf q8_0
build/release/bin/oklm quantize --dry-run model.gguf model-q4_k_m.gguf Q4_K_M # the size the file would have; nothing is written (convert --dry-run too)
build/release/bin/oklm quantize --progress-json - model.gguf model-q4_k_m.gguf Q4_K_M # for a program: the write as JSON lines, a line a tensor (convert too)
build/release/bin/oklm imatrix -m model.gguf -f data/tinyshakespeare.txt -o model.imatrix.gguf # importance matrix
build/release/bin/oklm quantize --imatrix model.imatrix.gguf model.gguf model-q4_k_m.gguf Q4_K_M
build/release/bin/oklm split model.gguf model-part --split-max-size 2G # shards llama.cpp loads too
build/release/bin/oklm merge model-part-00001-of-00003.gguf model.gguf # the shards back into one file
build/release/bin/oklm tokenize --model model.gguf --pieces "Hello world"
build/release/bin/oklm train-tokenizer --vocab 4096 --out tok.gguf data/*.txt
build/release/bin/oklm embed -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf "What is the boiling point of water?" # one normalized vector a text
build/release/bin/oklm rerank -m models/ms-marco-MiniLM-L-6-v2 -q "How many people live in Berlin?" "Berlin has 3.5 million inhabitants." "Paris is in France." # passages scored by a cross-encoder, best first
# run, chat, measure
build/release/bin/oklm run -m model.gguf -p "The capital of France is" --temp 0
build/release/bin/oklm run -m <hf_dir> -p "Hello" # a safetensors checkpoint, no conversion
build/release/bin/oklm run -m model.gguf -p "Hi" --backend cpu # default: auto (OpenCL when available)
build/release/bin/oklm run -m big-moe.gguf --n-cpu-moe all -p "Hi" # routed experts on the CPU, the rest on the GPU
build/release/bin/oklm run -m moe.gguf --expert-cache 4 --expert-loaders 4 --expert-predict router -p "Hi" # experts read from disk ahead of need
build/release/bin/oklm expert-pack -m moe.gguf # moe.gguf.experts: each expert's matrices together, one read a miss
build/release/bin/oklm run -m moe.gguf --expert-cache 4 --expert-pack moe.gguf.experts --expert-mirror /mnt/b/moe.gguf.experts -p "Hi" # reads alternate between two drives
build/release/bin/oklm run -m big-moe.gguf --expert-gpu auto -p "Hi" # the hottest experts on the GPU too, by routing counts
build/release/bin/oklm run -m model.gguf -p "The capital of France is" --temp 0 --ablate-scan blocks # each block's attention and feed-forward write taken out in turn: how far the next token's distribution moves (--ablate 3:attn,9:head:4 for chosen parts, --ablate-mean for their mean in place of zeros)
build/release/bin/oklm run -m model.gguf -p "The capital of France is" --observe run.jsonl --observe-lens 4 # with each line, the 4 tokens the output head would give after each block
build/release/bin/oklm run -m model.gguf --backend cpu -p "The capital of France is" --observe run.jsonl --observe-attn 5,23 # what each block wrote, the experts picked, blocks 5 and 23's attention: a JSON line a token
build/release/bin/oklm run -m model.gguf --doc report.docx -p "Summarize this report."
build/release/bin/oklm ingest report.pdf notes.html # a document's text, as run --doc reads it
build/release/bin/oklm rag index docs README.md --out docs.gguf # a retrieval index: chunks, vectors, BM25; updated by file hash
build/release/bin/oklm rag index docs --out docs-bge.gguf -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf \
--query-prefix "Represent this sentence for searching relevant passages: " # vectors of an embedding model
build/release/bin/oklm rag search docs.gguf "how is the KV cache quantized" -k 5 # hybrid: vectors and keywords fused (weighted; --fusion rrf for reciprocal rank)
build/release/bin/oklm rag search docs.gguf "how is the KV cache quantized" --rerank models/ms-marco-MiniLM-L-6-v2 --rerank-top 50 # the 50 best reranked
build/release/bin/oklm rag info docs.gguf # the encoder, the chunker, the counts
build/release/bin/oklm rag index docs README.md --out docs.gguf --progress-json - # for a program: each file's fate and the embedding's advance as JSON lines
build/release/bin/oklm rag info docs.gguf --sources --json # every source, and whether its file is unchanged, changed or missing now
build/release/bin/oklm rag search docs.gguf "how is the KV cache quantized" --json-full # the mode, the times, each passage with its dense, sparse, fused and rerank scores
build/release/bin/oklm rag eval data/beir/scifact --index build/scifact.gguf -m models/bge-small-en-v1.5 --chunk-size 0 # nDCG@10, recall@k per mode
build/release/bin/oklm run -m model.gguf --rag docs-bge.gguf -p "How do I keep the prefix cache across restarts?" # answered from the index, sources cited as [n]
build/release/bin/oklm chat -m model.gguf --rag docs-bge.gguf --rag-k 8 --rag-mode hybrid --rag-crag on --rag-log rag.jsonl # each turn retrieved, judged, logged
build/release/bin/oklm chat -m model.gguf --rag docs-bge.gguf --rag-rewrite on --rag-followup both --rag-route on # follow-ups made standalone, searched with the previous question; the model decides: no retrieval, one, several
build/release/bin/oklm run -m model.gguf --rag docs-bge.gguf --rag-rerank models/ms-marco-MiniLM-L-6-v2 --rag-rerank-top 20 -p "..." # candidates reranked
build/release/bin/oklm serve -m model.gguf --rag docs-bge.gguf # chat requests opt in with "rag": true or {"k": 5, "crag": false, "route": true}, streamed or not
build/release/bin/oklm serve -m model.gguf --rag docs-bge.gguf --rag-dir indexes/ # and may name an index of the directory, "rag": {"index": "NAME"} (/rag NAME in the dashboard's chat)
build/release/bin/oklm agent -m model.gguf --rag docs-bge.gguf # the Retrieve tool over an index
build/release/bin/oklm rag index docs --out docs-graph.gguf -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf --graph --gen-model model.gguf # entities and relations kept
build/release/bin/oklm rag search docs-graph.gguf "where was the designer of the Varnel Bridge born" --rag-graph on # the graph walked, two hops
build/release/bin/oklm run -m model.gguf --rag docs-graph.gguf --rag-graph on --rag-critique on -p "..." # answer checked against its sources, one retry
build/release/bin/oklm chat -m model.gguf --rag docs-bge.gguf --rag-critique flag # replies stream, a verdict line after each retrieved turn
build/release/bin/oklm agent -m model.gguf --rag-project --rag-critique on # a final answer citing unsupported passages sent back once
build/release/bin/oklm agent --url http://127.0.0.1:8080 --rag-project --rag-critique on # judged, rewritten and critiqued by the served model (/v1/decide)
build/release/bin/oklm rag index docs report.pdf --out docs-fig.gguf --caption-model models/smolvlm2-2.2b/SmolVLM2-2.2B-Instruct-Q8_0.gguf # figures captioned
build/release/bin/oklm rag index docs --out docs-fig.gguf --caption-model models/smolvlm-256m/SmolVLM-256M-Instruct-Q8_0.gguf --caption-prompt @prompt.txt # a small captioner, its own question
build/release/bin/oklm agent -m model.gguf --rag-project --rag-memory # Retrieve over the project's docs (kept indexed) and earlier sessions
build/release/bin/oklm agent -m model.gguf --rag-project --rag-inject route # passages given with each message when the router asks for them
build/release/bin/oklm chat -m model.gguf # /system TEXT, /reset, /decide A | B, /exit; /help lists the commands
build/release/bin/oklm chat -m model.gguf --kv-save --resume # go on with the last chat, its cache reloaded
build/release/bin/oklm chat --url http://127.0.0.1:8080 # the same chat on an OpenAI-compatible server (oklm serve, llama.cpp's server...): its model if it lists one, a list to choose from if several
build/release/bin/oklm chat --url https://host/v1 --api-key KEY --remote-model NAME # a server with a key (else OKLM_API_KEY), its model named; /model lists and changes it
build/release/bin/oklm chat -m SmolVLM-256M-Instruct-Q8_0.gguf --mmproj mmproj-SmolVLM-256M-Instruct-f16.gguf # with the model's projector, /attach photo.jpg (a picture, a sound or a video) goes with the next message; a catalogue model that carries one (oklm chat -m smolvlm-256m) needs no option
build/release/bin/oklm chat -m model.gguf --chat-template vicuna # a file that carries no chat template: a built-in format's name or a Jinja file (run, serve, agent, describe too)
build/release/bin/oklm ppl -m model.gguf -f data/tinyshakespeare.txt -c 512
build/release/bin/oklm ppl -m model.gguf -f data/tinyshakespeare.txt --progress-json - # for a program: a JSON line a window, the figure in the end line
build/release/bin/oklm tune -m model.gguf --backend cpu # tune the CPU work split once, cached
build/release/bin/oklm tune -m moe.gguf --backend cpu --expert-cache 4 # the expert cache's loaders, read-ahead, O_DIRECT for this disk
# serve: OpenAI and Anthropic APIs, decisions by scoring, the dashboard on http://127.0.0.1:7430/
build/release/bin/oklm serve -m model.gguf # --port N, --api-keys FILE, --parallel N, --lora-module ...
build/release/bin/oklm serve -m model.gguf --spec-grammar # the text a JSON schema or tool call forces, drafted and verified
build/release/bin/oklm serve -m model.gguf --prefix-dir cache/prefix # keep the prefix cache on disk: a restarted server finds long prompts cached
build/release/bin/oklm serve -m model.gguf --deterministic # temperature 0: the same tokens whether the prompt was cached or not
build/release/bin/oklm serve -m model.gguf --backend cpu --observe # GET /api/activations: the blocks' writes, the experts and the attention of the token decoded last; ?serial=S for an earlier one (--observe-lens 4: what the output head would say after each block too); drawn in a browser at http://127.0.0.1:7430/activations
build/release/bin/oklm serve -m model.gguf --embed-model models/bge-small-en-v1.5 --rerank-model models/ms-marco-MiniLM-L-6-v2 # /v1/embeddings, /v1/rerank
build/release/bin/oklm serve -m model.gguf --transcribe-model whisper-base --upload-max 64 # /v1/audio/transcriptions (Whisper); documents and sounds of 64 MiB at most
curl -s http://127.0.0.1:7430/v1/audio/transcriptions -F file=@speech.wav # a sound to text, OpenAI's route
build/release/bin/oklm serve -m model.gguf --backend vulkan --mmproj mmproj.gguf --transcribe-model whisper-large-v3-turbo --transcribe-backend vulkan --speak-model outetts-0.2-500m --speak-backend vulkan # the image encoder, the recognizer and the synthesizer on the model's device (cpu by default for the two sound models)
build/release/bin/oklm serve -m Qwen2-VL-2B-Instruct-Q4_K_M.gguf --mmproj mmproj-Qwen2-VL-2B-Instruct-f16.gguf --ffmpeg # images and videos in chat requests (image_url, video_url as base64); --ffmpeg: containers other than Ogg Theora
build/release/bin/oklm serve -m model.gguf --speak-model outetts-0.2-500m # /v1/audio/speech (OuteTTS); --speak-vocoder, --speak-voice, --speak-voices DIRS (GET /v1/audio/voices), --speak-model-id, --speak-backend cpu|opencl|vulkan|auto, --speak-max 4096
build/release/bin/oklm serve -m model.gguf --speak-model soprano-1.1-80m # the same route with Soprano: the sound at 32 kHz (X-Sample-Rate), one voice, a first byte in under a second on a CPU
build/release/bin/oklm serve -m model.gguf --transcribe-model whisper-base --speak-model outetts-0.2-500m # both: a spoken conversation in the dashboard's chat (the Talk switch, /talk: a pause ends your turn, the reply is spoken, speaking interrupts it)
build/release/bin/oklm serve -m model.gguf --transcribe-model whisper-base --speak-model outetts-0.2-500m --preload # the sound models loaded at start (without it: checked at start, loaded by the first request)
curl -s http://127.0.0.1:7430/v1/audio/speech -H 'Content-Type: application/json' -d '{"input":"The weather is nice today."}' -o speech.wav # a text to a WAV file, OpenAI's route ("response_format":"pcm" streams raw samples)
curl -s --data-binary @report.pdf 'http://127.0.0.1:7430/api/ingest?name=report.pdf' # a PDF, DOCX, HTML or JSON file to text
build/release/bin/oklm serve -m model.gguf --embed-model models/bge-small-en-v1.5 --ingest-keep 7200 # a document too long for the dashboard's chat is kept for retrieval instead of cut (/api/ingest with a conversation, /api/ingest/search), two hours after its last question
curl -s http://127.0.0.1:7430/api/model -d '{"model": "qwen3-4b"}' # a running server loads another installed model in place of its own (/model NAME in its dashboard; oklm serve --fixed-model refuses it)
build/release/bin/oklm serve --list # the servers of this user that run: pid, port, model, address (--json)
build/release/bin/oklm serve --stop 7430 # stop one of them, by its port or its pid
build/release/bin/oklm serve --keep smol -m model.gguf --port 7440 # the same server kept across logins: a systemd user unit, started now, at each login and when it fails
build/release/bin/oklm serve --rekeep smol -m model.gguf --port 7440 -c 8192 # change the command kept under a name: its unit written again, the server started again when it ran
build/release/bin/oklm serve --start-kept smol # start a kept server that does not run (what Windows runs at login, from the user's Run key)
build/release/bin/oklm serve --kept # the kept servers: running, stopped or failed, with the reason, the unit and the log (--json)
build/release/bin/oklm serve --unkeep smol # stop a kept server and keep it no longer: its unit and its log deleted
curl -s http://127.0.0.1:7430/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Hello"}]}'
# the coding agent
build/release/bin/oklm agent -m model.gguf # interactive, in this project
build/release/bin/oklm agent -m model.gguf -p "fix the failing test" # one request, then exit
build/release/bin/oklm agent -m model.gguf --listen # no terminal: other programs attach on loopback (oklm-gui's Connect)
build/release/bin/oklm agent --stop-listening 7481 # ask the agent listening on that port to end: it closes its listener and keeps its session
build/release/bin/oklm agent sessions # this project's sessions: id, date, turns, model, title (--json)
build/release/bin/oklm agent sessions export ID --format json -o session.json # one as JSON, or as Markdown (the default); sessions rm ID removes it
build/release/bin/oklm agent --url http://127.0.0.1:7430 --remote-model NAME # through an OpenAI-compatible server
build/release/bin/oklm agent -m model.gguf --tool-rag 8 # many MCP tools: the 8 that fit each message retrieved and shown with it
build/release/bin/oklm permissions Bash "rm -rf build" # what the permission rules decide for a call
build/release/bin/oklm audit --decision deny # the audit log of the agent's tool calls
# train
build/release/bin/oklm train -f data/tinyshakespeare.txt --out build/nano/nano --vocab 2048 --steps 1200 # a GPT-2 from scratch
build/release/bin/oklm train -f data/tinyshakespeare.txt --out build/nano/llama --vocab 2048 --arch llama --steps 600 # a Llama
build/release/bin/oklm train -m <hf_dir|model.gguf> -f data/tinyshakespeare.txt --out build/ft/ft --lr 1e-4 # fine-tune
build/release/bin/oklm train -m model.gguf -f data/tinyshakespeare.txt --out build/lora/a --lora 16 # a LoRA adapter
build/release/bin/oklm train -m model.gguf -f data/tinyshakespeare.txt --out build/lora/a --lora 16 --backend rocm --matmul-precision bf16 # products on the matrix units
build/release/bin/oklm train -m model.gguf -f data/tinyshakespeare.txt --out build/ft/f16 --backend rocm --matmul-precision f16 --loss-scale dynamic # F16 products, loss scaled
build/release/bin/oklm train -m model.gguf -f data/tinyshakespeare.txt --out build/ft/b16 --backend opencl --act-type bf16 --opt-type bf16 # BF16 activations and AdamW moments, on a GPU
build/release/bin/oklm run -m model.gguf --lora build/lora/a-000200.gguf -p "ROMEO:" # run with it
build/release/bin/oklm lora-merge model.gguf build/lora/a-000200.gguf merged.gguf # or merge it
build/release/bin/oklm rl -m model.gguf -f tasks.jsonl --out build/rl/rl --steps 100 # reinforcement learning on checked tasks
build/release/bin/oklm train -f data/tinyshakespeare.txt --out build/nano/nano --vocab 2048 --progress-json build/nano/progress.jsonl # the run as JSON lines (rl, abliterate, refusals too)
# images, sound, refusals
build/release/bin/oklm describe -m SmolVLM-256M-Instruct-Q8_0.gguf --mmproj mmproj-SmolVLM-256M-Instruct-f16.gguf -p "What is this?" photo.jpg
build/release/bin/oklm describe -m Qwen2-VL-2B-Instruct-Q4_K_M.gguf --mmproj mmproj-Qwen2-VL-2B-Instruct-f16.gguf --vision-id -p "What differs between picture 1 and picture 2?" before.jpg after.jpg # the pictures numbered as Qwen's template does when asked (Picture 1: , Picture 2: )
build/release/bin/oklm describe -m SmolVLM-256M-Instruct-Q8_0.gguf --mmproj mmproj-SmolVLM-256M-Instruct-f16.gguf -p "What is this?" --kv-save photo.kv photo.jpg # keep the conversation's cache, the image's rows included
build/release/bin/oklm describe -m SmolVLM-256M-Instruct-Q8_0.gguf --mmproj mmproj-SmolVLM-256M-Instruct-f16.gguf --kv-load photo.kv -p "What colour is it?" # a later run: the next question, the image not evaluated again (--then asks it in the same run)
build/release/bin/oklm describe -m SmolVLM-256M-Instruct-Q8_0.gguf --mmproj mmproj-SmolVLM-256M-Instruct-f16.gguf --kv-load photo.kv -p "Is this the same place?" other.jpg # the next question with a picture of its own (--then QUESTION --then-with other.jpg in one run)
build/release/bin/oklm transcribe -m models/whisper-base speech.wav # a Hugging Face Whisper checkpoint; the language is found
build/release/bin/oklm transcribe -m models/whisper-small --format srt talk.ogg > talk.srt # timed segments as subtitles (srt, vtt, json, --timestamps)
build/release/bin/oklm transcribe -m models/whisper-small --language fr --translate fable.ogg # a given language, written in English
build/release/bin/oklm transcribe -m models/whisper-large-v3-turbo --backend vulkan talk.ogg # the encoder and the decoder on a device (opencl, vulkan, auto)
build/release/bin/oklm speak -m outetts-0.2-500m -o speech.wav "Hello world." # text to speech (OuteTTS 0.2, 24 kHz WAV); or the text on standard input
build/release/bin/oklm speak -m outetts-0.2-500m --speed 1.25 --first-words 5 -o speech.flac < chapter.txt # faster, the first sound sooner, a FLAC file
build/release/bin/oklm speak -m outetts-0.2-500m --backend vulkan -o speech.wav "Hello world." # the language model and the vocoder's network on a device (outetts-0.3-500m: the same)
build/release/bin/oklm speak -m soprano-1.1-80m -o speech.wav "It costs $2.47, Dr. Smith." # Soprano: 32 kHz, one fixed voice, faster than real time on a CPU
build/release/bin/oklm refusals -m model.gguf # how often the model refuses a prompt set
build/release/bin/oklm abliterate -m model.gguf --harmful refused.txt --harmless answered.txt --out abl.gguf # refusal direction removedBuild options and every environment variable OKLM reads are listed in docs/build.md.
Kernel tuning: oklm tune times a 512-token prompt of a
model with other work splits of the CPU kernels (rows per matrix-product
task, attention tile sizes) for up to 60 s and keeps the fastest whose
logits are the defaults' bit for bit, in
~/.cache/oklm/tuning/ (or
$XDG_CACHE_HOME/oklm/tuning/), keyed by the CPU, backend,
model shapes and types, and kernel sources. Contexts apply a cached
result; OKLM_TUNE=0 turns that off and deleting the
directory is safe. oklm run --tune tunes first when the
cache has nothing. The output never changes (docs/tuning/). With
--expert-cache GB|auto (or --streaming, the
auto size) it tunes a MoE model's streaming settings instead:
--expert-loaders (0, 2, 4, 8),
--expert-predict (off, last, router) and
--expert-direct, one at a time from the defaults, each
timed over a 256-token prompt and 64 greedy tokens with that cache
(median of 3, a cold cache each run) and kept only when every logit
equals the defaults' run and it is 3 % faster, then again against the
defaults with their runs interleaved. The result,
stream-<key>.json beside the kernel tuning, is keyed
by the model, the cache size, the CPU, its threads and the disk holding
the model (device number and file system type); contexts with that cache
size apply it to the flags not given (a flag given always wins,
--no-expert-direct turns direct reads off),
OKLM_TUNE=0 turns it off too (docs/moe.md).
Compatibility across versions
From version 1.0 on, within the 1.x versions:
Kept compatible. A command line or a file that works with one 1.x version works with the later ones.
- The command line: the commands and options this page and
docs/describe keep their names and their meaning. Commands and options are added; one that has to go is first announced in the release notes and kept for a minor version. - The HTTP routes under
/v1/ofoklm serve(OpenAI's chat, completions, embeddings, audio and rerank routes, Anthropic's/v1/messages): they follow those APIs. - Standard files: GGUF (versions 2 and 3) and safetensors are read as their specifications say, and what OKLM writes in them stays readable.
- OKLM's own files that carry a version number (the KV file of
--kv-save, versions 1 and 2 today): a later 1.x reads the versions an earlier one wrote. - Project and user settings of the agent (
.oklm/,~/.config/oklm/).
May change between minor versions (1.1, 1.2, ...), each change named in the release notes:
- The C API in
include/oklm/. A program that links the library checksOKLM_VERSION_NUMBERat build time andoklm_version()at run time, and is built again for a new minor version. - OKLM's own routes under
/api/and the dashboard that uses them. - The output written for programs (
--json,--progress-json,/api/stats): fields are added; a field is not renamed or removed without a line in the release notes. - Caches and derived files (a retrieval index, the documents' cache, the expert cache, a training run's optimizer sidecar): one written by an older version may be refused with a message and has to be built again; it is never read as something else.
- Results to the last bit: a kernel made faster may round differently, so a sampled text may differ from one version to the next. The checks against the references (docs/testing.md) hold in every version.
Platforms. Linux on x86-64 with AMD and Intel GPUs
is what every version is tested on. Windows is built with MinGW and
tested under Wine, not on a real Windows machine; macOS branches are
written and were never compiled; NVIDIA GPUs are untested
(tools/gpu/report.sh writes the report an NVIDIA owner can
send back).
Windows: the signature
The Windows programs (oklm.exe,
oklm-gui.exe), the DLLs beside them, the installer and its
uninstaller are signed by NiloreaStudio when the build had the
publisher's key. The certificate is self-signed: no authority vouches
for it, so Windows shows the signature as not trusted, and SmartScreen
still warns, until the certificate is installed on the machine. That is
optional. The programs run the same without it. A build without the key
signs nothing, and never makes a key of its own; the installer CI
attaches to a release is signed only if the server was given the key (docs/build.md).
The certificate is NiloreaStudio.cer, in the programs'
folder. It is the public half only and cannot sign anything. Check that
it is the right one before trusting it:
certutil -hashfile NiloreaStudio.cer SHA256
must print
b9976986aebbfa925b019ee453d9834549b03581fd6ffa28c68f88e6df4a6acd
Then, in a console run as administrator, from the programs' folder:
certutil -addstore Root NiloreaStudio.cer
certutil -addstore TrustedPublisher NiloreaStudio.cer
To undo it:
certutil -delstore Root NiloreaStudio
certutil -delstore TrustedPublisher NiloreaStudio
Installing it makes the machine accept every program signed with that
key, so it is the administrator's decision on a machine that has one.
The certificate is limited to signing code and cannot issue other
certificates. A build made from the sources elsewhere is not signed,
unless its builder made a key on purpose
(sh tools/windows_sign.sh new-key), which has another
fingerprint: the build refuses a key whose fingerprint is not the one
above until WIN_SIGN_FINGERPRINT says otherwise.
Documentation
Start at docs/index.md. The C API is
documented in the headers under include/oklm/
(Doxygen).
New to LLM internals? docs/how-it-works.md explains what each command does, how and why, with commands to try.
make docs-html converts this page and docs/
to a browsable HTML set in build/doc/docs-html (page list,
per-page table of contents, offline search; needs pandoc). A bilingual
(French / English) slide deck is in presentation/: open
presentation/index.html.
License
Apache-2.0, like nilorea-library. See LICENSE and NOTICE.