Roadmap: done
What the roadmap planned and is finished, each item with its result and its links (the proof), moved here from the roadmap on 2026-11-11 so that the roadmap holds only what is left. The open items some blocks still mention (G5, G6, I4, and the "Left" lines) are scoped again in the roadmap, under "Next".
Status: maintained (finished items move here from the roadmap).
Milestones
| # | Milestone | Proof | State |
|---|---|---|---|
| M0 | Skeleton builds, tests, sanitizers, static analysis, docs | make test, make asan-test,
make tsan-test, make check,
make doc clean |
done |
| M0b | Core tensor engine on the CPU | every op against double references; quantized SIMD bit-identical to scalar; thread-count independent | done |
| M1 | oklm info parses every model of the public corpus (all
quant types, sharded files); oklm quantize output loads in
llama.cpp |
GGUF / safetensors I/O | done: corpus parses, quantize/convert output is tensor-for-tensor identical to llama.cpp's own files |
| M2 | Tokenizer bit-exact with Hugging Face on GPT-2, Llama 2 and Llama 3 vocabularies | tokenizer | done: identical ids on 10 000 entries for GPT-2, Llama 2, Llama 3.2, Qwen2.5 and SmolLM2; trainer identical to the reference |
| M3 | oklm chat with published Q4_K_M / Q8_0 GGUFs (SmolLM2,
Qwen2.5-0.5B, Llama-3.2-1B, TinyLlama) and a safetensors repository,
same greedy output as llama.cpp |
inference | done: oklm run/chat/ppl on SmolLM2, Qwen2.5, Qwen3,
Llama 3.2, TinyLlama and GPT-2 GGUFs and on the SmolLM2 safetensors
checkpoint loaded directly; F16/BF16 continuations identical to
llama.cpp's, quantized ones identical up to near ties (llama.cpp's own
margin at most 0.15 where they part); chat templates rendered by a Jinja
interpreter identical to jinja2 |
| M4 | Same on OpenCL, faster decode | OpenCL backend | done: every op runs on OpenCL (kernels for all inference ops, CPU
fallback for the rest), every kernel checked against the CPU, quantized
models match the float64 reference to 1e-4, generation 4 to 13 times and
prompts 17 to 71 times faster than the CPU on the corpus, layers split
between CPU and GPU with --gpu-layers |
| M5 | OKLM-Nano trained from scratch generates Shakespeare-like text; GPT-2-124M fine-tune matches llm.c | training | done: a 5.3M-parameter GPT-2 with its own 2048-token tokenizer trained on Tiny Shakespeare in 18 minutes on the CPU (validation loss 3.99 per token, 1.39 per character) writes Shakespeare-like verse; GPT-2 124M logits, loss, every gradient and ten AdamW steps within llm.c's tolerances of its PyTorch reference, and llm.c's published 40-step Tiny Shakespeare run reproduced (validation 4.10773 against 4.10778); checkpoints are GGUF models plus an optimizer sidecar (ADR-0008) |
| M6 | Agent ticks a plan using skills and tools on real files, runs background jobs and agents, permission modes and rules enforced and audited | harness | done: permission rules, modes and settings (ADR-0010) with an audit
log, Read / Write / Edit / Glob / Grep / Bash tools, task lists and plan
acceptance, skills, background jobs with a journal (ADR-0012),
sub-agents in the foreground or in the background with contexts of their
own, a Linux sandbox of commands (ADR-0011: namespaces including process
ids, deny-rule paths hidden, resource limits) and the agent loop
(oklm agent); Qwen3-4B keeps and ticks a three-step task
list while editing, building and running a C program in the sandbox,
follows a skill, delegates a search to a sub-agent, edits a file while a
background agent searches, and fails to write outside the project in
bypass mode (harness.md) |
| M7 | Image, audio and video questions answered locally | multimodal | done: SmolVLM reads images (SigLIP / idefics3 embeddings within 5e-4 of llama.cpp's), Ultravox hears speech, Whisper transcribes exactly, documents (text, JSON, HTML, DOCX, PDF) read as text, videos as frames and sound (Ogg Theora natively, the rest through ffmpeg when allowed, ADR-0007), on Linux and Windows (multimodal.md) |
| M8 | HTTPS API with dashboard; native GUI | server, GUI | done: oklm serve (ADR-0018) answers OpenAI's own Python
client (chat, streams, tool calls, JSON schemas) with API keys and TLS,
batches concurrent requests on one context (2.2 to 2.5 times the
throughput, the same answers), and serves a dashboard (server.md); oklm-gui, an Allegro 5
window over the API (its own server, or one at a URL), streams Markdown
replies with nested lists and code (gui.md) |
| M9 | v0.1.0 release with OKLM-Nano/Micro weights and tokenizers | hardening | done: OKLM-Nano (5.3 M, Tiny Shakespeare), OKLM-Micro (33.6 M, TinyStories then SmolTalk) and OKLM-Small (88.1 M, FineWeb-Edu then SmolTalk) trained from scratch on one GPU, released as v0.1.0 in F16 and Q8_0 with their tokenizer, licence and checksums, loading in OKLM and llama.cpp (release-notes-0.1.0.md, training.md) |
| M10 | oklm agent on a par with a full coding-agent CLI (agent_skill_chat_spec.md) |
harness, terminal UI | in progress: M0 (audit, design), M1 (backends: in-process,
OpenAI-compatible HTTP, mock fixtures; stream events; Esc / Ctrl-C
interrupts), M2 (lenient fuzzed parser, schema validation, message
metadata, parallel reads, loop detection, steering, Read / Glob / Grep
in full), M3 (Write / Edit read-before-write, MultiEdit, persistent
shell, head and tail output), M4 (Shift+Tab, dangerous-pattern list,
questions with diffs, saved rules, feedback, /add-dir), M5
(system prompt from templates with a snapshot test, output styles,
TodoWrite activeForm, reminders) M5b (memory files: scopes,
imports, lazy sub-directories, /init, #,
/memory) and M6 (settings merged by priority, environment
variables, /config, /permissions) M7 (context
accounting, /context, old tool results cleared, summaries,
/compact, /clear) M8 (sessions: transcripts,
--continue, --resume, rewind of the
conversation and the files, /export) M9 (terminal UI:
Markdown as it streams, compact tool calls, spinner, status line,
several-line input, @ / !, themes, trust) and
M10 (slash commands, the commands spec's C1–C3: registry, parser, menu,
picker, prompt commands as skills) M11 (hooks: nine events, exit codes,
JSON decisions, /hooks) and M12 (sub-agents: types and
definitions, restricted tools, several at once, SendMessage,
/agents) and M13 (skills in full: checked headers, the
.agent and built-in places, a prompt budget, allowed tools
per request, forked and preloaded skills, reload,
skills validate; ToolSearch and deferred tools) and M14
(headless: JSON and stream-json output, stream-json input, piped input,
the spec's flags, exit statuses; custom tools and hook functions in the
library) done; M15 (MCP): configuration scopes, trust, the stdio and
Streamable HTTP transports, tools, resources, /mcp,
oklm agent mcp done; server prompts as commands,
@server:uri mentions, the legacy SSE transport, the
official filesystem server end to end, OAuth (discovery, registration,
PKCE, tokens kept and refreshed, /mcp login) done; M16
(WebFetch, WebSearch through SearXNG, NotebookEdit, AskUserQuestion) and
M19 (plugins: commands, skills, agents, hooks and MCP servers bundled,
installed from a directory, a repository or a marketplace) done; user-guide.md and extending.md written (M20's docs); M17 (the
debug log, a price table and costs, the OpenTelemetry export,
/bug and /feedback) and M18 (the sandbox's
network filtered by domain through a proxy) and M20 (the eval harness:
twelve tasks in tests/evals/, make eval, a
report of pass rate, turns, tokens and time; Qwen3-4B 7 of 12) done |
| M11 | Background processes, Monitor, language servers and service integrations for the agent (process_spec.md) | harness | in progress: P0 (audit, design note, its four decisions taken
2026-11-03), P1 (the job manager as the process manager: kinds, labels,
owners, environment markers, a limit, readers with filter and tail,
rotation, restart policy, stop commands and grace, reverse close) P2
(readiness probes, ports and their holders, WaitFor) and P3
(ProcessOutput / ProcessStop / ProcessList, commands moved to the
background at their timeout or by Ctrl+B, PowerShell,
/tasks) P4 (orphans of crashed sessions found and stopped,
the exit question, what stops and what stays said) P5 (Monitor, events
coalesced and rate-limited, told at each turn, waking the idle agent
with a cap, turns of their own in headless runs) and P6 (language
servers as piped jobs over a JSON-RPC module, new diagnostics after
edits, the LSP tool) and P7 (MCP servers as jobs of kind mcp, their
crashes told and reconnected, ProcessStart / ProcessExit / MonitorEvent
hooks, background sub-agents in /tasks) and P8
(services.json: dependency order, readiness, monitors, the
Service tool, /services, the /init proposal)
and P9 (Docker / Podman, dev servers, databases, Playwright as skills,
the session's containers stopped at its end, image and host-network
dangers; checked with a fake docker) and P10 (Kubernetes context safety,
Android device safety, their skills; the emulator checklist on a real
emulator) and P11 (cloud CLIs read-only by default, tunnels saying what
they expose, the vms, cloud and tunnels skills) and P12 (resource
guardrails, the spec's settings, processes.md; the evals report and cgroup
limits left) done |
| M12 | Git for the agent: context, safe commits, branches, PRs, reviews, worktrees (git_spec.md) | harness | in progress: G0 (audit, design note, its four decisions taken
2026-11-03), G1 (detection, the snapshot in the prompt, branch and HEAD
changes told), G2 (git and gh commands read apart: reads allowed, plan
mode included, interactive forms refused, state-dependent dangers) G3
(secrets block a commit, unwanted and large files told, the commit
rules, /commit) and G4 to G6 (protected branches and dirty
switches asking, the conflict workflow, bisect reset at the request's
end, /diff with the agent's edits marked) done; G7
(/pr) and G8 (/code-review,
/security-review, /simplify,
/ultrareview, /bisect) as prompt commands (the
seeded-bug measure on Qwen3-4B: 2 of 6, then 0 of 6, against 80% asked);
G9 (worktrees: --worktree, sub-agent isolation,
/batch, /doctor; EnterWorktree / ExitWorktree,
setup commands and the worktree hooks done 2026-11-03); G10
(/autofix-pr, git-history,
/install-github-app); G11 (shallow, submodules and LFS said
in the snapshot, git.md) done but its evals |
| M13 | Slash commands: registry, menu, prompt commands as skills, the full catalog (commands_spec.md) | harness, terminal UI | done (the gaps left are in gap-analysis.md): C0 to C3 (with M10) and
C4 (the session's commands: /btw, /copy,
/todos, /recap, /rename,
/branch, /resume, /autocompact,
/goal, /clear NAME, /undo) and C5
(/model, /effort, /sandbox,
/plan TEXT, /config KEY=VALUE,
/update-config, /fewer-permission-prompts) and
C6 (/cd, /import, /skill-doctor,
/statusline, /color) and C7 (the git and
review commands with G7 to G10, /run, /verify,
/run-skill-generator) and C8 (ScheduleWakeup,
/loop, /schedule, background sessions:
/background, /fork, `oklm agent agents |
| M14 | vLLM-style serving: paged KV cache, continuous batching without fixed sequences, prefix caching across requests, preemption (serving_spec.md) | model, server | done: S0 done (the "Serving" audit in gap-analysis.md, serving-design.md, ADR-0019 accepted with
amendments, the baseline measured with tools/bench_serve.py
at 1 to 64 clients; decisions reviewed 2026-11-02); S1 done
(non-streamed requests cancelled on disconnect, --workers
over --parallel + --queue, bursts admitted
within the free sequences, view headers sized by the sequences; 64
clients: TTFT median 8 s to 0.4 s); S2 done: the pool, its allocator and
tables, slot mapping, copy-on-write, attention through the block table
on the CPU and OpenCL (step B, after gathering measured at half the
throughput), --kv-mem and the paged cache by default in the
server, the same greedy answers, memory following the tokens; S3 done:
FCFS admission by free blocks, decode reservation, preemption by
recompute (answers unchanged), prompt pieces oldest first capped by
--prefill-chunk, 64 sequences on OpenCL and 16 on the CPU
by default, /api/stats preemptions; 64 clients served at
once (TTFT 7.9 s to 0.13 s, no 503), throughput 1.3 to 2.4 times the
baseline, its OpenCL bound the per-sequence attention launches (S6); S5
done: block hashes chained on the parent, the
hash -> block map with the tokens compared, the LRU of
released blocks, attachment of a cached prefix, admission waiting for a
prefix being computed; 16 cold agent clients find 1297 of 1319 tokens
cached (2.4 times the throughput on OpenCL, 6 on the CPU); S4 measured:
a 38,872-token prompt among four streams stalled them 3.5 s a token with
fixed pieces; pieces now sized from measured costs
(--prefill-share) keep them at 40 ms against 17.6; flat
needs faster long-context attention; S6 in progress: one key and one
value write per layer, one attention launch for every sequence's decode
token, the step's metadata uploaded once (16 clients: 750 to 1078
tokens/s, 64: 746 to 1354, 1.9 to 2.4 times the baseline); greedy
sampling in place, the argmax and the candidates of a draw on the device
with only them read back (64 clients on OpenCL: 5289 tokens/s greedy,
4917 at temperature 0.8, 9 times the baseline); S6 done: decode steps
compute-bound, overlap measured unneeded (2 %); S7 done: the cache's
use, the prefix hit rate, the mean wait and the preemptions in
/api/stats and on the dashboard, blocks given back as a
request ends; S8: quantized caches checked on the paged cache (exact),
speculative decoding by prompt lookup in the server
(--spec-ngram, 4 by default: a rewrite 253 to 569 tokens/s,
the same answers); images and audio through the server
(--mmproj, base64 parts in chat and raw completions, their
rows cached by the media's content hash: the same picture again 4.43 to
0.13 s); a draft model in the server (--draft, measured
against plain decoding and rested when slower: Qwen3-4B with a 0.6B
draft 11 to 13 % faster on the CPU), sampled requests drafting too, the
same answers; S8 done; the partials closed (2026-11-04): the pool within
the device's free memory, sequences bounded by memory (256 clients: TTFT
4.57 to 0.22 s), swapping to host memory (--swap-mem),
prompt attention twice as fast at depth (key tiles shared by a KV head's
query heads, half tiles), pieces sized at their depth (no stall), the
pool's invariants checked under a stress test; S8.5 done (2026-11-04):
LoRA adapters served beside the weights, several in one batch (ADR-0021,
--lora-module; 64 clients: 73 to 80 % of the model's
throughput with q,v or q,k,v,o adapters, mixed adapters as fast as
one) |
| M15 | ROCm/HIP training backend with LoRA: hipBLASLt products on the
gfx1100, LoRA forward and backward by GEMM composition, then LoRA in the
trainer, adapter files, oklm train --lora,
oklm run --lora (rocm_spec.md) |
training, backends | done: R0 done (the audit, rocm-design.md); R1 done: hipBLASLt measured
no faster than the OpenCL product kernel on gfx1100, rocBLAS 1.7 to 3.5
times and approved in its place (loaded at run time,
tests/train/test_rocblas.c); R2 done:
--backend rocm, the trainer's large products through
rocBLAS on OpenCL shared virtual memory (Nano 38 to 26.5 ms a step,
SmolLM2-135M 390 to 310 ms, the same losses); R3 to R5 and R7 done: LoRA
in the trainer (CPU, opencl, rocm), adapter
files in llama.cpp's format (ADR-0020), oklm train --lora,
oklm lora-merge, --lora on run / chat / ppl /
serve (SmolLM2-135M: validation loss 3.83 to 3.30 in 200 steps, 30 s; a
LoRA step faster than a full one); R6 done: the baseline (LoRA 1.3 to
1.6 times a full step; Llama-3.2-1B trains with LoRA only, 344 ms a step
on rocm), gradients and moments held for the trainable
floats only, rocBLAS's default solutions kept, no custom kernel needed;
done, the large base in "Default models" (hybrid-layer training is
"Done", item 1); activation recompute added after (2026-11-04,
--recompute: the blocks recomputed and the head in chunks,
activations 3 to 8 times smaller for 21 to 51 % more time a step;
Llama-3.2-1B LoRA trains on 2 x 2048 tokens in 4.3 GB of
activations) |
| M16 | MCP tool-use fine-tuning: a LoRA adapter that drives an MCP tool server through the agent (schema-valid tool calls, multi-step orchestration), on the general default base (mcp_training_spec.md) | training | done (2026-11-07): T0 (audit, design note), T1 (the tool zoo, 117 conversations captured and gated), T2 (two QLoRA rounds on the Q4_K_M base), T3 (the loop, measured with the grammar off: trained tools 8 to 11 of 15 tasks, the held-out tools no better) (mcp-training-design.md) |
| M17 | Retrieval-augmented generation: embeddings, a local index, hybrid search and reranking, retrieval in run / chat / serve / agent with corrective fallback, tool selection by retrieval, then agentic, adaptive, conversational, graph, self-critique and multimodal retrieval (rag_spec.md) | RAG | done: the spec written, K0 (audit and design: rag-design.md, ADR-0022) and K1 to K6
(embeddings, the index, hybrid search, reranking, retrieval in
generation with corrective fallback, tool selection by retrieval) and K7
(agentic, adaptive and conversational retrieval) and K8 (graph,
self-critique, figures) and K9 (the evaluation report) done (2026-11-08
to 2026-11-09, rag.md). Result: SciFact nDCG@10
71.27 dense (MTEB's figure), 73.04 hybrid, reproduced by
make rag-eval; Qwen3-8B answers 92 % of 38 doc questions
with corrective retrieval against 16 % without; tool top-1 61.8 %
against 27.0 %; agent doc tasks 8 of 8 against 2 (rag.md) |
Done (2026-11-17)
The chats: the window on a server that is not OKLM's (C11)
Points 2 and 4 of C11,
and the window's part of point 3: the chat window on any
OpenAI-compatible server, such servers kept with their key, and the
models a server lists chosen in the window (gui.md, gui.md). Nothing existing changes
meaning: the window on oklm serve asks five questions more
when it is linked and behaves as before.
- Found out once. Linked to a server at an address,
the window asks each of OKLM's own routes once
(
GET /api/stats,/api/context,/api/commands,/api/model,POST /api/compactwith an empty count): a status other than 200 or an answer that is not OKLM's JSON is a route the server lacks, not asked again for that connection. A server with neither figures, nor a context, nor commands is not OKLM's: the side panel saysNot an OKLM server: it gives no figures, no memory line, no model to load in place of its own and no index, and does not tell its context. The window counts on 32768 tokens, its own setting: ..., the figures are no longer asked every two seconds, the memory line of the model settings says the same, and nothing is said to be out of reach. - A server that is not up yet. One that does not answer when the window is linked (or refuses the key) is taken for neither kind: the side panel says it cannot be reached, its figures go on being asked as an OKLM server's are, and the first request that gets through has the five questions asked again, the window then showing what a link to that server shows.
- A reply without a count. The count of the tokens is
asked for the standard way (
stream_options), and no longer of a server that refuses the member. When a reply ends without one the window counts the conversation by estimate, three tokens for ten bytes, says so once, and compacts by itself from it. The terminal chat on a server (oklm chat --url) already counted four bytes a token when the server told nothing, and compacts by itself there (run on the stand-in withoutusage:compacting the conversation: 6 of its 7 messages become a note, the context being nearly full); it is unchanged, as is whatoklm agentsends. - A member a server refuses. One rule for the chat
window,
oklm chat --urland the agent's backend (src/client/request_members.c): a 400 or a 422 that names a member of the request an answer can do without (stream_options,chat_template_kwargs,top_k,top_p,min_p,seed,repeat_penalty) is asked again without it, the member is no longer sent for that connection, and the chat says once that the option has no effect on this server; a refusal that names none stays an error, shown as it is. - The context is a setting. 32768 tokens (the agent's
for such a server), or what is kept with the saved server; the slider
with Apply context and
/context Nchange it without a request, and the line under the slider follows it, red when the conversation holds more than the size shown (with 256 it will be cut at 256 tokens). A refusal that tells the server's context (llama.cpp'sn_ctx) makes that the window's size. - A full context. The server's own wording of a
prompt longer than its context is read as one; a compaction counts by
estimate, its request sent again without
chat_template_kwargsto a server that refuses the member; a piece that goes on with a reply leaves it open as on OKLM, and is asked for in a message once the server refused that. The requests name the model (the first listed, the one kept, or/model NAME). - The models a server lists. Read once when the
window is linked, by the function the terminal chat shares
(
oklm_client_remote_models()): a server that lists several has a chooser of them in the side panel, in place of a name typed; the one chosen is asked for from the next message and kept with a saved server./model NAMEtakes a name the server lists and refuses another;/modeland/modelslist them, the list read again then and when a name is not in it; the list above the message completes/modelwith them, and Tab after/modelinoklm chat --urlcompletes among the names the server lists, where it offered this machine's catalogue and files. In the Connect window, List models asks the server in the fields with the key typed so far and offers its models beside the Model field, where a name can still be typed; a server that is not there, or refuses the key, is said in one line without the key. - Attachments, speech. A picture goes as it goes to
oklm serve(image_url), the server not asked first; a sound as an OpenAIinput_audiopart, a WAV or an MP3 as it is and any other format as the WAV of its samples (an Ogg file used to go with"format":"ogg", which that part does not name), one a server refuses being left out afterwards with the server's reason, as a picture is; a video is refused by the window with the reason; Record,/listenand Talk are the window's own Whisper, Read aloud its own speech model. - Servers kept with their key. The Connect window has
a Name, a Model and a
Context beside the address and the key, which is
masked, and a Save that keeps a server without
connecting. The entry (
serversofgui-servers.json:url,key,ca,verify, and nowname,model,ctxwhen set) is listed under the saved servers of the Connect window and of a start with nothing named, where choosing it connects with the key kept; a context or a model changed later is written back in it; Forget removes it with its key. The file is made with mode 0600 and given it again at each write; on Windows it is given an access list of its owner alone (oklm_file_owner_only(),oklm_w8_owner_only()insrc/core/win_utf8.c: protected, one entry, the owner's). The key is written nowhere else. Under Wine (tests/io/test_sys.cof the Windows build, and thesavedcase oftools/check_gui_wine.sh) the calls succeed, the file is still read and written by its owner, and the window saves and forgets a server; Wine keeps no access list, so what the list holds is Windows' own to show. - Measured on llama.cpp's server (the test
reference's build, Qwen3-0.6B Q8_0, CPU, loopback,
--api-key): a message answered (24 tokens in 0.60 s, the prompt 20 tokens), the reply cut at 24 tokens gone on with in a request that leaves it open (the prompt 44 tokens), the saved reply one text;/context 4096without a request; a compaction by estimate (2 of 4 messages into a note of 117 bytes) and the message after it answered (the prompt 124 tokens); the server saved from the window, the file at mode 600, chosen at a start with nothing named and answered with the key kept (4096 tokens, kept with it); the key in no output. With SmolVLM-256M and its projector a picture is answered (the prompt 149 tokens); with the text model it is refused by the server, which the note says.samples/jfk.wavspoken to the window is transcribed by the window in 0.5 s and answered. - Went through, each command once on llama.cpp's
server:
/help,/system,/reset,/retry,/rewind,/copy,/stop,/stats,/temp,/top-k,/top-p,/min-p,/seed,/max-tokens,/thinking,/attach(a PDF, a picture),/detach,/save,/continue,/compact,/autocompact,/autocontinue,/context,/model,/speak,/pause,/listenwork (/stophad no reply to stop in that run);/modelslists this machine's catalogue and names the server's model;/decidesays the server has no/v1/decideroute,/ragno retrieval index here;/videosets a sampling no video can use there, since a video is refused;/talkwas not run;/tooland/tuiare the terminal chat's, as everywhere. - Checked.
tests/client/test_request_members.c(34 checks) andtests/harness/test_cli_chat_url.c(127) for the rule of a member refused;tests/client/test_gui_remote.c(88 checks,apps/gui/chat_remote.h);sh tools/check_gui_remote.sh, part ofmake gui-test: 115 checks on a stand-in server (tools/standin_openai.py) and onoklm serve, 15 on llama.cpp's server where it is built (gui.md). The test aids--press fill:FIELD=VALUE,save,forget:URL,apply-context,model:NAME,list-modelsandpick-model:NAMEare new.
Done (2026-11-16)
T11: a model its device holds with less than one context of cache
One point T11
left, decided for 1.0.1 (memory-plan.md, gui.md, server.md, testing.md). The rule that sizes the cache for
the server and for the plan (oklm_kv_cache_room() then
oklm_kv_cache_rule()) has the case; the tools hold no copy
of it.
- Fewer layers in place of a refusal. A model whose
weights go whole on the device and leave less than one context of cache
beside them was refused by the memory line
(
what is left for the cache holds 0.7 contexts, less than one) and started byoklm servewith a cache no conversation of that length fits. With no--gpu-layersand no--kv-mem, the rule now places the most layers that leave the device one context of their cache, as it does for a model larger than its device, the other layers and their cache in host memory (oklm_kv_cache_ask::part_cache,oklm_kv_cache_size::part_cache,oklm_mem_plan::cache_part_cache,serve.part_cacheofoklm plan --serve --json). The line saysThe device holds the model with less than one context of cache beside it: 28 of its 36 layers go there with 14.3 GiB of the cache, the other 8 keep 4.1 GiB of it in memory, slower than the whole model on the device. It fits with 28 layers on the device...,oklm plan --servethe same in itscacheline, and the server's log names the layers. - Measured on the RX 7900 XT, Qwen3-8B Q4_K_M at
131072 tokens, on OpenCL and on Vulkan: 28 layers and 133664 tokens
planned, 28 layers and 133664 tokens served
(
GET /api/context;kv_blocks8354), from the form and from a terminal; 128 tokens answered in 7.1 s (OpenCL) and 7.6 s (Vulkan), against 1.5 s and 1.3 s for the whole model on the device (--gpu-layers 36, whose cache holds 0.7 contexts). In a chat window,/modelto Qwen3-8B with-c 131072kept for it serves it with 28 of 36 layers, and with the device held to 8 GiB at 32768 tokens with 27 of 36 and a cache of 34000 tokens. - Unchanged. A device with one context or more beside
the whole model keeps every layer and the cache it had, to the token
(97664 for Qwen3-8B at 40960 and at 4096, 161216 for Qwen3-0.6B).
--gpu-layersand--kv-memmean what they meant: every layer asked for at 131072 tokens is stillIt does not fit, in the form, in the chat window's Apply and for settings kept for a model. When no count of layers leaves the device a context the whole model stays there, refused as before; when host memory does not hold what stays there the plan does not fit, and the line says both memories are short. - A server that runs keeps its layers.
POST /api/contextand the Context slider of a selected server size the cache beside the layers in place: a server whose model is whole on the device still answers a larger context with the cache that is left (its line red above it), and one started with fewer layers keeps them at a smaller context (oklm_kv_cache_room()goes by a placed model's layers). - Checks.
test_device_short_of_a_contextintests/server/test_server.c, on the counting device: the layers and the cache of the rule, of the plan and of a server started there; every layer asked for, a cache size, a device with one context, host memory short; a resize of a split server and of a whole one.test_gui_settingshas the line's sentences.make gui-device-teststarts Qwen3-8B at 131072 tokens from the form on each backend (layers, cache, an answer), keeps a refusal with--gpu-layers 36in the form, the chat window and the kept settings, and has the form's two verdicts beside the running server's red line.
oklm agent -p
ends with status 1 when the message was not answered for want of
room
A message the context had no room for was said on stderr and the run ended with status 0 in the text output, while the JSON outputs told the same message as a failed request, status 1. A script could not tell an answer from none.
- What changed.
oklm agent -p "..."with the text output (the message alone, or after what is piped in,-p -included) ends with status 1 whenoklm_agent_send()returnedOKLM_ERR_OVERFLOW, which is two cases:(no room for a conversation in the context: ...), the agent's system prompt and tools leaving less than a turn's room by the model's count; and(the conversation no longer fits the context; /compact or /clear), a prompt the model or the server refused for its length once a larger context and a summary were tried. The message on stderr is the one it was. This is the one exit status that differs from 1.0.0. - What did not.
--output-format jsonandstream-json(and--input-format stream-json) ended with status 1 in both cases already, the text told as theresultof a request with"is_error":true. An answer that was written is status 0, cut by--max-tokensor not. A session at a prompt goes on as before.oklm chathas no one-message form: what is piped in is a conversation's lines, a message the full context leaves unanswered is said and the next line is read (/continueanswers it), and the status stays the conversation's. - Only a refusal that happened. Both cases follow the
model's own count of a prompt (in process, the tokens it made of it;
with
--url, the two counts anoklm serverefusal tells): the agent's estimate refuses nothing (agent-sessions.md).test_chat_template_reachranoklm agent -p Hion a model of a 512-token context, which counts the agent's prompt at 16359 tokens: that message was not answered (the test read only the rendered prompt of--save-training), so its two runs now have a context that holds the prompt (-c 20480) and are answered.
Checked: test_agent_no_room_status
(tests/harness/test_cli_chat.c, in process, the CPU, a
model of one layer and a 512-token context): status 1 with the message
once for -p Hi, for a message after piped input, for
--output-format json and stream-json
("is_error":true, the text as the result), and for 36 kB
piped into a context of 20480 that holds the agent's part
(no longer fits); status 0 with
-c 20480 --max-tokens 8, in text and in JSON
("subtype":"success").
T11 and C2: what the memory line did not size, a model larger than its device, a switch that never ends with nothing loaded
Three of the points T11
and C2
left (memory-plan.md, gui.md, server.md, testing.md). One rule still sizes the cache for
the server and for the plan (oklm_kv_cache_room() then
oklm_kv_cache_rule()), what loads beside the model is still
sized by oklm_server_beside_plan(), and the tools hold no
copy of either.
- The four things the memory line did not size.
- A projector's encoder on the CPU backend is its file's
weights and the arena its activations are computed in
(
oklm_vision_host_bytes_at(), the sizes ofoklm_vision_load()andoklm_vision_encode_pair();oklm_audio_host_bytes()), which the server tells ashost_memory.visionandaudio. SmolVLM-256M with its projector on the CPU:Counted beside the model: 284 MiB of memorywhere the file is 181 MiB; the 108 003 328 bytes the line counted for the arena are the server'shost_memory.vision. - A model beside the model on a device that is not the
model's (a recognizer, a speech, an embedding or a reranking model;
or any device when the model is on the CPU) is counted on that device by
the functions that count it on the model's, and that device is judged
for it (
oklm_server_beside::other):On vulkan1: 94 MiB of 84.1 GiB, for the embedding model., and, when it is short, a red line that names it (It does not fit: vulkan1 has not the room for what is asked of it.).oklm plan --serveprints a line a device and ends with status 3 when one is short. The server tells what it holds there inside_devicesofGET /api/stats. The choosers of the four backends offer the other devices of this machine (vulkan1). Qwen3-0.6B on the RX 7900 XT with bge-small on the Intel UHD 750 (OKLM_VK_DEVICES=all): 98 637 824 bytes counted by the line, 98 637 824 told by the server, and the model's cache as planned. - In the chat window, the embedding and the reranking models
the window finds by itself for a long document are counted as what it
will load, and the line says so
(
Counted: the models a long document is searched with, loaded when one is attached.), as the speech model is. - A server tells what its speech model holds of the host's
memory:
host_memory.speak(its context and its vocoder's copies,oklm_speech_host_held()) andspeak_call, the longest call of its vocoder. That call is no longer counted at 2 048 frames: it is the longest the synthesizer can hand over, a whole prompt's codes (oklm_vocoder_host_call()at the language model's context). Qwen3-0.6B with OuteTTS 0.2 on the CPU, loaded at the start: the 757 346 304 bytes the line counted beyond the files are the server'sspeakandspeak_calltogether, on Vulkan and on OpenCL.
- A projector's encoder on the CPU backend is its file's
weights and the arena its activations are computed in
(
- A model larger than its device no longer gets a
cache of one block. Where the device's budget does not hold every
layer's weights, the rule finds the layers that go there (those
--gpu-layersasks for, else the most that leave the device one context of their cache) and sizes the cache for them, four contexts at most, the other layers keeping theirs in host memory. The line saysThe model is larger than the device: 22 of its 36 layers go there with 407 MiB of the cache, the other 14 keep 259 MiB of it in memory. It fits with 22 layers on the device.... Qwen3-8B Q4_K_M at 4096 tokens on the RX 7900 XT held to 4 GiB (OKLM_GPU_MEM_MB=4096), on Vulkan and on OpenCL: 22 layers and 4736 tokens planned, 22 layers and 4736 tokens served, 3 929 860 096 bytes on the device by both; 11.7 to 11.9 tokens a second, against 93 for the whole model on the device. A model the device holds whole keeps its cache (97664 tokens for Qwen3-8B at 4096, 161216 for Qwen3-0.6B), and--kv-memand every layer asked for keep their meaning. - A model switch never ends with nothing loaded. The
server plans the new model with its own settings before it frees
anything (
oklm_mem_plan_loaded()): what the memory does not hold, or a model that does not take the settings, is refused with nothing touched; when the memory holds both models with the same cache and layers, the new one is loaded before the old one is freed ("loaded_first": true); else the old one goes first, and a load that fails then restores it, the error sayingthe previous model, NAME, was restored and serves as before, or, when that fails too, that no model is loaded. On the RX 7900 XT, Vulkan and OpenCL, from Qwen3-0.6B at 4096 tokens: Llama 3.2 1B loaded first with--kv-mem 512(0.31 and 0.33 s); freed first with the cache left to what fits, the new cache the planned 559680 tokens; Qwen3-8B refused by the plan with--kv-mem 16000 --gpu-layers 99(it takes 20.4 GiB on vulkan, which has 18.3 GiB), Qwen3-0.6B answering on. The switch had not been run on Vulkan: its cache was one block, the driver giving memory back 150 ms after the buffers are released; the server now waits for the device's own figure before it sizes a cache after a switch or a resize. - Tests.
test_kv_cache_ruleandtest_cache_room_whole(test_mem_plan);test_larger_than_device,test_switch_memoryandtest_side_device(test_server, on a counting device whose size and allocations the test holds);test_cache_sizeandtest_beside(test_gui_settings); groups 11 and 12 oftools/check_gui_form.sh device.
T11 and C10: the window that asks, the other options, the Advanced window's width, a full context beside the text
What was decided, point by point, of the items T11 and C10 listed as left for the chat window and the graphical tools (gui.md).
- A model chosen at a bare start is judged, then loaded in the
window. Load this model started the window
again on the model. The settings it would be loaded with (the window's,
with what is kept for the model) now go through the memory line first,
by the code that judges a model loaded in place of another
(
switch_fits()): settings that do not fit load nothing, the reason is written beside the list and the window that asks stays; settings that fit load the model on the thread a restart takes and the window's own server is made in place (start_load()), the conversation and the message being written kept. The models beside it are found as a start with-mfinds them, a Whisper model or a catalogue projector that cannot be opened being left out as at a start. - A wider list. After the servers of this machine:
the saved servers and the listening agents, which are the Connect
window's own targets (
targets_scan(): one search fills both lists), a saved server being reached with the key and the certificate settings kept with it and handed to the Connect window when it refuses them; and a row that opens the system's file dialog for any GGUF file, the file then listed with a row for its projector (a file that is an installed catalogue model's is loaded as that model).--press choose:URL,choose:file[:PATH]andchoose:mmproj[:PATH]select the new rows. - The options without a control, and the MoE disk options, in
the chat window. Its Advanced window has the
Other tab: Apply sends an option that
has a control to it, by the options' table (
so_parse()), gives the window's own server those that say how it works and the expert cache's (more_apply()), and refuses the others with the reason. The text is named by the line under the sampling and by the question asked before a restart, kept for the model as the memberotherof what the settings file keeps, read by the next window and at a switch, forgotten by Defaults. The MoE disk tab is the chat window's too, and--expert-cache,--pin-gband--expert-packare options of its command line. Nothing is kept for a server's model: that stays. - The Advanced window is 740 by 420 pixels, where it was 880 wide, wider than the server manager's smallest window. Its tabs and its controls took their widths from the window's already: nine tabs of 80 pixels, the label's column of 250.
- A compaction for a server at an address counts by the
server's tokenizer.
POST /api/compacttakescount, an array of texts, in place ofmessages, and answers{"window":N,"counts":[...]}, nothing compacted (server.md). The window asks it for the plan's messages in one request, then for the texts of the summary's requests; a server that does not answer leaves the estimate. - Under Wine.
tools/check_gui_wine.shhas the groupstart. - Three runs that had not been made, and what they
found:
- read aloud across a continuation. With a larger context the
voice followed the reply. With room made by a compaction it did not: the
note is inserted before the reply being written, and the reply the voice
follows was still found by its place before, so nothing of the second
piece was read (594 tokens of a reply of 1299 with SmolLM2-135M), and a
message longer than what had been read would have been read from its
middle. The index moves with the reply now. A piece asked for by
Continue after the reply ended was not read at all: the
sentences' stream is kept at a reply's end and fed again
(
speak_stream_end()leaves the line being written where it is; what was given at the end is found again by its words when a mark left open is closed by what follows). The times written of a reading were counted from the last piece, and negative: they are counted from the message. - pictures. A conversation of six exchanges, two pictures and two questions compacted with SmolVLM-256M: the message that carries the pictures and its reply stay where they are, eight older messages become the note, and the next message is answered from a prompt of 1206 tokens after 1486. Nothing to fix.
- on a device. Qwen3-0.6B, its 28 layers on the device, on OpenCL then Vulkan: a reply cut at 256 tokens gone on with after the context was made 512 in place (0.2 s on OpenCL), and, with a cache held to one context, room made by a compaction of 12 of 13 messages, the reply gone on with and the next message answered after a second compaction. Nothing to fix.
- read aloud across a continuation. With a larger context the
voice followed the reply. With room made by a compaction it did not: the
note is inserted before the reply being written, and the reply the voice
follows was still found by its place before, so nothing of the second
piece was read (594 tokens of a reply of 1299 with SmolLM2-135M), and a
message longer than what had been read would have been read from its
middle. The index moves with the reply now. A piece asked for by
Continue after the reply ended was not read at all: the
sentences' stream is kept at a reply's end and fed again
(
- The memory line counts the expert cache. The plan
already sized a context's expert cache
(
oklm_context_plan(),plan_judge()); the tools did not give it the two options.so_context_params()now does, for the server manager's form and the chat window alike, the window's own server is made from the same parameters, and the line saysThe routed experts are read from the file: an expert cache of 117 MiB is counted in place of their 697 MiB.(Granite 3.1 1B A400M on the CPU with--expert-cache 0.1 --pin-gb 0.02: 1.2 GiB of memory where 2.5 GiB without; a server started so tells 123 052 032 bytes, 117 MiB, inexpert_cache.bytesofGET /api/stats). - A reason that was lost. A model that could not be
served in place, with nothing kept for it, said
the server could not be made: the reason of the load was overwritten by that of a second try that was not made.
Checked on 2026-11-16:
sh tools/check_gui_start.sh, part ofmake gui-test: 42 checks. SmolLM2-135M chosen isjudged to fit, loaded in this window, the window's first model withthe 5 bytes of the message being writtenkept, and answers;tiny.ggufgiven as the file dialog gives it is listed, loaded and answers, withtiny-mmproj.ggufas its projector when one is chosen; in a window started with--kv-mem 1 -c 16384it isnot loaded: with a context of 16384 tokens it does not fit: that cache holds 0.5 contexts, less than one; a context of 16384 tokens takes 2 MiB., and the next message is not sent; a server started with--api-keys, saved with its key, links and answers, and with another key is handed to the Connect window; an agent that listens is attached to.sh tools/check_gui_advanced.sh, part ofmake gui-test: 38 checks. The nine tabs of the server manager and the eight of the chat window saved in a window of 760 by 480 pixels, the seventeen pictures read: no control cut, none over another.--settings "--workers 3 --deterministic -ctk f32"asks about-ctk, the other options, restarts the model (now -c 512 -ctk f32 --workers 3 --deterministic) and keeps"other":"--workers 3 --deterministic"; the next window reads it;--insecure,--workers manyand--expert-loaders 2without an expert cache are refused; the tiny mixture of experts started with--expert-cache 0.001 --pin-gb 0answers and restarts with--expert-cache auto --pin-gb 0.0001 --expert-loaders 0 --expert-predict off.test_compact_route:{"count":[...]}gives the countsoklm_tokenize()gives for the served model, an empty list an empty list, an entry that is no string a 400.make gui-test: a window on a server at an address sayscompaction: the server's tokenizer counted 8 messages, 462 tokensand compacts six of them. By hand, against a server without the member:the server does not count texts: an estimate, 1.00 tokens a byte, and the same compaction.sh tools/check_gui_full.sh, part ofmake gui-test: 19 checks. The words given to the voice are those of the reply, once and in order (tools/gui_said_words.py): 383 words across a context made larger, 1036 across a compaction, 383 after Continue; a spoken conversation whose reply is cut at 192 tokens is said to its end and the window listens again; the pictures' case above.- By hand, with the installed OuteTTS 0.2 as the window's own voice on
the CPU: the reply of 383 words, cut and gone on with, made 108.5 s of
sound in 25 sentences (325.9 s), which Whisper base wrote down as 387
words, 360 of them the reply's in order; across the cut it wrote
sailing with him. As Emily's father was lost,. BACKENDSofmake gui-device-test(RX 7900 XT, under the machine's GPU lock): the two cases of a full context pass on OpenCL and on Vulkan.CHECKS=start bash tools/check_gui_wine.shaftermake MINGW=1 BUILD=build/mingw tests apps gui, Wine 11, the prefix underbuild/mingw: 13 checks in 34 s, the last two the system's file dialog answered from the keyboard with the file's name and the file loaded. A dialog's title held parentheses, which a search of windows by name reads as a pattern: the titles areA model: a GGUF fileandThe model's projector: a GGUF file.test_gui_speak(the stream fed again after its end, with and without a mark left open),test_gui_settings(the MoE disk options among those the chat window's command line takes and keeps).ONLY="gui-chat-start gui-servers-advanced" sh tools/docs/gui_shots.sh: the two pictures whose content changed.
What is left is under T11 and C10.
T11: what shares the memory, the presses of every tool, a model loaded with nothing kept
Three of the points T11 left (gui.md, memory-plan.md, testing.md).
- The memory line counts what shares the memory. A
server takes its draft model, its encoder and its sound models out of
the cache's room before it sizes the cache; the line planned as if the
model were alone.
oklm_server_beside_plan()(oklm/server/server.h) sizes each part from its file, with nothing placed, by the function the server sizes it with once loaded, where the server places it: the projector's encoder, the transcription model and the speech model with its vocoder on the model's device when their backend is that device, with the backend's own reserve, else their weights in host memory; the embedding and reranking models in host memory. The plan takes the two sums (oklm_mem_plan_params::beside_dev,beside_host) and gives the device's tooklm_kv_cache_rule()as the server does; the draft model's context is made byoklm_serve_draft_params(), which the server calls too. The line saysCounted beside the model: 590 MiB on the device, and names what it cannot size (Not counted: the speech model (no vocoder named)).oklm plan --servetakes--transcribe-model,--speak-model,--embed-model,--rerank-model, their backends and--tiles. - A model loaded in place does not run with the settings of
the model before. The options are of two kinds
(
so_model_owned,so_fresh_model()): the model's (the layers on the device, the experts on the CPU and their tier, the cache's size, the expert cache, the draft model, the chat template, the picture side) are automatic again at a switch; the machine's and the user's (the backend, the context, the cache's types, the sequences, the sampling) stay. A context longer than the model was trained with is reduced, and the switch's note says what changed. What is kept for the model is read over the result. The memory line judges the settings before the model before is stopped: what is kept and does not fit is left aside, and when the window's alone do not fit the model is not loaded and the one before goes on serving, with the line's reason. - Every group of the Wine check on the tools'
presses. The launcher, the training window, the sessions
browser and the retrieval manager gained
--press(open:TOOL,docs;stop,merge,export,export:FILE,chat;select:ID,select:live,attach,resume,export,export:FILE,delete,yes,no;sources:PATH,build,eval:DIR), carried by one piece of the shell (gui_presses) in which a press waits for what it presses; the chat window gainedexport,export:jsonandattached:N.tools/check_gui_wine.shholds no place of a window: its chat, launcher, train, sessions, rag and media groups are rewritten on the presses, the system's dialog answered from the keyboard.tools/check_gui_press.sh, part ofmake gui-test, drives the same presses on Linux.
Choices made where the item was open. The parts beside the model are
sized by loading their headers as the server does at its start, not by a
second formula: a transcription model's device share is
oklm_whisper_device_bytes() of the checkpoint read, 0.2 s
for the whole plan of a model with a projector, both sound models and a
draft model. An encoder is counted on the model's device whenever the
backend is a device, as the server reserves it, the model's layers there
or not. On the CPU a part is its file's weights: the mapped pages it
would hold. A model's options that the command line named stay at a
switch, as every option of the command line does. A switch whose
settings are the window's is still the server's own, in place; one whose
settings differ stops the window's server and makes another. The
judgement of a switch is made by the drawing thread before anything is
stopped (a fifth of a second): nothing is to be taken back when it
refuses. A press that opens the system's dialog has a twin that answers
it (export:FILE), for the runs without a keyboard; the Wine
run keeps the dialog, which is what it is there to check.
Found on the way and fixed. The usage text of oklm
passed the length a C99 compiler is required to take in one string once
plan's options were in it: it is printed in one more
piece.
Checked on 2026-11-16, the device runs on an RX 7900 XT, one at a time:
test_mem_plan, 8577 checks (test_serve_beside);test_gui_settings, 574 checks (501 before:test_beside,test_fresh_model);test_server, 2585 checks; the other eighttest_gui_*pass.make gui-device-teston OpenCL and on Vulkan, 110 checks (68 before). SmolVLM-256M with its projector at 8192 tokens, started from the form: the line plans 806992 tokens of cache on Vulkan and 820096 on OpenCL, the server tells the same (cache_tokens), where the line planned 833872 before, 3.2 % and 1.7 % more than the server made; the encoder's share the line counted is the server'sdevice_memory(vision249934848 andbackend_own369098752 on Vulkan, 67108864 on OpenCL). Qwen3-8B Q4_K_M with Qwen3-0.6B Q8_0 as its draft model at 8192 tokens: 59072 tokens planned and made on both backends, 7.2 contexts, where the line planned 97664, 11.9 contexts. A chat window on Qwen3-8B with 20 of its 36 layers on the device, then/modelto Qwen3-0.6B:28 of 28 layers on a device,automatic again: --gpu-layers. With the device held to 8 GiB (OKLM_GPU_MEM_MB), a window on Qwen3-0.6B at 32768 tokens asked for Qwen3-8B:not loaded: with a context of 32768 tokens it does not fit: what is left for the cache holds 0.6 contexts, and Qwen3-0.6B answers the next message.make gui-test, twice: its flows of a model loaded with nothing kept (the layers and the cache's size of the model before automatic again, the context of 256 tokens, the cache's types and the sampling kept; a context of 1024 tokens reduced to the 512 the model was trained with),tools/check_gui_form.sh tiny(18 checks) andtools/check_gui_press.sh(34 checks).sh tools/check_gui_kept.sh, 21 checks against the stand-in.tools/check_gui_wine.sh, whole, under Wine 11 in a prefix of its own aftermake MINGW=1 BUILD=build/mingw tests apps gui: 58 checks passed in 186 seconds, the sound group skipped for want ofultravox-1b.ONLY=gui-chat-restart sh tools/docs/gui_shots.sh: the one picture whose content changed, the model settings of a window's own model, whose line saysCounted beside the model: 276 MiB of memory(the transcription model found for the window).
What is left is under T11.
T11: the models beside the model where they compute, a model's options of the command line, three lines of one server
Five more of the points T11 left (gui.md, memory-plan.md, server.md).
- An embedding or a reranking model on a device is counted
there.
--embed-backendand--rerank-backendhave a control each, Its backend, in the Advanced window's Retrieval tab, kept in the profiles.oklm_server_beside_plan()sizes the context the embedder or the reranker makes (oklm_embed_context_params(),oklm_rerank_context_params(), which the two now make their contexts with) on the model's device when its backend is that device, by the function the server tells it with (oklm_context_device_bytes()), with what it keeps in host memory. The server loads these two before it sizes its cache: their share comes out of the device's budget (oklm_kv_blocks_fit_beside()), for the server and for the plan (oklm_mem_plan_params::beside_held), where the server's cache was the model's alone before and the two used the device's reserve.oklm plan --servetakes the two options. - What computes on the CPU is counted with what it works
with. An embedding or a reranking model: its weights as they
are resident and its context's cache, activations and scratch
(
oklm_mem_plan_model()of that context); the transcription model: its checkpoint, the two arenas of its passes and the copies it makes of its weights (oklm_whisper_host_bytes()); the speech model: the language model with the context the synthesizer opens, the vocoder with the copies of its weights and a call of its network (oklm_speech_host_bytes(),oklm_vocoder_host_bytes(), the call's arena sized by the function the call sizes it with). The draft model's context was counted. A server tells what the first three hold (host_memory:embed,rerank,transcribe, counted inheld). - A speech model the chat window will load. A speech
model that is only installed is opened by the window itself, on the CPU,
at the first reading: the line counts it as that until the window has
loaded it, and says
Counted: the speech model, loaded at the first reading.The transcription model was not such a case: the window's own server is given the one found for the model, and the line counted it. - A model's option given on the command line is that
model's. The chat window's command line takes the options its
model settings keep (
--gpu-layers,--n-cpu-moe,--expert-gpu,--kv-mem,-ctk,-ctv,--devices,--draft,--draft-n,--spec-ngram,--prefix-dir,--chat-template,--tiles): it took-cand--backendalone, so that a model's option could not be given at start at all. Those of the model's (so_model_owned) are automatic again when another model is loaded in place, as the same option set in the panel is, and the note of the switch lists them; what is kept for the other model is read for them; switching back does not bring them back; a window started again on another model leaves them out. The machine's and the user's (the backend, the context, the cache's types, the devices, the prompt lookup) stay the command line's. - Three lines of one server say the same. The server
manager now judges the context a server has as it judges another size,
as the chat window did: a server whose cache ends before its own context
was red in the chat window and said nothing in the server manager. Both
tools say
It stops near N tokens, where the cache endsfor a cache in host memory too, where neither did while the form's line was red.--press context:Nof the server manager drags the selected server's slider and writes its line; the chat window writes the line of its own.tools/check_gui_form.shchecks the form, the server manager and a chat window attached to the server against each other and against the server'scache_tokens.
Choices made where the item was open. The backends of the embedding and the reranking models are a server's: in the chat window, where the two are the window's own and computed on the CPU already, the controls are greyed and the line counts them on the CPU. They are kept in profiles, not for each model: nothing of the models beside the model is. What is loaded before the cache is sized comes out of the device's budget, not of the cache's room, because the device's free memory counts it by then: taking it twice would make the cache smaller than the server makes it. A speech model already loaded by the window is not counted apart: the machine's own figures have it. The embedding and the reranking models the window finds by itself for a long document are not in the line: they are loaded with that document. The chat window gained the options on its command line because the decision had nothing to apply to without them.
Found on the way and fixed. The projection of an embedding context
(oklm_context_mem_estimate()) left out the rows it keeps, a
whole batch's where a batch holds several texts, and
oklm_context_mem_used() counted a context's where a batch's
are allocated; on a device both counted a row of logits a token where an
embedding context reads one value back (250 MiB too many for
bge-small).
Checked on 2026-11-16, the device runs on an RX 7900 XT, OpenCL then Vulkan, one at a time:
test_mem_plan, 8715 checks (test_embedding_context, the estimate of an embedder's and a reranker's context against its allocations;oklm_kv_blocks_fit_beside());test_gui_settings, 650 checks (574 before: the two backends' controls, each part's working memory against the function that sizes it, the speech model loaded later, a model's options of the command line);test_server, 2566 checks; the other eighttest_gui_*pass.make gui-device-test, 174 checks (110 before). Qwen3-0.6B at 8192 tokens with bge-small-en-v1.5 Q8_0 on the device: the line counts 94 MiB there, 98637824 bytes, which the server tells to the byte (device_memory.embed), and plans a cache of 160352 tokens, which the server makes, where the model alone has 161216; the 14433792 bytes counted for what that context keeps in host memory are the server'shost_memory.embed. With bge-small and MiniLM-L6 F16 on the CPU:232 MiB of memory, the cache planned and made 161216 tokens, and the 162264064 bytes counted for the two contexts are the server's (embed81582080,rerank80681984), where the line counted their files' 79 MiB. The three lines: Qwen3-0.6B at 8192 tokens,fits, both tools grey for a cache of 161216 tokens, the server's; Qwen3-8B at 40960 tokens,fits with a smaller cache, both grey for 97664 tokens, the server's, and at 131072 tokens the formdoes not fitand both toolsIt stops near 97664 tokens; Qwen3-0.6B at 16384 tokens with a cache of 64 MiB, 576 tokens, both tools red at its own context. The figures are the same on the two backends.make gui-test: the flow of a model's option given on the command line (--kv-mem 1 -ctk f32 -ctv f32 --spec-ngram 2: a cache of 4096 tokens;/modelto another model,automatic again: --kv-mem, 2048 tokens with the same types and prompt lookup;/modelback, 2048 tokens still, nothing of it in the settings file), andtools/check_gui_form.sh tiny, 22 checks (18 before: a cache in host memory that ends before the context, red in the three lines).- By hand, on the canary models: a chat window on Qwen3-0.6B with
Whisper base and OuteTTS 0.2 installed says
Counted beside the model: 1.8 GiB of memory. Counted: the speech model, loaded at the first reading.; with--no-speak,609 MiB of memory, the transcription model's. ONLY="gui-servers gui-servers-stop gui-chat-restart" sh tools/docs/gui_shots.sh: the three pictures whose content changed (the line under a selected server's slider; the model settings' line).
What is left is under T11.
The chat window started with nothing named loads no model
Started with neither -m, --url nor
--agent (the menu entry, the launcher's
Chat, a double click), the chat window took the first
installed chat model of the catalogue and loaded it into a server of its
own. With a large model already loaded in a server of the machine the
memory was taken twice, and the window's own cache got what was left (gui.md).
- Nothing is loaded.
chat_start()loads a model only when one is named. The window opens on What should this window talk to?: the servers answering on this machine, found by the search of the Connect window (targets_scan()), then the installed chat models of the catalogue; the text beside the list says what a press does, and, for a model, how many servers already run that it would be loaded beside. Connect to this server links the window as Connect does; Load this model starts the window again on the model, as Use this model does in a window with no server of its own. Nothing is selected at start, also when one row is all there is. With no chat model installed and no server answering, the window opens on its Models window, as before. - While nothing is chosen. A message stays written
and the status line says what to choose
(
Not sent: nothing is loaded yet. ...), the window that asks opened again when it was closed; Send is live for that, where it was off. A scripted--sendwrites the same on the error stream and the window is saved, not left waiting. The side panel, the Context slider, the model settings and the commands were already those of a window with no model (nothing installed). - The options are as they were.
-mloads at once,--urland--agentconnect; the server manager's Chat passes--url, the training window's, the retrieval manager's, the conversion tool's and the model manager's pass-m, the sessions browser's--agent. The model manager's Read aloud on a speech model starts a window with nothing named and--read-aloud: it now asks what to talk to, the option kept for what is chosen. - The test aid.
--press chooseopens the window that asks,choose:PORTandchoose:NAMEselect a server or an installed model in it,usepresses its button. A window that starts again on a model no longer carries the script's presses with it.
Checked:
sh tools/check_gui_start.sh, part ofmake gui-test, 20 checks on the CPU beside a server of the tiny model: with SmolLM2-135M installed in the check's own folder and that server running, the window started with nothing named leaves the loopback ports answering as they were (its own server would take one) and names no model on its error stream;--send "hello"is refused with the line;--press choose:PORT --press uselinks it (chosen: the server at http://127.0.0.1:7480: nothing is loaded by the window) and the message after is answered (answered: 8 tokens);--press choose:smollm2-135m --press usestarts it again, and a port more answers withSmolLM2-135M-Instruct-Q8_0; with an empty folder and no server,0 servers on this machine, 0 installed modelsand the Models window.ONLY=gui-chat-start sh tools/docs/gui_shots.sh: the picture of the window that asks, a server of Qwen3-0.6B selected above five installed models.
What is left is under T11.
A notice of layers left on the CPU that no context had
oklm serve -m qwen3-8b -c 40960 on a 20 GiB device
logged
context: 23 of 36 layers fit in 20464 MiB of device memory; the others run on the CPU,
and GET /api/context then answered
"n_gpu_layers":36 with a cache of 97664 tokens. The
server's account was the right one.
- Where the notice came from. Before it starts its
server,
oklm serveprojects the memory the model needs, to warn when it exceeds what the machine has (warn_memory()). It projected the contiguous cache of four sequences of the whole context, which is what--kv-mem -1makes: 163840 tokens of cache, more than the device holds beside the weights, hence 23 layers. The server then made the paged cache its rule sizes (oklm_kv_cache_rule(): what the device has left beside all the layers), and the context placed the 36 layers. The count of the projection was logged byfit_layers(), which a projection and a placement share. - A projection says nothing.
fit_layers()andctx_shared_layers()log only for a context being placed:oklm_context_mem_estimate()counts the same layers silently. The notice is the placement's, once. - The warning is of the cache the server makes.
oklm servenow projects the paged cache by the server's rule (the draft model's share taken first), and the contiguous one only with--kv-mem -1or weight streaming: a warning of host memory for layers that the server does not leave on the CPU is not printed either.
Checked:
test_gpu_fit(tests/model/test_infer_public.c, OpenCL, Llama 3.2 1B, the device held to 1200 and 700 MiB byOKLM_GPU_MEM_MB): the projection of each size writes no notice and counts the layers the context then places (16, 11 and 0 of 16); the context writes one notice when layers stay on the CPU and none when all fit. 24 checks.- On the RX 7900 XT, once:
oklm serve -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf -c 40960(Vulkan) writes no notice of layers, andGET /api/contextanswers"n_gpu_layers":36, "n_layer":36,"cache_tokens":97664, as before.
Done (2026-11-15)
T11.
The options of oklm serve in the server manager and the
chat window
The server manager's form passed the model, the backend, the context,
the sequences, the port, keys, TLS, adapters and one index, and left the
rest to a text field; the chat window's own server got the model, the
backend and the sequences. Both now have controls for 51 options of
oklm serve, from one table (gui.md, the model settings).
- One table (
apps/gui/serve_opts.h): for each option its name and its other spelling, what its value is, its range, what the server does without it, its group and when it applies. The command line, a profile'soptionsobject, the reading of a command line or of an old profile's text back into the controls (what the table does not know stays as text, untouched), the line that says what the Advanced window has set, the memory plan's parameters and the sampling of a chat request are all made from it. The rest ofoklm servestays in a text field. - One component (
apps/gui/settings.c): the offload row, the sampling block and the Advanced window, each control a line of the table, kept in step with the values at each timer event, so that two controls of one option follow each other. What does not apply is greyed and says why as text: in the field, on the line at the foot of the tab, as the tooltip of a tab whose name is dimmed. - The server manager. Under Backend,
the offload row: Layers (
--gpu-layers: as many as fit, then none to all), Cache (off, 8-bit, 4-bit:-ctkand-ctvat once), MoE CPU (--n-cpu-moe). The memory line isoklm plangiven all of them, and names the cache's types, the layers on the device and the experts on the CPU. Advanced... opens the window, nine tabs (Cache, Sampling, Vision, Speech, Retrieval, Speed, MoE disk, Network, Other); the line beside it is what the window has set. A profile keeps everything, and one kept before is read as it is. - The chat window. The model's name opens the model
settings: the backend, the offload row, the context, the same memory
line, the sampling. The sampling goes with each request at once.
Anything else asks first
(
Restart the model with these settings? The conversation is kept; the model loads again, with the last load's time, and Restart, Cancel, Do not ask again in this window), then the window's server is stopped and made again with the settings, the conversation kept; a context alone is resized in place; settings that do not hold bring the ones before back. The Advanced window is there without its Network and MoE disk tabs, with a button that opens the server manager on the same settings. The transcription, speech, embedding and reranking models and the projector, options ofoklm-guiuntil now, have their fields on its tabs. - A server's settings told to a client.
GET /api/contextgainedbackend,n_gpu_layers,n_layer,cache_type_k,cache_type_vandsampling(temperature,top_k,top_p,min_p,repeat_penalty): no route told a server's sampling defaults (server.md). The chat window takes them as its own when it attaches and shows where the server computes, greyed; a server that tells nothing leaves the window's values.
Choices made where the item was open. The row is one line where the window is 760 pixels wide or more and two lines in the form, which is 400. A tab that does not apply is dimmed and stays pressable, so that its reason can be read. The Advanced window's text field is the form's former More. The cache's types are told lower case, as the command line names them. In the chat window the Other tab is left out (its server is in the process: an option without a control has nowhere to go), and a change of the embedding or reranking model needs no restart.
Checked on 2026-11-15, the device flows on the RX 7900 XT:
test_gui_settings, 354 checks: the table, the values, the command line, what applies, the reading back, a profile's object, the offload row's mappings, the plan's parameters, the sampling, a server's answer, the restart's question.test_server, 2562 checks at the last run,test_context_resizewith the new members.make gui-test, six flows more: the form's command line with the offload row and three tabs set and the profile kept; the profile loaded in another window; a profile written as the tool kept them before; the sampling set in the panel in the request; the cache's type changed between two messages (the question,restarted in 0.1 s; 2 messages kept, both answered); a server started with--temp 0.25 --top-k 11, its defaults in the panel and in the request.- On a device, Qwen3-0.6B Q8_0:
--gpu-layers 20 -ctk q8_0 -ctv q8_0between two messages on OpenCL, restarted in 1.0 s, the panel showing 20 of 28 layers;-ctk q8_0 -ctv q8_0on Vulkan, 0.5 s. Qwen3-8B Q4_K_M in the form with 20 layers asked:On the device: 3.7 GiB of 18.3 GiB, 20 of 36 layers there, as asked. - By hand with xdotool on the CPU: the question answered
Cancel, then Restart with the box
checked, then a restart without the question; Restart
pressed while a reply of 1024 tokens was being written
(
The reply being written is waited for, then the model restarted and the reply stayed); the two cache types set apart and the row's chooser readingother (Advanced); the tabs, their greyed fields and reasons; the form's Layers and MoE CPU on Granite 3.1 1B A400M.
What is left is under T11.
T11, what was left
The entry above left seven things; each was done the same day (gui.md, server.md).
-nand--seedare a server's defaults.oklm serveread both and used neither. They are now what a request leaves out on the three routes that generate:--seed Sis the seed of each request that names none, the same for all, so the same request gives the same text;-n Nthe length of a request that gives none, and/v1/messagesthen takes a request withoutmax_tokens. A request's own value wins.GET /api/contexttells both insampling, and the chat window takes them with the other defaults at an attach.- The index's reranker in the chat window.
--rag-rerankhad a field and went nowhere: the window's server is now given the model, loaded beside the chat model and freed with the server.ragof/api/statsgainedreranker, which the side panel shows. - The memory line on a device. The plan read the
device's free memory with the window's model still on it (about 1.5 GiB
of 20, the server's cache taking what is left) and let everything
through.
device_memoryof/api/statsgainedmodelandheld; a memory plan takes what its caller frees first (oklm_mem_plan_params::dev_reclaim); the line counts it and says so, and its verdict decides again. - The settings kept for each model. Where a model of
the window's own computes and how it samples are kept in the window's
settings file (
gui-servers.json, a membermodelsbeside the servers), read when the window serves that model again, with Defaults to go back. The command line's options win over what is kept, what is kept over the program's defaults, a server's defaults over both at an attach. - The buttons pressed.
oklm-gui --tool servers --pressandoklm-gui --open-serverspress Start server, Re-keep with form and Open in the server manager... without a person;tools/check_gui_form.shchecks the three,make gui-testruns it. - The pictures.
tools/docs/gui_shots.shtakes the server manager's four pictures and four of the chat window's on the canary models; they were retaken with it. - Windows.
make MINGW=1 testsno longer built (test_cli_chat.c, three unused functions) andtest_gui_settingsfailed five checks under Wine (it expected POSIX quotes of a Windows command line): both fixed.
Choices made where the item was open. --seed is the seed
of every request, not of the first one: a server whose replies should
vary is started without it. /v1/messages keeps
max_tokens required unless the server has a length of its
own. What is kept is keyed by the model file's whole path, and holds
what differs from the window's own start, so that a default that changes
later is not frozen in the file. An option given on the command line is
neither read from what is kept nor written to it. Nothing is kept for a
server's model, nor for the models beside the model. A model loaded in
place of another is offered its kept compute settings instead of being
loaded twice. What a server holds is counted for a plan on the same
backend, or on a device of the same size through another one. The
service manager of the Re-keep with form check is a
stand-in, so that no unit of the user's is written.
Checked on 2026-11-15, the device runs on the RX 7900 XT:
test_server, 2592 checks,test_request_defaults: on each of the three routes the same request twice gives the same text and the server's 10 tokens at most,max_tokens3 and 40 win,seed8 gives another text andseed7 the server's; a server without them gives four different replies and refuses a Messages request withoutmax_tokens.test_rag:"reranker":null, then"cross-encoder"with the tiny cross-encoder.test_mem_plan,test_dev_capacity: a device's capacity with what is freed first, 12 cases.test_gui_settings, 441 checks (354 before).make gui-test: the settings kept (three windows), the index's reranker, andtools/check_gui_form.sh tiny, 18 checks;canary, 22 checks, a server of SmolVLM-256M started from the form with every tab set (the server agrees with the form: --backend -ctk -ctv -c --fixed-context --temp --top-k --top-p --min-p --repeat-penalty --seed -n) and one of Granite 3.1 1B A400M with the MoE disk tab.- The memory line, Qwen3-0.6B and Qwen3-8B on Vulkan and on OpenCL:
1.5 GiB free now and the 18.2 GiB the model served now holds; an 8-bit cacheIt fits; Qwen3-8B at 32768 tokens with an 8-bit cache14.3 GiB of 18.3 GiB ... It fits; at 40960 tokens with all layers asked27.3 GiB of 18.3 GiB ... It does not fit; at 262144 tokens the host's 144 GiB against 116 GiB. The server's 18.2 GiB against 18.0 to 18.4 GiB more in use on the card while it served. A real Apply on Vulkan: restarted in 0.6 s with 20 of 28 layers, then settings the line refuses left the model as it ran. - Under Wine 11, the MinGW build: the nine
test_gui_*andtest_mem_planpass,test_server.exepasses with 2575 checks (its Vulkan and OpenCL tests included), and the options' flows of the two windows, run without a pointer on the tiny models, pass 17 checks: the settings kept in%APPDATA%, the index's reranker, Open in the server manager..., Start server agreeing with the form, Re-keep with form on a server kept in theRunkey (gui.md).
What is left is under T11.
T11, the leftovers: a switch with the kept settings, the device options, the scripts, the pictures, the memory line
Five things T11 left, done the same day (gui.md, memory-plan.md, testing.md).
- A model loaded in place takes its kept compute
settings.
/modeland Use this model made the server of the other model with the settings of the one before and offered what was kept for it, for a second load. The switch now stops the window's server and makes one of the other model with its kept backend, layers, context and cache: one load, no question. Settings the memory line refuses, or that do not hold once loaded, are left aside, kept as they are, and the model is loaded with the window's current ones, which the note says; a model that cannot be served leaves the one before. - What was set in the window stays at a switch, and
a message that is not answered says so.
make gui-testfailed in its switch flow: a model loaded in place with nothing kept reset the sampling to the window's start, the reply after it ran to 1024 tokens in a context of 512, and the next message was refused with nothing but a status line the following message replaced. The window's current values now stay where the new model keeps none; a request that is refused or not reached leaves a note,Not answered, in the conversation and a line on the error stream. - The options only a device has, checked from the
form.
tools/check_gui_form.sh device(make gui-device-test, not part ofgui-test): on each device backend that answers, a server at a time started from the form, compared with what it tells and stopped:--gpu-layers 20on Qwen3-0.6B,--n-cpu-moe 8 --expert-gpu 0.25on Granite 3.1 1B A400M,--devices 2where two devices answer.GET /api/contextgainedn_devices,n_cpu_moeandexpert_gpufor it. - The two scripts that clicked.
--pressof the server manager may be given several times and gainedkeep:NAME,profile:NAME,restart:T,unkeep:NAME,ask-rekeep:NAME,dashboard:T,chat:T,yesandno.tools/check_gui_kept.shand the servers group oftools/check_gui_wine.share rewritten on them, a window a step, and hold no place of the form. The first one runs by default against a stand-in service manager (tools/standin_systemctl.py: units and services of the check's own),REAL=1against systemd's. - Every picture of the chat window from the script.
The chat window gained
--press(speech,models:NAME,connect:PORT,link,own,context:N,paste) and--shot-when(resizing,recording,speaking,talking,commands:TEXT), the repository the three files of the examples (tests/fixtures/gui, made bytools/docs/gui_fixtures.py), andtools/docs/gui_shots.shtakes the fourteen pictures it left to a person, on the canary models.gui-chat-qwen2.5-1.5b.png, which no page showed, is deleted. - The memory line judges as the server does. One
function,
oklm_kv_cache_rule(), sizes the cache ofoklm serveand of a memory plan made as a server (oklm plan --serve [--kv-mem MB]): the weights first, then the cache takes what is left beside them on a device, four contexts on the CPU, or the size asked for. The lines of both tools are that plan, with three verdicts: it fits; it fits with a smaller cache (2.4 contexts where four are kept when there is room); it does not fit (less than one context, or memory short). The host memory the model served now holds is counted as the device's is (host_memoryof/api/stats,ram_reclaim). - Cache size. A control of the offload row of both
tools,
--kv-memin MiB, empty forauto; it replaces the field of the Advanced window's Cache tab. Profiles, the settings kept for a model, a command line read back and the table follow.
Choices made where the item was open. The cache's size is in MiB: it
is what --kv-mem takes, and a count of contexts would
change meaning with the Context slider beside it. The
rule is shared by moving it to the model layer, under the server and the
plan: neither holds a copy. On a device the server's cache is everything
that is left, not four contexts at most: the line says how many contexts
that is (a cache of 23.8 contexts), and fits
means four or more. A size asked for above the device's room is made as
asked, the context then placing fewer layers there, so the line judges
it by the memory, not by the room. oklm plan without
--serve means what it meant. A switch asks no question: it
is the decision. The stand-in service manager really starts and ends the
kept server, so that the checks of Stop,
Start and Restart hold.
Found on the way and fixed. GET /api/context told as
cache_tokens the cache that would be made now beside the
one in place (4080 tokens on a device whose memory the cache of 97664
held): it tells the cache the server has. A restart of the chat window's
model on a device made its server before the driver had given back what
the stopped one held, and got a cache a quarter smaller than the line
said (75184 tokens for 97664): the restart waits for the device, three
seconds at most.
Checked on 2026-11-15, the device runs on an RX 7900 XT (and an Intel UHD 750 for the second device), one at a time:
test_mem_plan, 8549 checks with the devices (test_kv_cache_rule,test_plan_serve);test_gui_settings, 501 checks (441 before:test_cache_size, the device members of a server's account);test_server, 2586 checks; the other eighttest_gui_*pass.make gui-test: the switch with the length set kept (three requests withmax_tokens8), a context that is full (not answered: error 400), a model loaded in place with its kept settings and with settings that do not hold, Cache size in the form (the option, the line, the profile, a size below a context refused) and in the chat window (the model restarted with the cache planned, 8192 tokens; a size refused), the chat window's presses and--shot-when.make gui-device-teston OpenCL and on Vulkan: the five starts of the device options; Qwen3-8B Q4_K_M at 40960 tokens, which the line refused before,It fits with a smaller cache: 2.4 contexts, the server'scache_tokens97664 as planned (0.0 % apart), the chat window's Apply restarting with the same cache; at 131072 tokensIt does not fit, Apply refused; Cache size 4096 the server's cache to the token; a model whose kept-c 131072does not fit loaded with the window's own.sh tools/check_gui_kept.sh, 21 checks against the stand-in; the servers group oftools/check_gui_wine.shunder Wine 11 in a prefix of its own, 9 checks.sh tools/docs/gui_shots.sh: the twenty-two pictures taken.
What is left is under T11.
The canary set of published models
Until 2026-11-15 models/ held about 130 GB of published
files and build/ref 15 GB of recorded references, and the
tests compared every one. The disk was cleared that day (housekeeping). What the tests
compare from now on (testing.md):
- The canary set,
tests/canary_models.txt: 66 files, 10.15 GB, one per kind of model. SmolLM2-135M (Q8_0, Q4_K_M, F16 and its checkpoint), Qwen3-0.6B Q8_0, Llama-3.2-1B Q4_K_M, Qwen3-8B Q4_K_M (the default model, Qwen's own file as the catalogue pins it), Granite 3.1 1B-A400M Q4_K_M (routed experts), LFM2-350M Q4_K_M (convolution blocks), SmolVLM-256M Q8_0 with its projector, Whisper base, OuteTTS 0.2 Q8_0 with WavTokenizer, bge-small-en-v1.5, ms-marco-MiniLM-L6-v2, and the small files (tokenizers, templates, the sharded tiny GPT-2). It stays on the disk;make testcompares it at every run. - Every other family is verified when its code is touched: its file fetched, its reference recorded, the comparison run once, the file removed. Its standing record is the figures of its entry in this file.
- Where a file is pinned.
tests/public_models.txtandtests/public_models_large.txtstay the only lists with a sha256 and a URL; the large one gained the canary's files it did not hold (Qwen3-8B, Granite, LFM2-350M, SmolVLM-256M, Whisper base, OuteTTS 0.2 and its vocoder,samples/jfk.wav). A set is a file of destinations picked from them. - Nothing large is fetched by accident.
sh tools/fetch_models.shfetches the canary set (it fetchedtests/public_models.txt); a list, a family (--family gemma3-1b), another set (--set) or both lists (--all) are asked for by name.make test-publicis the only make target that downloads a model and takes the canary set. - CI downloads no model in the jobs a push starts.
The manual
public-modelsjob takes a set of its own,tests/ci_models.txt: 28 files, 214 MB (SmolLM2-135M Q8_0, 145 MB, and the tokenizers, templates and tiny files). It tooktests/public_models.txtwhole before, several GB with Qwen3-4B. - The references recorded again with the pinned
llama.cpp and the Python environment, 2.5 GB under
build/ref: gguf-py's dequantization of the three SmolLM2 files, the tokenizers' ids (11 tokenizers) and the trainer's merges, llama.cpp's tokens and logits of the eight canary language files, the float64 passes of SmolLM2 (three types), Granite and LFM2-350M, llama.cpp's embeddings of SmolVLM-256M's tile, jinja2's renderings of nine templates. - Measured on the canary files
(
test_infer_public, CPU, against llama.cpp on its own tokens, largest logit difference and choices of 160): SmolLM2 F16 0.044 and 160, Q8_0 1.77 and 157, Q4_K_M 2.13 and 157; Qwen3-0.6B Q8_0 0.89 and 159; Qwen3-8B Q4_K_M 1.61 and 159 (the published file, where the earlier figures were those of the file made bytools/make_qwen3_8b.sh; the same bounds hold); Llama-3.2-1B 1.03 and 156; Granite 2.91 and 157; LFM2-350M 1.2 and 157.oklm quantizeis byte for bytellama-quantizeon the 22 types of the F16 file, on six requantizations of Qwen3-0.6B and with an importance matrix. - Skips name their file. With the canary set alone
every test binary passes, and a part that wants another file prints the
path it misses. Two parts skipped without a word and now name it: the
published templates of
test_jinjawhose rendering was not recorded, and the comparison of a Qwen2.5-VL answer with the model's own code intest_mm_public.tools/gen_infer_ref.shstopped at the first SmolLM2 type that is not there and now names it and goes on;tools/check_quantize.shnames the importance matrix it misses.
The roadmap cleared of finished work
roadmap.md held, inside its open items, the account of what had been done on them: it now lists what is left, each item with a link to its entry here, and the leftovers of L1, L3, L5, L6, L7, L8, L9 and T10 are grouped by theme instead of by the entry that left them. No code changed and nothing was measured.
- Taken out as finished, each with a line under the roadmap's "Closed": section 3 (RAG, M17), T0 to T9 of the graphical tools, L4 (DeepSeek-Coder), and L9's entries that left nothing (Qwen2.5-VL's windowed blocks, gpt-oss-20b's perplexity between the backends).
- Decisions, no longer listed as leftovers (the
roadmap's "Left out on purpose" has a line for each): 4 096 embeddings
at most for Qwen2-VL and Qwen2.5-VL,
--fps 1 --frames 8and the 4.49.0 form of Qwen2.5-VL's time, no partial sums in OpenCL's tiled products, the wide products as Vulkan's default for the encoders, LLaVA's encoder as the model defines it. - The order of work of 2026-11-13 was followed to its
end but for G6, I4's other architectures and G5's
glm5-next, which the roadmap keeps with the files and measures that wait for disk. - Judged out of date and dropped: "the server not run
with an image on a device" under L1's LLaVA and Gemma 3 (the published
files went through
oklm serveon OpenCL and Vulkan for the stream gaps; what stays is that their answers there were not compared). - Done facts the roadmap alone held, kept here:
- The choices taken when the graphical tools were planned (2026-11-12): one binary rather than several; the servers and jobs the tools start are plain background processes (they end with the session, no system service); abliteration sits in the training tool, behind an advanced tab; the inspector shows the static layout and the live figures a server already serves, not per-neuron activations; Linux first, Windows after.
- The choices taken when the chats were planned (2026-11-13): speech both ways (recognition and synthesis); the microphone as well as dropped recordings; the window reads documents itself, whether its server is its own or a remote one, and only the dashboard uploads them to a route; as many commands as are useful, the agent's included where they make sense in a plain chat. Out of date in the chats' "Left" and dropped: "a device path for the vocoder" (the vocoders' networks are on the device since 2026-11-14).
- F1 (2026-11-11): the one read-back is done, and streamed layers keeping their experts was measured as the faster choice. H4 (2026-11-11): the next router on the device.
- F2 (2026-11-11): Granite at two threads under full load, once measured slower with the free-core cap (51 -> 45 t/s), is level or ahead over eight runs each (44.3 against 40.6): the runs' spread (benchmarks.md).
- L9, the video of Qwen2-VL 2B on a device (2026-11-14): the clip of 384 embeddings and 418 prompt tokens, its describe prompt in 0.47 s on OpenCL and 0.36 s on Vulkan against 8.59 s on the CPU.
- L9, the Intel UHD 750 (2026-11-14): Qwen2-VL 2B's tokens at 15 (Vulkan) and 4 (OpenCL) a second against 28 on the CPU.
L9. The device and server leftovers, and the tables measured again
What entries 4, 8, 17, 18, 19 and 20 of L9 still left on the devices and in the server, taken in the evening of 2026-11-14 and the night after it, each measured first; then the tables the day's changes had left behind, measured again on the last code.
- WavTokenizer's group normalizations and attention block on
the device (entry 8;
src/mm/vocoder.c, speech.md). No op added. A frame holds a group's channels side by side, so a view [channels a group, frames, groups] of the stream names them: a copy through it lays each group out as one row, a layer normalization of those rows is the group's mean and variance over its channels and every frame, a copy through the same view of the destination puts the values back, and the scale and shift by channel are a product and a sum. The one head of 768 values, wider than Vulkan's attention shaders take, is two products, a softmax and a copy on both backends: the keys read as a matrix times the queries, the softmax of each frame's scores, the values copied by channel times the weights, over the frames rounded up to 32 with rows of zeros. Nothing is read back but the spectrum. The network for a sentence of 7 s (507 to 527 frames): 43.9 to 44.6 ms -> 14.2 to 14.3 ms on Vulkan, 32.9 to 33.8 ms -> 13.5 to 13.7 ms on OpenCL; the 16-bit files one step from those with the normalizations on the CPU in 207 and 272 samples of 168 640 and 162 240; in floats 124 dB (OpenCL) and 110 dB (Vulkan) from the CPU's vocoder (test_speech, which asks for 80). The two forms are checked as ops against the CPU's arithmetic (check_group_views()intest_conv_views: RX 7900 XT and Intel UHD 750, both backends). - A text encoder's batch in one launch a block (entry
4;
src/model/arch_bert.c, opencl.md, rag.md). The profile ofoklm embedon 364 lines showed a launch a sequence and a block on OpenCL's first attention kernel, 4 308 of them, a third of bge-small's kernel time. A batch of several sequences is now one op on a device, each row given its sequence's rows as its span of keys (oklm_attention_spans(), the table after the positions in the context's buffer); host memory and a device whose kernels do not take the op attend sequence by sequence, to the CPU's bits. bge-small: 4 368 launches and 55.8 ms of attention -> 108 launches and 11.0 ms (0.162 -> 0.118 s of kernel time); nomic-embed-text: 81.2 -> 21.7 ms (0.380 -> 0.331 s); the command 0.76 -> 0.71 s and 1.15 -> 1.08 to 1.10 s on OpenCL, 0.25 to 0.36 -> 0.23 s and 0.59 -> 0.57 s on Vulkan; the vectors 2.7e-7 to 4e-7 from the CPU's. - The single rows of Whisper's cross-attention, by the figures (entry 4): a launch reads the 1 500 keys and values of 20 heads of 64, 15.4 MB in F32, in 38 us: the memory's rate. Large-v3 on jfk.wav: 2 112 launches for the decoder's self- and cross-attention, 47.7 ms, and 1 056 merges, 5.4 ms, of 0.49 s of kernel time. No kernel changed.
- Intel's OpenCL on the UHD 750, by the figures
(entry 17; multimodal.md): of
the 196 s of Qwen2-VL 2B's prompt, 92.5 % are matrix products
(
gm_f16115 s in 194 launches,gm_q4_k55 s,gm_q6_k10.7 s) and 4.2 % the encoder's bidirectional tiles; nothing is handed back to the CPU. WithOKLM_CL_GEMM_TILES=0the row kernels take as long (195.6 s): about 17 GFLOP a second either way. No kernel changed. - Published files on the UHD 750 with the encoders' ends on
the device (entries 9, 10 and 21; one run each, 2026-11-15):
Whisper base on jfk.wav, the CPU's text on both backends, 2.75 s on
Vulkan and 5.41 s on OpenCL against 0.37 s on the CPU; SmolVLM-256M, the
overview tile's embeddings 2.3e-6 (Vulkan) and 2.6e-6 (OpenCL) from the
CPU's, 47 and 48 of 48 greedy choices the CPU's in the test, its
oklm describeprompt in 18.3 s and 59.0 s against 2.9 s (the answer the CPU's for ten words, then another reading, the same on both backends); Qwen2-VL 2B, the CPU's 24 greedy tokens on both backends, its prompt in 40.2 s and 196 s against 6.5 s. - A stream of passes, and a recognizer's cross keys and
values (entry 18;
src/core/backend.c,src/server/sched.c,src/mm/audio.c, server.md). A device was shared for a yield while a pass waited or for a quarter of a second after another thread's pass. A thread can now open a stream of passes on a device (oklm_backend_pass_stream_all(),oklm_context_stream()), and the scheduler keeps one open on its model's devices while it has requests to decode: a model slower than four steps a second is seen between two steps (test_pass_stream). The keys and values of the cross-attention of a recognizer's 32 decoder layers are cut after each layer's: the largest gap in a stream while jfk.wav arrives for large-v3 0.024 -> 0.013 s on Vulkan and 0.031 -> 0.018 s on OpenCL. - The gaps not measured (entry 18): Qwen2.5-VL 7B 0.16 s (Vulkan) and 0.17 s (OpenCL), the step of its 391 image rows; LLaVA 1.6 0.047 and 0.055 s; SmolVLM2-2.2B 0.017 and 0.024 s; a picture of 3 024 embeddings for Qwen2-VL 2B 0.22 and 0.19 s, one block of its encoder; a sentence of OuteTTS 0.2 with its vocoder's call 0.025 and 0.030 s; Ultravox 0.020 and 0.022 s. One gap of 0.30 s was seen once on OpenCL with LLaVA 1.6 and once with the large picture.
- A spread context's pass cut between its devices
(entry 20;
forward_yield()insrc/model/model.c): where the stream goes from one device's layers to the next's it is in host memory, and the pass lets the waiting passes of every device in. Qwen2-VL 2B over the RX 7900 XT and the UHD 750, a stream while three images arrive: a request with a new image 7.77 to 7.78 s -> 7.20 to 7.23 s; the stream's largest gap, its own step through the UHD 750's layers, 5.71 s before and after. - The device memory counted (entry 19;
src/server/server.c, server.md). A backend says what it holds of the device for itself and what that is left (oklm_backend_iface::own_memory: 352 MiB on Vulkan, 64 MiB on OpenCL); with an encoder or a sound model on the device, the part not held yet comes out of the cache. Gemma 3's first picture took 297 MiB of Vulkan that no count held and now leaves nothing pending; Qwen2-VL's picture of 3 024 embeddings takes 51 MiB over the count where it took 351. The embedding and the reranking model are counted and shown (embed,rerankindevice_memory): bge-small 154 MiB counted, 161 and 164 taken; MiniLM-L6 113, 121 and 124 (oklm_context_device_bytes()priced a text encoder a cache it does not keep: 729 MiB for bge-small before). A vocoder's activations are counted as the twelve a call makes, at its longest call as before: OuteTTS 0.2 with its vocoder 1 161 MiB counted and 1 070 to 1 078 taken, Soprano 348 and 270 to 274. - The tables measured again (RX 7900 XT, one job at a
time, the load average with each): the stream gaps of server.md, every row on both
backends (Gemma 3 4B 0.076 and 0.093 s, Qwen2-VL 2B 0.042 and 0.049 s,
LLaVA 1.5 0.19 and 0.19 s, Whisper large-v3-turbo 0.014 and 0.018 s,
Soprano 0.016 and 0.017 s); the prompts of
oklm describeand the encoders alone on the CPU, OpenCL and Vulkan, the large pictures, Whisper's window and Ultravox (multimodal.md, multimodal.md); theoklm speaktable (speech.md: a sentence of 7 s in 1.48 s on Vulkan and 1.90 to 1.92 s on OpenCL against 13.7 s on the CPU, the vocoder 0.02 s of it); the captions of the nine figures (rag.md: 7.3 to 7.4 s on Vulkan, 9.0 to 9.1 s on OpenCL). - Checked (RX 7900 XT, once on the last code):
test_threads,test_multi_device,test_vision,test_audio,test_speech,test_soprano,test_embed,test_rerank,test_mem_plan,test_serverandtest_mm_publicwhole;test_conv_viewsand the tiny vocoder on the Intel UHD 750 too.
Left: in the roadmap under L9, entries 4, 8, 17, 18, 19 and 20. Not
run under ThreadSanitizer: the stream counter of the pass lock
(pass_lock.m, taken after registry_lock, by
the scheduler under its own lock), the yield of a spread pass (the
devices' pass locks given back in the reverse of their order and taken
again in it), and the backends' run locks in
own_memory.
L4. DeepSeek-Coder 6.7B against a float64 pass
What L4 left. tools/infer_ref.py (numpy float64, the
weights dequantized by gguf-py, activations not rounded) ran TheBloke's
deepseek-coder-6.7b-instruct.Q4_K_M.gguf whole: the model
in float64 is 54 GB and the machine has 124 GB, so no layer-by-layer
pass was needed. The first three prompts of llama.cpp's recording, eight
greedy tokens each, as the 1.3B's reference was made
(tools/gen_infer_ref.sh now names the file,
build/ref/infer/deepseek-coder-6.7b-Q4_K_M.json). On the
CPU (tests/model/test_infer_public.c,
check_model_if(): skipped without the file) OKLM makes 23
of the 24 choices of the pass, logits within 6.99: three steps are 6.69
to 6.99 from it and the other 21 between 0.33 and 1.00; the choice that
differs is at a step where the pass's own top two are 0.03 apart.
llama.cpp's recording against the same pass, on the 22 steps where its
tokens are the pass's: 21 choices, logits within 6.98 (6.41 once a
step's common shift is removed), the same step differing. On a prompt's
first step OKLM is 1.00, 0.59 and 0.54 from the pass and llama.cpp 0.86,
0.69 and 0.61: the two programs are equally far from the model computed
without rounding, as on the 1.3B. The pass's logits span -11.6 to 37.6.
The file was on the disk from 2026-11-14's device runs and stays there.
Left: nothing of L4.
I4. The audit of colibrì's other architectures, and MiMo-V2.6's distillation
I4's list was looked at one model at a time, as L5's was: the first
24 to 32 MB of each published GGUF file (its metadata and tensor table),
the pinned llama.cpp's sources and the names of colibrì's. The Qwen3.6 /
3.8 hybrids of the list were already running (qwen35,
qwen35moe, L5 and L6); the roadmap's line said otherwise
and is corrected. The others (mimo2, inkling
and inkling2, deepseek4,
deepseek41, kimi-k3,
mistral-large4) are each a new architecture, and none has a
published file under 74.8 GB (Inkling-Small at UD-IQ1_S;
DeepSeek-V4-Flash 82.5 GB, MiMo-V2.6-Flash 105.9 GB, the others 233 GB
and more) where the disk had 28 to 32 GB free: the table, with what each
adds, what the pinned llama.cpp has for it and the order to take them
in, is in the roadmap (Scoping of the open
items). Two findings beside the sizes: the pinned llama.cpp knows
neither inkling nor the name mistral-large4
the published Mistral-Large-4 files carry (its mistral4 is
deepseek2's graph and lacks the indexer, the scalable
softmax and the shared keys those files have), so these two have no
second program to be compared with; and mimo2 is the
smallest to write, the llama block with a value width of its own.
The one file of the list that fits: MiMo-V2.6-Distill-Qwen-9B
(bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF, Q4_K_M, 5.84 GB,
SHA-256 4bca6f18...8c81), Xiaomi's fine-tune of Qwen3.5-9B
on text written by MiMo-V2.6, a qwen35 file of the 9B's 427
tensors with a chat template of its own. No code changed. On the CPU
against the pinned llama.cpp (tools/gen_llama_ref.sh,
tests/model/test_infer_public.c): 5 of 5 prompts tokenized
alike, 158 of 160 greedy choices on llama.cpp's tokens, logits within
2.85, three of the five continuations identical (120 of 160 tokens
before parting) and the others parting where llama.cpp's own two logits
are 0.08 apart. Its template renders as jinja2 renders it on the 12
conversations of tools/gen_jinja_ref.py
(tests/model/test_jinja.c). oklm chat answers
a question after an empty thought and stops; with --tools
it writes
<tool_call>{"name": "get_weather", "arguments": {"city": "Paris"}}</tool_call>
after a thought, parsed in the hermes format. 5.3 tokens/s
over 64 tokens on a shared machine. In the catalogue as
mimo-v2.6-distill-qwen-9b (code; MIT), marked run; the
generator ran whole and moved no other entry. The file was removed after
its runs. Left: OpenCL and Vulkan on this file (the GPU was in use; the
9B's layout runs there), its vision projector, and every architecture of
the table.
I4. mimo2 on a tiny
model
The first of I4's architectures, taken because it is the smallest to
write; no published file of it fits the disk (105.9 GB at least), so it
is checked on a tiny model and on the header of a published file, as
glm4moe and muse-glimmer first were.
- What it is (llama.cpp's
src/models/mimo2.cpp, MiMo-V2.6-Flash's metadata): the llama block with one fusedattn_qkvwhose queries and keys have heads of 192 and whose values have heads of 128; a window of 128 positions and a count of key heads given block by block (a first block without window, then five window blocks out of six; 4 key heads without window, 8 with); the window blocks with their own rotation base and attention sinks; the projected attention output times 0.707; one dense block, then 256 experts, 8 a token, chosen by sigmoid with a selection bias. - How (
src/model/arch_mimo2.c,src/io/hf_config.c, inference.md): the attention op and the cache take one head size, so the values are written into heads of the keys' size, zero past their own, and the first places of each output head are given to the output projection: the same sums, exactly, with no change to an op or to the cache. The two lists are read as leading blocks without window (n_swa_lead_full, new), a period, and a count of key heads per kind of block (gemma4's fields); the test of a window block is now one function (oklm_hparams_is_swa()). The count of dense blocks is taken from the tensors. A context asked on a device runs on the CPU and says so. - Proof.
tiny-mimo2(seven blocks, heads of 16 and values of 8, 1 and 2 key heads, a window of 4, sinks, a dense block then 8 experts) against the float64 reference (tools/infer_ref.py, which now has the architecture): 36 / 36 greedy choices, logits within 1.4e-5; with an F16 cache 36 / 36 and 0.042; batches and several sequences on the contiguous and the paged cache equal to single tokens bit for bit; a KV file saved and loaded, contiguous and paged, gives the same logits; the memory plan equals the context's allocations (tests/model/test_model.ctest_mimo2_model,test_kv_file.c,test_mem_plan.c). The pinned llama.cpp on the same model (build/llama_ref): the reference's 36 / 36 tokens, logits within 0.062; its cache is F16 for this architecture with or withoutLLAMA_REF_KV=f32, and the float64 pass with queries, keys, values and attention weights rounded to F16 is within 1.2e-3 of it. With other seeds the experts' choices flip under that rounding in both programs alike (llama.cpp 0.846, 1.26 and 0.51 from the pass where OKLM with an F16 cache is 0.848, 1.27 and 0.515), which is why the fixture's seed was chosen. The CPU's other fixtures are unchanged (14 architectures within their bounds, gemma4's three and the two gated ones with their caches and batches). - The published file, without its weights. The first
megabytes of both parts of
MiMo-V2.6-Flash-RL-Q2_K(126.4 GB), completed by holes to the listed sizes:oklm infosays loadable (472 tensors, 308.8 B parameters, Q2_K, MXFP4, Q8_0); the model loads and every tensor binds (48 blocks, 64 heads, 4 and 8 key heads, heads of 192 and 128, 64 rotated, a window of 128 on five blocks out of six after the first, bases 1e7 and 1e4, 256 experts of 2048, one dense block);oklm planat 8 192 tokens: 117.68 GiB of weights (111.62 GiB of experts), 120.48 GiB in all against 124.64 GiB of memory with 3.78 kept free, "does not fit", and 8.85 GiB resident with the experts streamed from disk. Its chat template renders as jinja2 renders it on 12 conversations. Its multi-token prediction file is refused with a message. - Left: in the roadmap under I4 (a published file run, the devices, the tokenizer on the published vocabulary, the tool-call form, the prediction blocks).
The chats: the small leftovers
Done 2026-11-15: what the "Left:" lines of C1 to C9 held that a machine without a person at a microphone, and without Windows or Safari, can do and prove (inference.md, gui.md, server.md, speech.md).
- One table for the voice's pause (C1).
/pauseis a command of the dashboard's chat as of the window's, and/speaktakespauseandresumein the window as in the page: one row each for every chat that reads aloud.test_chat_commands,test_gui_commands,test_gui_speak, the page's parser cases (tests/server/dashboard_cases.tsv), and the page driven in Chrome (CHECKS=voice sh tools/check_dashboard.sh:/pauseon a held voice, then on a playing one). - Tab in
oklm chat's line mode (C1). On a terminal, line mode reads key by key with the full-screen prompt's line editor: Tab completes a command's name and its argument, the matches listed under the line; the arrows move in the line and through the lines sent before. Piped input is read whole lines as before.tools/check_chat_tui.shunderscript:/hecompleted and run,/moto/modeland its two commands listed,/export jto/export json, Up bringing a line back, the agent's keys (Ctrl-T, Shift+Tab) doing nothing there. The same check's model selection is run on 200 rows: the catalogue had grown past the 50 of its terminal, and its last group's title was never drawn. --rag-*with--rag-diralone (C2).oklm serve --rag-dir DIR --rag-k 1 --rag-mode sparse --rag-crag offis accepted and the directory's indexes are opened with those settings; the other commands still refuse them without--rag.test_cli_rag, andtest_rag_dirintest_server: an index of two passages gives one withrag_k1 and both without.- Files copied in a file manager, pasted (C3). Ctrl+V
in the chat window asks the clipboard for a list of files
(
text/uri-list, thenx-special/gnome-copied-files) before the text targets and attaches eachfile:address of this machine, as a drop does; a list without a local file is left to the field that pastes text.att_clip_next()intest_gui_attach; under Xvfb withtools/gui_copy_files.pyowning the clipboard as a file manager does, a PDF and a PNG copied and pasted were both in the row, and a copied text went into the message. X11 only. - A question about the whole document (C4).
"Summarize this.", "What is this file about?", an outline, the main
points (
oklm_ingest_whole_question(): English words, andrésum) no longer get the passages nearest those words: the chat window (docs_find()) and the server'sPOST /api/ingest/search("spread", told by the question unless the request says) give passages at equal steps over the document, its first and its last among them, as many as the room orkandmax_tokenstake; the page's line above them says so.test_gui_docs(the first and the last passage, in order, no gap above twice the mean, within the room) andtest_ingest_retrievalintest_server(five passages ofk5 at chunksi (n - 1) / 4, fewer in a budget,"spread": falsefor the nearest). - A sound for a model that hears, from the dashboard
(C5). A group of the page's check (
CHECKS=sound): on a server with Ultravox's projector over Llama 3.2 1B, on the CPU,jfk.wavdropped on the chat goes as aninput_audiopart of base64 before the text, is encoded once for two questions, and the model repeats it: "And so, my fellow Americans ask not what your country can do for you, ask what you can do for your country." A picture is refused there. - Translation from the window (C7). Write
what is said in English in the Speech window,
--transcribe-translateon the command line: Whisper's translation task on the window's own model,POST /v1/audio/translationson a server.test_gui_listen(the path);make gui-teston the tiny Whisper fixture:language found: French (fr), 75 % sure, written in English. - A pause at every sentence's end (C8). OuteTTS's
passes no longer join the sentences that hold together in 24 words: a
sentence's end is always a pass's end, with its 0.3 s of silence when
another sentence follows under the prompt, so a text is as long with
--first-wordsas without. The tiny fixture's "Hello there, world. One two three." is 14 540 bytes with the option absent, 2 or 5 (it was 140 bytes without it); the published OuteTTS 0.2 on the 64-word text, its language model on OpenCL: 1 016 684 bytes and 21.18 s both ways, 2 prompts of 1913 tokens, a first sound after 0.86 s (five passes) and after 0.56 s with--first-words 5(six), the text after 5.7 s. A sentence the model does not go on to is spoken under a prompt of its own, where it was lost inside a pass: OuteTTS 0.3 on "Hello. Yes." with the default sampling (the commit before, built and run again the same day: 1.71 s under one prompt, Whisper large-v3 hearing "Hello."; since: two prompts, 2.77 s, "Hello. Yes."). At temperature 0, which this entry first named, the sound of 0.3 is close to silence before and after and no recognizer hears a word in it: that run shows the second prompt (one prompt and 1.32 s before, two and 2.22 s after), not a sentence spoken.test_speech,test_speech_routeintest_server. - One speech model in the examples (C8).
outetts-0.2-500m: the Speech section of server.md and the README's device example named OuteTTS 0.3. speech.md says why it and not Soprano (the examples about voices need speaker files, the transcripts were made with it, 24 kHz, and its language model runs on a device since L6), and a Soprano line stands beside the examples that start a server or a window.
Run once on the last code: test_chat_commands,
test_gui_commands, test_gui_speak,
test_gui_attach, test_gui_docs,
test_gui_listen, test_gui_talk,
test_cli_rag, test_cli_chat,
test_ingest, test_speech,
test_soprano and test_server whole;
tools/check_chat_tui.sh;
tools/check_dashboard.sh with
CHECKS="chat docs voice sound" in Chrome;
make gui-test.
Left: in the roadmap under C1 to C9. Not done among what was looked
at: a picture or a sound attached in the terminal chat (its turns were
token ids compared with the cache's, where a picture's rows need
oklm describe's evaluation: a rewrite of the chat's turn,
which is the next
entry); the window's index of a long document kept between windows;
the words while speaking (the recognizer reads a whole recording); a
model switch in a running server.
The chats: the model switch, the words while speaking, the documents kept
Done 2026-11-15, after the small leftovers: what the "Left:" lines of C1 to C9 still held that this Linux machine can do, each with its run (server.md, gui.md, speech.md).
- A model switch in a running server (C2).
POST /api/model {"model": NAME}loads another installed model of the catalogue, or a file of the model directory, in place of the one served (oklm_server_switch()). A reload in place rather than the server manager's stop and start: the server already had a gate every request enters under and a resize closes, and a scheduler that alone holds the context, so the switch is a resize that also replaces the model. The new file is opened and checked first (it loads, its template reads, a draft model shares its vocabulary, its projector is its width), the requests under way are waited for (wait, three seconds by default) or the switch answers 409, the scheduler, its cache and the old model with its weights on the device are freed, and the scheduler is made again with the server's settings: backend, context (a resize's included), cache budget, the encoders' share and its refusals, the draft model. A model that cannot be served leaves the one before loaded again and serving (507).oklm serve --fixed-model,--loraand--lora-modulekeep the model. The dashboard's/model NAMEand a model control beside the chat ask for it and the conversation goes on; the chat window's/modeland Use this model load the model in the window's own server in place of starting the window again, and ask a server at an address that says it switches. On the RX 7900 XT (OpenCL,-c 4096): Qwen3 0.6B to Qwen3 4B in 7.96 s, back in 2.81 s, SmolVLM 256M with its projector in 0.31 s; a stream of 400 tokens running,"wait": 0.2answered 409 and the stream ended whole,"wait": 120waited and loaded; with--kv-mem 12000and a Whisper large-v3 still to load on the device, Qwen3 8B Q8_0 was refused by the memory plan and Qwen3 0.6B answered the next request.test_model_switchintest_server, the page'sswitchgroup,make gui-test. - The words while speaking (C7, C9).
partial=trueon/v1/audio/transcriptionsand/v1/audio/translations: the text of a recording still coming, never made to wait (503 at once while the recognizer works), counted apart. The dashboard and the chat window send the recording so far every two seconds of new sound (twice the last answer's time when that is more), show the words two answers agree on as said and the rest as heard, and past twenty seconds keep the segments said and send the sound after them; a turn of a spoken conversation shows them too. The final text is the whole recording's, transcribed once at its end, as before. Cost (tools/bench_partial.py, the recognizer's seconds per second of recording,jfk.wavof 11 s and a French fable of 82.9 s): Whisper base 0.14 and 0.27 on the CPU, 0.030 and 0.061 on OpenCL, 0.024 and 0.050 on Vulkan; large-v3-turbo 0.15 and 0.19 on OpenCL, 0.12 and 0.15 on Vulkan, and on the CPU 0.92 and 0.52, asked 2 and 6 times only, a word waiting up to 33 s: it cannot follow a voice there. A word is shown 2.1 to 3.0 s after it was said otherwise.test_partialintest_gui_listen,test_transcriptionsintest_server, the page'sspeechandtalkgroups,make gui-test. - Documents (C4). The chat window keeps a long
document's index between windows, in
~/.cache/oklm/docs, named by the text's hash and the embedding model's identity:docs/rag.md(180 KiB) indexed in 5.0 s with bge-small by a first window, found by the second with nothing embedded. An installed reranker (or--rerank-model) puts the 24 passages found first in its order, in the window, and on a server started with--rerank-modelforPOST /api/ingest/search("reranked",rerank_score). The window tells the model when the passages are spread over the document for a message about the whole, as the dashboard's line does. An attached agent tells its context inGET /agent/stateand is no longer sent the whole text of a document longer than what is left: 16 passages of 295 (14 871 bytes) where 184 123 bytes had answeredsize overflow. A fault found on the way: a passage's line of place was counted 12 tokens whatever it held, and 19 passages under long headings were refused by the server for a tenth too many tokens; it is counted by its length.test_cache,test_rerankintest_gui_docs,test_ingest_retrievalintest_server,test_listen,make gui-test. - The OuteTTS tables measured again (C8). With every
sentence a pass of its own, on the CPU, OpenCL and Vulkan (speech.md):
the 64-word text with OuteTTS 0.2 is 20.86 s of sound on the CPU (four
threads, load 3 to 5), 21.18 s on OpenCL and 20.75 s on Vulkan, 2
prompts of 1913 tokens everywhere; its first sound comes after 6.6 s,
0.87 s and 0.65 s (4.6 s, 0.58 s and 0.44 s with
--first-words 5), the whole text after 40.9 s, 5.8 s and 4.3 s: 0.51, 3.7 and 4.8 seconds of sound a second. Soprano's Q8_0 file was measured beside it (12.6 to 13.1, 39 and 44 to 47), its F16 file is not on this machine. The notes that said what had moved are gone. And a claim of the entry before corrected: OuteTTS 0.3 on "Hello. Yes." was said to have dropped the second sentence at temperature 0 before the cut and to speak both after. The commit before the cut, built and run: at temperature 0 the sound is close to silence before (one prompt, 107 tokens written, 1.32 s) and after (two prompts, 156 tokens, 2.22 s), and Whisper base, large-v3 and large-v3-turbo hear no word in either; what the cut changes there is the second prompt. The drop is real at the default sampling: before, 1.71 s under one prompt and Whisper large-v3 hears "Hello."; after, two prompts, 2.77 s, "Hello. Yes.". - Translation from the window on real speech (C7). A
French fable read for LibriVox (public domain, 82.9 s; downloaded for
the run and deleted):
oklm-gui --transcribe-model whisper-small --transcribe-translate --mic-file, the window's own server on the CPU, wrotelanguage found: French (fr), 98 % sure, written in English: The Renard and the bust. Fable number 14 of the Fontaine, book 4. ... It was a hollow bust and bigger than nature. ... How many great masters are busts in this point? ...in 6.3 s, the partial texts in English meanwhile;POST /v1/audio/translationsgave the same text with Whisper small (0.78 s on Vulkan) andthe fox and the bust fable number 14 of la fontaine book iv ... it was a hollow bust and larger than nature ...with large-v3 (1.92 s). - Files copied in the Explorer, pasted (C3). The
Windows clipboard's
CF_HDROPlist is read as a list of addresses (att_hdrop_uris(), which reads the block as bytes and is checked on every system:test_hdropintest_gui_attach) and attached as X11's is; the Windows glue that copies the block (gui_clip_files()) is cross-compiled, not run.
Run once on the last code: test_server whole,
test_gui_commands, test_gui_listen,
test_gui_docs, test_gui_attach and
test_listen; make gui-test;
tools/check_dashboard.sh with
CHECKS="chat served switch keys docs speech voice talk" in
Chrome; oklm.exe, oklm-gui.exe and those tests
built with MinGW, not run.
Left: in the roadmap under C1 to C9. Not done among what was looked
at: the line editor of oklm chat's line mode on Windows
(console modes and waiting on the console handle inside the terminal
view of apps/oklm.c: nothing of it a unit test reaches
without a Windows console).
C2. A picture, a sound or a video in the terminal chat
Done 2026-11-15: oklm chat takes a projector and
/attach a picture, a sound or a video, kept in the
conversation with its rows in the cache (inference.md, multimodal.md). It was
the item C2 left in the terminal: /attach took documents
only.
- The projector.
--mmproj FILE|NAME|none, with--tiles,--fps,--framesand--ffmpegasoklm describehas them; an installed catalogue model that carries a projector gets it without the option, as in the chat window. One whose embeddings are not of the model's width is refused at start. - The chat's turn. A turn's prompt was the
conversation rendered and tokenized, its ids compared with the cache's.
It is now rows (
chat_rows()inapps/oklm.c): without a projector the same tokens; with one, each message's media as markers written byoklm_mm_question(), the conversation byoklm_mm_format_chat(), and the rows byoklm_mm_build(), the functionsoklm describeand the server's chat routes write and build a turn with. A media row is compared by its item's content hash and its index within it; a media item is never cut by the common prefix; what follows it is evaluated as its model reads it (tokens, plain rows, rows that see each other, an image's grid, a video's grids). No third path: the files are decoded bydescribe_turn_load(). - Encoded once. An item's embeddings are kept by its
hash after its first turn, the item is named by the hash
(
oklm_mm_item.hash) and its pixels or samples are freed; the same picture attached again is not encoded. - The commands.
/retry,/rewind,/reset,/detach,/context N,/stats,/save,/export, retrieval with its critique and a tool's result work with media in the conversation;/decideworks on the text and says so;/modelis refused while the chat holds media and changes the projector with the model otherwise.--chat-filenames each message's media files and--kv-savekeeps their rows (oklm_kv_save_media(), appended);--continuereads the files again, encodes them once and takes the rows back without evaluating them. A conversation too long for the context, a file the encoder refuses and a template with no place for a picture are said, and the chat goes on. - Measured on the CPU,
--temp 0, the machine shared with other work. SmolVLM-256M (Q8_0), a newspaper's front page at 640 x 488: the picture's turn 348 rows (320 the picture's), 4.3 s to encode and 0.22 s to evaluate, "A newspaper with an old picture of an astronaut on the front."; the next question 14 rows evaluated of 375 in 0.02 s, nothing encoded, "Men walk on moon.";/retry1 row;/rewindand another question 11 rows. Ultravox v0.5 on Llama 3.2 1B (Q8_0),jfk.wav: 228 rows (188 the sound's), 8.5 s to encode, the sentence transcribed; the next question 18 rows of 294;/retry1 row. Qwen2-VL 2B (Q4_K_M), the same page: 418 rows (391 the picture's); the next question 15 rows of 473; a second picture later 292 rows of 773 with one item of two encoded;--continue --kv-savein a new process 19 rows of 832, 813 from the file (24.1 MB), both pictures encoded and not evaluated. That turn with the second picture gives the same text with 481 rows from the cache as with the conversation evaluated from nothing (/context 8192before it). Gemma 3 4B (Q4_K_M), whose picture is 256 rows that see each other: 272 rows, the headline read; the next question 22 rows of 312;/retry1 row. - Tests.
test_cli_chat(test_media, 278 checks in the file) on a tiny model with a tiny vision encoder and a tiny audio one made by the test: the rows of each turn as--statstells them through/retry,/rewind, a picture attached twice,/decide,/context Nand/reset; the conversation's file, the export,--continuewith and without the projector; a sound; retrieval with its critique; a tool's result; a conversation longer than the context; a projector of another width; the catalogue's projector and/model.tools/check_chat_tui.sh: a picture attached in the full-screen UI on SmolVLM-256M.test_chat_commandsis unchanged and passes.
Left: in the roadmap under C2 (not run on a device or under wine;
/model with media in the chat; /decide on the
text only; --continue encodes again; the context not grown
for a picture; no --vision-id; LLaVA and Qwen2.5-VL not run
in the chat).
Done (2026-11-14)
L9. The leftovers of the chats, the server's videos and saved conversations
Entries 12, 15 and 22 of L9 left six things: a video decoded again at
every request, no sampling to choose and no Transcribe for a video in
the graphical chat, the page's video case run in Chrome alone, no
published file run with a video's frames as still images, the labels of
Qwen's template never written, and a next turn after
--kv-load and --then that was text only.
Everything below ran on the CPU.
- A media part sent again is not decoded (server.md, Decoded
once). The server's embedding cache is keyed on the decoded
pixels and samples, so a part was decoded at every request before the
cache could answer, and a chat window sends every earlier part again
with each message. The server now remembers the last 128 parts it
decoded by what was sent (a hash of the encoded bytes and their count,
the kind, a video's
fps,framesand the ffmpeg switch, the slicing side, the bound on an image's embeddings), each with the hashes of the items it became and what a prompt needs of it beside their embeddings (sizes, a video's times and length). A part found there with its embeddings still kept gives items named by their hashes (oklm_mm_item.hash), whichoklm_mm_build()lays out with no pixel read; the request copies the embeddings when it finds the part, so nothing the cache drops meanwhile can miss. Pictures and sounds were measured first and take the same path, their decoding costing more than a video's here. A request sent a second time (max_tokens1): the fixture video on the tiny model 1.7 ms -> 0.9 ms; an 8-frame 1920 x 1080 Theora video 0.153 s -> 0.0075 s, a 4000 x 3000 JPEG 0.202 s -> 0.009 s, ten minutes of Ogg Vorbis 1.76 s -> 0.0125 s (the tiny model again); the published Qwen2-VL 2B with the four-frame clip of the entry below 0.070 to 0.084 s -> 0.044 to 0.055 s (eight requests each, two rounds, one binary after the other on a machine other work was using), the rest being the model's two rows and one token.media_decodedandmedia_knownin/api/statscount the parts decoded and known.test_server: the second request oftest_video_describe,test_mediaandtest_sound_describedecodes nothing (1 decoded, 1 known), another sampling decodes again; every case ofdescribe_items()builds the same rows and embeddings from items named by their hashes, their pixels gone, and is refused (OKLM_ERR_STATE) without the embeddings. - The labels of Qwen's template (multimodal.md). The
template in the files of Qwen2-VL and Qwen2.5-VL writes
Picture N:andVideo N:before a conversation's pictures and videos when itsadd_vision_idvariable is set, and nothing sets it unless the caller does: no label by default in the template, and none in the pinned llama.cpp, which never sets it. The default stays no label. The labelled form isoklm describe --vision-idandchat_template_kwargs.add_vision_id(oradd_vision_idbeside the messages) in a chat request, written byoklm_mm_content_numbered(), whichoklm_mm_content()calls without a numbering.test_vision'stest_vision_id: four conversations rendered by the files' template, with and without labels, equal jinja2's rendering of the same conversations given in parts (tools/qwen_vision_id_ref.py, the same output from the 2B's and the 7B's file).test_server: a request with the variable on the tiny Qwen projectors has the rows of describe's numbered prompt, and the rows of the plain one on LLaVA's. Qwen2-VL 2B, two pictures: 425 prompt tokens with the labels against 415. - A next turn with media of its own (multimodal.md).
The files named with
oklm describe --kv-load FILE -p QUESTIONgo with that question, and--then-with FILEgives one to the question of--then: a picture, a sound or a video, laid out as in a first turn by one loader, their rows at the positions that follow the conversation's.test_kv_file'stest_next_turn_media, the tinyqwen2vlmodel: a turn with a picture saved, then a text, a second picture or a video and a text after the load, contiguous and paged either way, give the logits and the position counter of the context that never left memory, bit for bit. Qwen2-VL 2B (Q4_K_M,--temp 0): a second picture (192 embeddings, 220 rows after the 227 loaded) and the clip (384 embeddings, 402 rows after 225) as the next turn from a file give the text of the one run with--then ... --then-with, byte for byte. - A defect found on the way.
oklm describe --thenand--kv-loadrefused every SmolVLM conversation (the model's chat template does not write a second question after the first answer as a continuation of them), the README's own example among them: SmolVLM's format ends its prompt atAssistant:and writes a kept answer after a space, which the check of the next turn's text did not allow. It does now: SmolVLM-256M answers a second question in one run, and one with a picture of its own from a file as in one run, byte for byte. - The graphical chat (gui.md).
/video [fps F] [frames N] | default, a row of the commands' table that is the window's own as/connectis (the terminal chat attaches no video), sets how the videos attached next are sampled: their parts carryfpsandframes, the window decodes them the same way for the frames its box counts, and a video keeps the sampling it was attached under. A video that has a sound track has the recording's Transcribe button: the track is read by the readeroklm describegives a video to an audio model with, transcribed as a recording is, and the video stays attached;--transcribe-attacheddoes it without a press. Run asmake gui-testruns the window (which has the two cases now):/video fps 2 frames 3then/attachgivestestsrc.ogv Ogg video, 14 KiB, 2.0 s, 3 framesand 323 prompt tokens where the server's own sampling gives 2 frames and 183; the fixture's track transcribed by the window's server with the tiny Whisper, its text in the message, the video still in the row with its button (the picture looked at).test_gui_attach(186 checks) has the words of/videoand the part's two members,test_gui_commandsthe command's places. - The page in Firefox.
OKLM_CHECK_BROWSER=firefox CHECKS=video sh tools/check_dashboard.sh(Firefox 153.5.1esr, headless, over WebDriver BiDi): the 14 checks of the video group pass as they do in Chrome; nothing differed, nothing was changed. - A file's frames as still images. SmolVLM-256M
(Q8_0, the smaller of the two on disk), the clip,
What does the video show?: 1 280 media rows throughoklm serveand throughoklm describe, 1 398 prompt tokens against 1 402, the 4 tokens of the file's name thatoklm describewrites in the video's line and a request has not (Video clip.ogv (4.0 s):is 11 tokens,Video (4.0 s):7). Both answered, differently: "The video shows a man walking on the moon." with the name, "The video is not clear." without; sent again, 1 392 of the 1 398 rows cached, the four frames' embeddings reused and the file not decoded.
Left: the Transformers processor's default for the labels was read in
the template, its code not being installed here;
/v1/messages writes no label; --vision-id
cannot go on from --kv-load (the file keeps no count of
pictures and videos), and the context after --kv-load is
not sized for a new picture (-c); a next turn's sound was
not run on a published audio model; the table of decoded parts is not
kept across a restart, and a part whose embeddings the cache dropped is
decoded again (that path is not driven by a test: the cache holds 256
MiB); for SmolVLM and LLaVA 1.6 a known picture's grid is found by
cutting an empty picture of its size; the page sends no fps
or frames and has no /video; the page's video
group was not run in WebKit (the WebKit walk drives one server without a
projector and has no video case); the press of
Transcribe on a video's box was not driven (the option
that calls the same function was), nor the window's own recognizer on a
video's track, nor the file dialog, a drop and the path's field with a
video; the frames as still images were not run on LLaVA 1.5 or Gemma 3,
and the two SmolVLM answers were not compared under one text (the
command always writes the file's name); nothing here ran on a
device.
L9. The paths no test took, and the leftovers later entries had settled
Several entries of L9 left a path "no test takes" for want of a small device, and several "Left:" still named what a later entry did.
- A device of a chosen size
(
tests/mm/count_backend.h): the tests' counting device takes a bound on what it holds (count_cap) and, ascount_backend_computing()makes it, the CPU backend's kernels behind a device's interface: buffers, uploads and downloads of its own, so an encoder, a recognizer, a vocoder or a context placed on it goes through its device path with the CPU's arithmetic. Registered as a backend, a server takes it by its name. It is not a GPU: what it proves is the path (what is asked for, in what order, what is done when it is refused), not a backend's kernels. - Encoders with room for their blocks alone (entries
9 and 10).
test_vision: each tiny vision encoder (the one the test writes, Gemma 3's, LLaVA's, Qwen2-VL's and Qwen2.5-VL's at two sizes) on a device with the roomOKLM_MM_DEVICE_ENDS=0takes, to the byte: the notice, one part on the device, the bytes held and the embeddings of that setting bit for bit (3.6e-7 to 5.4e-7 from the CPU's); with one byte less, refused, nothing held, the CPU's bits.test_audio: the tiny recognizer (test_whisper_small_device: the CPU's text each way) and the tiny Ultravox encoder (test_audio_enc_devices: both notices and the embeddings ofOKLM_MM_DEVICE_ENDS=0bit for bit; with room for all but the convolutions, the projector on the device; 1.6e-6 from the CPU's). - Found there (
src/mm/audio.c): a device with the room ofOKLM_MM_DEVICE_ENDS=0was refused whole, the encoder then on the CPU, in two cases. The audio encoder asked for its projector's weights before its activations, so a projector that fitted left the activations none; and the activation buffer was sized for the first convolution's windows whether or not they would be there, the larger part for a narrow encoder (the tiny ones; the published encoders' blocks need more).wenc_act_alloc()asks again without the windows,oklm_audio_use_backend()then with the projector given back, each with its notice. - A vocoder (entry 8) has no split of that kind: its
network is on the device whole or not at all. What it has is a call
whose activations find no room:
test_speechgives the tiny vocoder, and WavTokenizer when the published file is there, a device with room for the weights alone: the call is computed on the CPU, to its bits, the network staying on the device. - A server's device (entry 19,
test_small_deviceoftest_server): a recognizer loaded at the start on a device with room for 4 KiB is refused ("the transcription model ... does not fit on count"), and starts once the room is there; with the model on the device too and the recognizer left to its first request, a--kv-memthat leaves it no room and a device of 1 MiB are refused by their messages; on a device of 64 MiB the server starts with 13 531 of the 15 388 blocks that fit (the recognizer's 3 802 880 bytes left out), shows that share as pending indevice_memoryand answers a completion. - Found there (
src/server/server.c):GET /api/contextand a dry run planned the default cache from what fits on the device alone (15 388 blocks here where the server had 13 531).oklm_server_context_plan()leaves the encoders' share out asmake_sched()does. - A picture in a prefix directory (entry 12,
test_media_describe): the tiny encoder of Qwen2-VL on the tests' language model, a request of 115 rows; a server started again on the directory loads the file, finds 112 rows cached and gives the same answer; another picture finds the one block of text before it. - Positions that hash right and make no sense (entry
12,
test_kv_file): the test computes the chunk's hash itself, changes a row's position and makes the hash that of the changed payload. On the tinyqwen2vlmodel a time position below zero and a position after a row that is not past the row's own are refused, nothing loaded; a change no rule forbids loads, which shows the refusals are the positions'. On the tiny llama model a row's position other than its index, and a position after it other than the next, are refused. - Matrices of 16 bits in an encoder's blocks (entries
4 and 21,
test_encoder_types): the encoder the test writes with F16 and with BF16 matrices, against the reference in doubles from the rounded weights (6.6e-7 and 4.7e-7) and through the device path on the counting device. An encoder with quantized matrices in its blocks is not put on a device at all; no tiny file has such matrices (the written encoder is 16 values wide, half a quantized block). - SmolVLM's embeddings on a device (entry 21) were
compared by
test_mm_publicsince entry 10 (the overview tile of SmolVLM-256M): the remark was stale. - The "Left:" lines of entries 2, 3, 5, 9, 10, 12,
13, 14, 15, 19, 21 and 22, and the bullets of L1 and L6 that repeated
them, say what is left now: what later entries did is named with its
entry, and the owner's decisions of 2026-11-14 are said as decisions
(the wide products the default, 4 096 embeddings at most, the 4.49.0
form of Qwen2.5-VL's time,
--fps 1 --frames 8, no partial sums in OpenCL's tiled products). - Checked once after the last edit, on the CPU and
the counting device
(
OKLM_CL_DEVICES=none OKLM_VK_DEVICES=none):test_vision(702 checks),test_audio(343),test_speech(2 906),test_kv_file(898) andtest_serverwhole (1 977), without a failure; the formatter on the C files changed.
Left: no real small device was used, and nothing here ran on OpenCL
or Vulkan (the device runs of these test files are the earlier
entries'); the refusal of an encoder with quantized matrices in its
blocks has no test; a served model with three-part positions in a prefix
directory has none either (no tiny file of one has a tokenizer); the
plan of GET /api/context reads the device's free memory
with the cache in place and leaves a draft model's room out;
test_soprano's copy of the vocoder's device test was not
given the counting device.
L9. OpenCL's decoding on the quantized files, and its start
OpenCL decoded Ternary Bonsai 2 4B at 138.6 tokens a second where Vulkan gave 169.3, gpt-oss-20b at 127 to 128 against 164 and Muse Glimmer at 28.7 against 36.7; Ternary Bonsai's prompts were at 0.75 of Vulkan's; and every OpenCL start on the RX 7900 XT compiled the float matrix-unit kernels, 0.3 s, for a model without a type that takes them too (opencl.md).
- Where a step went (RX 7900 XT,
OKLM_CL_PROFILE=1, each launch waited for; thenbench_llama's 128 tokens with each kernel's launches left out in turn). No decoder reads a table a weight: with the 16-entry codebooks computed in registers as Vulkan's are, gpt-oss-20b's indexed MXFP4 product went from 31.9 to 35.9 us and its prompt tiles from 1186 to 1220 us, so OpenCL keeps its constant arrays. The products are at the memory's rate (Ternary Bonsai's head 159 MB in 299 us, gpt-oss-20b's four experts 17.6 MB in 32 us and its head 579 MB in 854 us, Muse Glimmer's IQ2_S 85 MB in 236 us). The rest is launches: 612 a step of Ternary Bonsai, 600 of gpt-oss-20b, 902 of Muse Glimmer, each 2.2 to 3.5 us of kernel time when its work is nothing and about as much again in the step (the host's part is 1.7 us a launch). - Ternary Bonsai 2
(
src/core/opencl/kernels.cl,src/model/arch_llama.c): the rotation of a folded weight's input ran a stage of butterflies a barrier, with integer divisions a pair, 8.5 us a launch for a block of 512; an item now computes three stages of 8 elements in registers (5.5 us, the same butterflies). A step's RMS norm and the rotation of its output are one launch (norm_fold, 65 launches fewer a step, the bits of the two ops). The two products of a Gated DeltaNet block that read the same normalized input share one rotation (24 fewer a step, on every backend). A step: 612 launches and 6.04 ms of kernel time to 523 and 4.87. - The end of a residual branch
(
res_norm): an item reads four elements before it writes them, where a load, the operations and a store an element waited for the memory once an element: 16.1 to 10.7 us a launch on gpt-oss-20b (49 a step), the same bits. - Ternary Bonsai's prompts: the Gated DeltaNet
recurrence was 44 % of a prompt's kernel time (2.14 ms a launch for 512
tokens), its state read and written in device memory at every token;
gated_delta_rows(from 8 tokens) keeps the state in registers over the tokens, in work-groups of 32 state rows as Vulkan's shader: 1.46 ms a launch. Its sums are in another order than the step kernel's: the CPU's values within the op test's tolerance. - The start: the float unit kernels are a second
program, built at the first product that takes them
(
units_ready();kernels.clwithOKLM_UNITS_ONLYcompiles to those 91 kernels alone).oklm run -n 1on SmolLM2-135M Q4_K_M: 0.87 s before, 0.56 s now (0.55 s before withOKLM_CL_WMMA=0; 1.12, 0.76 and 0.74 s in a round with the CPU busy); Ternary Bonsai 2 4B with a prompt that takes the units 1.78 and 1.76 s. - An IQ4_NL cache on Vulkan (the arrays entry 6
left):
norm_store_rows,norm_get_rowsand the attention's cache rows read IQ4_NL's grid from constant arrays, some three thousand reads a block quantized; the grid is nowiq4nl_grid()ofcommon.glsl, from registers. Llama-3.2-1B Q4_K_M with an IQ4_NL cache: decoding 93.5 to 154.5 tokens a second (the same at 64, 256 and 1024 tokens of context: the writes), a prompt of 512 tokens 12 190 to 12 610 (a Q4_0 cache 447 to 456 and 15 230 to 15 370) (vulkan.md). - Figures (prompt of 512 / decode of 128 tokens, tokens a second, the best of three to five runs, before and after in one session): Ternary Bonsai 2 4B on OpenCL 2522 to 2582 / 137.7 to 138.6 before, 3041 to 3050 / 163.1 after, Vulkan 3420 / 180.9 (169.3 before the shared rotation): 1.11 of OpenCL's decode rate, and OpenCL's prompts at 0.89 of Vulkan's. gpt-oss-20b on OpenCL 3304 / 127.3 to 128.1 before, 3323 to 3346 / 129.9 to 130.6 after, Vulkan 3656 / 163.6. Muse Glimmer on OpenCL 858 / 28.7 before, 875 / 29.2 after, Vulkan 831 / 36.7. Llama-3.2-1B Q4_K_M on OpenCL 14 771 to 15 338 / 398.3 to 408.9 before, 15 097 to 15 151 / 396.1 to 411.1 after.
- Checks.
test_openclon the RX 7900 XT and the Intel UHD 750 (92231 checks: the rotation's other pass counts, the norm with its rotation against the two ops apart and the launch counts, the residual end against the ops apart, the recurrence of a step then of eight tokens against the CPU),test_opencl_wmma(by default and withOKLM_CL_WMMA=1) andtest_opencl_q8on the RX 7900 XT,test_vulkan_normandtest_vulkan_attentionon both devices. Ternary Bonsai 2 4B (test_bonsai_public) on OpenCL against llama.cpp's recording: 160 of 160 choices, logits within 0.652, 0.0116 on average; against the CPU: 159 of 160, within 0.174, 0.0114 on average, the figures before on both (and the three 2B files pass); the CPU's perplexity over 254 tokens is 32.9058 before and after the shared rotation, OpenCL's 32.8443 before and 32.8405 after (the prompt recurrence's order). gpt-oss-20b on OpenCL (test_archs_public): 158 of 160 choices llama.cpp's, logits within 4.05, 0.0806 on average; 155 of 160 the CPU's, within 2.88, 0.067: the figures before. Muse Glimmer on OpenCL: 156 of 160 choices llama.cpp's, logits within 3.66, 0.171 on average; from the CPU: 157 of 160, within 4.36, 0.163, the figures before.test_modelpasses. - gpt-oss-20b's file (11.6 GB, SHA-256
c2753664...568aeb4f) and Muse Glimmer's (10.7 GB,3e9bd0e6...00875ad9) were downloaded, run and removed, one on the disk at a time. - Left: in the roadmap (L9, entries 6 and 7).
L9. gpt-oss-20b's perplexity between the backends: its router on the products' rounding
gpt-oss-20b Q4_K_M gave 64.33 on the CPU, 61.61 to 61.90 on OpenCL and 63.20 on Vulkan over the same 1275 tokens, where Muse Glimmer and Ternary Bonsai agree within 0.2 %. It is the model's sensitivity, not a kernel's defect: no code changed (inference.md).
The reference (
oklm ppl -c 512on the first 14 000 bytes ofdocs/harness.md: five windows, 255 tokens scored in each;llama-perplexity -c 512 -b 512of the pinned llama.cpp on the same file): llama.cpp 62.6978, inside OKLM's range. The mean negative log-likelihood of each window:1 2 3 4 5 perplexity llama.cpp (CPU) 3.2330 4.7215 4.3648 4.4630 3.9093 62.6978 CPU 3.2316 4.7181 4.3493 4.6091 3.9119 64.3276 CPU, F32 cache ( -ctk f32 -ctv f32)3.2347 4.7197 4.3752 4.5605 3.9145 64.1297 OpenCL (default) 3.2266 4.7158 4.3584 4.4123 3.8908 61.6072 OpenCL, F32 tiles ( OKLM_CL_WMMA=0)3.2281 4.7213 4.3436 4.4391 3.8952 61.8951 OpenCL, float kernels ( OKLM_CL_Q8=0 OKLM_CL_WMMA=0)3.2331 4.6779 4.3255 4.5715 3.9097 63.0249 Vulkan (default) 3.2338 4.6722 4.3255 4.5936 3.9065 63.2014 Vulkan, F32 tiles ( OKLM_VK_Q8=0 OKLM_VK_COOPMAT=0)3.2336 4.6781 4.3237 4.5724 3.9040 62.9504 The CPU's windows are 0.001, 0.003, 0.016, 0.146 and 0.003 from llama.cpp's: the distance is the fourth window's, where the eight rows spread from 4.41 to 4.61, llama.cpp among them; on the first every row is within 0.008 (perplexities of 25.2 to 25.4).
The same arithmetic gives the same figure. With both devices on their float kernels (the last OpenCL row and the last Vulkan row: neither rounds a prompt's activations) the perplexities are 0.12 % apart, a token's log-probability differs by 0.011 on average, and the experts chosen differ in 0.54 % of the (token, block) pairs, none in blocks 0 and 1. The attention with sinks and its window, the cache, the indexed products, the gathered biases and the gated unit are the same kernels in those runs as in the default ones: what parts the default rows is how each backend rounds the activations of a product. The CPU quantizes them to Q8_K for a K-quant's dot product (llama.cpp's own form), OpenCL's 8-bit kernels do the same and its float matrix units take F16 inputs for MXFP4, Vulkan's units take F16 inputs for every type but F32. Block 0's router scores (standard deviation 1.0) are 0.0033 RMS from the CPU's on Vulkan and on OpenCL's float kernels, 0.0004 on OpenCL's default, 0.0003 on Vulkan with its products handed to the CPU kernels.
What the router does with it (the CPU's scores and choices of every block recorded, each device run compared and optionally made to take the CPU's choices; an instrumented build, not kept). The router keeps 4 experts of 32 on scores whose fourth and fifth are 0.05 to 0.11 apart at the median, by block; they are within 0.01 for 6 to 14 % of the tokens and within 0.05 for 27 to 50 %. The scores' differences between two backends grow from block to block (CPU and Vulkan: 0.0033 RMS in block 0, 0.013 in block 4, 0.028 in block 12, 0.066 in block 23; CPU and OpenCL: 0.0004, 0.010, 0.025, 0.104). The chosen sets differ in 9.10 % of the 61 440 (token, block) pairs between the CPU and Vulkan (1.25 % in block 0, 13.5 % in blocks 12 and 23), 8.55 % between the CPU and OpenCL, 9.49 % between the two devices: 72 to 77 % of the tokens have another set in at least one block. The CPU against itself with an F32 cache instead of F16: 8.0 %.
Per token the log-probability differs by 0.082 on average between the CPU and Vulkan, 0.109 between the CPU and OpenCL, 0.082 between the CPU's two cache types. With the CPU's choices forced on a device: 0.030 on Vulkan (perplexity 64.0649, 0.4 % from the CPU's) and 0.038 on OpenCL (63.5934, 1.1 %); 4.4 and 4.1 % of the device's own choices still differed before being replaced (the weights and the states keep their rounding).
Not the window, not the attention, not the cache. The share of differing choices does not step at the window of 128 (by 64 positions, CPU and Vulkan: 7.4, 7.7, 8.8, 10.4, 9.4, 8.9, 9.9, 10.3 %). With windows of 128 tokens (no query beyond the window; 1323 tokens scored) Vulkan and OpenCL give 75.5581 and 75.5095, with 256 (1270 tokens) 56.4098 and 57.2277: the distance grows with the length of the context the differences build up over. With the attention op of both devices handed to the CPU's kernel: 63.0258 and 62.5174. With F32 caches: 62.8665 on Vulkan, 62.6151 on OpenCL's F32 tiles. Every switch moves the figure by as much as the backends differ.
Why the mean moves here and not elsewhere. Tini-Cybersec-8B-A1B (Q4_K_M, routed experts too) has the same scatter per token (0.091 between the CPU and Vulkan, 0.088 between the CPU and OpenCL) and its perplexities stay together: 50.3877 on the CPU, 50.2366 on Vulkan, 50.1217 on OpenCL with the units and with the F32 tiles, no window more than 0.025 apart. gpt-oss's differences keep one sign over a window: in the fourth, OpenCL's largest logit is 0.52 above Vulkan's on average and the target's 0.54 (a sharper distribution, 0.18 less negative log-likelihood), and block 23's router scores part by 0.11 to 0.28 RMS in its second half where the other windows have 0.01 to 0.09. Which block's choice starts it in that window was not traced.
The short recordings do not show it because they compare 160 choices and the largest logit difference, both of which gpt-oss passes as the other models do (153 of 160 choices equal to llama.cpp's on the CPU, logits within 3.96), not a sum of log-probabilities over 1275 tokens.
Nothing changed in the runtime, so
test_archs_publicand the rates were not run again. The file (11.6 GB, SHA-256c2753664...568aeb4f) was downloaded, run and removed.Left: nothing on this entry. A perplexity of gpt-oss-20b over a thousand tokens is known to about 2 % either way, on one backend as between two.
L9. The CPU's image encoder at large pictures
A picture of 4 015 embeddings had taken OKLM's CPU encoder 467 s (Qwen2-VL) and 169 s (Qwen2.5-VL) where llama.cpp's whole prompt had taken 159 s and 206 s, and why was not looked into.
- Where the time went (
OKLM_CPU_PROFILE, Qwen2-VL at 4 015 embeddings, 16 060 patch rows, 389 s in the ops): 290 s in the attention and 95 s in the products. The attention was the tiled one the text models' prompts use, threaded; each tile of 32 rows gathered its head's keys and values from the tensors again (a row of 80 floats every 5 120 bytes, 16 060 times, twice), about as long as its arithmetic. The products ran one dot product a row: F16 weights had no kernel of several rows, 48 GFlop/s on a thread where the blocked kernel ofoklm_matmul_dense()does 112. - Attention head by head
(
attention_dense(),src/core/cpu/attention_cpu.c): from 512 float keys and 32 rows a thread, a head's keys are copied once eight side by side and its values row after row; every thread works on that head; the scores of eight rows come from one kernel that writes them in the keys' order (oklm_avx2_attn_qk8()); a tile takes fewer rows when its scores would pass 512 KiB. The four-row kernel of the values keeps its eight sums in registers. One block at 4 032 rows: 0.41 s to 0.14 s. - Products (
src/core/cpu/matmul_cpu.c,simd_avx512.c,simd_avx2.c): F16 weights get a kernel of eight rows on AVX-512 and of four on AVX2, each row the sums of the one-row kernel, and a product of 256 rows or more is cut by blocks of its activation rows, the weights then being what the threads share. 1 280 x 5 120 over 16 060 rows: 0.82 s to 0.30 s; 5 120 x 1 280: 1.63 s to 0.49 s. - The bits are the same: every score, every output of
the attention and every dot product sums in the order it did
(
test_bidirectional,test_many_rowsandtest_vec_dot4compare the paths bit for bit), and the embeddings of the picture at 1 008 and at 4 015 embeddings are byte for byte those of the kernels before. No recording and no bound moved. - Against llama.cpp's CPU encoder, one after the other on a machine with other work at times (eight threads, Core i7-11700K): Qwen2-VL at 1 008 embeddings 16.5 s (32.4 to 34.7 s before) against 16.8 s, at 4 015 164 s (389 s) against 136 s; Qwen2.5-VL 7B at 1 008 13.5 s against 12.4 s, at 4 015 66.3 s (169 s) against 83.9 s (multimodal.md).
Left: Qwen2-VL at 4 015 embeddings is a fifth behind llama.cpp, its attention over 16 060 rows at 370 GFlop/s where 4 032 rows reach 590 (a tile reads its head's 10 MiB of keys and values from the last-level cache; the softmax taken in the passes over the keys was tried and measured no faster on a busy machine); the products whose rows are 3 420 or 5 120 elements long gain least (an inner dimension in blocks needs each output's sums carried between them); every figure was taken with other work on the machine; an AVX2-only processor has the four-row kernel, not measured.
L9. Qwen2.5-VL against its own code: whole answers, the 3B, a recording
The float64 pass of the transformers code had compared the logits of an answer's first token, on the 7B alone, and kept nothing for the tests.
- Whole answers
(
tools/qwen2vl_ref.py --answer): the tokens of a recorded answer follow the prompt and one causal pass gives the logits before each (80 s for the 7B's 418 rows and 48 tokens, 42 s for the 3B's). Q4_K_M, the 48 tokens of llama.cpp's answer, largest difference over the vocabulary and the mean: the 7B's one-picture answer, OKLM 1.26 and 0.169 from the pass, llama.cpp 2.96 and 0.403, and 1.12 and 0.143 from the pass with an epsilon of 0; its text alone, OKLM 1.03 and 0.156, llama.cpp 2.30 and 0.233. 46 or 47 of the 48 best tokens are the pass's in both programs; where OKLM's is not, the pass has the two within 0.055 to 0.175. - The 3B (fetched, checked against its SHA-256,
removed): its embeddings of
<|im_start|>and<|im_end|>have mean squares of 1.91e-4 and 1.99e-4, so the epsilon changes its first norm by 0.3 % as the 2B's, and llama.cpp is as far from the pass with the file's epsilon as with 0 (first token 0.970 and 0.968, OKLM 1.12; the answer 4.21 and 4.26, OKLM 4.58, means 0.211 and 0.220). The 3 to 4.6 are single logits from the 38th token on, in both programs at the same tokens (1.61 and 1.50 at most before it). - A recording for the tests (
--record, 96 KiB an answer: the pass's 256 largest logits and their tokens before each token;tools/gen_mm_ref.shwrites it for the 3B and the 7B).test_qwen25vl_answerandtest_qwen25vl_texthold OKLM's CPU logits to it where it is present: within 0.943 (0.157 on average) on the 7B's picture, 0.689 (0.148) on its text, 3.17 (0.225) on the 3B's picture, bounded at 1.5, 1.5 and 5.0 and a mean of 0.35. llama.cpp's distance from the same logits is printed (2.31, 1.95, 4.21) and its own comparison keeps its bounds (multimodal.md).
Left: the 32B and the 72B against the pass (18.5 and 22.1 GiB, not on the disk); whole answers on the BF16 file and of the prompts with several pictures.
L9. Qwen2.5-VL 7B's distance from llama.cpp: its RMS norms' epsilon
The 7B's logits parted from llama.cpp's as a prompt took pictures, on the float file too, where Qwen2-VL 2B's did not; which program was nearer the model was not known.
- Where they part (the same rows and embeddings in
both, one picture, 418 rows; OKLM's steps from
OKLM_DUMP, llama.cpp's from its evaluation callback in a copy oftools/mtmd_ref.cpp): after the first block already, in the rows of<|im_start|>and<|im_end|>(after the first norm a relative difference of 0.76 to 0.82 there, 0.001 to 0.002 in the other rows of the text; Q4_K_M), and from the fourth block the row whose residual stream grows to thousands is the first in llama.cpp and the third in OKLM. The positions are not it: both give every row the same three (time 15, rows 15 to 31, columns 15 to 37 for the image, the text after from 38; 38, 53 and 76 after three pictures), which are those of transformers'get_rope_index(). - The cause: llama.cpp at the pinned commit reads no
attention.layer_norm_rms_epsilonfor theqwen2vlarchitecture (src/models/qwen2vl.cpp) and evaluates its norms with 0; the files say 1e-6, the model'srms_norm_eps, which OKLM uses. The 7B's embeddings of the two tokens have a mean square of 3.4e-8 and 6.2e-8, under the epsilon, so its first norm scales them 5.5 and 4.1 times less than a norm without it; the 2B's are 1.6e-4 and 2.0e-4 (0.3 %). - The arbiter (
tools/qwen2vl_ref.py): the language model in numpy float64 from the transformers code, weights by gguf-py. On the 7B's BF16 file (fetched, checked against its SHA-256, removed) the first token's logits of a text, one, two and three pictures: OKLM within 0.007, 0.014, 0.149 and 0.013 of that pass (2.2e-4 for the two pictures with an F32 cache), llama.cpp 1.13, 3.60, 6.42 and 4.87 from it, and within 0.055, 0.089, 0.150 and 0.228 of the same pass with an epsilon of 0. On the Q4_K_M file: OKLM within 0.84 to 1.31 (8-bit activations against none), llama.cpp 1.32 to 8.53. - Nothing changed in the runtime: OKLM computes the model. The tests keep llama.cpp as the reference with their bounds, and say why (multimodal.md).
Left then, and taken since for the 3B, whole answers and a recording (Qwen2.5-VL against its own code): the 32B and the 72B against the float64 pass.
L9. The room of the encoders and the sound models on a server's device
A server's cache takes what its device has free, less a share for
what computes beside the model there. That share was an estimate from
the files' sizes for the two sound models (the checkpoint and 1 GiB; the
speech model's file, a quarter more and 512 MiB; the vocoder's file and
128 MiB), the image encoder's buffer was that of 1 280 embeddings
whatever the server accepted, and nothing was set aside with
--kv-mem.
- Counted as allocated:
oklm_whisper_device_bytes()(the weights as the device holds them, and the one buffer of the passes),oklm_speech_device_bytes()(the language model's context throughoklm_context_device_bytes()with the parametersoklm_speech_open()gives it, and the vocoder's network),oklm_vision_device_bytes_at()(the activations of a picture of a given size) andoklm_vision_device_held(); the audio encoder's and the vocoder's accessors were there. A device that only counts (tests/mm/count_backend.h) is given the tiny recognizer, the tiny audio encoder and the tiny Qwen encoders: each asks for its count to the byte and gives it all back. - The server (server.md): the sound models'
weights are read once at the start to be counted (0.1 s for
large-v3-turbo and OuteTTS 0.2), the image encoder is counted at the
largest picture the server accepts (3 072 embeddings at
-c 4096: 2 491 MiB for Qwen2-VL 2B where 1 782 were kept), and what none of them holds yet comes out of the cache, with--kv-memtoo./api/statsgives the counts, the device's total and free memory and what is pending indevice_memory. - Refused at start, with the figures: a
--kv-memthat does not fit with that share ("--kv-mem 15000 does not fit on vulkan with what shares it: ... --kv-mem 11916 at most, or 0 for what is left"); a device that held one context of-ctokens beside the model and does not once the share is taken, or has nothing left; a recognizer asked for on the device with--preloadthat does not fit there (it moved to the CPU with a warning). - Against the device (RX 7900 XT, the free memory before and after a first request of each kind, counted / Vulkan / OpenCL, MiB): Qwen2-VL 2B's encoder 1 782 / 1 782 / 1 784, its buffer for 3 024 embeddings 690 / 991 / 692; Gemma 3 4B's 1 148 / 1 148 / 1 150; LLaVA 1.5's 658 / 658 / 658; Ultravox's 1 410 / 1 418 / 1 414; Whisper large-v3-turbo 1 726 / 1 733 / 1 728; OuteTTS 0.2 with its vocoder 1 145 / 1 056 / 1 058; Soprano 366 / 273 / 278. One server with Qwen2-VL 2B, the recognizer and OuteTTS on the device and the default cache kept 1.3 GiB (Vulkan) and 1.7 GiB (OpenCL) free after a request of each kind, the large picture included; nothing moved to the CPU.
- Checked:
test_audio,test_vision,test_speech(the counts; the speech model's against the device's own figure, 903 + 243 MiB counted where the device gave 898 to 1 036 MiB before the vocoder's first call);test_device_encoders(device_memory, the two refusals by their messages, the same recognizer on the CPU starting with the cache that was refused).
Left: in the roadmap under L9, entry 19.
L9. A spread context holds every device it uses
A context whose layers are spread over several devices took the pass lock of its head's device only: a pass of another thread on the device of its first layers could run between its ops there.
- The pass (
run_items(),src/model/model.c): one pass on the head's device and on each device of the earlier layers (oklm_backend_pass_begin_all(),oklm_backend_pass_end_all()). The locks are taken in the order of the devices' registration, sorted from the backends given whatever their order, a device named twice taken once, and given back in the reverse order: no two passes can each hold a device the other waits for. An encoder holds its own device alone and yields between its blocks as before (oklm_backend_pass_yield_all()gives back and retakes every device of a pass; no caller on several devices yet). - Checked:
test_threads(test_pass_devices, fake devices): a pass naming {second, first} waits for the first without holding the second, which another thread takes meanwhile; a yield of a pass on both lets it in; four threads naming the devices in opposite orders run 8 000 passes one at a time.test_multi_device(check_shared): a context spread over two devices decoding 32 tokens, a context on the head's device decoding 32, and the tiny Gemma 3 encoder on the other device encoding 12 times, in three threads: logits and embeddings bit for bit those of each alone, on two OpenCL instances, two Vulkan instances, and the RX 7900 XT with the Intel UHD 750. - By hand
(
OKLM_VK_DEVICES=all oklm serve, Qwen2-VL 2B over the RX 7900 XT and the UHD 750): nine new images answered, three during a 400-token stream. The stream's largest gap there, 6.4 to 7.2 s, is the step of an image's 416 rows through the UHD 750's layers (0.3 to 0.7 s with--prefill-chunk 16), not the encoder.
Left: in the roadmap under L9, entry 20.
L9. An encoder's pass cut between its blocks
An image, a sound or a vocoder call was one pass on the device in
oklm serve, and the requests that were decoding waited for
all of it: 0.20 to 0.70 s on Vulkan and 0.19 to 0.75 s on OpenCL,
measured again before the change (the encoders' ends are on the device
since).
- The lock (
src/core/backend.c): a device's pass lock was a recursive mutex, which the thread that releases it takes again before a waiting thread wakes. It is now a ticket lock: a pass draws a number and runs at its turn, so the passes of a device run in the order they asked for it.oklm_backend_pass_waiting()says how many wait. - The yield (
oklm_backend_pass_yield()): called by the vision encoders once the stream is made and after each block, by Whisper's and Ultravox's encoder after the convolutions and after each block, by the vocoders after their first stack. On a device another thread uses (a pass of its waits, or ended within the last 0.25 s) it waits for the work the pass queued (synchronize: no op held back, no open command buffer, nothing queued) and, when a pass waits, gives the device to it and takes it back behind the others. The wait is what bounds the gap: an encoder queues its blocks in a millisecond, and a step let in behind them would wait for them all. Alone on its device an encoder does nothing at these points. - Measured (server.md; RX 7900 XT, a
stream of 400 tokens while three requests arrive, the largest gap
between two tokens, before -> after): Gemma 3 4B's image 0.70 ->
0.08 s on Vulkan and 0.75 -> 0.10 s on OpenCL; Qwen2-VL 2B's 0.25
-> 0.04 s and 0.29 -> 0.05 s; jfk.wav by Whisper large-v3-turbo
0.20 -> 0.015 s and 0.28 -> 0.02 s; by Ultravox 0.20 -> 0.03 s
and 0.27 -> 0.03 s; a sentence by OuteTTS 0.2 0.04 -> 0.03 s and
0.07 -> 0.03 s, by Soprano 0.02 s either way. LLaVA 1.5 7B's stays at
0.19 s on both: its encoder is 0.04 s, the gap is the step that reads
the image's 576 rows (0.06 and 0.09 s with
--prefill-chunk 64). - What the encoder pays: nothing alone (a request
with a new image or sound: Gemma 3 0.89 -> 0.87 s, Qwen2-VL 0.39
-> 0.39 s, Whisper 0.29 -> 0.29 s on Vulkan; 0.96 -> 0.96, 0.43
-> 0.43, 0.36 -> 0.37 s on OpenCL; medians of five); among streams
a quarter to a half more (Qwen2-VL 0.39 -> 0.51 s on Vulkan, 0.44
-> 0.63 s on OpenCL), the time of the steps it lets in. Hence a
default,
OKLM_PASS_CUT=0turning it off. Waiting for the queued work at every block on a device nobody shared cost 1 to 5 % alone and was dropped. - Checked:
test_threads(test_pass_handoff,test_pass_devices: fake devices, no GPU): the passes in the order asked, a yield handing over to three waiting passes in their order and keeping the nested depth, ending and restarting a pass behind a waiting one, no wait for queued work alone or with the cut off;test_device_encoders: the answers of the CPU's encoders, sixteen requests at once as alone, and a greedy stream of 393 events while six new pictures arrive, at most 5.5 ms (OpenCL) and 0.8 ms (Vulkan) between two events on the tiny fixtures, 3.2 and 2.0 ms on the Intel UHD 750 (bound: 2 s).
Left: in the roadmap under L9, entry 18.
L9. Published vision files on the Intel UHD 750
The fixtures alone had run on the integrated GPU
(test_vision).
- Run
(
OKLM_CL_DEVICE=UHD OKLM_VK_DEVICE=Intel,test_mm_publicwith SmolVLM-256M, SmolVLM-500M and Qwen2-VL 2B alone in its directory: 3 508 checks, none failed): the encoders on both backends against the CPU's embeddings, then each answer four ways a backend (every layer; half with a paged cache; half with the contiguous cache; half streamed). - Embeddings: SmolVLM-256M's within 2.6e-6 of the CPU's, Qwen2-VL 2B's within 6.2e-5, on both backends.
- Answers: 46 to 48 of the CPU's 48 greedy choices in every layout (logits within 2.5), Qwen2-VL 2B's 46 of llama.cpp's 48 as on the CPU.
- Times (the machine otherwise idle,
oklm describe, Qwen2-VL 2B, 391 embeddings): the prompt in 9.78 s on the CPU, 40.4 s through Vulkan, 196 s through OpenCL; 23 tokens in 0.82 s, 1.50 s, 5.69 s. The encoder of a picture of 273 embeddings: 6.9 s on the CPU, 25 to 32 s on Vulkan, 104 to 122 s on OpenCL. SmolVLM-256M's language model at 28.7 (OpenCL) and 42.9 (Vulkan) tokens/s against the CPU's 157. - No defect, no refusal: the UHD 750 is slower than the CPU for every one of these, by 2 to 40 times, which the pages now say.
Left: Qwen2.5-VL, Gemma 3, LLaVA and SmolVLM2-2.2B were not run there, nor a large picture, several pictures or a video; the test's rates were taken while other runs used the machine (the describe times were not); why Intel's OpenCL takes four times Vulkan's time on the encoder's attention was not looked into.
L9. Qwen2.5-VL 3B, 32B and 72B
The test had the 3B's checks waiting for its file; the 32B and the 72B had never been fetched.
- The files (one on the disk at a time, each removed
after its runs; URLs and SHA-256 sums in
tests/public_models_large.txt, the catalogue'sqwen2.5-vl-3b,qwen2.5-vl-32b,qwen2.5-vl-72b):ggml-org's Q4_K_M for the 3B (1.8 GiB) and the 32B (18.5 GiB);bartowski's IQ1_M for the 72B (22.1 GiB), the largest that left a margin with 26.8 GiB free: its 2-bit files start at 23.7 GiB, which with the projector's 1.3 GiB left under 2 GiB. - Against llama.cpp on the CPU (the 7B's tests,
test_qwen25vl,test_qwen25vl_answer): the prompt's 418 rows are its rows for the three; 47 of 48 greedy choices for the 3B (logits within 4.93 from the same embeddings, 1.96 against its whole path), 16 of 16 for the 32B (3.40), 15 of 16 for the 72B (1.18). The encoders within 1.2e-5 of the float64 pass. - On the devices (RX 7900 XT, the layers that fit, then half of them with a paged cache): the 3B whole, 47 and 48 of the CPU's 48 choices, 141 and 193 tokens/s (OpenCL, Vulkan) against the CPU's 14.5; the 32B with 61 of 64 layers, 16 of 16, 13.7 and 14.9 tokens/s against 1.5; the 72B with 62 of 80, 14 of 16 (the two tokens 0.18 apart where they part), 2.7 tokens/s against 0.7.
- Found on the way: Vulkan kept a freed model's weights until its next wait, so the 32B placed twice in one process went past the card's memory (above); the tests' bound on an encoder's embeddings on OpenCL goes from 1e-4 to 2e-4, the 32B's reaching 1.1e-4 for one picture with its blocks alone on the device; a device's choices that part from the CPU's at near ties alone (0.25) are not held against it, two of them being more than a tenth of a 16-token answer.
Left: one quantization a size; the 72B at more than 1.75 bits needs a
disk with 30 GiB free; no video and no several-picture prompt on these
sizes; the 32B and the 72B were not run on the CPU through
oklm describe, and the 72B's rates were taken with another
test running.
L9. Pictures above a megapixel for Qwen2-VL and Qwen2.5-VL
A picture was reduced to 1 280 embeddings (a megapixel) unless
--tiles asked for more; llama.cpp goes to 4 096, and the
size above 1 280 had been encoded once, without a reference.
- The limits (read on 2026-11-14). The processor
class: 4 to 1 280 embeddings. The
preprocessor_config.jsonpublished with Qwen2-VL 2B and Qwen2.5-VL 3B, 7B, 32B and 72B: 3 136 to 12 845 056 pixels, 4 to 16 384 embeddings. llama.cpp at the pinned revision: 8 to 4 096. - The default (
src/mm/vision.c): 4 to 4 096 embeddings, the processor's minimum and llama.cpp's maximum, the file's keys first when it has them.--tiles SIDEcaps the area under it (1002 for the former 1 280). The models' 16 384 is not offered: no reference, and sixteen times the attention of 4 096. - Where the context is fixed:
oklm_vision_limit_tokens()bounds an encoder's images;oklm serveand the captioner ofoklm rag indexset it to three quarters of a sequence's context, so a photograph is reduced where its request would have been refused (3 072 embeddings with-c 4096). The bound is part of what names a picture among a prompt's rows.oklm describewithout-csizes its context to the pictures (oklm_vision_image_tokens()). - Device memory: the buffer counted for what shares
the device stays that of 1 280 embeddings (1.74 GiB with the weights for
Qwen2-VL); a larger picture's is asked for when it comes (2.62 to 2.80
GiB in all at 4 015 embeddings), and a device without room encodes that
picture on the CPU, now with a notice.
OKLM_MM_DEVICE_ACT_BYTEStests the path (test_vision). - Against llama.cpp
(
test_qwen_large_pictures; its test image enlarged to 2.0 and 5.2 megapixels: 2 552 and 4 015 embeddings, its sizes and rows): Qwen2-VL 2B 16 of 16 choices at both sizes, logits within 0.94 and 0.93 (same embeddings; 0.80 and 1.03 against its whole path); Qwen2.5-VL 7B 15 of 16, within 3.92 and 3.60. - On the devices (RX 7900 XT): the encoder of a picture of 4 015 embeddings in 9.87 s on OpenCL and 11.6 s on Vulkan for Qwen2-VL (467 s on a busy CPU), 2.92 s and 2.63 s for Qwen2.5-VL (169 s), within 2.9e-4 and 1.7e-4 of the CPU's embeddings; the language model 16 of 16 choices the CPU's for the 2B, 15 or 16 for the 7B.
Left: the models' own limit of 16 384 embeddings is not reachable; a device's embeddings are 1e-4 to 3e-4 from the CPU's at these sizes (1e-5 to 5e-5 for a small picture), growing with the rows; OKLM's CPU encoder is slower than llama.cpp's at these sizes (467 s against the 159 s of llama.cpp's whole prompt, both on a busy machine) and was not looked into; the buffer of a large picture is not counted in a server's memory plan.
L9. Vulkan: a freed model's memory is released before the next is placed
Found while Qwen2.5-VL 32B (18.5 GiB on a card of 20) was run in
several layouts in one process (test_mm_public): after the
layout with 61 of its 64 layers on the device, the one with 32 layers
and a paged cache failed in its first evaluation once, and once ran its
prompt at 11 rows/s and its tokens at 0.8 a second, under the CPU's 23
and 1.5.
- The cause
(
src/core/vulkan/vk_runtime.c,oklm_vk_buffer_free()): a freed buffer is kept until the device is idle, and was destroyed at the next wait, or when 64 of them waited. A model's weights are a few buffers: nothing waited, and the next model's were allocated beside them, past the device's memory. - The fix: the free waits for the device and destroys what waits as soon as 256 MiB of it does.
- After: the same layout at 42 rows/s and 2.9 tokens/s (OpenCL: 43 and 2.8), 16 of 16 choices the CPU's.
Left: a process that holds two models near the card's size at once is not helped; nothing checked how OpenCL's driver returns such memory (its runs were at their speed).
L9. Several pictures in one prompt on the published Qwen files
Two and three pictures in one prompt had been checked on the tiny
qwen2vl model alone.
- The reference
(
tools/mtmd_ref.cpp --answer): several pictures, comma-separated, one a marker; with given embeddings each image's chunk takes its own from the file, one after the other. The context grows to 8 192 rows for a prompt that needs it. - The tests (
tests/mm/test_mm_public.c):items_answer()renders a conversation with a marker a picture through the file's template, compares its rows with llama.cpp's, feeds llama.cpp's 32 greedy tokens on the CPU (from the same embeddings, then against its whole path) and runs the same rows four ways on OpenCL and on Vulkan (answer_devices()). Four prompts a model: two and three pictures in one user turn, two and three across two turns (test_qwen2vl_several_images,test_qwen25vl_several_images). The comparison with a recording is one function for every answer (answer_against()), which now prints the mean logit difference too. - Figures (CPU against llama.cpp given the same embeddings, of 32 choices, largest logit difference). Qwen2-VL 2B: 31, 32, 32, 31 (0.75 to 3.04); every layer on OpenCL 30 to 32 of the CPU's choices, on Vulkan 29 to 32 (logits within 3.39). Qwen2.5-VL 7B: 28, 29, 28, 31 (4.57 to 12.8); on OpenCL 29 to 32 of the CPU's, on Vulkan 30 to 32 (within 2.68). The rows are llama.cpp's in the eight prompts (518 to 817 rows, 482 to 755 of the pictures).
- No defect found in OKLM: the rows and positions are the same code for the two models, and the 2B is llama.cpp's to a near tie.
Left: the 7B's distance from llama.cpp with several pictures (the logit of the first token of the three-picture answer is 31.0 there, 22.9 here) is not explained. The CPU and both devices agree with each other (within 2.7). It is not the quantization: with the 7B's BF16 language model (fetched for this, then removed) the two programs are within 1.71 on a text (0.16 on average), 3.70 with one picture (0.44), 6.30 and 9.43 with two and three in one turn (0.49, 1.26), the choices parting at near ties only (29 to 31 of 32). Qwen2-VL 2B shows none of it (0.11 on average on a text and on 4 015 image rows). Which program is nearer the model on the 7B is not known: no third computation of its language model was run. The "Picture N:" labels the models' template can add are not written.
L9. An image with the layers split between a device and the CPU
Qwen2-VL and Qwen2.5-VL had run with an image on a device whole, and
with half their layers there on the paged cache; the contiguous cache
with a split and the CPU's layers streamed to the device for the prompt
(--gpu-stream) had not, nor a video on a device at all.
- The tests (
tests/mm/test_mm_public.c).answer_devices()runs a prompt four ways on each backend: every layer on the device, half of them with a paged cache, half with the contiguous cache, and half with the CPU's layers streamed for the prompt (batches of 16 rows and more; the tokens after it on the CPU's kernels). Every model that answers about a picture goes through it: Qwen2-VL 2B, Qwen2.5-VL 7B, Gemma 3 4B, LLaVA 1.5 and 1.6, the three SmolVLM files.answer_eval()gives each media item's rows as its kind asks (an image's grid, a video's grids and time step, rows that see each other), so a prompt may hold several.test_qwen_video_devicesis a video of four frames (two grids of 192 embeddings) the same four ways. - Figures (RX 7900 XT, llama.cpp's test image, 418 rows, 48 tokens; choices the CPU's and the largest logit difference from the CPU's, OpenCL then Vulkan). Qwen2-VL 2B: contiguous split 47 and 48 (1.12, 1.38), streamed 45 and 46 (1.71, 2.73), the prompt at 989 and 987 rows/s then 2 256 and 2 067 streamed. Qwen2.5-VL 7B: 47 and 46 (1.14, 1.11), streamed 47 and 47 (1.20, 1.26), 225 and 223 rows/s then 534 and 488. The video: 48 of 48 every way for the 2B, 46 to 48 for the 7B (logits within 2.25). The other models: multimodal.md.
- No defect: nothing was changed outside the tests. The split's logits with the contiguous cache are the paged cache's to the digits printed.
Left: the 2B's answer to "What happens in this video?" parted from the CPU's at up to 5 of 48 tokens (logits within 1.9, as the other runs): the test asks what the newspaper says, whose answer is the page's text; a split over two devices and the server's batches with a video were not run here.
L9. The vocoders on a device
oklm speak --backend put the language model on a device
and left the vocoder on the CPU: 0.30 s of a sentence of 1.67 s with
OuteTTS 0.3 on Vulkan, 0.16 of Soprano's 0.28 s.
- Where a call went (RX 7900 XT host, a sentence of 6.4 s of sound in two passes, milliseconds): WavTokenizer's network 97 and 150, its inverse transform 20 and 32; Soprano's network 39 and 53, its inverse transform 26 and 37. The transform was a fifth of the first and two fifths of the second.
- The network (
src/mm/vocoder.c,oklm_vocoder_use_backend(),vd_forward();oklm_speech_open()gives the vocoder the device the language model's layers are on, sooklm speak --backendand a server's--speak-backenddo it). No op was added: a convolution of kernel k is k products, one a position, the kernel copied to the device as that many matrices, each product over a view of the rows its position reads, accumulated, with the activations at F32's precision; an activation a convolution reads is kept between rows of zeros. The depthwise convolution of a ConvNeXt block is a product by a vector and a sum a position, in the CPU's order. WavTokenizer's group normalizations have no op and no such form: the stream is read back for each (ten a call), normalized on the CPU and written again. Its attention block of one head of 768 values is attended on the device by OpenCL and handed back to the CPU by Vulkan. - The inverse transform stays on the CPU, its frames on the worker team, 64 at a time, added to the sound in their order: the same samples to the bit (the 16-bit files of a sentence are the same bytes), 52 ms -> 9 ms and 63 ms -> 11 ms for the two sentences.
- Figures (RX 7900 XT,
--seed 1, the vocoder then the sentence; the CPU vocoder as it was, with the transform on the team, the network on the device). OuteTTS 0.3 on Vulkan: 0.30 s and 1.67 s, 0.25 to 0.26 s and 1.61 to 1.62 s, 0.06 s and 1.42 s; on OpenCL: not run, 0.26 s and 2.07 to 2.10 s, 0.05 s and 1.89 to 1.90 s. Soprano on Vulkan: 0.16 s and 0.28 s, 0.10 s and 0.23 s, 0.02 s and 0.14 s; on OpenCL: not run, 0.10 s and 0.25 s, 0.02 s and 0.16 to 0.17 s. The networks alone: WavTokenizer's 97 and 150 ms -> 23 and 29 ms (Vulkan), 17 and 25 ms (OpenCL), of which its "posnet" stack 16 and 21 ms and 10 and 16 ms; Soprano's 39 and 53 ms -> 4.8 and 5.2 ms, 3.5 and 3.9 ms. - The sound. The 16-bit files of the sentence differ
from the CPU vocoder's by one step at most, in 263 to 1 267 of 153 280
to 200 704 samples. In floats: WavTokenizer on the codes of "Hello
world" within 1.2e-6 (OpenCL) and 5.9e-6 (Vulkan), 125 and 110 dB;
Soprano's decoder on a sentence's states 3.5e-6 and 3.3e-6, 112 and 113
dB; the tiny files 117 to 132 dB. The language model's tokens are
unchanged (
test_published_devices: 92 of 96, 94 or 95, 95 of the CPU's best tokens, as before). - Checks.
test_conv_viewsintests/core/test_opencl.candtests/core/test_vulkan_gemm.c: the convolutions' products (kernels of 3 and 7, strides of 1 and 2, 96 to 1 280 channels) against the convolution in doubles, within 6.8e-8 (OpenCL) and 1.8e-7 (Vulkan) of the sum of the terms' magnitudes, the depthwise form and the sums of views against the same sums in floats; on the RX 7900 XT and the Intel UHD 750, both backends.test_speechandtest_soprano: the tiny and the published vocoders on each backend against the CPU's samples (80 dB asked), a shorter then a longer call, a second call's bits, back on the CPU the CPU's bits, a device with no room, a backend without buffers,OKLM_MM_DEVICE_ENDS=0; the tiny files on the Intel UHD 750 too (118 to 131 dB).test_serverwithOKLM_TEST_ONLY=test_device_encoderspasses. - Left: in the roadmap (L9, entry 8).
L9. The encoders' convolutions on a device
With the blocks of Whisper's encoder on a device, the host computed the spectrogram and the two convolutions before them, and Ultravox's projector after them.
- Measured first (jfk.wav, RX 7900 XT, Vulkan, large-v3-turbo's encoder, milliseconds): the spectrogram 24.0, the first convolution 6.4 and 0.6 for its windows, the second 23.7 and 2.7, their GELUs 1.8 and 1.4, the blocks 200; Ultravox's projector 28.0.
- The change (
src/mm/audio.c:wenc_convs_place(),wenc_device_convs(),ultravox_project_device()). The first convolution is the product it was, over the windows the host unfolds (their rows widened to a multiple of 32 values), uploaded in place of the stream. The second, of stride 2, reads the first's output on the device: its kernel is three matrices there, one a position, and it is three products over views of every second row from that position on, the first written and the others accumulated, after a row of zeros for the padding. Ultravox's projector reads the encoder's output where it is, with four rows of zeros after it, as 188 rows of eight frames. No op and no kernel were added. - Figures (in host memory -> on the device, three
runs each).
oklm transcribeof jfk.wav: large-v3-turbo 0.31 s -> 0.27 s on Vulkan and 0.38 to 0.40 s -> 0.35 s on OpenCL; large-v3 0.48 to 0.49 s -> 0.44 to 0.45 s and 0.63 s -> 0.59 s; base 0.08 to 0.09 s -> 0.08 s and 0.10 s -> 0.09 s. The prompt ofoklm describewith Ultravox: 0.33 s -> 0.26 s and 0.40 to 0.41 s -> 0.33 s; its encoder alone 0.298 s -> 0.230 s and 0.363 s -> 0.298 s, its embeddings 5.5e-5 -> 7.2e-5 and 3.1e-4 -> 2.4e-4 from the CPU's. - Checks. The texts of jfk.wav with base, small,
large-v3-turbo and large-v3 are the CPU's on both backends
(
test_mm_public); the tiny checkpoints' texts, segments, times and languages both ways on both backends and the tiny Ultravox encoder within 1.9e-6 (test_audio, 1 314 checks; 2.0e-6 on the Intel UHD 750); the second convolution's form as an op, with 1 280 channels (test_conv_views, the RX 7900 XT and the Intel UHD 750). - The spectrogram was measured and left: 24 ms of 0.27 s, in doubles on the worker team; on a device it would be a product by a table of cosines and a logarithm, which no backend has as an op.
- Left: in the roadmap (L9, entry 9).
L9. The patch embedding and the projector on a device
A vision encoder with its blocks on a device computed its patch embedding before them and its projector after them in host memory.
- Measured first (llama.cpp's test image, RX 7900 XT, Vulkan, milliseconds of one tile or image; the patch embedding, the projector, the whole encoder): Gemma 3 4B 21.5, 6.8, 721; Qwen2-VL 2B 7.7, 79.3, 341; LLaVA 1.5 7B 2.7, 67.0, 108; SmolVLM-256M 5.6, 4.1, 42; SmolVLM2-2.2B 4.0, 11.7, 89.
- The change (
src/mm/vision.c:dev_stream(),dev_project()). The host cuts the image into the patches' rows as before and uploads them in place of the stream; the patch kernel is on the device with its rows widened to a multiple of 32 values, the embedding is a product there (two for a pair of frames, the second accumulated), then the bias, the class token, the positions and CLIP's norm. The projectors are the ops they are made of: LLaVA's two layers over the rows after the class token, the mergers' over each group of four rows read as one; SmolVLM's pixel shuffle and Gemma 3's pool as views of a stream whose rows the device takes group by group (the positions copied in that order: every row sees every row, so nothing else changes), the pool being fifteen additions of views of every sixteenth row and a scale. The projector's rows are read back. No op and no kernel were added. - Figures (the blocks alone on the device -> the
whole encoder; the prompt of
oklm describe, three runs each). Vulkan: LLaVA 1.5 7B 0.40 to 0.41 s -> 0.33 to 0.34 s, LLaVA 1.6 1.87 to 1.89 s -> 1.53 to 1.54 s, Qwen2-VL 2B 0.44 s -> 0.35 s, Qwen2.5-VL 7B 0.55 s -> 0.45 s, SmolVLM-256M 0.29 to 0.30 s -> 0.25 s, SmolVLM2-2.2B 0.60 to 0.61 s -> 0.49 s, Gemma 3 4B 0.83 to 0.84 s -> 0.80 s. OpenCL: 0.40 to 0.42 s -> 0.33 to 0.36 s, 1.82 to 1.88 s -> 1.49 to 1.52 s, 0.47 s -> 0.38 s, 0.60 to 0.61 s -> 0.48 to 0.51 s, 0.33 s -> 0.28 s, 0.70 to 0.71 s -> 0.59 to 0.60 s, 0.89 to 0.90 s -> 0.86 s. The encoder alone on Vulkan: LLaVA 1.5 0.109 s -> 0.040 to 0.042 s, Qwen2-VL 2B 0.234 s -> 0.171 s, Qwen2.5-VL 7B 0.199 to 0.202 s -> 0.124 to 0.125 s, Gemma 3 0.733 s -> 0.689 s. - Checks. The embeddings against the CPU's
(
test_mm_public, two pictures, OpenCL then Vulkan): Gemma 3 1.4e-5 and 4.3e-5, 1.2e-5 and 2.8e-5; LLaVA 1.5 1.4e-5 and 2.1e-5, 2.5e-5 and 4.8e-5; Qwen2-VL 2B 6.4e-5 and 5.1e-5, 2.1e-5 and 3.0e-5; Qwen2.5-VL 7B 1.5e-5 and 1.1e-5, 8.0e-6 and 9.8e-6; SmolVLM-256M's overview tile 2.7e-6 and 4.0e-6, compared by the test since. The greedy answers from the device encoder's embeddings against the CPU embeddings': Qwen2-VL 2B 46 of 48 (OpenCL) and 47 (Vulkan), Qwen2.5-VL 7B 48 and 48.test_vision: every tiny encoder both ways (the whole encoder, the blocks alone) on both backends within 8e-7, 697 checks (9.6e-7 on the Intel UHD 750). No op of the new parts is handed back to the CPU on the published files. - Left: in the roadmap (L9, entry 10).
L9. OpenCL's prompts on Muse Glimmer's 2-bit matrices
OpenCL read a prompt of Muse Glimmer 30B UD-IQ2_XXS at 253 tokens a second where Vulkan reads it at 834 to 839.
- Where the prompt went (RX 7900 XT,
OKLM_CL_PROFILE=1, 512 tokens): 94 % of the kernel time in the tiled products of its four grid types,gm_iq2_s49.0 % (18.6 ms a launch),gm_iq3_xxs28.0 % (10.6 ms),gm_iq2_xxs11.7 %,gm_iq3_s5.4 %. These are the F32 tiles, one launch a matrix, the codebook decoded once a group of 32 weights and a tile of 64 columns: no kernel was missing, they are the slow family. The 8-bit matrix-unit kernels take six types (Q4_K, Q5_K, Q6_K, Q8_0, Q4_0, Q5_0) and every other quantized type was left on the F32 tiles, where Vulkan gives every prompt product its cooperative matrices with F16 inputs. WithOKLM_CL_WMMA=1(every product on the float units) the same prompt ran at 892. - The change
(
src/core/opencl/backend_cl.c,on_units()): on a gfx11 device the prompt products of quantized weights that have no 8-bit kernel take the float matrix units by default (w2_<type>,wm_<type>, andwi_<type>for the indexed products of a MoE prompt), F16 inputs; the six types keep the 8-bit kernels and the CPU's bits, float weights and the products asked at F32's precision (oklm_matmul_wide()) keep the F32 tiles.OKLM_CL_WMMA=0gives the F32 tiles back,1,f16andbf16mean what they meant; a compiler that builds the 8-bit kernels and not the float ones keeps the 8-bit ones (opencl.md, build.md).oklm_opencl_wmma_quant()tells a test which it is. - Figures (prompt of 512 / decode of 128 tokens,
tokens a second, the best of three; the F32 tiles, the default, Vulkan):
Muse Glimmer 253.3 / 29.4, 865 to 868 / 29.5, 834 to 839 / 37.1 to 37.3
(the CPU 20.8 / 2.41); a launch of
iq2_s18.6 -> 4.62 ms, ofiq3_xxs10.6 -> 2.42 ms. Ternary Bonsai 2 4BPQ2_0, which shares the kernels: 1276 / 136.7, 2544 / 137.3, 3411 / 172.4. gpt-oss-20b (its MXFP4 experts' indexed products), downloaded a second time for this: 1680 / 126.1, 3313 / 126.4, 3738 / 163.9. Llama-3.2-1B Q4_K_M, no such type: 14 568 to 14 644 / 391 to 406 withOKLM_CL_WMMA=0, 14 545 to 14 643 / 404 to 407 by default. - Checks.
tests/core/test_opencl_q8.c(the process with the default policy): the 21 types without an 8-bit kernel at 200 rows by 150 columns in a batch of two and by 40 columns (K split, the W2 and WM kernels), seven of them at 1030 by 1000 (whole and partial tiles), indexed products of IQ2_XXS and MXFP4, against a double reference: within 1.1e-4 of the sum of magnitudes (the limit 2e-3); the same products asked at F32's precision within 3.1e-7 and other bits (the route).test_opencl(the F32 tiles of every type,OKLM_CL_WMMA=0now set there) on the RX 7900 XT and the Intel UHD 750,test_opencl_wmmaon the RX 7900 XT. Muse Glimmer on OpenCL and Vulkan against llama.cpp's recording and the CPU: 156 and 157 of 160 choices, logits within 3.66 and 3.63, 0.171 on average; 157 and 156 from the CPU, 4.36, 0.163: the figures of L6 on both. gpt-oss-20b on OpenCL: 158 of 160 and 155 from the CPU as before, the largest differences 4.05 and 2.88 (4.03 and 2.50 before).test_bonsai_publicandtest_modelpass. Perplexity over 1275 tokens (the CPU, the F32 tiles, the default, Vulkan): Muse Glimmer 30.7146, 30.7098, 30.7160, 30.7671; Bonsai 4B 47.9792, 47.9281, 47.9281, 47.9271; gpt-oss-20b 64.3276, 61.8951, 61.6072, 63.2014. - Muse Glimmer's file (10.7 GB, SHA-256
3e9bd0e6...00875ad9) and gpt-oss-20b's (11.6 GB,c2753664...568aeb4f) were downloaded, run and removed, one on the disk at a time. - Left: in the roadmap (L9, entry 7).
L9. gpt-oss decoding on Vulkan
gpt-oss-20b decoded at 96 tokens a second on Vulkan where OpenCL gives 127, the one routed model below OpenCL's rate (Tini-Cybersec-8B-A1B: 340 against 256).
- Where the step went (RX 7900 XT,
bench_llama, 128 tokens, each kind of op left out of the step in turn): of 10.34 ms, 6.67 were the 72 indexed products of the experts (four experts of 2880 x 2880 in MXFP4 each: 92 us a product, 190 GB/s of weights). The attention with sinks took 0.25 ms, the routing 0.4, the gathered biases and the gated unit less than the spread; nothing is read back, and the step replays. OpenCL's kernel for the same product takes 32 us (OKLM_CL_PROFILE=1). Tini's experts (Q4_K, Q6_K) go through the same shader and dispatches: the cause was the weight type's. - The cause
(
src/core/vulkan/shaders/dequant.glsl): MXFP4's decoder took each weight's value from a constant array of 16 integers indexed by its code, which RADV compiles to a memory load per weight. NVFP4 read the same array, IQ4_NL and IQ4_XS another one. - The fix: the two codebooks computed from constants
in registers (
dq_fp4(): the magnitudes as the 4-bit fields of one word, bit 3 the sign;dq_iq4nl(): the grid as signed bytes in four words), to the integers the arrays held, so every product has the bits it had (vulkan.md). - Figures (prompt of 512 / decode of 128 tokens, tokens a second, the best of three runs): gpt-oss-20b on Vulkan 2160 / 96.7 before, 3738 / 163.9 after, OpenCL 1681 / 126.3 in the same session; a product 34 us, the step 6.10 ms (2.43 the indexed products, 0.8 the rest of the expert layers, 2.9 the attention blocks and the head). Tini-Cybersec-8B-A1B on Vulkan 7859 / 339.7 before, 7824 / 342.2 after (OpenCL 7149 / 255.6). SmolLM2-135M IQ4_XS on Vulkan 23 790 / 391 before, 33 600 to 35 450 / 650 to 707 after.
- Checks. gpt-oss-20b on Vulkan
(
test_archs_public): 157 of 160 choices equal to llama.cpp's, logits within 4.05, 0.081 on average; 156 of 160 equal to the CPU's, within 2.69, 0.067 on average; a paged cache 16 of 16, half of the blocks on the CPU 15 of 16: the figures of the day's first run.test_vulkan_moe,test_vulkan_matmul,test_vulkan_gemmandtest_vulkan_q8pass on the RX 7900 XT and the Intel UHD 750 (MXFP4, NVFP4, IQ4_NL and IQ4_XS among their types). - The file (11.6 GB, SHA-256
c2753664...568aeb4f) was downloaded, run and removed. - Left: in the roadmap (L9, entry 6).
L9. The chat windows attach a video
The server took a video in a chat request; the graphical chat attached pictures, recordings and documents only, and the page's change that attaches one had been checked for syntax alone.
- A kind in one place
(
apps/gui/chat_attach.h). An attachment is a picture, a recording, a video or a document, and what depends on that is asked of five functions: whether it is a part of its own in a request (att_kind_media()), which of a link's two answers (images, sounds) it waits for (att_kind_asks()), whether the link takes it (att_kind_taken(): a video where either is taken), which answer tells why not, its letters in a box and its name in a sentence. The window askedkind == ATT_IMAGEorkind != ATT_DOCin each place; it calls these. - A video (gui.md) is told by its
extension (
.ogv, and with ffmpeg MP4, M4V, Matroska, WebM, QuickTime, AVI, MPEG, WMV, FLV;.oggstays a recording,.tsa text), through every way a file is attached: the file dialog (it lists all files), a drop, the path's field,/attachand--attachare one function. The window talks to a server, its own or another: the video goes whole as avideo_urlpart (server.md) and the server samples it, one video item for Qwen2-VL and Qwen2.5-VL, frames as images for the other image projectors, the sound track for an audio one; the window holds no copy of that code. It decodes the file once with the library's reader (oklm_video_load(), the server's default sampling and its limit on decoded frames) for the thumbnail, which is the first frame, and for the length and the frames shown in the box and in the conversation (clip.ogv Ogg video, 73 KiB, 4.0 s, 4 frames). - ffmpeg.
oklm-gui --ffmpeg(new) allows it for the window's decoding and for the window's own server (oklm_server_params.video_ffmpeg); without it another container is refused when attached, with the decoder's sentence. On a server at an address the file is attached without a thumbnail: that server may have ffmpeg, and its refusal is shown if not. - The page was right as written.
tools/check_dashboard.mjshas a group for it (CHECKS=video) and a line in the chat group. - Checked (CPU).
test_gui_attach(167 checks): the kinds by extension, each video extension one the library's reader names a video, the five functions over every pair of answers, the request'svideo_urlpart beside an image's, a refused video left out, the line of a video with and without its length.make gui-testhas the case: the fixture video on the tiny Qwen2-VL projector,Attached: testsrc.ogv Ogg video, 14 KiB, 2.0 s, 2 frames, the reply's 183 prompt tokens, the window saved. By hand, the same way: on the tiny Ultravox projector 33 prompt tokens (what the fixture's sound alone gives) and an answer; without a projector refused before sending; a PNG, an Ogg sound and a PDF attached and answered as before, and their two refusals; the clip below as MP4 refused without--ffmpeg, 1 579 prompt tokens with it, and the same 1 579 throughoklm serve --ffmpegat an address with the window started without (no thumbnail, one item encoded). - Published file (Qwen2-VL 2B Q4_K_M, the CPU, the
clip of the entry below,
/temp 0,/max-tokens 64,What does the video show at its start and at its end?): 418 prompt tokens and the 64 tokensoklm describeandoklm servegive ("The video starts with a close-up shot of a newspaper with the headline "Men Walk on Moon" [...] The video ends with the same black and white"), in 21.7 s. - The page in a browser (Chrome 157 headless,
CHECKS=video sh tools/check_dashboard.sh, the tiny model with the tiny Qwen2-VL projector): 14 checks pass. The fixture video dropped is attached with the file's base64, its row has its name and no picture, the request holds avideo_urlpart withdata:video/ogg;base64,and the file before the text and nofpsorframes, the server encodes one item and the reply is written, the next message sends the part again,/export jsonkeeps it, a file without a type is a video by its extension, bytes that are no video come back with the server'sthe video cannot be decoded (only Ogg Theora ...). The chat group (156 checks pass) refuses a video on a server without a projector.
Left: the window's file dialog, a drop and the path's field were not
driven with a video (they hand a path to the function
--attach calls); the window sends no fps or
frames (the server's defaults) and offers no Transcribe for
a video's sound; a video above the limit on decoded frames and one above
20 MiB are refused by code that was not run; the page's video group was
not run in Firefox or WebKit; nothing here ran on a device.
L9. The encoders' embeddings on Vulkan
With an encoder's blocks on Vulkan the embeddings were 2.6e-3 to 1.5e-2 from the CPU's (relative norm; Qwen2-VL 2B the worst, its activations reaching 80) where OpenCL's are 1.4e-5 to 5.3e-5: the prompt products run on cooperative matrices, which take the activations rounded to F16.
Where the error enters (vulkan.md). The units accumulate in F32 and every result is stored as F32: it is the rounding of the activations on the way in. On Qwen2-VL 2B, with one group of a block's products left on the rounded activations and the others not (two pictures): the queries, keys and values 1.3e-3 and 2.5e-3, the attention's output 3.5e-3 and 4.2e-3, the MLP's first product 1.8e-2 and 1.2e-2, its second 4.1e-3 and 5.1e-3; all six rounded 1.5e-2 and 1.1e-2, none 1.9e-5 and 3.0e-5. With
OKLM_VK_COOPMAT=0(every product on the F32 tiles) 5.3e-5 and 4.6e-5. The merger and the patch embedding are host work.oklm_matmul_wide()(core.md):OKLM_OP_MATMULwithiparams[1] = 1, a product whose activations need F32's precision. The blocks of the vision encoders (src/mm/vision.c) and of Whisper and the Ultravox encoder (src/mm/audio.c: the encoder's products, the keys and values of the cross-attention) ask for it on a device. The CPU and OpenCL run it as any product; the language models never set it and keep their shader variant,gm_tohalf's values and their bits (the logit figures oftest_mm_public's language models on Vulkan are the same to every printed digit before and after).Two F16 parts on the units (
cm.glslwith the specialization constantWIDE,gm_tohalfwithpc.wide): the half nearest each activation and, in a second copy, what that rounding dropped; each 16 x 16 x 16 block multiplies the weights by both into one F32 accumulator, and the accumulators join a second set every 128 values of k. The candidates, on Qwen2-VL 2B (embeddings from the CPU's, the encoder alone on 273 embeddings, the prompt ofoklm describe):Embeddings Encoder Prompt activations rounded to F16 (before; OKLM_VK_WIDE=0)1.5e-2, 1.1e-2 0.18 s 0.37 to 0.38 s two F16 parts, one running sum 1.7e-4, 3.2e-4 0.22 s two F16 parts, partial sums every 64 values of k 2.2e-5, 2.2e-5 0.24 to 0.25 s two F16 parts, every 128 (the default) 1.9e-5, 3.0e-5 0.23 to 0.24 s 0.43 to 0.45 s two F16 parts, every 256 2.8e-5, 4.8e-5 0.23 to 0.24 s the F32 tiles for these products ( OKLM_VK_WIDE=f32)5.3e-5, 4.6e-5 0.28 to 0.29 s 0.50 s The RX 7900 XT lists no cooperative matrix with F32 inputs, so F32 activations on the units were not a candidate. A block's six products at width 1 280 over 1 092 rows take 1.42 ms rounded, 3.25 ms as two parts and 4.30 ms on the F32 tiles (
test_vulkan_gemm,OKLM_VK_BENCH=1 OKLM_VKT_BENCH_WIDE=1). The first form ofgm_tohalfwrote the dropped part as x - float(half(x)): RADV gives 0 for it, so the nearest half is made on the F32's bits. Weights that are not F16 (BF16, quantized: their own rounding would remain) take the F32 tiles for such a product, as does a device with less than 32 768 bytes of shared memory.Every encoder (multimodal.md; RX 7900 XT, one build,
OKLM_VK_WIDE=0-> the default):Embeddings from the CPU's Encoder alone Prompt ( oklm describe)Qwen2-VL 2B 1.5e-2, 1.1e-2 -> 1.9e-5, 3.0e-5 0.18 -> 0.23 to 0.24 s 0.37 to 0.38 -> 0.43 to 0.45 s Qwen2.5-VL 7B 6.3e-3, 4.8e-3 -> 1.1e-5, 9.6e-6 0.15 to 0.16 -> 0.20 s 0.48 to 0.50 -> 0.55 to 0.56 s Gemma 3 4B 2.6e-3, 5.1e-3 -> 1.2e-5, 2.8e-5 0.63 -> 0.73 s 0.74 -> 0.84 s LLaVA 1.5 7B 5.6e-3, 7.2e-3 -> 2.3e-5, 5.0e-5 0.11 -> 0.11 s 0.40 to 0.41 -> 0.41 to 0.42 s LLaVA 1.6 Mistral 7B 1.82 to 1.83 -> 1.89 s SmolVLM2-2.2B 0.49 to 0.50 -> 0.60 s SmolVLM-256M 0.26 -> 0.29 to 0.31 s Ultravox on jfk.wav 9.6e-3 -> 5.5e-5 0.23 -> 0.30 s 0.27 -> 0.33 s Whisper large-v3-turbo, jfk.wav transcribed 0.24 to 0.25 -> 0.31 to 0.32 s Whisper large-v3, the same 0.41 -> 0.49 s At most about 1.3 times an encoder's time and 0.10 s a picture, so it is the default for all of them;
OKLM_VK_WIDE(build.md) gives the rounded products (0) or the F32 tiles (f32). The tiny files oftest_visionare 3.0e-7 to 6.0e-7 from the CPU's (1.3e-4 to 4.1e-4 before).The answers. With the language model on Vulkan, the newspaper answer from the Vulkan encoder's embeddings against the CPU's: 47 of 48 greedy choices for Qwen2-VL 2B (47 before; the two tokens 0.24 apart where they part) and 47 for Qwen2.5-VL 7B (48 before; 0.11 apart). The language model's own prompt products round their inputs to F16, a difference of 1e-5 as one of 1e-2: the two runs' logits are as far apart as before (within 1.07 and 0.93; 0.90 and 1.02) and a near tie falls either way. Whisper's texts of jfk.wav are the same; large-v3-turbo's probability of English is 0.9499 (the CPU 0.9498, the rounded products 0.9508).
Tolerances.
test_vulkan_gemmhastest_wide: products with activations of +-80 within1e-6 sum |w x| + 1e-6of a double sum (partial tiles, a split, a batch, strided columns, accumulated; F16 by default and withOKLM_VK_WIDE=f32, F32, BF16, Q8_0 and Q4_K), andOKLM_VK_WIDE=0must leave outputs beyond it where the device has the units (1 964 to 54 327 of them; worst 1.8e-5 to 7.4e-5 of sum |w x| against 8e-8 to 3.0e-7).ENC_VK_TOLoftest_mm_public2e-2 -> 2e-4, Vulkan's bound intest_vision2e-3 -> 1e-4 and intest_audio2e-3 -> 1e-4 (OpenCL's).Checked once after the last edit:
test_vulkan_gemmon the RX 7900 XT and the Intel UHD 750 (69 183 checks; the UHD 750 has no cooperative matrices and runs these products on its F32 tiles, 3e-8 to 1.4e-7 of sum |w x|),test_wideagain withOKLM_VK_GEMM_SG=32,test_visionon both devices (537 checks each),test_audio(794),test_mm_public(5 632),test_device_encodersoftest_server(285), all without a failure.
Left: in the roadmap under L9, entry 21.
L9. Three leftovers of
oklm describe
--kv-saveruns the encoders once (multimodal.md).oklm_mm_eval()did not say which rows it evaluated were a media item's, sooklm describe --kv-savelaid the prompt out a second time (oklm_mm_build()), encoders included.oklm_mm_eval_rows()evaluates and names each row on the way, a token or an item's hash and index, without keeping the embeddings;oklm_mm_eval()is that call without the names. Checked (CPU):test_media_describe,test_video_describeandtest_sound_describe(test_server) hold the names tooklm_mm_build()'s rows, one for one, on the tiny projector of each family, a video item and a sound. Qwen2-VL 2B (Q4_K_M,--temp 0 -n 16, the newspaper picture, 391 embeddings): 19.5 s of wall time before, 10.4 s after, the two KV files the same bytes (434 rows, 12.5 MB); the next question from the new file in a new process answers; a video with--thenand--kv-savewrites its 459 rows.- The default question names what is shown (multimodal.md). Without
-p,oklm describeasked "Describe this image." about a sound and about a video. It asks "Describe this sound." of an audio model and "Describe this video." of a vision model given a video. Run (CPU, greedy): Ultravox v0.5 1B on jfk.wav without-p, 228 prompt tokens, an answer about the sentence it heard and not about a picture ("The sound you've described is a phrase that has become iconic in American culture, [...]", which it then attributes to the wrong speaker: the 1B's own); Qwen2-VL 2B on the clip of the video entry without-p, 409 prompt tokens, "The video shows a close-up of a newspaper with the headline "Men Walk on Moon." The text below the headline reads, "Astronauts land on plain, collect rocks, plant flag."". - A video's sound where the model's order puts a
sound (multimodal.md). For
an audio projector
oklm describewroteVideo NAME (D s):,Sound:and the sound's marker, then the question: the sound before the question, which is not the order of Ultravox's own prompt. The video is now its sound track, a sound item like another, whichoklm_mm_format()puts after the question and a space for Ultravox, as the server's chat routes do with a video part; a video without sound is refused with a message where it was asked about with nothing to hear. No model here takes a video's frames and its sound together (a projector file holds one encoder), which the page now says. Run (Ultravox v0.5 1B, the CPU, greedy, "Transcribe this.", jfk.wav as the Vorbis sound track of an Ogg Theora video): 243 prompt tokens before and an answer that speaks of the quote first ("The given text appears to be a quote from John F. Kennedy's inaugural address [...]"); 228 after, the count of jfk.wav alone, and the sentence ("[...] "And so, my fellow Americans ask not what your country can do for you, ask what you can do for your country." This is a famous quote from John F"); the silent clip is refused ("has no sound track, and ... has an audio encoder").
Left: none of the three run on a device (they change no computation
there); a default question in the server's chat routes (a request always
has its own text); oklm describe has no test binary of its
own, the three were run by hand on the published files.
L9. A video in the server's chat requests
oklm describe gave a video to a model; the server's chat
routes took images and sounds only.
- The request (server.md): a
video_urlpart, base64 in adata:URL, the shape of the OpenAI-compatible servers that take video;fpsandframeson the part (or in itsvideo_urlobject) areoklm describe's--fpsand--frames, with its defaults (1 and 8, at most 32),--tilesthe server's own, andoklm serve --ffmpegits--ffmpeg./v1/messagestakes avideoblock shaped as Anthropic'simageblock (the format has none). - One decoder, one layout.
oklm_video_decode()reads the request's bytes with the readeroklm describeuses (multimodal.md); the seconds between the frames kept (oklm_video_frame_seconds()) and the lines announcing the frames to an encoder of still images (oklm_mm_video_text()) moved from the command to the library, and both call them. Qwen2-VL and Qwen2.5-VL get the video as one item (frame pairs, the time position), the other image projectors the frames as images afterFrame at M:SS:lines, an audio projector the video's sound track as a sound. - The scheduler gave a media item's rows to the model
with their grid's columns only (
oklm_eval_batch_grid()): it gives a video's with the rows of a grid and the time step too (oklm_eval_batch_video()), whatever pieces its steps cut the video into. - A bound on what a request decodes.
oklm_video_params.max_pixels: the frames asked for times a frame's width and height, checked from the stream's headers before a frame is decoded (the first picture's header through ffmpeg). The server allows 66 355 200 pixels (8 frames of 3840 x 2160, 32 of 1920 x 1080;oklm_server_params.video_pixels) and answers 400 with the figures above it; 33 frames or more, anfpsof 0, bytes that are no video, a remote URL, a video for a server without a projector and a video without sound for an audio projector are each refused with their own message. - The caches. A video item's hash is of its frames, their count, their spacing and the slicing side, so the server's embedding cache and the prefix cache find the same video with the same sampling and not another sampling; as still images each frame is hashed alone, and the frames two samplings share are found again. The prefix directory keeps a head holding a video by the same hashes.
- The chat windows. The page the server serves
attaches a video as it attaches an image (by its type or extension, a
video_urlpart).oklm chatattaches documents only, no image either. The graphical chat is left (below). - Checked (CPU; the device case is written, not run).
test_video_describe(test_server), on the fixture video (4 frames atfps2) and the tiny projectors: the chat route's rows are describe's (331 forqwen2vl_merger, 75 forqwen2.5vl_merger, one item; 1 123 for LLaVA 1.5, four images), the same greedy answer; the request sent again finds every row but the last in the prefix cache and its items' embeddings in the server's (1 encoded, 1 reused; 4 and 4);fps1 encodes one more video item, and no more image for LLaVA (its two frames are among the four);/v1/messagesgives the same row count; a server started again on its prefix directory finds 320 of the 331 rows and gives the same answer; the refusals above, each by its message; a file cut in its middle is answered or refused, the server unharmed.test_video: the file from memory gives the file's frames byte for byte; the limit refuses 8 frames of 64 x 48 one pixel under their size and passes them at it, natively and through ffmpeg. - Published file (Qwen2-VL 2B Q4_K_M, the CPU,
greedy; a clip made from llama.cpp's test image, four frames a second
apart of 448 x 336, the page for two and its top left corner enlarged
for two,
ffmpeg -loop 1 -framerate 1 -t 2 -i test-1.jpeg -loop 1 -framerate 1 -t 2 -i test-1.jpeg -filter_complex "[0]scale=448:336[a];[1]crop=iw/2:ih/2:0:0,scale=448:336[b];[a][b]concat=n=2:v=1" -r 1 -c:v libtheora -q:v 8 clip.ogv): throughoklm serveand throughoklm describe --temp 0, 384 embeddings in 2 pairs, 418 prompt tokens and the same 64 tokens ("The video starts with a close-up shot of a newspaper with the headline "Men Walk on Moon" and the text "Astronauts Land on Plain, Collect Rocks, Plant Flag." It then cuts to a black and white image of a man walking on the moon. The video ends with the same black and white"); sent again, 416 of the 418 rows cached and the embeddings reused;/v1/messagesthe same text. The clip as H.264 in MP4: refused without--ffmpeg(the message names both ways out), 418 prompt tokens and the same first 24 tokens with it. Ultravox v0.5 1B given the fixture video (a 440 Hz tone) after a question: 229 prompt tokens, an answer; given the silent clip: refused, no sound track.
Left: not run on OpenCL or Vulkan (test_device_encoders
runs test_video_describe on each device found, the encoder
then on the model's device); the graphical chat attaches no video (its
attachments are a picture or a recording, told apart by one test in a
dozen places, and a picture's row draws its thumbnail: a third kind is a
change of its own); the page's change was checked for syntax only, not
in a browser (tools/check_dashboard.mjs has no case for
it); a video is decoded again at every request (its hash is of the
frames); the processors' own sampling (two frames a second, at least
four frames) is not applied; a video's sound and its frames are never
given together (no projector file here holds both encoders); a published
file with the frames as still images (SmolVLM, Gemma 3, LLaVA) was not
run through the server.
L9. A KV file of a sequence that holds an image
oklm_kv_save() refused a sequence of a
qwen2vl model (Qwen2-VL, Qwen2.5-VL) that held an image:
the file kept a token id a row, the row at the position of its index,
where an image's rows have three positions and the token after an image
does not take the position its row count gives.
- What a context keeps of such rows
(
src/model/mrope.c): the keys in the cache are already rotated, so of the three positions it keeps, per row, the time position and the position a token after the row takes. A cut, a continued image and the next token need no more. The file keeps those two per row, not the grids' descriptions: they are the context's own state, 8 bytes a row beside key and value rows of tens of kilobytes, and a load that stops inside an image needs nothing worked out again. - Version 2 of the format (inference.md,
src/model/kv_file.c): after a chunk's token ids, per row a media hash (64 bits), its time position and the position after it (32 bits each), under the chunk's hash; the header's padding word says 1. It is written only for a sequence with a row that has a media hash or a position that is not its index. A sequence of tokens alone is written as version 1, byte for byte as before, so the builds before this one read it; they refuse a version 2 file as not a KV file, which is the safe answer (they would read its rows at the wrong positions). Both versions are read. A version after 2 is refused withOKLM_ERR_UNSUPPORTEDand a log line naming the versions. - The media hashes
(
oklm_kv_save_media(),oklm_kv_load_media(),oklm_kv_file_rows()): a paged cache that loaded a file recorded every row as a token, an image's rows too, under the index the caller gave them. With the hashes in the file it records them as the media rows they are, and its blocks keep their positions (model_mrope_blocks_store()), so the prefix cache serves them to a request with the same media. This holds for the models whose image rows take one position each as well (Gemma 3, LLaVA, SmolVLM): they were saved and loaded before, as tokens, and had no test; they have one now (plain embedding rows on the tiny llama model, rows that see each other on the tiny gemma3 one). - Who writes such files now.
oklm serve --prefix-dirkeeps the heads of requests holding media (it skipped them), the file's name hashing the media hashes with the tokens (server.md).oklm describe --kv-save FILEwrites the conversation after the answer,--kv-load FILE -p QUESTIONasks the next question in a later run without the picture,--then QUESTIONasks it in the same run (multimodal.md).oklm chat --kv-saveand the agent's hold no media; they go through the same writer and still write version 1. - Checked (the CPU):
test_kv_file, 870 checks: the tinyqwen2vlmodel saved after an image, after a video, after an image of one column, inside the image and the video and mid-way through the text, loaded into a fresh context, contiguous and paged either way, the next logits and the position counter those of the context that never left memory, bit for bit; version 1 for the text before the image, the file written again as version 2 when the image comes, then appended to; a torn append gives the previous save with its positions; a row section cut in the hashes or in the positions, a changed hash, time position or next position, versions 0 and 3: refused, the sequence left empty.test_model'stest_mrope: 100 saves over the 14 cases of the numpy reference, each loaded contiguous and paged, the logits the reference's within 4e-5 as before. On the published Qwen2-VL 2B (Q4_K_M,--temp 0):oklm describeof a picture saved after its answer (480 rows, 13.8 MB), then a second question in a new process from the file, give byte for byte the text of one run asking both (64 and 8 tokens); the server, started again on its prefix directory, reported 400 cached tokens of the request's 416 rows and gave the same answer.
Left: not run on OpenCL or Vulkan (the tests have the device cases:
test_kv_file's test_three_part and
test_model's test_mrope run them where a
device is found); oklm describe --kv-save runs the encoders
a second time to learn which rows are the media's
(oklm_mm_eval() does not return them); a next turn with a
picture of its own after --kv-load or --then
(text only); the server's prefix directory with an image has no test of
its own (one run by hand; the cache it rests on is tested); the check of
positions that hash right and make no sense has no test (the file's hash
is not reachable from a test); sounds and videos in a saved conversation
were not run on a published file (an image was); a hybrid stack's kept
state in a KV file is as before (L5).
L9. Videos as Qwen2-VL and Qwen2.5-VL define them
oklm describe gave a video's frames to these two models
as still images, each with its time written before it. The models define
a video otherwise: frames two by two through a convolution over two
frames, one run of embeddings, and a time position that advances along
it.
What the models' code does, read in transformers (
video_processing_qwen2_vl.py,modeling_qwen2_vl.py,modeling_qwen2_5_vl.pyat 4.57.1, the latter at 4.49.0 too): the frames at one size, 128 to 768 embeddings a frame; an odd count repeats its last frame; the patch embedding's first kernel on a pair's first frame, the second on its second; the encoder's blocks and Qwen2.5-VL's windows within a pair's grid only; one<|video_pad|>an embedding between<|vision_start|>and<|vision_end|>, no text for the time; inget_rope_index()a row's height and width are the start plus its row and column in its pair's grid, its time the start plus the pair's index (Qwen2-VL) or the index times the seconds a pair covers timestokens_per_second= 2, truncated (Qwen2.5-VL); the text after the video continues from the largest position plus one.The encoder (multimodal.md):
oklm_vision_encode_pair()keeps the file's two kernels apart for a pair and uses their sum, as before, for a frame that stands twice (a still image, the last frame of an odd count): the image path computes what it computed.oklm_vision_frame_size()andoklm_vision_frame()give a frame the video processor's size (--tiles SIDEfor a smaller one),oklm_vision_time_step()the positions a pair advances.The positions (
src/model/mrope.c):oklm_eval_embd_video()andoklm_eval_batch_video()take the rows with the columns and rows of a grid and the time step;oklm_eval_embd_grid()is the case of one grid. The context's two numbers a row still suffice: a video given in pieces, cut in its middle, copied, or found in the paged cache's prefix cache has its start back from its last row's time less what that row's grid has advanced. A video's grids hold at least two embeddings (one is refused: its rows could not be told from tokens).The prompt (
src/mm/prompt.c): a media item may be a video's frames (oklm_mm_item.frames, the seconds between them); for these encoders it becomes the pairs' grids between the two vision tokens, the item carrying the grid and the step (oklm_mm_media.grid_h,t_step); another encoder refuses the item, andoklm describegives it the frames as images as before. One or two frames are one grid, placed as an image's rows.Checked (CPU; the device cases are in the tests and were not run).
test_vision: on the tiny Qwen2-VL and Qwen2.5-VL files, a video of three frames against the numpy rewrite of the models' code (tools/gen_vision_fixtures.py): max difference 5.2e-7 and 3.8e-7 on values up to 2.1 and 1.8; the odd last frame equal to its still image bit for bit, a frame given twice by its pixels within 4.8e-7 of it; the frame size of the processor's rule; in a prompt, 270 and 30 rows for three frames (the processor's count), the two vision texts around them, the grid and step on the item, one frame's rows those of the image.tools/vision_ref.py --second(the reference for published files) agrees with that rewrite on the tiny files to 3e-8.test_model: the tinyqwen2vlmodel on the cases oftools/gen_mrope_fixture.py, six of them videos (steps of 1, 2, 4, 1.5 and 0.5 a grid, a video then an image, one grid), seven ways each (one call, pieces of 4 rows, a batch with another sequence on a contiguous and on a paged cache, a cut in the video's middle, a copy, rows from the prefix cache): logits within 3.9e-5 of the numpy pass, the next position the reference's. The generator asserts its layout againstget_rope_index()written again statement for statement, with torch's float32 arithmetic for Qwen2.5-VL's time: on its cases and on 98 videos of the published models' grid (23 x 17 embeddings, 1 to 64 pairs, 0.5 to 30 frames a second, both models).Published files (CPU, greedy, a clip made from llama.cpp's test image: four frames a second apart of 448 x 336, the page for two and its top left corner enlarged for two; 384 embeddings in 2 pairs, 423 prompt tokens): Qwen2-VL 2B, prompt 8.7 s, "The video starts with a close-up shot of a newspaper, specifically the New York Times, and then transitions to a black and white image of a man walking on the moon. The video ends with a shot of the moon."; Qwen2.5-VL 7B, prompt 10.8 s, asked for the two parts: "The first part of the video shows a newspaper front page from The New York Times dated July 20, 1969. The headline reads "MEN WALK ON MOON," with a subheading that says "Astronauts Land on Plain; Collect Rocks, Plant Flag." [...]", cut at 110 tokens before the second part. The still page through the 2B gives the answer and the counts it gave before the change ("The headline is "Men Walk on Moon."", 192 embeddings, 218 prompt tokens).
Found on the way: an image of one column (or of one row) takes as many positions as it has rows, so the position after it is the row count, which is what
oklm_kv_save()looked at to tell a sequence of tokens: such a sequence was written to a KV file, whose one position a row then gave the image's rows a token's positions when read back. The check now reads every row's time and next position; the fixture has the case (a column of four after four tokens), refused as the other images are.
Left: OpenCL and Vulkan were not run (the encoder's blocks take a pair as they take an image, and the rotary operation takes any three positions: the tests' device cases say whether that holds); the server has no video part in its chat requests; the processors' own sampling (two frames a second, at least four frames) is not applied; no reference for a video on the published files, where only the answers were read; Qwen2.5-VL's time when a pair covers a fraction of a second follows transformers 4.49.0 (the published form), where 4.57.1 truncates the seconds first.
L9. A question about a sound in Ultravox's own order
oklm describe -p "Transcribe this." jfk.wav with
Ultravox v0.5 (Llama-3.2-1B) answered with a quote of "My Country, 'Tis
of Thee" under greedy decoding, on every backend; the server's chat
route given the question then the sound transcribed the sentence.
- The cause: the order. Both wrote the turn through
oklm_mm_content();oklm describeput the sound first and the question after a new line, the request had them the other way. No system message on either side, the same template, the same marker. Through the server, greedy, on the CPU: the sound then the question gives the hymn (228 tokens of prompt), the question then the sound the sentence (227), the sound alone a text about the Gettysburg Address. Withoklm describe's default sampling the sentence came out of the first order too, which hid it. - The reference: Ultravox's own pipeline
(
ultravox_pipeline.py) appends the sound to the user's text after a space (prompt += " <|audio|>"), and adds no system message. The language model's template takes a string and says nothing of a sound. - The change (multimodal.md, server.md): the
ultravoxrow of the projectors' table insrc/mm/prompt.c: a space before a sound that comes right after a text, andoklm_mm_format()(oklm describe) writes the text, then the sounds. A text after a sound keeps its new line: the pipeline never writes one. In the server a text part followed by a sound gains the space (228 tokens). - Checked (CPU):
test_sound_describe(test_server), on the tiny audio encoder: a chat request holding the question then the sound has describe's 41 rows, the same greedy answer, and describe's prompt sent after it as a raw completion finds 40 of them in the prefix cache;test_content_parts(test_vision): the contents. The published file, jfk.wav, "Transcribe this.", greedy:oklm describeand the chat route 228 tokens of prompt each and the same 40 tokens ("Here's the transcribed text: "And so, my fellow Americans ask not what your country can do for you, ask what you can do for your country." This is a famous quote from John F"). A test helper cut an answer at a byte 0 the random model wrote; it reads the answer's length now.
Left: not run on a device (the prompt's text does not depend on one);
a video's sound in oklm describe keeps its own lines
("Sound:", then the question); oklm describe's default
question for a sound is still "Describe this image.".
L9. A picture and a question as each model's template writes them
oklm describe, the server's chat routes and the
retrieval's captions wrote a new line between an image and the text
after it for every projector but Qwen's. That is LLaVA's form. The
owner's decision (2026-11-14): Gemma 3 and SmolVLM follow their
templates.
- One table, by projector (multimodal.md, server.md):
oklm_mm_content()reads what follows a marker from one table insrc/mm/prompt.c.idefics3,gemma3,qwen2vl_merger,qwen2.5vl_merger: nothing.mlp(LLaVA 1.5 and 1.6) and a projector not known: a new line. - The references, each checked. The template of each
published file rendered with content in parts (an image, then a text):
User:<image>What is this?for SmolVLM-256M, 500M and SmolVLM2-2.2B,<start_of_image>What is this?for Gemma 3 4B,<|vision_start|><|image_pad|><|vision_end|>What is this?for Qwen2-VL 2B and Qwen2.5-VL 7B. LLaVA 1.6's template takes a string and LLaVA 1.5's files have none: the form is the model's own conversations',<image>\n. llama.cpp at the pinned revision joins a marker and a text with nothing for all of them (common/chat.cpp,mtmd-cli). Gemma 3's Transformers processor writes two new lines before the image's tokens and two after, which neither the template nor llama.cpp does: not followed, and the page that said it put "an empty line there" now says what it writes. - The recording (
tools/gen_mm_ref.sh, CPU): llama.cpp's answer for Gemma 3 4B made again from the new prompt (273 rows where there were 274), and a second one from the embeddings OKLM computes, as Qwen's have. SmolVLM has no recorded answer; every other file underbuild/ref/mmcame out byte for byte the same. The reference tool given the earlier prompt writes the earlier recording bit for bit: the change below is the prompt's. - Checked (CPU,
test_mm_public, 48 greedy tokens about llama.cpp's test image). Against llama.cpp's whole path the figures are worse than with the new line: 44 choices equal where there were 47, the first 10 in a row (23), logits within 3.85 (3.64), llama.cpp's two best 0.21 apart where they part (0.32) and 1.11 at the widest of the four choices that differ, above the 1.0 this check allowed. Given OKLM's embeddings, llama.cpp's own language model leaves its whole-path answer at the same eleventh token: its encoder's embeddings are 0.3 to 0.7 % from OKLM's, and this prompt's answer turns on less. Against that answer OKLM gives 47 of 48, the first 12 in a row, logits within 3.78, 0.84 apart where they part. The test now holds Gemma 3 as it holds Qwen2-VL: 90 of 100 choices and ties under 1.0 on the same embeddings, 80 and 3.0 on the whole path. SmolVLM-256M and 500M 348 rows (349), SmolVLM2-2.2B 433 (434).test_content_parts(test_vision): each projector's content.test_media_describe(test_server): the chat route's rows are describe's on the tiny projector of each family (253, 71, 286, 558, 115, 67), the same greedy answer, every row but the last found again by describe's prompt.oklm describe --temp 0on the published files: SmolVLM-500M the same nine tokens as before ("The newspaper says "The New York Times"."), Gemma 3 4B "Here's the text from the New York Times front page as it appears in the image: Headline: MEN".
Left: not run on a device (the device tables of Gemma 3 and SmolVLM in multimodal.md are the earlier prompt's); the retrieval's caption figures not measured again with SmolVLM's new prompt (rag.md); texts that follow each other are joined with nothing, where llama.cpp puts a new line between them.
L9. OpenCL's remaining distance to Vulkan on the encoders
With its bidirectional attention on tiles, OpenCL still took 0.86 s
for Gemma 3's encoder where Vulkan took 0.63 s, 0.38 s against 0.24 s
for jfk.wav with Whisper large-v3-turbo and 2.06 s against 1.79 s for
Qwen2-VL 2B's prompt on a picture of 2 000 x 2 000 pixels. The kernel
times said where (OKLM_CL_PROFILE=1): attention 0.50 of the
0.90 s of Gemma 3's prompt and 0.91 of the 1.65 s of Qwen2-VL's, the
matrix products (gm_f16) 0.14 of the 0.29 s of Whisper's
window.
One buffer for a tile's keys and values (opencl.md):
attention_bi64/80/128kept a tile of keys, a tile of values, the scores and the probabilities in local memory: 30 KB a work-group for heads up to 80, and a compute unit runs as many work-groups as its local memory holds. The keys now leave the buffer to the values once every score is taken, and the scores stay in registers (a row's four items share their maxima alone): 16 KB (14 KB for heads up to 64, 23 KB up to 128). Each item reads its share of the values while the scores are computed and of the next tile's keys while the values are added. The arithmetic and its order are what they were.Measured (RX 7900 XT, 2026-11-14, one session, the two builds run in turn, before -> after, Vulkan beside; a second job used the CPU at times):
OpenCL Vulkan Gemma 3 4B, the prompt of oklm describewith llama.cpp's test image1.05 to 1.11 s -> 0.89 to 0.90 s 0.74 s its 4 096 rows through the encoder alone ( test_mm_public)0.91 s -> 0.77 s 0.63 to 0.66 s its kernel time: all, the encoder's attention, gm_f160.90 s, 0.50 s, 0.22 s -> 0.82 s, 0.40 s, 0.23 s Qwen2-VL 2B, a picture of 2 000 x 2 000 pixels (1 225 embeddings) 2.07 to 2.08 s -> 1.89 to 1.91 s 1.78 to 1.96 s its kernel time: all, attention, gm_f161.65 s, 0.91 s, 0.40 s -> 1.48 s, 0.72 s, 0.42 s Whisper large-v3-turbo, jfk.wav ( oklm transcribe)0.40 to 0.41 s -> 0.39 to 0.40 s 0.30 to 0.32 s its kernel time: all, attention, gm_f160.29 s, 0.095 s, 0.14 s -> 0.28 s, 0.085 s, 0.14 s Ultravox, the prompt of oklm describeon jfk.wav0.40 to 0.41 s -> 0.40 to 0.42 s 0.27 to 0.28 s The encoders alone on the test image (
test_mm_public): Qwen2-VL 2B 0.26 s (0.18 s on Vulkan), Qwen2.5-VL 7B 0.23 s (0.15 s), LLaVA 1.5 0.13 s (0.10 s), Ultravox on jfk.wav 0.36 s (0.23 s). Gemma 3's attention on the variant of 128 (23 KB a work-group): 0.46 s.What is left is the products' arithmetic. OpenCL's tiled products are F32 (14 to 15 Tflop/s on these encoders); Vulkan's are on the matrix units, with the activations rounded to F16, which is why its embeddings are 2.6e-3 to 1.5e-2 from the CPU's and OpenCL's 1.4e-5 to 5.3e-5.
OKLM_CL_WMMA=1gives OpenCL the same units and rounding:gm_f160.14 -> 0.035 s (Whisper), 0.22 -> 0.050 s (Gemma 3), 0.40 -> 0.11 s (Qwen2-VL), and the three runs above in 0.35 to 0.36 s, 0.72 to 0.73 s and 1.55 s. It stays opt-in. The launches themselves are not the distance: 0.005 to 0.008 s of a run is spent inclEnqueueNDRangeKernel.Ultravox's 3.0e-4 (multimodal.md): it is the tiled products' sum, each output one running sum over the 1 280 or 5 120 terms of a product, through the 32 blocks of Whisper's encoder, whose terms cancel more than a vision encoder's. With a partial sum every 32 terms the embeddings are 4.5e-5 from the CPU's, every 128 terms 1.1e-5 (3.1e-5 on the row kernels). Rounding, not a defect, and the partial sums cost 7 % of every float prompt product (
gm_f16228 -> 245 ms on Gemma 3's encoder, 140 -> 149 ms on Whisper's, 414 -> 444 to 455 ms on Qwen2-VL's large picture; added into the output buffer instead of registers, 313 ms on Gemma 3's): not taken,AUDIO_CL_TOLstays at 1e-3.Checked:
test_openclon the RX 7900 XT and on the Intel UHD 750 (89 467 checks: bidirectional attention and its spans on the tiles as before, within 2e-6 + 2e-4 relative of the CPU's),test_opencl_wmma,test_vision,test_audio,test_device_encoders(test_server) on the RX 7900 XT;test_mm_public: the embeddings where they were (Gemma 3 1.45e-5 from the CPU's, LLaVA 1.5 1.41e-5, Qwen2-VL 2B 5.0e-5 to 5.3e-5, Qwen2.5-VL 7B 1.7e-5, Ultravox 3.05e-4), 47 of 48 greedy choices the CPU's embeddings give for both Qwen files with the encoder on OpenCL (the two tokens 0.07 and 0.02 apart). One check of it fails, with and without this change and not on OpenCL: Qwen2.5-VL 7B with its encoder on Vulkan parts from the CPU's answer at a token whose two candidates are 0.59 apart, over the test's 0.5 (47 of 48).
Left: in the roadmap under L9, entry 5.
L9. What OpenCL's first attention kernel still takes
After the prompts and the encoders' rows had their tiles, OpenCL's
attention kernel (one work-group a head and token) still
took 35 ms of a Gemma 3 prompt with an image, and a list of operands
nobody had counted: rows that see each other, bidirectional rows with
BF16 or block-quant keys, heads over 128 or not a multiple of 4, fewer
than 16 rows or 128 keys.
What reaches it (RX 7900 XT,
OKLM_CL_PROFILE=1, every launch waited for and its operands printed;oklm describewith llama.cpp's test image or jfk.wav and 8 tokens,oklm transcribeon jfk.wav,oklm embedandoklm rerankon the lines of a page of these docs):Model Launches on attentionKernel time Gemma 3 4B, an image (8 heads of 256 on 4, F16 cache) the image's 256 rows over 261 keys, 34 17.5 to 18.3 ms of 0.95 s 8 decode steps and two single rows, 340 15.3 ms a 10-row and a 4-row piece of text, 68 2.0 ms LLaVA 1.5 7B, SmolVLM-256M, SmolVLM2-2.2B, Qwen2-VL 2B, Qwen2.5-VL 7B, Ultravox decode steps, pieces under 16 rows or 128 keys: 144 to 600 3.1 to 17.4 ms of 0.22 to 0.56 s Whisper large-v3-turbo (decoder of 4 blocks) cross-attention, one row over 1 500 F32 keys: 116 4.4 ms of 0.30 s Whisper large-v3 (32 blocks) the same: 1 024 39.1 ms of 0.62 s bge-small-en-v1.5, nomic-embed-text-v1.5 (12 heads of 32, of 64) 16 rows or more over fewer than 128 keys: 5 316 46 to 57 ms of 0.20 to 0.28 s fewer than 16 rows: 840 5.6 to 6.4 ms No launch had bidirectional BF16 or block-quant keys: the callers of bidirectional attention (the vision and audio encoders, Whisper's cross-attention, the text encoders) give their activations, F32. No encoder on disk has heads over 128 or not a multiple of 4 (32, 64, 72, 80).
Rows that see each other on the prompt tiles (opencl.md, multimodal.md):
attention_tile64/128/256,attention_tile64h/128hand the matrix units'attention_wm64h/128htake a last key (-1: causal). With one, every block of queries walks the keys up to it and masks none for coming after a query; the window, the block table of a paged cache, the cache types and the split of the keys are the causal rows'. The thresholds too: heads up to 128 by default, so Gemma 3 27B's image rows (heads of 128) are on the tiles.Left on the first kernel, by measurement: Gemma 3 4B's and 12B's heads of 256. The image's rows and the 10 rows after them, 34 layers: 20.0 ms on
attention, 25.6 ms onattention_tile256with the keys split (OKLM_CL_ATT_BLOCKS=168), 29.7 ms (64), 35.5 ms without a split (1). A text encoder's rows over fewer than 128 keys: 8.7 to 10.6 us a launch there, 10.9 to 20.1 us onattention_bi64(OKLM_CL_ATT_BLOCKS=1). Single rows (decode, Whisper's cross-attention at 38 us a launch) are what that kernel is for.Checked:
test_attention_mutual(test_opencl, RX 7900 XT and Intel UHD 750, within 2e-6 + 2e-4 relative of the CPU's): rows that see each other on the float tiles of 64, 128 and 256, the halves' (8 heads of 128 on 4, 256 rows over 261 keys), BF16, Q8_0 and Q5_1 caches, a window shorter than the rows, a soft-cap, partial blocks and tiles, few blocks whose keys are split and merged;test_attentionoftest_opencl_wmmawith three such cases on the matrix units (within 1e-3). A text model's rates are where they were (Llama 3.2 1B Q4_K_M,bench_llama, another job on the CPU: pp512 12 973 to 14 889 t/s before, 13 821 to 15 356 after; tg128 376 to 381 and 378 to 396).
Left: Gemma 3 27B is not on disk, its image rows on the tiles are checked by the op tests alone. No kernel was written for the operands nothing produces.
L9. The chat route's prompt with an image is describe's
A chat request holding an image part and a question wrote the
question right after the image's marker, where
oklm describe writes a new line between them: one or two
tokens fewer (Gemma 3 4B 273 against 274, LLaVA 1.5 7B 592 against 594)
and answers a word apart.
- One function (multimodal.md, server.md):
oklm_mm_content()writes a message's content from its parts, a marker for each image or sound, a new line after a marker that another part follows (nothing for Qwen2-VL and Qwen2.5-VL), nothing before a marker or between two texts.oklm_mm_format()(oklm describe, the retrieval's captions) calls it with the items then the text, which gives the prompts it gave; the server's chat routes (/v1/chat/completions,/v1/messages) call it with a message's parts. A message without media is its texts as before. What changes in the server: the new line, for every projector but Qwen's (SmolVLM and Ultravox included, whose answers happened to agree or were not compared). - Which form the references write. LLaVA's own
conversations write
<image>\nbefore the question:oklm describe's form. Gemma 3's and SmolVLM's templates, given content in parts, write the text right after the image, and so does llama.cpp at the pinned revision (mtmd-cliand its server join a marker and a text with nothing); Gemma's processor puts an empty line there. For these two the new line is OKLM's own;oklm describekeeps it, the answers recorded from llama.cpp having been made with the prompts OKLM writes. - Checked (CPU):
test_media_describe(test_server), on the tiny projector of each family (idefics3, gemma3, LLaVA 1.5 and 1.6, Qwen2-VL, Qwen2.5-VL): the chat route's prompt has describe's count of rows (254, 72, 286, 558, 115, 67), gives the same greedy answer (16 tokens at most), and describe's prompt sent after it as a raw completion finds every row but the last in the prefix cache (253, 71, 285, 557, 114, 66);test_content_parts(test_vision): the contents, andoklm_mm_format()against the conversation of that content for six projector types. Published files, llama.cpp's test image, "What does the newspaper say?", greedy,--deterministic: Gemma 3 4B 274 prompt tokens by either way and the same 24 tokens ("Here's the text from the New York Times newspaper, based on the image you sent: Headline: MEN"); LLaVA 1.5 7B 594 and the same 11 tokens to its end of sequence ("The newspaper headline reads "Men Walk on Moon."").
Left: not run on a device (the prompt's text does not depend on one); SmolVLM, the Qwen files, LLaVA 1.6 and Ultravox on their published files, not run again through the server.
L6. The retrieval's image encoder on a device
oklm rag index --caption-model captions the figures of
documents with a vision model. --gen-backend opencl|vulkan
put its language model on the device and left its image encoder on the
CPU: about 10 of the 11 s of a caption with SmolVLM2-2.2B.
- The same rule as
oklm describe(rag.md):oklm_rag_vlm_open()gives the encoder the device the model's layers are on (oklm_vision_use_backend(), the path of the vision encoders' blocks: no op added).oklm_rag_vlm_encoder_device()puts it there or back on the CPU andoklm_rag_vlm_encoder_status()says where it is;oklm rag indexprintsthe vision encoder runs on the CPUwhen the device had no room. No option was added:--gen-backendis the one. - Measured (RX 7900 XT, 2026-11-14, the figure set of
tests/rag/figures_set, 9 figures, SmolVLM2-2.2B-Instruct Q8_0 with its F16 projector, another job on the CPU): the captioning 98.3 s -> 10.1 s on OpenCL and 95.4 s -> 7.6 s on Vulkan; 176.2 s with--gen-backend cpu. - Checked: on each backend the index is the one built
with the encoder on the CPU: the top five chunks of the set's 20
questions the same in the same order, the captions those hits show (6 of
the 9) the same text.
test_vlm_devices(tests/rag/test_rag.c, RX 7900 XT and Intel UHD 750, both backends): a tiny model with a tiny projector found beside it; the caption with the encoder on the device is the one the same context gives with it on the CPU.test_graph(test_captions) andtest_cli_ragpass.
Left: in the roadmap under L6, Vision.
L6. The Ultravox encoder on a device
The audio encoder of an --mmproj file (Ultravox:
Whisper's encoder and a projector) computed on the CPU whatever the
model's backend: 5 of the 5.3 to 5.9 s of a prompt with 11 seconds of
speech, the language model on the RX 7900 XT.
- The recognizer's path (multimodal.md):
oklm_audio_use_backend()copies the blocks' weights and the final norm's to a device, andoklm_audio_encode()then runs each 30-second chunk's blocks there as one pass, through the code that runs a recognizer's (oklm_whisper_use_backend(); the copy of the weights is one function for both). No op, kernel or shader was added or changed. The spectrogram, the two convolutions and the projector stay on the host.oklm_audio_device_bytes()is what it takes (the blocks, about the file's 1.4 GB, and 0.1 GB of activations). - Where (server.md):
oklm describeandoklm serve --mmprojgive the encoder the device the model's layers are on, as they do an image encoder; the server's cache leaves it its share and/api/statssaysaudio_device. A device without room leaves it on the CPU with a notice (the audio encoder runs on the CPU), quantized blocks without one. - Measured (RX 7900 XT, 2026-11-14, Ultravox v0.5
with Llama-3.2-1B Q8_0, jfk.wav, 188 embeddings; another job on the CPU
meanwhile): the prompt of
oklm describe5.77 s -> 0.41 s on OpenCL and 5.32 s -> 0.26 s on Vulkan (the CPU backend: 5.58 s); a chat request of the server with a new sound and 48 tokens 5.80 to 5.88 s -> 0.57 to 0.60 s on OpenCL and 5.65 to 5.79 s -> 0.42 to 0.43 s on Vulkan; the encoder alone 0.37 to 0.38 s and 0.23 s against 4.8 to 5.6 s on the CPU. The 48 tokens ofoklm describeand of three requests (the recording, its first 8 and 5 seconds) are the same text before and after on each backend. - Checked: the embeddings against the CPU's. The tiny
file (
test_audio_enc_devicesintest_audio, a random encoder and projector written bytests/mm/tiny_audio_mmproj.h, two chunks): 1.9e-6 on OpenCL and 1.7e-6 on Vulkan (RX 7900 XT), 2.0e-6 and 1.8e-6 (Intel UHD 750); the same device given twice, the CPU's bits again once taken back, a device without room and a backend without buffers refused. The published file (test_mm_public): 3.1e-4 relative on OpenCL (the largest difference 3.1e-3) and 9.6e-3 on Vulkan (8.1e-2).test_device_encoders(test_server, both backends, both GPUs): a sound's answer is the one computed with the encoder on the CPU, four new sounds and four completions at once answered as alone. Whisper's own device tests (test_whisper_devices, the four published checkpoints on jfk.wav) pass through the shared copy.
Left: in the roadmap under L6, Speech and Serving.
L6. Qwen2.5-VL's windowed blocks in one launch a block
A block of Qwen2.5-VL's encoder that attends within windows (28 of 32) ran the attention op once a window on a device, on views of the window's rows: 24 launches a block for a picture of 273 embeddings, 30 for the test image's 391, 81 for one of 1 225, no kernel taking a list of windows.
- A span of keys a query (core.md, multimodal.md):
oklm_attention_spans(), bidirectional attention with a table of two values a query, its first key and one past its last. The encoder uploads the table once a picture (a row's window, the rows being ordered window by window) and a windowed block is one op. The CPU kernel attends each run of queries with one span on views, the bits ofoklm_attention()over them; the encoder's CPU pass is the host loop it was. - On both backends' bidirectional tiles (opencl.md, vulkan.md): OpenCL's
attention_bi64/80/128and Vulkan'sattn_tileread the table. A block of 32 rows walks the tiles of keys from the first key any of its rows sees to the last, each row keeping the scores of its own span, so the windows need not be of one size nor fall on the blocks' or the tiles' boundaries, and spans that are not windows are taken too. Without a table the kernels compute what they did. What the tiles do not take is refused withOKLM_ERR_UNSUPPORTEDand the encoder attends window by window as before: on OpenCL BF16 or block-quant keys, heads over 128 or not a multiple of 4, fewer than 16 rows,OKLM_CL_ATT_BLOCKS=0; on Vulkan fewer rows than half a block, or a block whose shared memory the device lacks (heads of 256 on the UHD 750). - Measured (RX 7900 XT, 2026-11-14, one build, a
launch a window -> a launch a block): the launches were a small part
of the time. On OpenCL the prompt of
oklm describewith Qwen2.5-VL 7B on llama.cpp's test image (391 embeddings) 0.61 to 0.63 s -> 0.58 to 0.59 s, its 3 661 launches 2 849, the windowed blocks' attention 0.029 s of kernel time in 840 launches -> 0.006 s in 28 (OKLM_CL_PROFILE=1); the encoder alone on 273 embeddings 0.25 -> 0.24 s; a picture of 2 000 x 2 000 pixels (1 225 embeddings) 1.80 to 1.83 s -> 1.72 to 1.76 s. On Vulkan, whose dispatches of a pass were already recorded together, nothing moved: 0.48 to 0.49 s -> 0.48 s, 0.14 to 0.15 s either way, 1.40 to 1.41 s -> 1.40 to 1.42 s. Qwen2-VL 2B (no windows) is where it was: 0.48 to 0.49 s (OpenCL) and 0.37 to 0.38 s (Vulkan) for the prompt, 0.27 s and 0.18 s for the encoder alone. - Checked: the op against the CPU's within 2e-6 +
2e-4 relative,
test_attention_spans(test_opencl, RX 7900 XT and Intel UHD 750) andtest_spans(test_vulkan_attention, both GPUs): windows of 1 to 100 rows, smaller than a tile and over several, blocks of rows that straddle windows, one window over everything, 2 to 16 heads, two a KV head, a soft-cap, spans that are not windows, the refusals;test_vision(the tiny files 5.2e-7 and 2.3e-4 from the CPU's);test_mm_publicfor the Qwen files: Qwen2.5-VL 7B's embeddings 1.7e-5 from the CPU's on OpenCL (1.3e-5 to 1.7e-5 before) and 4.8e-3 to 6.3e-3 on Vulkan (4.6e-3 to 5.0e-3 before; 1.3e-5 to 1.7e-5 withOKLM_VK_COOPMAT=0: the F16 rounding of its matrix units moves with the order of the sums), 47 (OpenCL, the two tokens 0.19 apart) and 48 (Vulkan) of the 48 greedy choices the CPU's embeddings give, where a launch a window gave 48 and 47;test_device_encoders(test_server) on both backends.
Left: in the roadmap under L6, Vision.
L6. Tiles for OpenCL's bidirectional attention
On OpenCL the attention of rows that see every key (an encoder's: the
vision encoders' blocks, Whisper's encoder) ran on the kernel of one
work-group per head and token, where the prompts had their tiles and
Vulkan its own for these rows. That kernel was 2.64 of the 3.08 s of
kernel time of a Gemma 3 prompt with an image
(OKLM_CL_PROFILE=1: 27 launches of 98 ms, no launch a row
and no readback), and 69 % of a Whisper window's.
- The kernels (opencl.md):
attention_bi64,attention_bi80andattention_bi128, the work-groups of the prompt tiles (128 items, 32 rows by tiles of 32 keys, the keys and values of a tile loaded once for every row, the heads of one KV head sharing a block, the keys split into ranges merged byattention_mergewhen the blocks are few) without position, window, mask or block table. Each item holds its row's query in registers and computes one exponential a score, where the prompt tiles compute a row's four times. F32 or F16 keys and values, heads a multiple of 4 up to 128: a head that is not a multiple of 16 (SigLIP's 72) is laid out as the next one. The variant of 80 is SigLIP's and Qwen2-VL's: Gemma 3's attention took 0.72 s on the 128 one, 0.53 s on it (less local memory a work-group), and 0.49 s with the queries in registers. From 16 rows andOKLM_CL_ATT_KEYSkeys;OKLM_CL_ATT_BLOCKS=0keeps the old kernel. The causal kernels are not touched. - Measured (RX 7900 XT, 2026-11-14, before ->
after, Vulkan beside): Gemma 3 4B's prompt with llama.cpp's test image
3.15 -> 0.97 s (0.74 s) and its 4 096 rows alone 3.03 -> 0.86 s
(0.63 s); Qwen2-VL 2B's prompt 0.88 -> 0.48 s (0.37 s), its encoder
on 273 embeddings 0.45 -> 0.27 s (0.18 s), a picture of 2 000 x 2 000
pixels (1 225 embeddings) 6.66 -> 2.06 s (1.79 s); SmolVLM2-2.2B's
prompt 0.98 -> 0.72 s, SmolVLM-256M's 0.51 -> 0.33 s, LLaVA 1.5's
0.42 -> 0.40 s; Whisper large-v3-turbo's encoder 0.69 -> 0.34 s
(each kernel waited for), a 30-second window 0.83 -> 0.47 to 0.49 s
(0.32 s), large-v3's 1.32 -> 0.97 s (0.65 s); through
oklm servethe largest gap of a stream while an image or a sound arrives 3.0 -> 0.83 s (Gemma 3; 0.60 s), 0.71 -> 0.30 s (Qwen2-VL 2B; 0.18 s), 0.65 -> 0.28 s (Whisper large-v3-turbo; 0.14 s). A decode row is where it was (Llama 3.2 1B Q4_K_M,bench_llama: pp512 14 802 to 14 989 t/s before, 14 814 to 14 847 after; tg128 398 to 403 and 406 to 409). - Checked:
test_openclon the RX 7900 XT and on the Intel UHD 750 (bidirectional attention on the tiles with rows and keys that are not multiples of 32, heads of 36, 64, 72, 80, 100 and 128, one head, several, two a KV head, a soft-cap, keys in 2 and 8 ranges; BF16 keys and a head of 66 on the old kernel; within 2e-6 + 2e-4 relative of the CPU's);test_vision,test_whisper_devices(test_audio) andtest_device_encoders(test_server) on the RX 7900 XT; the published files' embeddings 1.4e-5 to 4.3e-5 from the CPU's (LLaVA 1.5, Gemma 3), 5.0e-5 to 5.3e-5 (Qwen2-VL 2B) and 1.3e-5 to 1.7e-5 (Qwen2.5-VL 7B), 48 of 48 greedy choices the CPU's embeddings give for both Qwen files (test_mm_public).
Left: in the roadmap under L6, Vision, Speech and Serving.
L6. The encoder of Qwen2-VL and Qwen2.5-VL on a device
The encoder that reads an image at its own size stayed on the CPU when the other encoders' blocks went to a device: its rows were turned by their patch's place and its windows cut in host loops. It was 8.4 of the 8.5 s of a describe's prompt with the 2B on a device.
- Positions through the rotary op (multimodal.md):
oklm_rope_parts_factors(), the rotation with three positions a row given a factor a pair, which the CPU kernel, OpenCL'sropeand Vulkan'snorm_ropealready took together. A base of 1 and the factors 10000^(i / (d/4)) give each band the model's frequencies from its own start; the positions are a patch's row, its column and 0. No kernel or shader changed. - Windows as views: a block with windowed attention
runs the attention op once a window, on views of that window's rows (the
rows are ordered window by window). The blocks, the gated MLP and the
RMS norms are those of the other encoders' device pass
(
vit_blocks()), which now takes every encoder; the CPU's pass of these two is the host loop it was, the same bits. - Measured (RX 7900 XT, 2026-11-14, llama.cpp's test
image, 391 embeddings): the prompt of
oklm describewith Qwen2-VL 2B 8.57 -> 0.43 s on Vulkan and 8.32 -> 0.88 s on OpenCL; with Qwen2.5-VL 7B 8.71 -> 0.59 s and 7.20 -> 0.65 s. The encoder alone on 273 embeddings: 4.8 to 5.7 s on the CPU, 0.45 s on OpenCL, 0.18 s on Vulkan (2B); 4.2 to 4.4 s, 0.27 s, 0.15 s (7B). - Checked: the op with factors and three positions
against the angles computed in float64 (
test_ops) and on each backend against the CPU's (test_opencl,test_vulkan_norm); the tiny files' embeddings on each device within 5.4e-7 (OpenCL) and 4.1e-4 (Vulkan) of the CPU's (test_vision); the published files' within 6.3e-5 and 1.5e-2 (2B), 1.7e-5 and 5.0e-3 (7B), and 47 or 48 of the 48 greedy choices the CPU's embeddings give, parting at near ties (test_mm_public). Throughoklm serveon both backends, five pictures at once beside transcriptions, streams and speech: each answer a reading of its picture, one (Vulkan) and three (OpenCL) of the fiveoklm describe's to the token (the 2B's wording moves at near ties with what shares its batches).
Left: in the roadmap under L6, Vision.
L6.
oklm serve with its encoders on a device
A server's image encoder, its recognizer and its synthesizer computed
on the CPU whatever the model's backend:
--transcribe-backend and --speak-backend took
cpu only, with the reason that one thread at a time uses a
device, the scheduler's.
- What made a device one thread's. Each call of a
backend already took its instance's lock, and Vulkan holds it from
step_begintostep_end; what was not kept together is a computation of several calls (a prompt's forward pass, an encoder's blocks: ops held back to fuse, a read-back between them).oklm_backend_pass_begin()andoklm_backend_pass_end()(a recursive lock per device, none on the CPU) now bracket a context's forward pass, the image encoder's blocks over a tile, a window of Whisper's encoder, its keys and values and each token of its decoder: threads share a device one pass at a time. The scheduler has no place for foreign jobs between its steps, and the synthesizer's loop is the library's: a lock the passes take covers both without either knowing the other. - The server (server.md): the image encoder
goes where the model's layers are, as
oklm describedoes;--transcribe-backendand--speak-backendtakeopencl,vulkanorauto(a backend the machine lacks stops the server at start);/api/statssays where each computes. The model's cache, which takes what the device has free, leaves the encoders their share: without it a transcription took 42 to 97 s on OpenCL (the device paging) instead of 0.76 s. - Measured (RX 7900 XT, 2026-11-14): an image through the server 22.5 s -> 1.26 s for Gemma 3 4B on Vulkan (22.7 -> 3.68 s on OpenCL), 1.97 -> 0.91 s for LLaVA 1.5 7B (2.07 -> 1.05 s), 3.48 -> 0.34 s for SmolVLM-500M (4.14 -> 0.59 s); jfk.wav by large-v3-turbo in 0.26 s (Vulkan) and 0.76 s (OpenCL); a sentence of 2.9 s spoken in 0.76 and 0.90 s by OuteTTS 0.2. A request streaming a completion meanwhile waits for an encoder's pass: 0.19 s (LLaVA's image, both backends), 0.59 s and 3.0 s (Gemma 3's, Vulkan and OpenCL), 0.14 s and 0.65 s (Whisper's window); not for the synthesizer, whose passes are single tokens.
- Checked:
test_device_encoders(tests/server/test_server.c) on both backends, the fixtures' answers those of the CPU's encoders, sixteen requests of four kinds at once answered as alone; by hand, twenty at once on Gemma 3 4B and LLaVA 1.5 7B with large-v3-turbo and a speech model on the same device: transcriptions, sounds (byte for byte) and images as alone, but LLaVA's image answer on OpenCL without--deterministic, a word apart as its batches fall (the embeddings are kept by content: the same ones). The image answer is the same with the encoder on the CPU or on the device (Vulkan: three models; OpenCL: all but Gemma 3 4B, one word at the 41st token), andoklm describe's for SmolVLM-500M. - Unchanged speeds (the pass lock is taken once a
forward pass; no kernel or shader changed): Llama-3.2-1B Q4_K_M,
bench_llama, before and after in the same session, best of five, twice: Vulkan prompts 16 460 and 16 560 or 16 530 tokens a second, decoding 509 or 508 and 516 or 508; OpenCL 15 080 or 15 060 and 15 060 or 15 000, 409 or 397 and 411 or 404.
Left: in the roadmap under L6, Serving.
L6. Vulkan's mixture-of-experts prompts on the prompt tiles
Vulkan's prompt rate on the routed models of 2026-11-13 was a half to a third of OpenCL's (Tini-Cybersec-8B-A1B 2523 against 7265 tokens a second, gpt-oss-20b 844 against 1685, Gemma 4 26B-A4B 1302 against 2220). The profile of a 512-token prompt gave one cause for the three, and for the older routed models, whose prompts were not level with OpenCL's either (OLMoE-1B-7B 3035 against 8790, Granite-3.1-1B-A400M 8365 against 17880).
- The cause. No op was handed back to the CPU (the
token lookup alone) and nothing was read back. The indexed products of a
prompt, grouped by expert on the device, ran on
mvid_<type>, the decode shader with up to 16 pairs a work-group: 2.7 to 2.9 ms each on Tini (2048 pairs over 32 experts), 180 ms of a prompt's 220, 5.6 TFLOPS where the dense products of the same prompt ran at 35 on cooperative matrices. The grouped form of the prompt products (oklm_vk_gemm_tiles()with a tile table and column id tables), written and tested for this, was called by nothing. - The fix (
src/core/vulkan/vk_ops_moe.c,moe_group.comp): from 16 pairs an expert on average,moe_groupwrites the tile table (expert, first, end; a tile's columns a chunk:oklm_vk_gemm_group_cols(), 128 on cooperative matrices, 64 on the F32 tiles) and, when the ranks share their token's row, each placed pair's token; the product is then one call ofoklm_vk_gemm_tiles(). Fewer pairs stay onmvid_<type>(the two meet at 8 an expert; at 4 the tiles are slower).OKLM_VK_MOE_TILES=0or1forces either (vulkan.md, build.md). - Figures (RX 7900 XT,
bench_llama, 512 and 128 tokens, the best of two or three runs), Vulkan's prompt before and after, then OpenCL's in the same session, then decode on Vulkan after (before): Tini 2522 to 7720, 6895 to 6958, 341.1 (340.2); OLMoE 3035 to 9060 to 9190, 8786, 477 to 485 (487.7: runs of 128 tokens in 0.264 to 0.269 s after and 0.262 to 0.268 s before, the decode dispatches being the same); Granite 8365 to 20500, 17880, 634 to 638 (636.6); gpt-oss-20b 844 to 2149, 1676, 96.3 (96.3); Gemma 4 26B-A4B 1304 to 2716, 2224, 106.4 (105.9). An indexed product of Tini's takes 0.65 to 0.73 ms on the tiles (21 to 23 TFLOPS). - Checks.
tests/core/test_vulkan_moe.c(RX 7900 XT and UHD 750): six weight types on the tiles against a double reference, on the F32 tiles at the float products' tolerance and on cooperative matrices at the prompt products', with the row shared or one per rank, an expert over several tiles, more rows than a tile, and the default policy. The published files on Vulkan against llama.cpp and the CPU kept their figures (their steps are decode steps: Tini 155 / 160 and 159 / 160, gpt-oss-20b 157 and 156, the 26B-A4B 152 and 154, the other layouts 14 to 16 of 16). Perplexity over 1530 tokens in 512-token windows, the CPU, Vulkan on the tiles, Vulkan onmvid_<type>: OLMoE 29.171, 29.151, 29.160; Tini 28.308, 28.336, 28.355. - gpt-oss-20b (11.6 GB, SHA-256
c2753664...568aeb4f) and the 26B-A4B (16.9 GB,f2c28b3d...f63f293f) were downloaded withoklm models pull, run and removed, one at a time.
L6. The steps where a device parts from the CPU
Three results of the published files on a device were left as "not traced block by block". They were traced with the records of observed runs (observe.md): a CPU context and a device one on llama.cpp's tokens, each block's writes and experts compared step by step. None is a defect of a device kernel or of the CPU's; no code changed.
- Gemma 4 26B-A4B, step 28 of the fourth prompt (the
whole vector of logits 9.5 from the CPU's on OpenCL and 7.1 on Vulkan,
the best token the same): a near tie of block 4's router (inference.md). The CPU gives its
eighth and ninth experts (95 and 107 of 128) logits of 3.370 and 3.353
and takes 95, as llama.cpp does; both devices take 107, the seven others
and every expert of blocks 0 to 3 being the same; after it 26 of the 30
blocks choose another set (2 to 16 on the prompt's other steps). A CPU
context made to exchange the two experts at that one place is 9.6 on
average from the plain one and 0.66 from OpenCL, 2.6 from Vulkan. One
rounding switch of the CPU moves expert 107's logit there by up to 0.06,
more than the 0.017 between the two. The move is an offset common to the
logits (9.5, 2.2 left without it).
test_gemma4_public's bounds stay (30 for the largest difference, 1.0 for the mean), with the cause in their comment; the test was run again on the file on both backends (OpenCL 155 of 160 choices equal to llama.cpp's, 22.8 at most, 0.680 on average; Vulkan 152, 20.7, 0.631). The file (16.9 GB, SHA-256f2c28b3d...f63f293f) was downloaded, run and removed. - Cyber-Prime 1.1 2.6B (149 to 152 of 160 choices on a device against the CPU's 154): ties, no block parts (inference.md). At each of the 12 (Vulkan) and 11 (OpenCL) steps whose choice differs the best logit is 0.001 to 0.78 above the second on every side; the stream's norms differ by 1.5 % at the median, growing along the blocks. Vulkan's 7.5 is one step after a token repeated seven times, where the CPU, llama.cpp and the device give three tokens.
- Spark-X2.6-4B, the step where the CPU is 16.2 from llama.cpp
and the devices 3.2 (inference.md): the
CPU's run, on a token the continuation repeats, has the feed-forward
parts of blocks 20 to 24 write 1.9 to 12 times what they write on a
device, and block 34 take it back. A CPU context with an F32 cache does
not, and the CPU with
OKLM_ATTN_F16=1orOKLM_Q8_ROUND_EVEN=1is as far from llama.cpp one step earlier (5.9 and 5.6 on average at step 14, where the plain CPU is 1.4; 0.45 and 4.9 at step 15, where it is 7.1). The CPU kernels are not off; its output was not changed. - Left: in the roadmap (L6): the Spark 1.7B's largest single logits.
L6. Speech on a device
Whisper and the speech models computed on the CPU only;
oklm speak refused another backend.
- Where the time went (CPU, one 30-second window of 69 tokens): Whisper large-v3-turbo 4.8 to 5.5 s in the encoder and 1.1 s in the decoder; large-v3 5.3 to 5.5 s and 6.2 s. A sentence of 7 seconds with OuteTTS 0.3: the language model 13.3 of 13.6 s, the vocoder 0.33 s; Soprano 0.34 and 0.18 s.
- Whisper (multimodal.md):
oklm_whisper_use_backend()copies the weights of the encoder's blocks, of the decoder and the output matrix to a device; the blocks of a window, the keys and values of both attentions of the decoder (its cache in the device's memory) and each token's logits are computed there through the ops the CPU path calls, the spectrogram, the two convolutions and the choice of each token staying on the host.oklm transcribe --backend cpu|opencl|vulkan|auto. A window of large-v3-turbo takes 5.92 to 6.58 s on the CPU, 0.83 s on OpenCL and 0.33 s on Vulkan (RX 7900 XT); of large-v3 11.6, 1.32 and 0.66 s. No kernel was added: bidirectional attention, the causal attention over a cache and the exact GELU were on both backends. - The text is the CPU's on the tests' samples: the
tiny checkpoints' texts, segments, times and languages on OpenCL and
Vulkan (
test_whisper_devices, RX 7900 XT and Intel UHD 750), and jfk.wav with base, small, large-v3-turbo and large-v3 (test_mm_public). No token differed; one could where two candidates are all but tied. oklm speak(speech.md):oklm_speech_params::backendis the language model's (--backend opencl|vulkan|auto), the vocoder computes on the CPU; a backend the machine does not have is refused. The sentence takes 13.62 s on the CPU, 2.25 s on OpenCL and 1.83 s on Vulkan with OuteTTS 0.3 (0.52, 3.07 and 3.72 s of sound a second), 0.52, 0.35 and 0.30 s with Soprano. The sound is not the CPU's sample for sample: on 96 greedy steps the device's token is the CPU's in 92 to 95, the CPU's margin where they differ 0.063 at most (test_published_devices), and a code that differs changes those after it.- Left on the CPU, with what it costs: the vocoder (0.31 to 0.33 s of a sentence) and Soprano's decoder (0.17 s): not of the size of the language model's work, and convolutions, group norms and an inverse transform that are not device ops; a server's recognizer and synthesizer (one thread at a time uses a device); the Ultravox encoder.
- Left: in the roadmap (L6).
L6. Observation of the later stacks on a device
The roadmap still said that the blocks a device runs were marked as not observed. They have been recorded since 2026-11-13 (what a pass leaves is copied aside on the device and read back once when it ends, observe.md); what was missing was a check on the stacks whose device paths came with L6.
- The tests
(
tests/model/test_observe.c):test_devicesandtest_device_batchedrun twenty-two tiny models where they ran nine: Qwen3.5 (linear blocks, experts, folded weights), Gemma 4 (two widths, shared caches, experts), gpt-oss, LFM2 and its experts, Spark, Muse Glimmer, StarCoder2 and Mistral 3 beside the first nine. Every block is recorded on OpenCL and on Vulkan, a linear or a convolution block included; nothing had to change insrc/model/observe.c. - Against the CPU's records (RX 7900 XT): with an F32 cache norms within 9.4e-6 (relative), attention weights 3.3e-5, expert weights 1.9e-5, the routings id for id, the logits with parts taken out 2.0e-5, over 792 records and 8640 attention rows a backend. With an F16 cache 3.9e-4, 5.8e-4, 8.7e-4 and 9.9e-4: Gemma 4 with experts, Spark, gpt-oss and Muse Glimmer amplify a key rounded to another half, so the F16 contexts are bounded apart (2e-3 where the F32 ones keep 2e-4).
- Once a pass: 1606 records the same bits whether the device is read once a pass or at each point of a block; 1331 reads for 1254 passes against 20866. The Qwen3.5 stacks keep ten sequences in the staging memory's room of eight.
- Not recording: the logits are the same bits with the recording on or off; a context that does not record reads nothing back (checked), replays as many Vulkan decode steps (411 and 411) and launches the same kernels (Llama 3.2 1B on OpenCL: 129 a token either way; the recording's own work is buffer copies and one read a pass).
- A published file: LFM2-350M Q4_K_M,
oklm run --observeon the CPU and on each device: the sixteen blocks recorded, 27 of 61 lines compared before the runs part, norms at a median of 4.3e-3 and attention weights within 0.065 of the CPU's (the quantized weights' arithmetic, as Llama 3.2 1B's 2.8e-3 and 0.040). - Left: in the roadmap (L6): the F16 cache's gap on four fixtures, the reads still made inside a pass.
L6.
Published files of lfm2, lfm2moe,
starcoder2, mistral3 and DeepSeek-Coder on
OpenCL and Vulkan
These architectures ran on a device on their tiny fixtures only. Their published files now run there against the CPU and llama.cpp (inference.md).
- Nothing failed: no kernel, no placement and no
notice had to change.
tests/model/test_archs_public.ctakes the seven files beside those it had (OKLM_PUB_FILES=lfm2,cyber-prime,tini,starcoder2,devstral,deepseek-coder). - Against llama.cpp and the CPU (RX 7900 XT, every block on the device, the 160 steps of the corpus on llama.cpp's tokens), choices equal to llama.cpp's on OpenCL and Vulkan (the CPU's), then the mean logit difference from llama.cpp, then the choices equal to the CPU's: LFM2-350M 158 and 158 (157), 0.137 and 0.127, 157 and 157; Cyber-Prime 1.1 2.6B 152 and 151 (154), 0.228 and 0.225, 150 and 149; Tini-Cybersec-8B-A1B 155 and 155 (156), 0.315 and 0.327, 157 and 159; StarCoder2-3B 159 and 157 (159), 0.193 and 0.187, 158 and 156; Devstral Small 2 24B 159 and 158 (158), 0.095 and 0.090, 157 and 158; DeepSeek-Coder-1.3B 159 and 157 (159), 0.581 and 0.534, 158 and 156; DeepSeek-Coder-6.7B 157 and 159 (159), 0.136 and 0.087, 156 and 158. The largest differences from llama.cpp are 2.13 and 1.82, 4.44 and 7.36, 7.30 and 5.65, 1.72 and 1.60, 4.01 and 4.57, 6.13 and 6.04, 3.73 and 1.28 (single logits far below a step's best).
- Other layouts, two sequences decoded together against the plain device context's choices on 16 steps: a paged cache 15 or 16 of 16 for every file on both backends; half of the blocks on the CPU 14 of 16 for Cyber-Prime, 15 or 16 for the others; Tini's experts in host memory 15 and 16.
- Speed, prompt and decode in tokens a second on the
CPU, OpenCL and Vulkan (
bench_llama, 512 and 128 tokens): LFM2-350M 2480 / 123.0, 29134 / 631.2, 38828 / 1007.9; Cyber-Prime 310.3 / 16.9, 6626 / 222.0, 8156 / 271.3; Tini 451.2 / 25.6 (while a file was being downloaded), 7265 / 250.9, 2523 / 342.3; StarCoder2-3B 202.2 / 14.3, 4838 / 139.4, 4952 / 203.0; Devstral (whole on the 20 GB device) 29.2 / 2.08 (128 and 32 tokens, one run), 1017 / 38.8, 1026 / 44.2; DeepSeek-Coder-1.3B 538.5 / 33.2, 10845 / 341.1, 11479 / 446.0; the 6.7B 109.1 / 7.10, 3083 / 111.2, 2961 / 135.3. - Tini (5.2 GB, SHA-256
a4d427e1...36217704) and DeepSeek-Coder 6.7B (4.1 GB,92da6238...a2013276) were downloaded and kept undermodels/; Devstral (14.3 GB,d14ba9ed...80aee496) was downloaded, run and removed. The llama.cpp recordings of StarCoder2-3B, Tini and Devstral were made again withtools/gen_llama_ref.sh's arguments. - The plan of an
lfm2model counts the final norm, which the files nametoken_embd_norm, with the head the device takes (oklm_weight_with_head(), memory-plan.md); a tied output stays one tensor of the file, counted with the embeddings, its device copy incontext.device_weights. - Left: in the roadmap (L6): Vulkan's prompt rate on Tini, a published file on the Intel UHD 750 (Cyber-Prime's choices on a device were traced since: above).
L6. The vision encoders' blocks on a device
With the language model on a device, a prompt with an image was the encoder's time on the CPU: 1.1 of 1.4 s for LLaVA 1.5, 6.7 of 7.8 s for LLaVA 1.6, 10.5 of 10.6 s for SmolVLM2-2.2B, 21 of 21 s for Gemma 3. The encoders computed through the ops already, on host tensors (multimodal.md).
oklm_vision_use_backend(): the blocks' weights copied to a device, a tile's stream uploaded, the blocks run there through the same ops and the stream read back for the projector. For the encoders that take tiles of one size (CLIP, SigLIP: LLaVA 1.5 and 1.6, Gemma 3, SmolVLM);oklm describecalls it when the language model has layers on a device. The CPU's pass is the same bits as before (Gemma 3's and LLaVA 1.5's embeddings at 8.9e-6 and 5.4e-6 of the float64 pass, as recorded).- What the devices lacked was little: bidirectional attention on OpenCL (the commit before this one), and rows of a multiple of 32 values for the matrix products, which SigLIP's inside width of 4 304 is not: the encoder pads that width with zeros on a device instead of having the product handed back.
- Measured (RX 7900 XT,
oklm describe's prompt with llama.cpp's test image): LLaVA 1.5 7B 1.40 s to 0.42 s on OpenCL and 1.50 s to 0.41 s on Vulkan; LLaVA 1.6 7.83 s to 1.94 s on OpenCL, 1.82 s on Vulkan; Gemma 3 4B 21.0 s to 3.15 s and 21.1 s to 0.74 s; SmolVLM2-2.2B 10.6 s to 0.98 s on OpenCL, 0.49 s on Vulkan. One tile through the encoder alone: LLaVA 1.5 1.16 s on the CPU, 0.15 s on OpenCL, 0.10 s on Vulkan; Gemma 3 22.0 s, 3.03 s and 0.63 s. - Checked:
test_visionruns the tiny gemma3 and mlp files with their blocks on OpenCL and on Vulkan against the CPU's embeddings (5e-7 on OpenCL; 3e-4 on Vulkan, whose matrix units round the activations to F16) and back on the CPU (the same bits), and the sized encoders refuse a device;test_mm_publicdoes it for Gemma 3 4B and LLaVA 1.5 7B on two pictures each: within 1.1e-5 to 2.7e-5 (relative norm) of the CPU's embeddings on OpenCL, 2.6e-3 to 7.2e-3 on Vulkan (llama.cpp's own distance from the float64 pass; 1.1e-5 withOKLM_VK_COOPMAT=0).
Left: the encoder of Qwen2-VL and Qwen2.5-VL on the CPU (8.4 of 8.5 s with the 2B on a device, 7.3 s with the 7B: rotary positions and windows in host loops); the server's and the retrieval's encoders on the CPU; OpenCL's bidirectional attention without tiles (3.0 s for Gemma 3's encoder against Vulkan's 0.63 s); Vulkan's F16 rounding in an encoder; a device's products still hand back rows that are not a multiple of 32 values.
L6. LLaVA and SmolVLM with an image on OpenCL and Vulkan
The language models of LLaVA 1.5, LLaVA 1.6 and the SmolVLM files had not been run on a device with an image's embeddings. Nothing was missing: the rows are evaluated as tokens' rows are, and all of them ran (multimodal.md).
test_mm_public evaluates each prompt's rows and 48
tokens on the CPU, on OpenCL and on Vulkan (llama.cpp's greedy tokens
for the LLaVA files, the CPU's own for the three SmolVLM files), every
layer on the device with the contiguous cache and half of them with a
paged cache, each on a model loaded for it, and holds the device's
logits to the CPU's as test_archs_public does (90 % of the
choices, 30 at most, 1.0 on average):
- LLaVA 1.5 7B (595 rows): 47 or 48 of the CPU's 48 choices, logits within 0.44 of the CPU's (0.04 to 0.05 on average), 47 or 48 of llama.cpp's choices; the prompt at 2 157 (OpenCL) and 2 011 (Vulkan) rows a second against 105 on the CPU, decoding at 101 and 127 tokens a second against 6.4.
- LLaVA 1.6 Mistral 7B (2 357 rows): the CPU's and llama.cpp's 48 choices in each of the four layouts, logits within 0.43 of the CPU's; 2 023 and 1 802 rows a second against 89, 82.9 and 113 tokens a second against 6.1.
- SmolVLM-256M, SmolVLM-500M, SmolVLM2-2.2B (349, 349 and 434 rows): 47 or 48 of the CPU's choices, logits within 1.48 (0.05 to 0.16 on average); the 2.2B's prompt at 3 455 and 4 012 rows a second against 295, decoding at 215 and 278 against 15.0.
Left: the server with an image, a published file on the UHD 750 and streamed CPU layers with an image were not run on a device.
L6. Gemma 3's image rows on OpenCL and Vulkan
An image's rows in Gemma 3 see each other, which the device backends handed back to the CPU attention, and no device had been run with an image. The device attention now computes it (multimodal.md).
- The last key the rows all see as a parameter:
OpenCL's
attentionkernel (one work-group per head and token) takes it beside each query's own position, which still gives the window's start; Vulkan'sattn_splitshader has a flag for it, the key count of its per-step data being the last query's. The tiled prompt kernels of both backends, whose blocks end at their own last query, are left as they were and not used for these rows. Not with per-token tables or attention sinks (handed back, as before; no model asks for it). - Checked:
test_openclandtest_vulkan_attentionhold the op to the CPU's (2e-6 absolute plus 2e-4 relative) at Gemma 3 4B's heads (8 of 256 on 4) with 256 rows, after cached keys, under a window shorter than the rows and one of 1 024, F32, F16, BF16 and block-quant caches, a soft-cap, three rows and one, and on Vulkan through a paged cache's block table; no hand-back is logged. The tiny model's five cases three ways each (the contiguous cache, a batch with another sequence, the paged cache) on OpenCL and on each Vulkan device, every block there and the last one alone, within 1e-6 of the numpy pass.test_mm_publicruns Gemma 3 4B's newspaper answer on each backend, every layer on the device and 17 of 34 with a paged cache: 46 or 47 of the CPU's 48 choices, logits within 4.19 of the CPU's (0.28 to 0.38 on average), 45 to 48 of llama.cpp's choices; the 274 rows of prompt at 2 800 rows a second on either device against 214 on the CPU, decoding at 105 (OpenCL) and 146 (Vulkan) tokens a second against 10.8.oklm describewrites the same headline on the three backends. - Unchanged speeds: Llama-3.2-1B Q4_K_M,
bench_llama, before and after in the same session: OpenCL prompts 15 500 and 15 510 tokens a second, decoding 404 and 412 (best of seven); Vulkan 16 620 and 16 620, 510 and 509 (best of five).
Left: the server with an image and the 12B and 27B files were not run
on a device (the encoder, 21 of the 22 s oklm describe took
on a device, is the entry above).
L6. Qwen2-VL and Qwen2.5-VL on OpenCL and Vulkan
A batch holding an image's rows was rotated by a function of the
model code on the CPU, and refused by a context with layers on a device
(and by oklm describe and the server before that). It is
now rotated where its block runs (multimodal.md).
- Three positions a row in the rotary op:
oklm_rope_parts(),OKLM_OP_ROPEwith the pairs of the first two bands as parameters and three positions a token in its position operand. The CPU kernel, OpenCL'sropekernel and Vulkan'snorm_ropeshader read the position of a pair's band (three more parameters; a rotation with one position a token computes what it did).test_opsholds the CPU op bit for bit to the one-position rotation at each band's position (both layouts, a partial rotation, linear scaling, no third band, in place),test_openclandtest_vulkan_normthe devices to the CPU at 2e-5 absolute plus 2e-4 relative (RX 7900 XT on both, UHD 750 on Vulkan). OpenCL's fused launches (norm_rope,rope_kv) keep one position a token and leave such a rotation to the plain kernel. - The model side: the three positions of a batch's
rows follow its one-part positions in each device's position buffer
(three more integers a batch row for this architecture, counted in the
memory plan);
model_mrope_rotate()is the op on that view, on the CPU too, whose logits are the same bits (the published files' figures against llama.cpp are unchanged: 46 of 48 and 1.76, 46 of 48 and 3.16). The refusals are gone fromrun_items(),oklm describeand the server's start. - Checked: the tiny model's seven cases seven ways
each (pieces of an image, batches with another sequence, the paged cache
and its prefix cache, a cut, a copy) on OpenCL and on each Vulkan
device, every block there and the last one alone, within 6e-5 of the
numpy pass.
test_mm_publicruns the newspaper question on Qwen2-VL 2B and Qwen2.5-VL 7B on each backend, every layer on the device and 14 of 28 with a paged cache, against the CPU's logits and llama.cpp's: 46 to 48 of the CPU's 48 choices, logits within 2.47 (2B) and 1.38 (7B) of the CPU's, 0.09 to 0.18 on average; 45 to 48 of llama.cpp's choices. The 2B decodes at 202 (OpenCL) and 303 (Vulkan) tokens a second against 28 on the CPU, its 418 rows of prompt at 6 100 rows a second against 480; the 7B at 91 and 115 against 6.5, and 1 910 and 1 750 against 111.oklm describewrites the CPU's sentence on both backends with the 2B. - Unchanged speeds: Llama-3.2-1B Q4_K_M,
bench_llama, best of five, before and after in the same session: OpenCL prompts 15 470 and 15 240 tokens a second, decoding 398 and 405; Vulkan 16 630 and 16 570, 510 and 517.
Left: the encoder computes on the CPU, which is then nearly all of a
prompt's time (oklm describe with the 2B: 8.5 s on OpenCL
against 9.7 s on the CPU, the language model's part 0.07 s);
oklm serve --mmproj with these models on a device, streamed
CPU layers with an image and a published file on the UHD 750 were not
run.
L6. Ternary Bonsai 2 on OpenCL and Vulkan
A context of a model with Walsh-Hadamard folded weights asked on a device was given the CPU. It now runs there, every block or the last ones (hybrid-layers-design.md).
The rotation as one device op:
oklm_hadamard_fold(),OKLM_OP_HADAMARDwith a sign per element (before the rotation for a product's input, after it for an embedding row turned back), a row's value heads reordered first, and groups up to 4096 (256 before). The CPU kernel takes the three; OpenCL has ahadamard_foldkernel and Vulkan ahadamard_foldshader beside the plain ones, which are untouched.test_opsholds the CPU op to its parts written out,test_openclandtest_vulkan_ssmthe devices to the CPU at 1e-6 absolute plus 1e-5 relative (blocks of 64 to 4096, in place, signs before and after, reordered heads; RX 7900 XT on both, UHD 750 on Vulkan).PQ2_0andPTQ1_0on the devices: decoded 32 weights at a time in every matrix-product family of OpenCL (mm_,mv_,mvg_,gm_,gi_,wm_,w2_,wi_,dequant_rows) and of Vulkan (mv_,gm_,cm_,mvid_), against a double reference intest_opencl,test_vulkan_matmulandtest_vulkan_gemm.Q2_0had its kernels.The model side (
src/model/fold.c,model.c): the sign vectors copied to each device with the weights; a folded weight recognized from a device copy's or a streaming slot's header, the joined query, key and value (gate and up) header included; the fallback ofoklm_context_new()narrowed to blocks above 4096 and to joined weights folded unlike. The CPU path is the same code as before.The fixtures (
test_model, test_fold):tiny-qwen35-foldandtiny-qwen35-fold-tiedon OpenCL and on each Vulkan GPU through the scenes of the plain fixture (every block on the device and the first on the CPU, batches, two sequences, a copy, a file, a cut, a paged cache) and with a streamed layer; within 4e-6 of the CPU, 1.9e-4 and 2.9e-4 of llama.cpp as on the CPU.The published files (
test_bonsai_public, now on the CPU, OpenCL and Vulkan; RX 7900 XT; 160 choices on llama.cpp's tokens, the pinned build forQ2_0, PrismML's fork for the two others):File Backend Choices llama.cpp's Largest (mean) logit difference from llama.cpp Against the CPU: choices, largest (mean) Tokens a second (CPU) 2B Q2_0OpenCL 160 / 160 0.21 (0.012) 160 / 160, 0.48 (0.012) 121 (27.6) 2B Q2_0Vulkan 160 / 160 0.21 (0.012) 160 / 160, 0.48 (0.012) 140 (28.2) 2B PQ2_0OpenCL 159 / 160 0.25 (0.012) 160 / 160, 0.22 (0.012) 123 (29.7) 2B PQ2_0Vulkan 159 / 160 0.25 (0.012) 160 / 160, 0.22 (0.012) 140 (30.1) 2B PTQ1_0OpenCL 159 / 160 0.21 (0.012) 160 / 160, 0.22 (0.012) 49 (15.2) 2B PTQ1_0Vulkan 159 / 160 0.21 (0.012) 160 / 160, 0.22 (0.012) 58 (15.2) 4B PQ2_0OpenCL 160 / 160 0.65 (0.012) 159 / 160, 0.17 (0.011) 77 (16.5) 4B PQ2_0Vulkan 160 / 160 0.65 (0.012) 159 / 160, 0.17 (0.011) 82 (16.5) The 100 greedy tokens after a chat turn and after plain text: 100 and 100 llama.cpp's everywhere but the 2B
PTQ1_0chat turn (99, a tie of 0.001 for llama.cpp, as on the CPU). A paged cache and half of the blocks on the CPU, two sequences together: 16 of 16 first tokens llama.cpp's on each file and backend. A device gives the three packings of the 2B the same logits to the bit (it does not round the activations to 8 bits as the CPU and llama.cpp do). The 4B's file (1.1 GB, SHA-256d406b1e9...b7f30409) was downloaded for the run and its reference recorded with the fork; the speeds are those of the 160 decode steps of the comparison, not a bench.The catalogue: the notes of
ternary-bonsai-2-2b,-4band-9bno longer say they run on the CPU.Left (roadmap.md, L6): the 9B and the other packings of the 4B and 9B on a device, a published file on the UHD 750, the multi-token prediction block on a device,
PTQ1_0's decode speed, the indexed products of the two types under a test.
L6. gpt-oss on
OpenCL and Vulkan
A device refused gpt-oss at its first batch. A context now runs there, every block or the last ones, nothing read back by a device layer (inference.md, "On OpenCL and Vulkan").
- The sinks in the attention kernels.
oklm_attention_params::sinksis a tensor in the memory the op runs in (the device copy ofattn_sinksfor a device layer). OpenCL:attention_mergetakes the sinks and adds each head's to its sum; an op with sinks always goes through it (the per-token kernel splits the keys in two ranges at least, the tiled prompt kernels write partials for one range too). Vulkan:attn_splitandattn_tileadd the sink where a row is normalized, its address a push constant, a flag in the shaders' options. An op either backend hands to the CPU reads the sinks back with its other operands (Vulkan's heads wider than 256). - A new op,
oklm_sigglu()(OKLM_OP_SIGGLU): the experts' gated unit,g x sigmoid(1.702 g) x (clamp(u, -c, c) + 1)withg = min(gate, c), on the CPU (ops_cpu.c), in OpenCL'sewkernel and in Vulkan'sewshader. - The experts on a device
(
src/model/arch_gptoss.c): the router's bias a broadcast add, the weights a softmax of the chosen scores, each bias gathered by the chosen experts and added (oklm_get_rows(),oklm_add()), the gated unit, the MXFP4 products that existed. The CPU layers keep their loops: the CPU's results are unchanged. A context asked withn_cpu_moe, an expert cache or a device expert tier is created without them (it failed at the first batch before). - The ops against the CPU
(
test_gptoss_shapesintest_openclandtest_vulkan_attention,test_activationsintest_ops; RX 7900 XT and Intel UHD 750): attention with sinks at 64 heads of 64 on 8, decode and prompts, a window of 128, a first token, a Q8_0 cache, heads of 128 to 512, a paged cache on Vulkan; the gated unit past its clamp; the gathered biases bit for bit. - The fixture (
test_gptoss_device):tiny-gptosson OpenCL and on each Vulkan device, every block there and the last two, F32 and F16 caches, batches, several sequences, a paged cache, streamed layers on OpenCL: 36 / 36 choices, within 2.9e-5 of the float64 reference (0.039 with an F16 cache); batches against single tokens bit for bit on OpenCL, within 1.6e-5 on Vulkan. - gpt-oss-20b Q4_K_M
(
tests/model/test_archs_public.c, RX 7900 XT, every block on the device): 158 (OpenCL) and 157 (Vulkan) of 160 choices equal to llama.cpp's, where the CPU has 153; mean logit difference from llama.cpp 0.082 and 0.081 (the CPU 0.081), the largest 4.03 and 4.05 (3.96); from the CPU 155 and 156 choices, 0.066 and 0.067 on average, 2.50 and 2.69 at most. A paged cache gives the plain context's choices (16 of 16), half of the blocks on the CPU 16 and 15 of 16. No op of the file runs on the CPU but the token lookup; Vulkan replays its decode steps. - Speed, prompt and decode in tokens a second: the CPU 7.08 / 5.62, OpenCL 1685 / 126.8, Vulkan 844 / 96.4. Models without sinks kept their rates within the spread of three runs (Llama-3.2-1B: OpenCL 15 062 / 409.5 before, 14 700 to 14 910 / 399 to 409 after; Vulkan 16 247 / 511.2, then 16 210 to 16 520 / 507 to 516; Qwen3-8B on Vulkan 2670 / 111.9, then 2688 / 112.1).
- The file (11.6 GB, SHA-256
c2753664...568aeb4f) was downloaded, run and removed; its llama.cpp recording was made again withtools/gen_llama_ref.sh's arguments. The catalogue note no longer says "on the CPU only". - Left: in the roadmap (L6): a fused indexed add, Vulkan's rates against OpenCL's, a file larger than the device, the fixture's narrow down product.
L6.
spark2_5 and muse-glimmer on OpenCL and
Vulkan
A context of a Spark-X2.6 or Muse Glimmer model asked on a device was given the CPU. It now runs there, every block or the last ones (inference.md, "On OpenCL and Vulkan").
- No kernel was missing, and no placement either: the
generic placement already copied
attn_gateto the device with its layer. The gate is a product, a sigmoid and a multiplication (broadcast over a head's values for one gate per head); the count of rotated dimensions and the base are parameters of each rotation op, the window of each attention op; Muse Glimmer's norms without weight, per head or with their own epsilon and its scaled, soft-capped logits are the existing norm and element-wise ops. The notice that sent these architectures to the CPU is removed (src/model/model.c). - The operand shapes
(
test_attn_gate_shapesintest_openclandtest_vulkan_attention, RX 7900 XT and Intel UHD 750, against the CPU kernels): 16 query and 4 key heads of 256 rotated as one tensor, 64 of the 256 values with a base of 5 000 000 (the others unchanged bit for bit) and the whole head with 10 000; the gate per head (16 heads of 256) and per value (32 heads of 128); the norms of 6656 values without weight, with an epsilon of 1e-8 and per head of 128; logits times 0.196 then soft-capped at 20; attention of 16 heads on 4 and of 32 on 2 under a window of 512, decode and prompt shapes. - The fixtures (
test_attn_gate_models):tiny-sparkandtiny-muse-glimmeron OpenCL and on each Vulkan device, every block there and the last two, F32 and F16 caches, batches, several sequences, a paged cache, the CPU layers streamed on OpenCL: 36 / 36 choices, within 1.6e-5 and 7.9e-6 of the float64 reference (0.011 and 0.006 with an F16 cache); batches against single tokens bit for bit on OpenCL, within 9.7e-6 on Vulkan. - The published files
(
tests/model/test_archs_public.c, RX 7900 XT, every block on the device), choices equal to llama.cpp's of 160 on OpenCL and Vulkan, then the mean logit difference from llama.cpp and from the CPU: Spark-X2.6-1.7B 158 and 160 (the CPU 159), 0.164 and 0.143, 0.155 and 0.155; Spark-X2.6-4B 158 and 158 (158), 0.172 and 0.174, 0.216 and 0.221; Muse Glimmer 30B UD-IQ2_XXS 156 and 157 (157), 0.171 and 0.171, 0.163 and 0.163. The largest differences from llama.cpp are 7.67 and 8.37, 2.89 and 3.19, 3.66 and 3.63 (single logits far below a step's best); at the 4B's step where the CPU is 16.2 from llama.cpp the devices are within 3.2 of it. A paged cache gives the plain context's choices (16 of 16), half of the blocks on the CPU 16, 15 and 15 of 16. - Speed, prompt and decode in tokens a second on the CPU, OpenCL and Vulkan: the 1.7B 482.4 / 22.6, 8030 / 215.7, 8179 / 285.3; the 4B 166.4 / 9.80, 3469 / 117.9, 3614 / 144.9; the 30B (whole on the 20 GB device) 20.9 / 2.42, 255.6 / 29.6, 831.7 / 35.6.
- Muse Glimmer's file (10.7 GB, SHA-256
3e9bd0e6...0875ad9) was downloaded, run and removed; its llama.cpp recording was made again withtools/gen_llama_ref.sh's arguments. The catalogue notes of the three models no longer say "on the CPU only". - Left: in the roadmap (L6): a published file on the Intel UHD 750, a file larger than the device, OpenCL's prompt rate on the 30B's 2- and 3-bit matrices.
Done (2026-11-13)
L6. gemma4 on
OpenCL and Vulkan
A context of a Gemma 4 model asked on a device was given the CPU. It now runs there, every block or the last ones (inference.md, "On OpenCL and Vulkan").
- No kernel was missing. The rotation and attention
kernels take the head size from their operands: OpenCL's attention heads
of 256 and 512 alike, Vulkan's attention group the window blocks' 256
and its latent-attention shaders the 512 of the blocks without window
(F32 and F16 caches; another cache type sends those blocks' attention to
the CPU). The values from the key projection are a copy and a norm
without weight, the block scale a number read from host memory, the
experts' scales a gather and a product, the joined gate and up product
an indexed product and two strided copies.
test_gemma4_headsandtest_gemma4_experts(test_opencl,test_vulkan_attention,test_vulkan_moe) hold these operand shapes to the CPU kernels on the RX 7900 XT and the Intel UHD 750. - The placement (
src/model/model.c): the joined Q, K, V matrix is not used for this architecture, the joined gate and up matrix is, at the block's width; the projection to the embeddings per layer goes with the head and the batch's embeddings have a device set of their own, copied from the host when the first blocks run on the CPU; a device allocates a cache for the blocks that keep one only (15 of the E2B's 35), which also ends the unused slots; the blocks that share a cache go to a device with the two blocks they read (a smaller count asked is raised, one that does not fit leaves the model on the CPU), on one device, without streamed CPU layers. A streamed layer's cache mirror has its block's heads. - The experts of the 26B-A4B
(
src/model/arch_gemma4.c): on their layer's device, or withn_cpu_moeon the CPU from the routing and the rows read back. A context asked with an expert cache or a device expert tier is created without it, with a notice (it failed at the first batch before). - The fixtures (
test_gemma4_models): the four tiny models on OpenCL and on each Vulkan device, every block there and one layer asked (five fortiny-gemma4e), F32 and F16 caches, batches, several sequences, a paged cache, the experts in host memory: 36 / 36 choices, within 1.7e-5 of the float64 reference (0.011 with an F16 cache); batches against single tokens bit for bit on OpenCL and within 8.9e-5 on Vulkan (its prompts and single tokens are two attention kernels, and these models do not scale their scores). - The published files
(
tests/model/test_gemma4_public.c, RX 7900 XT, every block on the device), choices equal to llama.cpp's of 160 on OpenCL and Vulkan, then the mean logit difference from llama.cpp and from the CPU: the 12B 148 and 152 (the CPU 148), 0.55 and 0.58, 0.58 and 0.62; the E2B 158 and 157 (160), 0.49 and 0.45, 0.45 and 0.44; the E4B 158 and 157 (156), 0.27 and 0.24, 0.26 and 0.24; the 26B-A4B UD-Q4_K_M 155 and 152 (152), 0.68 and 0.63, 0.64 and 0.62. The largest differences are the CPU's against llama.cpp but for one step of the 26B-A4B, where a device's whole vector of logits moves (22.8 at most, the best token unchanged). A paged cache, half of the blocks on the CPU and the experts in host memory give the plain context's choices but for near ties (14 to 16 of 16). - Speed, prompt and decode in tokens a second on the CPU, OpenCL and Vulkan: the 12B 64.8 / 3.90, 1464 / 53.8, 1616 / 68.5; the E2B 285.7 / 17.7, 5029 / 120.8, 6042 / 141.2; the E4B 154.1 / 9.07, 3110 / 80.4, 3598 / 115.0; the 26B-A4B (whole on the 20 GB device) 120.8 / 7.94, 2220 / 71.8, 1302 / 106.9.
- The 26B-A4B's file (16.9 GB, SHA-256
f2c28b3d...f63f293f) and the E4B's (5.0 GB,85a896a0...cd1fab87) were downloaded, run and removed; their llama.cpp recordings were made again withtools/gen_llama_ref.sh's arguments. - Left: in the roadmap (L6): the 26B-A4B's expert cache and device tier, a quantized cache under the wide heads on Vulkan, the joined Q, K, V product (the 26B-A4B's one step was traced on 2026-11-14 to a near tie of its routing: above).
L6.
qwen35 and qwen35moe on OpenCL and Vulkan
A context of the published hybrid layout asked on a device was given the CPU. It now runs there, every block or the last ones, the linear blocks' states in device memory (hybrid-layers-design.md).
- The recurrence: the
gated_deltakernel (OpenCL) and shader (Vulkan) take the decay per token beside the test layout's decay per head (exp(factor * softplus(projection + bias)), the key headh mod G, the query oversqrt(dk));test_openclandtest_vulkan_ssmhold both forms to the CPU kernel at 1e-5 absolute plus 1e-4 relative, on the RX 7900 XT and the Intel UHD 750. - The gated queries needed no kernel (a product, two strided copies, a sigmoid and a product); the device placement no longer joins a gated query matrix with its key and value matrices.
- The fixtures (
test_model):tiny-qwen35andtiny-qwen35moeon OpenCL and on each Vulkan device within 1.9e-4 and 2.9e-4 of llama.cpp, 36 / 36 choices, with every block on the device and with the first on the CPU, batches, two sequences, a copy, a file, a cut and a paged cache; the expert variant also with its experts on the CPU, streamed, in the expert cache and in the device tier. - Kept states with the linear blocks on a device (design):
the states are read back to host memory at the same places and written
back when a cut takes one; no journal is written there, a cut is a
forward pass from the nearest kept state.
test_state_keepruns its scenes on OpenCL and on each Vulkan device (within 4e-5 of a fresh evaluation). Decoding Qwen3.5-9B costs the same with and without them (83.0 and 83.3 tokens a second on OpenCL, 104.2 and 104.0 on Vulkan). The same change ends a crash: a hybrid context whose CPU layers were streamed to a device for prompts (stream_batch) read its journal rows from a device address. - Qwen3.5-9B Q4_K_M (
test_qwen35_public, each backend): 158 of 160 choices on llama.cpp's tokens on OpenCL and on Vulkan (159 on the CPU), the mean logit difference 0.122 (0.116) and the largest 13.7 (13.8); against the CPU 157 and 159 of 160 choices, 0.113 and 0.119 on average, 3.65 and 3.55 at most; a prompt in batches of 64, two sequences, the state through a file, a copy and a cut, a paged cache and 16 of 32 blocks on the CPU. Prompts at 1839 and 2300 tokens a second and decoding at 83.0 and 104.2, against 79.3 and 5.46 on the CPU. - Qwen3.6-35B-A3B UD-IQ3_XXS (13.2 GB, SHA-256
9c964e65...07b88fbe, downloaded, run and removed): whole on the 20 GB device, 156 of 160 choices on llama.cpp's tokens on each backend (158 on the CPU), logits within 2.74 and 8.35 (5.14), llama.cpp's two logits at most 0.16 apart where a choice differs; prompts at 1222 and 1536 tokens a second and decoding at 64.7 and 108.6, against 97.0 and 11.65.test_infer_publicruns both files on a device when they are undermodels/(measured through a driver calling the same check, the 35B's file being removed after). - Left: in the roadmap (L6): the 27B and the tunes on a device, a file larger than the device, a journal on a device, the multi-token prediction block there.
L7. The published files that waited for disk
On the CPU, one file on the disk at a time, each compared with
llama.cpp at the pinned commit (which read them all) and removed;
tools/gen_llama_ref.sh records their references again and
test_infer_public compares a file found under
models/.
BugTraceAI-CORE-Ultra 27B
(BugTraceAI/BugTraceAI-CORE-Ultra-27B-Q4, Q4_K_S, 15.6 GB),
with no change to the runtime: 5 of 5 prompts tokenized alike, 157 of
160 greedy choices on llama.cpp's tokens, logits within 3.59, three of
five continuations identical and the others parting where llama.cpp's
two logits are at most 0.08 apart; 99 of 100 choices twice over 100
greedy tokens; oklm chat opens a thought whose first 80
tokens are llama.cpp's. Bounds 6.0 / 0.95 / 0.6.
Muse Glimmer 30B
(unsloth/Muse-Glimmer-30B-GGUF, UD-IQ2_XXS, 10.7 GB): the
forward pass checked until then on a tiny model needed no change (5 of
5, 157 of 160, logits within 5.51, three of five continuations, 0.10 at
most where they part; 100 of 100 and 96 of 100 over 100 tokens). The
chat did: the model writes a reply as messages with a recipient
( to=self<|message|> its reasoning
<|eom|>, then
<|start|>assistant to=user<|message|> the
answer <|eot|>), the file types the four tokens as
control tokens, and the reply stopped at the first. The three tokens
written inside a reply are now read as added tokens that are not special
(recipient_tokens(), src/tok/tok_gguf.c), the
reply ends at <|eot|> as in llama.cpp, and the
reasoning splitter takes the message to self as reasoning
(src/harness/think.c; oklm chat,
/v1/chat/completions, /v1/messages).
oklm chat then gives the first 69 tokens of llama.cpp's
greedy reply and answers a second turn. In the catalogue as
muse-glimmer-30b; bounds 9.0 / 0.95 / 0.6. Not done: the
perception encoder, the template's tool calls, the devices.
The three tunes of Qwen3.6-35B-A3B (Q4_K_M, 21.2 GB
each), with no change to the runtime: Uraion Cyber V1.0 5 of 5, 159 of
160, logits within 4.40, four of five continuations, 100 of 100 twice;
Imperum-CybersecurityLLM v1.0 5 of 5, 154 of 160, 2.59, two of five, 97
and 95 of 100; CyberStrike-OffSec-35B (five shards) 5 of 5, 154 of 160,
2.30, one of five, 97 and 98 of 100. The last two have the closest
logits of all the files and the flattest choices (llama.cpp's two best
0.01 to 0.26 apart where they part); their bound on agreement is 0.93.
12.3 to 12.5 tokens a second on 6 threads. CyberStrike's
multi-token prediction block runs as the draft of
oklm run --mtp: 195 drafts, 64 accepted (32.8 %), the same
100 tokens, at 8.8 tokens a second against 12.5 without.
L7. Gemma 4 26B-A4B on its file, and the 12B's gap against llama.cpp
Done with no change to the forward pass (inference.md, testing.md).
The 12B's gap (148 of 160 choices, logits within
12.5) is rounding. Its whole stack was compared with llama.cpp's hidden
states (tools/llama_dump.c, OKLM_DUMP,
tools/dump_compare.py): the first step that departs is the
first block's attention (1.5e-4), then its GELU; each block on
llama.cpp's stream (OKLM_DUMP_FEED) adds 7e-4 to 1.8e-2,
none more than its neighbours; each step on llama.cpp's operands
(OKLM_DUMP_FEED_OPS=1) agrees within 6e-6 over the 48
blocks but attention (2.5e-4) and the GELU-gated product (1.8e-4).
llama.cpp rounds the queries and the probabilities to half precision in
its products with the F16 cache (OKLM_ATTN_F16=1 does the
same: 2.8e-8) and reads the GELU from a half-precision table
(OKLM_GELU_F16=1: 2.0e-6). On the same 160 steps llama.cpp
with flash attention against itself gives 153 of 160 choices and logits
within 11.7, with an F32 cache 150 and 12.3, and OKLM's scalar kernels
against its AVX kernels (attention summed in another order, 5.5e-7 in
the first block) 157 of 160 and 14.3: the figures are those of two runs
of one program. The 12B was also run with its tool calls, its thinking
channel and its suppress list (test_gemma4_suppress: the
best logit after an opened image is <image|>, which
the list holds).
Gemma 4 26B-A4B
(unsloth/gemma-4-26B-A4B-it-GGUF, UD-Q4_K_M, 16.9 GB: Q8_0
matrices, Q4_K and Q5_1 experts), run on the CPU and removed: 5 of 5
prompts tokenized alike, 152 of 160 choices, logits within 12.1, 72 of
160 free-running tokens before parting, a chat turn with and without
thinking, 26.9 GiB resident at 8192 tokens, 7.1 tokens/s. On llama.cpp's
hidden states the router, the eight experts of every token and their sum
agree within 4.2e-6 over the 30 blocks. Its first block's queries are
4.6e-4 from llama.cpp's: an exact half of a Q8_0 activation goes away
from zero in OKLM (ggml's portable code) and to the even step in
llama.cpp on x86 (OKLM_Q8_ROUND_EVEN=1 for comparisons).
The widest gap (5.2) is a step where the half-precision roundings
decide: after four lines of Python llama.cpp indents by twelve spaces
and OKLM by eight, and with the two variables OKLM writes twelve.
llama.cpp against itself on this file: 155 of 160 and 10.1 with flash
attention, 150 and 15 with an F32 cache. In the catalogue as
gemma4-26b-a4b, marked run; the test's bounds are 20 / 0.9
/ 6.5.
No cache rows for blocks without attention
A context reserved a key and a value cache for every block, the
linear blocks of qwen35, qwen35moe and
qwen3next and the convolution blocks of lfm2
and lfm2moe included, which never read a row. It reserves
rows for the blocks that attend only: on the CPU, in the paged pool (a
block of the pool costs the attention blocks' rows: four times the
blocks for the same budget on Qwen3.5-9B), on a device (its allocations
and the layer fit), in the copy of a sequence, the swap and the
--gpu-stream mirrors. oklm plan counts each
layer's own cache (8 of 32 layers x 32.0 MiB; each layer's
kv_bytes in the JSON, which add up to the cache of either
side on every fixture). The KV file format is unchanged: it never held
rows for these blocks, and a file written before loads as it is.
| Model | Context | Cache before | now | Peak RSS before | now |
|---|---|---|---|---|---|
| Qwen3.5-9B (8 of 32 blocks attend) | 8192 | 1.00 GiB | 256 MiB | 12.66 GiB | 11.90 GiB |
| 32768 | 4.00 GiB | 1.00 GiB | 15.66 GiB | 12.66 GiB | |
| Cyber-Prime 1.1 2.6B (8 of 30) | 8192 | 480 MiB | 128 MiB | 4.25 GiB | 3.90 GiB |
| 32768 | 1.88 GiB | 512 MiB | 5.65 GiB | 4.28 GiB | |
| LFM2-350M (6 of 16) | 8192 | 256 MiB | 96 MiB | 944 MiB | 784 MiB |
| 32768 | 1.00 GiB | 384 MiB | 1.67 GiB | 1.05 GiB |
Nothing computed changes: the text and the --observe
record of every block are the same bytes before and after on the three
models (CPU, and OpenCL for the LFM2 files);
test_qwen35_public and the llama.cpp comparison of the LFM2
files give the figures they gave.
test_rows_of_attention_blocks
(tests/model/test_mem_plan.c) checks the projection against
the context block by block on the five hybrid fixtures, contiguous and
paged, and on OpenCL with all, one, two and three layers on the device.
Details: hybrid-layers-design.md.
Left: the recurrent state is still reserved for the attention
blocks.
L8. Qwen3.5-9B's sensitive steps
The steps where single logits move by 2 to 13.8 between OKLM and
llama.cpp are the model's own sensitivity. llama.cpp run against itself
with only its last places changed (F32 keys and values, the prompt a
token a pass or in batches of 3, its other attention kernel; thread
counts change nothing) moves its logits by 0.17 to 0.23 on average over
100 steps and by 13.7 to 14.9 at most, on the same steps, where OKLM is
0.22 and 13.8 from it; the 13.8 of step 13 is the recorded run alone
against every other run of either engine (within 0.9 of each other).
Block by block: block 0, whose input is the same, agrees to 1e-7 through
the recurrence and its state after 43 tokens, so the gated delta blocks
lose no precision; the first differences are jumps of 1e-5 to 7e-3, the
size of one 8-bit step of an activation; from 2e-3 after block 1 the
stream of two runs is 4e-2 to 6e-2 apart at blocks 12 to 16 on an
ordinary step, and up to 1.0 on a sensitive one (step 13: one value
channel of block 30, -3.8 in the recorded run and +4.7 to +8.4 in the
others; steps 37, 38, 45: blocks 8 to 24), in llama.cpp against itself
as in OKLM against it. The build for Windows is the Linux build to 1e-8
until such a jump. Its other choice at step 38 comes from that
amplification, but no llama.cpp run of seven made it (they move that gap
by 2.3 at most), so its bound stays the measure (5.0; Linux 0.6).
tools/llama_ref.c records llama.cpp with its last places
changed (LLAMA_REF_KV=f32, LLAMA_REF_ONE=1),
tools/gen_llama_ref.sh the continuation a second time, and
test_llama_itself() bounds OKLM's distance from the
recording by 1.5 times llama.cpp's own (measured 1.19 on Linux, 1.17 on
Windows). The measure is in hybrid-layers-design.md.
Small leftovers: the speech fixture, the chat template, re-keeping a server
- The speech fixture. The tiny fixture's 14 540 bytes
with
--first-words 5are the 48 samples of its six words and the 7200 of the pause after the first sentence, which the shorter first pass ends on; neither the split nor the fixture was wrong, the text of the window's check was. Held bytest_speech_routeandtest_split_first. --chat-template. A template asked for is never replaced by the LFM2 tools template (oklm_tokenizer_chat_template_given(),oklm_tool_format_model_template(), shared by chat, serve and the agent); the agent's small model gets the catalogue's format or the option's when its file carries none;oklm run --ragand--rag-critiqueheld to the option's template by a test (test_chat_template_reach,test_lfm2_template).- Re-keep with form in the server manager gives the
selected kept server the form's command under its name
(
oklm_kept_replace()), asked once more, a stopped server left stopped;tools/check_gui_kept.sh15 checks.
Graphical tools: the recorder's device reads batched (T10)
What a block on a device records (the token's row of the stream, its
queries, a block's sinks, its routing) is copied on the device into
staging memory and read back in one round trip when the pass ends, with
the keys the attention rows need; the records are the same bits as
reading at each point (test_device_batched: 657 records a
backend compared with memcmp, 558 reads for 513 passes
against 5918; the files of oklm run --observe the same
bytes before and after on Llama 3.2 1B and Granite 3.1 1B, both
backends). A recording context's decode steps are replayed again on
Vulkan. Tokens a second with --observe on the RX 7900 XT,
medians of five pairs on a busy machine: Vulkan 104 to 235 (Llama 3.2
1B) and 60 to 284 (Granite); OpenCL 169 to 196 and 118 to 129, inside
the noise. oklm_observe_params::unbatched,
oklm_observe_device_reads(). observe.md.
The chats: a spoken conversation in the dashboard (C9's dashboard part)
The dashboard's chat holds a conversation by voice on a server
started with --transcribe-model and
--speak-model (server.md):
- The reply spoken while it is written. With Read aloud on, or in a conversation, the sentences of a reply go to the voice as they end: those no later text can change (whole lines, and of the line being written the sentences followed by a blank with no mark of Markdown left open; a code block told once, the reasoning never), one request at a time, all the sentences that wait in the next one, the first request in passes of five words as the window's; a 503 is asked again six times, two seconds apart, then said.
- Talk and
/talk [on|off]: a row of the shared table (a choice, for the window and the page, needing a recognizer and a voice; written for the page). The microphone stays open; a turn starts when the level stays above 0.25 for 0.25 s and ends after a pause of 1.2 s (both set in the page's row); it is transcribed and sent without a press; the reply is spoken; the page listens again when the voice has ended. The row showslistening,transcribing,thinkingorspeaking. The switch,/talk offor Escape leaves and releases the microphone. Without a recognizer, a voice or a secure origin the switch is disabled with the reason, and/talkanswers it. - Interrupted by speaking: while a reply is written
or spoken, a voice above one and a half times the level for 250 ms stops
the voice, drops the reply's request and starts the next turn. The
microphone is asked for with
echoCancellation,noiseSuppressionandautoGainControltrue, and a warning tells to wear headphones when the browser reports no cancellation. - The voice is the server's, named beside the switches, with its title saying it is chosen when the server starts and cannot be changed nor made from a recording.
- Reasoning sent apart
(
reasoning_contentin the stream's deltas, a harmony model's analysis) is folded above the reply as a<think>block is, kept out of the message, and never said: checked on a stream made in the page, the server's own harmony case being a model of its test program. - The page grew from 69,672 to 81,513 bytes.
Proof: tools/check_dashboard.sh in a headless Chrome
157, 318 checks in all, three runs in a row (CHECKS="talk"
runs the 38 of the conversation alone), against SmolLM2-135M with
Whisper base and OuteTTS 0.2 500M (Q8_0) on the CPU, the two sound
models loaded at start (--preload). Chrome's microphone is
a file the script builds from jfk.wav: the sentence (11.0
s) and silence. With the pause set to 1.5 s through the row, the turn
ends 12.6 s after the microphone opened, not at the two silences of a
second inside the sentence; Whisper's text ("And so my fellow American,
ask not what your country can do for you, ask what you can do for your
country.") is the message sent, one sound posted and one request made;
the states are
listening transcribing thinking speaking listening; the
first sound came 3.8 s after the message was sent (the first sentence
asked for in passes of five words); then four seconds of silence send
nothing, with the track still live; Escape leaves no live track and a
closed context. A reply made long for the check (the request asks for a
string of 9000 characters at least: 3800 tokens) has its first sound
scheduled 5.2 s after the message, while it is still being written
(said.first before said.end, which comes 15.4
s later); /talk off cancels the server's synthesis and the
reply goes on as text. For the interruption a second browser is given
the sentence, silence, and the sentence's end again 28.7 s after the
start (10 s after the first browser's first sound of the long reply):
the state is speaking and the reply still written when the
voice is cut 0.4 s after the second voice starts; nothing more is
scheduled, the reply's request is aborted with its text kept,
speak_stats.cancelled goes up, and what Whisper heard of
the second voice ("Ask not what your country can do for you ask what you
can do for your country") is the next message. The sentences ready of 20
replies cut at each of their 1050 positions are the first ones of the
whole reply's; the turn detector answers 10 rows of made levels
(silence, noise under the level, a short sound, a turn, a short silence
inside a turn, two turns, and the higher level and longer hold of an
interruption) and follows the two settings. The time to the first sound,
read at the end against read while written: a reply of 1500 tokens, 36.7
to 74.9 s against 11.2 to 11.4 s (four runs each); three sentences (53
tokens), 4.9 to 23.0 s against 4.8 to 19.4 s (nine runs each), which is
the spread of a CPU shared with other work: a short reply of a fast
model gains nothing. test_dashboard_chat holds the page's
elements, the table's row and the page's entry for it,
tests/server/dashboard_cases.tsv nine lines of
/talk.
Not verified: echo cancellation with a real loudspeaker and microphone (the fake microphone hears nothing the page plays); Firefox and Safari.
Server and tool-call defects of the day
Defects the day's model and voice work found and left, each fixed with a test that fails without the fix.
- LFM2 tool calls
(
src/model/tool_calls.c, thelfm2format). Liquid's LFM2 writes a call as a Python list between two control tokens,<|tool_call_start|>[get_weather(city="Paris")]<|tool_call_end|>; OKLM ended the reply at the first token and returned an empty answer. The format is recognized from the template (<|tool_call_start|>) or, for a file converted with the plain ChatML template, from the vocabulary's two tokens; the opening token is the call grammar's trigger, so it no longer ends the reply; the grammar is the list (names,key=valuearguments typed by the tools' schemas); the parser turns the Python literals into JSON arguments (strings with either quote and their escapes, numbers,True/False/None, lists, tuples, dicts; an argument by position; the list alone when a client removed the tokens). A file whose template drops the tools is rendered byoklm serveandoklm chat --toolswith a template that writes what Liquid's does (oklm_tool_format_template()).--tool-format's usage and error list every accepted name (harmonyandlfm2were missing;oklm_tool_format_names()). Proof:tests/model/test_grammar.c(test_tool_calls_lfm2: 19 replies and every prefix of each, the detection, the rendered conversation, the grammar's accepted and refused calls;test_lfm2_tiny_model: a call generated under the grammar by a random model parses back); on the published files throughPOST /v1/chat/completionswith one tool, CPU, temperature 0, plain and streamed and withtool_choice: "required": LFM2-350M (Q4_K_M, the plain ChatML template) and Cyber-Prime 1.1 2.6B (Q4_K_M, its own template) both returnget_weather {"city":"Paris"}withfinish_reason: "tool_calls", and answer the second turn from the tool's result ("The current weather in Paris is 17 degrees Celsius with light rain.");/v1/messagesreturns the same call as atool_useblock. Left:oklm agenton an LFM2 file whose template says nothing of tools (it detects the format from the template alone and keeps the model's template). - gpt-oss's harmony markup where the text was not
split.
/v1/chat/completionsreturned the channels' markup incontent, andoklm chatprinted it on standard output. The chat-completions route had no reasoning field on its answers (a<think>block stays incontent, which the clients here read); for a model that writes harmony messages it now gives the analysis channel's message asreasoning_contentand the final channel's ascontent, whole and streamed (reasoning_contentdeltas first; with tools, a message that is not the final one is held back until the call is parsed).oklm chatshows the reply through the same splitter on standard output (dimmed on a terminal, between<think>and</think>otherwise) and keeps the answer alone in the conversation, so the next turn's prompt no longer holds the last reply's markup twice.oklm_think_split_text()splits a whole reply. Proof (the published file, 11.6 GB, is not on disk and was not downloaded):tests/harness/test_think.c(test_whole: eight replies recorded as text);tests/server/test_server.c(test_harmony_chat: a harmony reply forced on the tiny model by a schema with one constant, on a server set to the harmony format:reasoning_content"The user greets me.",contentthe answer, no<|in the body, whole and streamed, cut by the token limit inside the analysis, a required call, and the same request on a server that is not harmony left alone);tests/harness/test_cli_chat.c(test_harmony_view:oklm chaton a model made by the test, whose one block turns each token of a harmony reply into the next: two turns shown as<think>,Hm,</think>,Ok, the conversation file holdingOktwice and no analysis). Left: the same on the published gpt-oss file; a commentary message that is not a call (a preamble) still shows its markup; the chat window reads noreasoning_contentfrom the stream, so it shows a gpt-oss answer without its reasoning. - A streamed WAV refused.
oklm_audio_from_wav()(src/mm/audio.c) stopped at a chunk whose length the file does not hold, so a WAV whose RIFF and data lengths are0xFFFFFFFF(whatPOST /v1/audio/speechwrites with"stream": true, and ffmpeg to a pipe) had no samples:oklm transcribeand/v1/audio/transcriptionsanswered "malformed data". A data chunk whose length is beyond the file is now the rest of the file (an incomplete last frame left out); another chunk in that state is still a refusal. Proof:tests/mm/test_audio.c(test_wav_streamed: a file made by the test with both lengths0xFFFFFFFFgives the samples of the file with its true lengths, throughoklm_audio_decode()too; an odd last byte, no sample at all, the true length on a cut file, a LIST chunk before the data); for real on the CPU: "The harbour of Brest opens on the Atlantic ocean." spoken byoklm serve --speak-model(OuteTTS 0.2 500M) with"stream": trueand saved (132 524 bytes, 2.76 s, both lengthsff ff ff ff), then heard by Whisper base as "The harbor of breast opens on the Atlantic Ocean." through/v1/audio/transcriptionsand throughoklm transcribe; the build before the change answers "malformed data" for the same file. - The sound models loaded when the server was
created. A server with
--transcribe-modeland--speak-model(the chat window's own among them) loaded both at its start: 253 MiB for Whisper base that nobody may record for, 1.6 GiB for the voice. They are now checked at start and loaded by the first request of each route (server.md): the check reads what is cheap (oklm_whisper_check(): the configuration, the safetensors header, the tokenizer;oklm_speech_check(): both GGUF headers, the vocabulary's OuteTTS tokens, the voice's codes against the vocoder's), so a wrong path, another kind of file or an unknown voice still stops the server at its start with the same messages;--preload(oklm_server_params.preload, added at the end of the structure) loads at start as before;/api/statssaysloadedandload_secondsintranscribe_statsandspeak_stats; a model that passed the check and cannot be loaded answers 500 with the reason and is tried again by the next request. Proof:tests/server/test_server.c(test_lazy_audio: neither loaded at start, each loaded by its first request and not by a refused one, both loaded at start withpreload, three wrong paths refused at start without it, files taken away after the start answered 500 on both routes while the chat goes on, then loaded once back); the existing route tests pass unchanged in the new default. Measured on the CPU with SmolLM2-135M as the chat model, Whisper base and OuteTTS 0.2 500M: 874 MiB at start against 2604 MiB with--preload; the first transcription loads in 0.18 s (1410 MiB after it), the first speech in 0.53 s (3025 MiB after it); the sound of the first and the second speech request are the same bytes. The chat window needs no change: its server is made with the defaults, and the window already says "Transcribing ..." and "the first sound comes once its first words are made" while a first request runs. Not measured: the window's own memory at start after the change. waitingcounted the text being spoken.speak_stats.waitingandtranscribe_stats.waitingread 5 with one request served and four in the queue. Neither the dashboard nor the chat window read the member (they readspeak_statsformax_inputand for its presence); the dashboard's scripted check did, and follows (tools/check_dashboard.mjs). Its meaning is corrected rather than documented:waitingis the queue alone (0 to 4) and a new memberactiveis 1 while a text is spoken or a sound transcribed. Proof:tests/server/test_server.c(test_speech_queue: one long text being spoken and four behind it read"active":1,"waiting":4, a sixth is answered 503, and once every client left both are 0 and the next text is spoken; the transcription figures have the two members);tools/check_dashboard.shin headless Chrome, the voice and speech groups: 95 checks pass, the busy server among them (1 / 4), and one fails,/listenended by Enter after 3.6 s (Whisper hears "America and" where the check wants "American"): it fails the same way with the binary built before these changes, on this machine while other work runs.- A client leaving while its speech request waits in the
queue had no test.
tests/server/test_server.c(test_speech_leave_waiting): a long text is being spoken, a request for a whole WAV file and a short one wait; the first waiting client hangs up and its request leaves the queue at once (cancelled1,waiting1, the long text stillactive), is never spoken, and its place is free (three more wait, one beyond is answered 503); then the long text's client leaves and the four that wait are each spoken in turn with the same samples. The test found nothing to fix; it andtest_speech_queueran nine times without a failure.
L1. Vision: Qwen2-VL and Qwen2.5-VL
oklm describe and oklm serve --mmproj run
Qwen2-VL and Qwen2.5-VL (multimodal.md); the
catalogue offers qwen2-vl-2b
(ggml-org/Qwen2-VL-2B-Instruct-GGUF) and
qwen2.5-vl-7b
(unsloth/Qwen2.5-VL-7B-Instruct-GGUF, the model L5 asked
for), Q4_K_M with their F16 projectors, all four files run here; Apache
2.0.
The encoder (
src/mm/vision.c): an image at its own size in units of 28 pixels (the model'ssmart_resize(), a bicubic stretch), the two kernels of the two-frame patch embedding summed, two-dimensional rotary positions on the queries and keys of every block, no learned positions; Qwen2-VL: layer norms and quick-GELU; Qwen2.5-VL: RMS norms, a gated MLP with SiLU, attention in windows of 4 x 4 merged embeddings except in every eighth block (clip.vision.n_wa_pattern); the merger: a norm, each 2 x 2 group of patches joined,mm.0, exact GELU,mm.2.oklm_vision_dynamic(),oklm_vision_sized_tokens(),oklm_vision_encode_sized();oklm_vision_tile()gives the image at its size (oklm_vision_tiles.w,.h).The language side (
src/model/mrope.c): the architectureqwen2vlisqwen2withrope.dimension_sections: a row has a time, a height and a width position, each turning its band of a head's rotated pairs.oklm_eval_embd_grid()andoklm_eval_batch_grid()take an image's rows with the columns of its grid, whole or in pieces; the text after an image continues from the largest position used plus one (oklm_seq_n_pos()). The context keeps each cached row's time position and the next position, through a cut (in the middle of an image too), a copy, a swap, and with the blocks of a paged cache, so the prefix cache returns an image's rows with their positions. A text goes through the usual rotary operation and gives bit for bit the logits of the same tensors asqwen2.The audit's findings: the two references agree on the model. They differ on the limits of the picture's area (the processor's defaults, 4 to 1 280 embeddings, which OKLM takes; llama.cpp 8 to 4 096;
--tiles SIDEasks for more, up to 4 096), and llama.cpp's embeddings are 0.3 to 2.4 % from the model's, which is its F16 rounding of each product's operand, concentrated on a few rows. The file of Qwen2-VL gives the language model's width as the encoder's MLP width: the width is read from the tensors.Proof, on the published files (CPU):
Check Qwen2-VL 2B Qwen2.5-VL 7B the pixels of a wide and a tall picture (588 x 364, 364 x 588) llama.cpp's, byte for byte llama.cpp's, byte for byte the 273 embeddings of each against a float64 numpy pass of the model's definition ( tools/vision_ref.py)relative difference 1.3e-5 and 1.3e-5 2.9e-6 and 2.4e-6 the same against llama.cpp 2.4e-2 and 2.0e-2 3.0e-3 and 1.0e-2 llama.cpp against the float64 pass with operands rounded to F16 and the tanh GELU ( --as-llama-cpp)2.9e-3 and 5.2e-3 1.1e-3 and 1.1e-2 (one row of 273 carries the second) a question without a picture, llama.cpp's 48 greedy tokens fed one by one 48 choices equal, logits within 0.79 47 equal, logits within 2.24; llama.cpp's two best 0.23 apart where they part "What does the newspaper say?" about llama.cpp's test image: the prompt's rows llama.cpp's 418 (391 of the image, 23 x 17) the same llama.cpp's language model given OKLM's embeddings at mtmd's positions, its 48 tokens 46 choices equal, the first 26 in a row, logits within 1.76; two best 0.09 apart where they part 46 equal, logits within 3.16; two best 0.40 apart where they part llama.cpp's whole path (its own encoder) 45 equal, logits within 2.34; two best 0.24 apart 46 equal, logits within 3.45; two best 0.26 apart the same rows at a text's positions (a control) 43 equal, logits within 9.8, a choice changed where the two best are 1.8 apart not run oklm describe --temp 0andoklm serve --mmproj, 64 tokensthe same text ("Men Walk on Moon. Astronauts Land on Plain; Collect Rocks, Plant Flag.") the same text (the date, "MEN WALK ON MOON", the subheading) the server's second request, and a second turn 416 of 418 rows from the prefix cache; 432 of 456 416 of 418; 480 of 497 Tests:
tests/mm/test_vision.c(tiny files of both layouts fromtools/gen_vision_fixtures.py: a wide and a tall image against numpy, the resize against Pillow, the area's limits);tests/model/test_model.c(tools/gen_mrope_fixture.py: a tinyqwen2vlmodel, seven cases seven ways each within 3e-5 of a numpy pass: one call, pieces, a batch on both caches, a cut inside an image, a copy, the prefix cache; a swap; the KV file refused; theqwen2twin bit for bit);tests/mm/test_mm_public.c(the table above, with the references oftools/gen_mm_ref.sh).Timing on the CPU, a machine shared with other work: 12 s for the 2B's prompt with the test image (1 564 patches, 418 rows), 13 s for the 7B's and 13 s for its 64 tokens.
Not done: see L1 in roadmap.md (a device backend, KV files with an image, videos as the models define them).
The chats: a spoken conversation in the chat window (C9's window part)
The chat window holds a spoken conversation (gui.md), reads a reply while it is written and chooses its voice (gui.md).
- The reply spoken while it is written. With Read
aloud on, each time the window draws, what was written since is read
(
speak_streaminapps/gui/chat_speak.h) and every sentence that nothing written later can change joins the queue of the thread making the sound: a request a sentence to a server, a call a sentence to the library. What is read of a reply is now worked out line by line (speak_line(), whichspeak_sayable()runs too), the whole lines once; the line being written is read again at each step, and a sentence of it is given when no character that may still be a mark is left before it and it stays the same with each of 16 closing marks appended, alone, after a letter and before one. The reasoning says nothing, a block of code is "Code omitted." when its fence opens, an emphasis or a link left open holds its sentence back. The voice is started when the message is sent, so that the window's speech model loads while the first sentence is written (and at once when Read aloud or Talk is ticked), and the first piece of a reading is made in two passes, five words then the rest. - Measured, on the CPU under a load average of 16 to
24, five runs each in turn, a reply of three sentences
(
Tell me three facts about the sea, one short sentence each.), the seconds from the message sent to the first sound: Qwen2.5-0.5B with the window's own voice, 19.6 to 34.8 (median 22.1) read after its end, 10.4 to 17.4 (median 13.9) read while written; with the window's server speaking, 11.8 to 32.3 (28.2) against 7.4 to 16.8 (15.2); SmolLM2-135M, 13.1 to 21.3 (18.2) against 8.0 to 12.1 (8.7). The replies' first sentence was written after 0.2 to 2.2 s and their end after 0.5 to 7.6 s: what is left is the voice's first pass (0.07 to 0.2 second of sound per second there, and the voice's prompt of 890 tokens before the first word). - The conversation. A Talk box,
--talkand/talk [on|off](a new row of the commands' table, for every chat that hears and speaks, written for the window on its own server and on a remote one). The microphone stays open; a turn starts when its level held above a threshold for 0.25 s (dips under 0.2 s not counting) and ends after a pause (listen_turninapps/gui/chat_listen.h: -40 dB and 1.2 s,--listen-thresholdand--listen-silence); its words are transcribed and sent without a press, the reply is spoken while it is written, and the window listens again once the voice ended. The row under the conversation saysListening,Listening: you speak,Transcribing what was said,ThinkingorSpeaking: speak to interrupt, with the time and the microphone's level; Escape, the box or/talk offleaves. Nothing is sent for a turn the level never held in, one under a quarter of a second or under -55 dB, or one in which Whisper heard no word; a turn that goes on while its words are found is sent as one message. The mode's states and what moves them are a function without a window (talk_step()inapps/gui/chat_talk.h). - Interrupted by speaking. While the reply is thought
of and spoken the same rule listens: the level held for 0.25 s stops the
voice (the queue dropped, the request closed) and the reply (the request
cancelled), and what is being said, from half a second before, is the
next turn. The window cannot tell the user's voice from its own played
on loudspeakers: the documentation says headphones, and
--no-interruptturns it off. - Voices. A Voice: chooser lists the
built-in voice and the speaker files (OuteTTS's JSON) of
~/.config/oklm/voicesand of the speech model's folder, read again every five seconds;--speak-voiceno longer needs--speak-model. The window's own synthesizer is loaded again with the voice at the next reading; its own server, which has the voice it was started with, is stopped and started again on the loaded model (0.5 s with the test's models), a file the model refuses leaving the voice as it was with the reason; a server at an address keeps its own, and the window says so. Cloning was not started: it needs WavTokenizer's encoder and an aligner. - Checked.
tests/client/test_gui_speak.c: a reply read whole, byte by byte and cut in two at every place gives the same sentences, on the 37 texts of its tables and on 2000 random runs of words and marks (a fixed seed); a run of 300 000 of them found one text, of marks only, that still gave a sentence the whole line says otherwise.tests/client/test_gui_talk.c: the turn detector on levels made for the test (silence, noise under the threshold, clicks, dips, the pause), a recording fed by an open microphone, the test source that gives several recordings at set times, and the mode (a turn, nothing to send, a reply spoken over, a turn that goes on, leaving at each moment). Under Xvfb, with Qwen2.5-0.5B, Whisper base andoutetts-0.2-500mon the CPU, two recorded questions fed as the microphone (--mic-file sky.wav --mic-file two.wav@voice+3) and the voice written to a file: the first question transcribed 0.8 s after its pause and sent, the first sound 10.6 s after, the second recording starting 3 s into the voice and stopping it 0.25 s later, its question transcribed, sent and answered, the window listening again; Whisper hears in the voice's fileThe sky is blown because 2 plus 2 is 4., the five words said of the first reply and the second reply.gui-testruns two turns on the tiny models; the voice chooser was driven with xdotool on them (a voice taken, a file that is no speaker file and a voice of codes the model does not know refused). - Left: cloning; echo cancellation; a person at a microphone and at loudspeakers; the pause and the threshold as controls; the words while speaking; an attached agent; and, in the speech module, the voice's prompt kept in the cache between two passes.
The model manager: Read aloud pressed
The model manager's Read aloud button, on a
speech-synthesis model, had been added without being pressed in a test.
Pressed under Xvfb
(oklm-gui --tool models --select outetts-0.2-500m, a click
on the button): a second window, OKLM - Chat, is started
with --tool chat --read-aloud on the first installed chat
model (SmolLM2-135M), its Read aloud box ticked and the
side panel reading Read aloud: on, outetts-0.2-500m; the
manager stays as it was. Nothing was wrong with it. Its message said the
speech model is loaded at the first reply: the chat window now reads it
as it opens when Read aloud is on, and the message says so.
The chat window: a saved conversation opened again
/save FILE wrote a file that
oklm chat --continue --chat-file reads and the window could
not open (gui.md).
- How.
--chat-file FILEat start, and such a file dropped on the window, which is what the dashboard's chat does with its own; no/loadwas added to the commands' table.cmd_read_chat()(apps/gui/chat_cmd.h) reads a JSON object withmessages: the window's/saveand/export json(the reasoning apart), the terminal's chat file, the dashboard's (parts for a message with a picture, asystemmessage, a tool's result left out). The conversation on screen is replaced, with the system prompt; a dropped JSON file that is no conversation is attached as a document, as before. - Checked.
tests/client/test_gui_commands.c: what the two writers write read back, the dashboard's form, and nine texts that are no conversation. Under Xvfb with Qwen2.5-0.5B:--chat-file saved.json --send "What is my cat called?"on a file whose first message names the cat answersYour cat is called Turing., the system prompt in the side panel; a file that is none says so and the window opens empty. - Left: the drop itself was not driven (the test
display has no drag and drop to script: it shares everything with
--chat-filebut the event); the files that went with a saved message are not kept by/save, so they are not there again.
The chat window: a large context no longer freezes it
/context 1000000 on SmolLM2-135M froze a chat window:
the memory plan said the cache fits (92 GiB of 108 free) and the resize
then ran on the thread that draws the window (gui.md).
- Where the time goes. In the server's allocation:
oklm_server_resize()frees the cache and makes the new one, and the new one is touched whole (the server's resident memory grows by the cache's size). Measured withoklm serveon SmolLM2-135M,POST /api/context: 0.14 s for 8192 tokens (0.7 GiB with four sequences), 0.24 s for 16 384, 0.64 s for 32 768, 1.0 s for 65 536 (6 GiB), 1.8 s for 131 072 (12 GiB); in another run, the machine giving back what it kept of files, 61 s for 262 144 (24 GiB) and 64.5 s for 1 000 000 (92 GiB, 77 GiB resident at the end). A cache that is not touched until it is used would make this instant; that is the server's (src/server/,src/model/), and was not changed here. - In the window. The resize runs on a thread of its
own (the window's own server asked directly, a server at an address by
its route, which blocked the same way). The button reads
Resizing..., the line under the slider says since when
in amber, the status line says that nothing is sent meanwhile, and the
end is told in the status line or in the note of the
/contextthat asked; a message, a change of link and a change of voice wait for it. The server's plan is not asked while its cache is being replaced. - The amber case. A size that fits but whose cache is
more than half of the free memory is said in amber under the slider
(
More than half of it: the server takes a minute or more to fill a cache this large.), before the red of a size beyond the trained context;/contextwrites it in a note before the wait. - Checked under Xvfb on SmolLM2-135M with four
sequences:
/context 262144(22.5 GiB) typed in the message box, the window saved every 0.4 s while the server worked (Resizing..., the time counting), the note at the end, then a message sent and answered; the slider dragged to 1 048 576 (90 GiB of 117 free) shows the amber line and nothing is applied.gui-testruns/context 1024on the tiny model. - Left: closing the window during a resize waits for the server; the server's own lazy cache.
The chats: replies read aloud in the chat window (C8's window part)
The chat window reads replies aloud (gui.md). It is the last piece of C8.
- The controls. A Speak beside the
name over each reply of the model (drawn in the conversation's widget
and hit-tested before the selection: a press on it starts no selection,
and dragging, a double click and copying were run again around it),
which reads Stop while its reply is read; a
Read aloud box beside Record
(
--read-aloud: on at start);/speak [on|off|stop|TEXT], withon,offandstopoffered by the list above the message although the row's argument is a text; while something is read, the row's meter saysPart 1 of 2: the first sound is made,Speaking, part 2 of 2with the time and the seconds of sound made, and a Stop reading button is at its end; Escape stops too when no recording and no list of commands take it. The row of the commands' table has the window's three letters: no command of the table is "not yet" anywhere now. - Who speaks. The window's own server when it was
started with
--speak-model NAME|FILE(--speak-vocoder,--speak-voice); a server at an address when its figures say"speak":true; in every other case the window itself with the library, the first installed speech model loaded at the first reading on the thread that makes the sound: its own server started without the option, a server without a voice (the reply's text is the window's to read), an agent, a session being read.--no-speakfor nobody. An installed speech model is not loaded by default: 1005 MiB for the window on SmolLM2-135M with Whisper base, 2627 MiB once it read a text itself, 2509 MiB from the start with--speak-model. - What is read is the dashboard's rule ported to C
(
speak_sayable()inapps/gui/chat_speak.h): the 16 texts oftools/check_dashboard.mjsgive the same words, and six more texts (line ends of another system, a fence inside a longer one, nested brackets, addresses, headings that are none) are in the test. A text above the server'smax_inputgoes in pieces cut at sentence ends (the page's rule, in bytes); the request is the page's (pcm,words12). - Playback. The thread that makes the sound gives the
samples to a sink, a small interface with two implementations: a queue
that a thread of its own feeds to an
ALLEGRO_AUDIO_STREAM(16-bit mono 24 kHz, four fragments of a twentieth of a second, silence when the queue is empty, muted while what it holds goes by after a stop, paused after a second of silence), and a WAV file (--speak-file, a test aid as--mic-fileis). A pass plays without a gap; between passes the stream plays silence until the next one is made. The device is opened before anything is asked: without one the window saysNo sound can be played: ... Nothing was asked of the speech model.in the note or the status line and in the controls' tooltips. - The client got
oklm_client_post_data(): a POST whose 2xx answer is passed to a callback as it arrives, a chunk a call, the body of another status kept whole.oklm_client_get_data()runs on the same code. - The models that are not chat models. The
speech-synthesis model was offered Use this model
(disabled) and "used from the command line": in the chat window's Models
window the button reads Read replies aloud with it and
turns Read aloud on; in the model manager Chat reads
Read aloud and opens a chat window with
--read-aloud, and the description says what the model is for.
Proof, on the CPU, under Xvfb, each window with its own state
directories, SmolLM2-135M as the chat model and
outetts-0.2-500m, the samples written by
--speak-file and transcribed with
oklm transcribe -m whisper-base:
| Run | Result |
|---|---|
/speak Hello there, this is a test of the chat window.,
by the window itself |
2.4 s of sound in 12.3 s; heard as "Hello there and this is a test of the chat window" |
--speak-model outetts-0.2-500m, Speak
pressed on a reply, by the window's server |
6.9 s of sound in 36.3 s |
the same window, /speak on, then a message |
the reply read by itself when it ended, 3.2 s of sound; the file's 10.1 s heard as "The sea, a vast and mysterious body of water that stretches across the globe, is a vast and mysterious body of water. The skies are brilliant blue with a subtle orange tint." (the replies: the first word for word, the second "The sky is a brilliant blue, with a subtle orange tint.") |
| the same window, Speak then Escape after 5 s | the server's speak_stats.cancelled from 0 to 1, its
waiting back to 0 |
oklm serve --speak-model outetts-0.2-500m --speak-max 120
on a free port, the window on it with --url,
/speak with four sentences |
two parts; Stop reading during the second: 8.6 s of
sound made, cancelled from 0 to 1, the file heard as "The
first sentence is about a quiet morning by the sea. The second sentence
tells of a small boat leading the harbor. The third one is short." |
the same server, six texts asked of it at once, then
/speak in the window |
not spoken: the server's voice is busy with other texts: ask again in a moment
(503) |
a server without --speak-model, the speech model
installed here |
read by the window itself, 1.9 s of sound in 6.2 s |
| the same server, an empty model directory | the note
Nothing here reads aloud: the server at URL has no speech model ... and none is installed on this machine;
--no-speak:
the window was started with --no-speak;
--speak-model outetts-0.2-500m there: the window does not
start, oklm models pull; a file without
--speak-vocoder: it does not start either |
attached to an agent (oklm agent --listen),
/speak TEXT, then --read-aloud and a
message |
1.9 s of sound in 4.3 s by the window itself; the agent's "Hello!" read when its turn ended, 0.8 s |
no sound server and no ALSA configuration, no
--speak-file |
No sound can be played: this machine has no sound device the window can open. Nothing was asked of the speech model.,
the window back after 1.3 s, no model loaded |
| the session's sound server, one sentence | 2.2 s of sound made in 6.7 s and played out by the stream (the window waited for its end); nobody heard it |
| a reply of 57 tokens, for the page's picture | 16.1 s of sound in 43.9 s; heard as the reply but one word ("scashing" for "scattering") |
tests/client/test_gui_speak.c (145 checks) runs what
needs no window: the texts made sayable, the cutting,
/speak's argument, who speaks, the figures, the request
received by a socket of the test and its chunked answer read through
oklm_client_post_data(), the refusals, the bytes made
samples for every size of piece, the queue, the WAV sink read back by
the library's decoder. tests/client/test_gui_commands.c and
tests/client/test_chat_commands.c have /speak
running in the window's three places and its three words completed.
make gui-test has one more line: the window's own server on
the tiny speech model the server test writes, /speak with
six words, the 48 samples counted in the file of
--speak-file.
Left: in roadmap.md, C8.
L5.
gemma4: Gemma 4 12B, and the 26B-A4B's form
Audited from the headers of the 12B, the 26B-A4B and the E2B (the
first 40 MB of each file by an HTTP range, extended sparse with
truncate) and from llama.cpp's
src/models/gemma4.cpp at the pinned commit (inference.md). Against Gemma 3: the
blocks are of two kinds with heads of different sizes
(256 in the window blocks, 512 in the others, 16 query heads in both, so
the attention of a block without window is twice as wide) and their own
count of key / value heads; a block without window has no
attn_v and takes its values from its key projection; the
values are normalized without weight; the scores are not scaled;
rope_freqs (1 for 64 pairs, 1e30 for the rest) divides the
frequencies of the blocks without window only; each block's output is
multiplied by layer_output_scale. The 26B-A4B keeps the
dense feed-forward and adds 128 routed experts beside it, 8 a token: the
router reads the stream through a norm without weight, 1 / sqrt(width)
and a scale per dimension, gate and up are one tensor
(ffn_gate_up_exps), each expert has an output scale, each
side its own norms. The E2B / E4B add an embedding per layer and blocks
sharing earlier blocks' keys and values: refused with a message, and
oklm info says so.
Code: hyper-parameters in src/io/hf_config.c
(gguf_gemma4()), the llama family's forward pass with six
hooks (src/model/arch_llama.c),
src/model/arch_gemma4.c for the binding and what the hooks
call, each layer's cache allocated in its block's shape
(src/model/model.c, model_kv_width() and
model_layer_heads() in
src/model/model_internal.h). The joined expert tensor is
bound as the gate stack with twice the rows; one indexed product gives
both halves.
Proof:
tiny-gemma4,tiny-gemma4moe,tiny-gemma4moe-split(6 blocks, a window of 4 on two blocks out of three, heads of 8 over 2 and of 16 over 1, one block withoutattn_v; 8 experts of 8, 2 a token, joined or apart) against the float64 reference: 36 / 36 choices each, logits within 8.0e-6 (0.011 with an F16 cache); batches and several sequences bit for bit on the contiguous and the paged cache (test_gemma4_models;test_modelwhole: 27 981 checks). llama.cpp on the same fixtures: the reference's 36 / 36 tokens, logits within 0.012 (dense) and 0.020 (experts). A first fixture had a router choice within 2e-4 of a tie, which llama.cpp's F16 cache turned the other way: the seed was chosen for a clear margin.- Gemma-4-12B-it Q4_K_M (
unsloth/gemma-4-12b-it-GGUF, 7.1 GB, SHA-256 that of the catalogue) against llama.cpp: tokenization 5 / 5, 148 / 160 choices on llama.cpp's tokens, logits within 12.5, 1 / 5 continuations identical, 66 / 160 tokens before parting, llama.cpp's margin there 1.64. Wider than the other families, so it was taken apart: withOKLM_GELU_F16=1(ggml's half-precision GELU table, added tosrc/core/cpu/ops_cpu.cfor this) OKLM's own logits move by up to 2.5 at a step and the figures become 149 / 160, 15.4, 2.09: one rounding moves this model as much as the two programs differ. Two of its blocks alone in F16 (tools/gguf_cut_blocks.py: blocks 4 and 5, 16 384 tokens, no cap, logits of 250 to 650): within 0.25 of llama.cpp, 96 / 96 choices, 3 / 3 continuations (gemma4-12b-cutintest_llama_cpp_gemma4); blocks 0 and 5: within 0.42, 96 / 96. oklm chatrenders the template and answers, thinking off (the default) and on (--template-vars '{"enable_thinking": true}': the thinking channel, then the answer);oklm infoloadable;oklm plan10.96 GiB resident at 8192 tokens; about 2 tokens/s on the CPU beside other work.
Catalogue: gemma4-12b (run), gemma4-26b-a4b
(not run: the file did not fit the disk). Left: in roadmap.md, L5.
L5.
spark2_5 and muse-glimmer, and the audits of
mllama and qwen4exp
The last four models of L5 that had not been looked at. Their headers
were read first (the first 16 to 32 MB of each file by HTTP range), then
llama.cpp's graphs at the pinned commit (8642541b), which has
spark2_5, muse-glimmer and
qwen4exp and no mllama.
spark2_5(Spark-X2.6), implemented and run. The llama block with one fusedattn_qkv, a window of 512 positions on three blocks out of four (named by a list of booleans), the NeoX rotation over the whole head (256) with the base 10 000 in the window blocks and over 64 dimensions with the base 5 000 000 in the others, GELU gating, and an attention output gate: each head's output times the sigmoid of a projection of the block's normalized input. That gate is the "one BF16 tensor a block" of the tensor list (attn_gate, 2560 x 16, kept at full precision by the quantizer; the 37th is the token embeddings). Code:src/model/arch_gate.c(the gate), hyper-parameters insrc/io/hf_config.c, and insrc/model/arch_llama.cthe count of rotated dimensions per kind of block. The pre-tokenizerspark2_5was already among the presets. Proof:tiny-spark(5 blocks, a window of 4, 16 and 8 rotated dimensions) against the float64 reference: 36 / 36 choices, logits within 2.6e-5; batches and several sequences bit for bit; llama.cpp on the same fixture gives the reference's 36 tokens, logits within 0.021;- Spark-X2.6-1.7B Q4_K_M against llama.cpp: tokenization 5 / 5 (the chat turn with its control tokens included), 159 / 160 choices on llama.cpp's tokens, logits within 4.53, 4 / 5 continuations identical, 129 / 160 tokens before parting, llama.cpp's margin there 0.03; its BF16 file (3.4 GB, removed after): 160 / 160, within 0.35;
- Spark-X2.6-4B Q4_K_M: 5 / 5, 158 / 160, within 16.2 at one step and 9.4 at another (2.76 and 0.69 there with an F32 cache, 3.88 at most and 159 / 160: sensitive steps, as Qwen3.5-9B has), 3 / 5 continuations, 122 / 160 tokens, margin 0.20;
- past the window, after 769 tokens: 14 / 16 and 16 / 16 choices, logits within 1.01 and 1.42 (a copy of the reference program with a longer context);
oklm chatrenders the template and answers (reasoning,</think>, the answer), about 8 tokens/s for the 4B; a reply that repeats its answer after a second</think>was checked to be llama.cpp's tokens too. In the catalogue asspark-x2.6-4bandspark-x2.6-1.7b(both files run, their SHA-256 those of the catalogue).
muse-glimmer(Muse Glimmer 30B), implemented on a fixture, no file run. A relative of the above and of Gemma 3: normalized embeddings (no weight), per-head Q/K norms, a window of 2048 positions on three blocks out of four, rotation in the window blocks only, a gate per attention output value, norms after the attention and the feed-forward with an epsilon of 1e-8, SwiGLU, logits timeslogit_scalethen soft-capped at 20.tiny-muse-glimmeragainst the float64 reference: 36 / 36, within 4.4e-6; llama.cpp on the fixture: the same 36 tokens, within 0.006. The smallest published file is 10.7 GB and the disk had 15 to 23 GB free with 12 to keep: not downloaded, so the model is not in the catalogue.oklm infoon its header says loadable.mllama(Llama 3.2 11B Vision), audit. The GGUF (leafspark/Llama-3.2-11B-Vision-Instruct-GGUF) is the one Ollama serves, in Ollama's own layout. Language file: 40 blocks, of whichmllama.attention.cross_attention_layers= 3, 8, 13, 18, 23, 28, 33, 38 are cross-attention blocks with no self-attention:attn_norm,cross_attn_q_proj[4096, 4096],cross_attn_k_projandcross_attn_v_proj[4096, 1024] (8 key / value heads of 128 reading image features of 4096), per-headcross_attn_q_normandcross_attn_k_norm,cross_attn_o_proj, thenffn_normand a SwiGLU feed-forward; two scalarscross_attn_attn_gateandcross_attn_mlp_gate, read through tanh, scale the two residual branches. The other 32 blocks are Llama 3.1 8B's (rope_freqsincluded). The embedding table has 128 264 rows for a head of 128 256: the image token and its neighbours are input-only. The image comes from a 1.9 GBmmprojof architecturemllamatoo: tiles of 560 pixels in patches of 14 (1601 positions with the class token), up to 4 tiles with gated tile and position embeddings, 32 blocks then 8 gated global blocks, the outputs of blocks 3, 7, 15, 23 and 30 joined to the last one (7680 values) and projected to 4096. Each cross-attention block reads those rows as keys and values; in the model's definition (transformers' Mllama, not checked against Ollama's code) there is no rotation there, a token attends the image that precedes it, the feed-forward of a text row that has no image is masked out, and without an image the eight blocks are skipped. A reference would be an Ollama build (Go and its own copy of ggml) or the checkpoint under transformers (21 GB); neither is here, so no code was written, not even the text-only path.qwen4exp(CYBER-FROST-3.8), header audit. 48 blocks of width 2560, three Gated DeltaNet blocks then a full-attention one (24 heads of 256 over 2, queries joined with their gate) asqwen35moe, the samessm.*keys and tokenizer, 512 experts of 640 with 10 a token and a gated shared expert. What it adds:hyper_connection.count4 residual streams (hc_attn_*andhc_ffn_*in every block: a grouped norm [10240], adown[10240, 320] /up[320, 10240] pair and aninject[10240, 4];output_hc_*in place of the final norm; no layer norm anywhere);ple.*, a per-layer embedding for block 1 gathered fromper_layer_token_embd[160, 320 001 536] (28.8 GB in Q4_0) through 16 hashes of the last three token ids, with a causal convolution and a key / value gate; a sigmoid in the linear blocks' output gate where Qwen3.5 has SiLU;attention.indexer.*tensors for a sparse attention whosecompress_ratiosare all 0 in this file (dense attention);rope.dimension_sections11 / 11 / 10 for images. Files of 80 to 123 GB, a multi-token prediction block and a projector in files of their own.
Tests: tests/model/test_model.c (the two fixtures in the
reference, batching and sequence checks,
test_attn_gate_models: the hyper-parameters, and a device
context computing on the CPU),
tests/model/test_infer_public.c (the two Spark files,
skipped without them). Not run: any device (hidden for this work, so the
device half of test_attn_gate_models was skipped), the
sanitizers.
Architectures:
qwen35moe on Qwen3.6-35B-A3B (L5)
The expert variant of the published hybrid layout, on the CPU (the
layout, compatibility.md).
oklm info answered "architecture 'qwen35moe' is not known";
it now says loadable, and the tunes asked for are in the catalogue
(uraion-cyber-v1.0,
imperum-cybersecurityllm-v1.0,
cyberstrike-offsec-35b, security), marked as not run.
- The audit: llama.cpp's
src/models/qwen35moe.cppat the pinned commit8642541bis itsqwen35.cppwith another feed-forward: a router (ffn_gate_inp, softmax, the 8 largest of 256 kept and renormalized, no scale), SwiGLU experts in separate gate, up and down tensors, and a shared expert whose output is multiplied by the sigmoid offfn_gate_inp_shexp. That isqwen3moe's routing withqwen2moe's shared expert, both of which OKLM runs; the name needed the renormalization chosen for it, and the published files give nofeed_forward_length, which the reader required: it now takes the experts' size when that is what a file has. - The fixture
tiny-qwen35moe(tools/gen_model_fixtures.py; 6 experts, 3 a token, a shared expert, nofeed_forward_length), whose reference is llama.cpp's: every logit within 2.9e-4, 36 / 36 greedy choices; a prompt in batches of 3 and token by token give the logits of one batch; the indexed expert products and the expert cache as on the other expert fixtures (test_modeltest_qwen35moe, test_moe_prefill, test_expert_cache). - The published headers (the first 24 MB of each
file, the local file extended to its full size):
unsloth/Qwen3.6-35B-A3B-GGUFUD-Q4_K_M,UraionLabs/Uraion-Cyber-V1.0-GGUFQ4_K_M (MIT),IMPERUM/Imperum-CybersecurityLLM-v1.0-GGUFQ4_K_M (Apache-2.0) and the five shards ofahmedandaloes/CyberStrike-OffSec-35B-GGUFQ4_K_M (Apache-2.0):oklm infosays loadable, the model loads with every tensor bound to its shape (733 tensors; 753 for CyberStrike, which carries a multi-token prediction block),oklm plangives 41.6 GiB resident at 8192 tokens for the tunes, of which 18.2 GiB are routed experts that can be streamed from disk. - Against llama.cpp on a published file
(
test_infer_public,tools/gen_llama_ref.sh): the disk had 27 GB free, which the 21 GB Q4_K_M does not fit with its margin, so the base model'sQwen3.6-35B-A3B-UD-IQ3_XXS.ggufwas used (13.2 GB, SHA-2569c964e65...07b88fbe; Q6_K, IQ3_XXS, IQ2_S and IQ4_XS tensors), loaded by one engine at a time and removed after. 5 / 5 prompts tokenized alike; 158 / 160 greedy choices on llama.cpp's tokens; logits within 5.14; four of the five continuations identical, the fifth parting after 6 tokens where llama.cpp's two logits are 0.08 apart.oklm plangives 31.5 GiB at 8192 tokens; 11.7 tokens a second decoding on 8 threads of a machine busy with other builds;--expert-cache 3gives the same text at 6.7;oklm chatwith the file's template thinks, then answers.
Left, on the roadmap under L5: a run of each tune's own file and of
the base model's Q4_K_M (with a catalogue entry for it), CyberStrike's
multi-token prediction block, ffn_gate_up_exps, the device
backends.
Architectures: Ternary Bonsai 2 (L5)
Prism ML's ternary Qwen3.5 models
(prism-ml/Ternary-Bonsai-2-2B-gguf, -4B-gguf,
-9B-gguf, Apache-2.0), on the CPU (the
layout, compatibility.md). In the
catalogue as ternary-bonsai-2-2b, -4b and
-9b: the PQ2_0 file of each, the one the model
card names as the default among the three packings, all of which OKLM
runs.
What was wrong: the
Q2_0files loaded onceqwen35existed and answered with noise ("The capital of France isPtmana atleast..."). Their matrices are stored multiplied by a vector of signs and a block Walsh-Hadamard matrix (theprism.hadamard.*metadata: blocks of 512 for the 2B and 4B, 4096 for the 9B; the token embeddings' rows rotated; the 2B and 4B with the rotated embeddings as their output head too; the 4B and 9B with the Gated DeltaNet output projection folded on value heads grouped by key head), and a reader that ignores the metadata computes another model without any error. OKLM now reads it (src/model/fold.c): each product with a listed matrix is given its input through the signs and a fast transform, a looked-up embedding row is turned back, and metadata the forward pass would not honour (another version or transform, an expert tensor in the list, a width without its signs) is refused at load. The pinned llama.cpp (8642541b) has the same runtime (load_hparams_hadamard(),build_hadamard_input()), which is the reference.Two tensor types: the
PQ2_0andPTQ1_0files use GGUF tensor types 142 and 143, which are not ggml's: PrismML's llama.cpp fork (github.com/PrismML-Eng/llama.cpp, branchprism, read and built at commit6447906b, CPU only) defines them as Q2_0's two-bit code and TQ1_0's base-3 digits with one scale per 128 weights (34 and 28 bytes a block). They are in OKLM's table under the same numbers (OKLM_DTYPE_PQ2_0,OKLM_DTYPE_PTQ1_0;OKLM_DTYPE_COUNTis 144), with a reader, a dequantizer, a quantizer and dot products against 8-bit activations (src/core/quant_p128.c), and AVX2 kernels for these two and for Q2_0, which had none (src/core/cpu/simd_avx2_tern.c: the 2B'sQ2_0file decodes at 17.9 tokens a second on 8 threads, 2.5 with the scalar kernel).tests/core/test_quant_p128.c: the dequantized floats of fixed random blocks and the quantized bytes of a fixed row hash to what the fork'sdequantize_row_*andquantize_row_*_refgive; each element against a decoder written from the layout; the dot products against a double reference; the AVX2 kernels the scalar ones' bits, one row and four at a time.Against llama.cpp on the published files (
tests/model/test_bonsai_public.c,tools/gen_llama_ref.sh; the pinned build forQ2_0, the fork for the two others; 5 / 5 prompts tokenized alike everywhere):File Greedy choices on its tokens Logit difference, largest (mean) 100 greedy tokens, chat turn / plain text 2B Q2_0(SHA-256c2893fa6...dc560650)160 / 160 0.30 (0.016) 100 / 100, 100 / 100 2B PQ2_0(d5f01a4e...c8b45928)159 / 160 0.32 (0.017) 100 / 100, 100 / 100 2B PTQ1_0(36061ad4...cd9929fa)159 / 160 0.29 (0.016) 97 / 100, 100 / 100 4B Q2_0(64a5183b...ab9433d4)160 / 160 0.32 (0.016) 100 / 100, 100 / 100 4B PQ2_0(d406b1e9...b7f30409)159 / 160 0.75 (0.016) 100 / 100, 99 / 100 9B Q2_0(1f884aa5...813a384d)159 / 160 0.29 (0.015) 100 / 100, 99 / 100 9B PQ2_0(7a89ab48...1fde23f9)159 / 160 0.44 (0.015) 100 / 100, 98 / 100 Where a choice of the 100-token runs differs, the reference's own two logits are 0.001 to 0.06 apart. Run freely, the five continuations of the 2B and 4B
Q2_0files are llama.cpp's token for token (160 / 160); the others part at such a near tie (134 to 141 of 160 tokens before it). The decoded tensors of the 2B'sQ2_0andPQ2_0files are equal value for value; OKLM gives thePQ2_0andPTQ1_0files the same logits to the bit, and theQ2_0file logits within 0.12 of them (the same products added in another order, then the rounding of the activations to 8 bits before each product, which both engines do).The fixtures (
tools/gen_model_fixtures.py,test_modeltest_fold):tiny-qwen35-fold,tiny-qwen35with every matrix folded (blocks of 16, grouped value heads, version 1), within 1.9e-4 oftiny-qwen35's reference (llama.cpp gives the two files the same logits within 4.3e-5);tiny-qwen35-fold-tied, version 2, within 2.9e-4 of llama.cpp on it; four files with one metadata key changed are refused.In use:
oklm infosays loadable,oklm plangives 1.47 GiB (2B), 2.85 GiB (4B) and 4.13 GiB (9B) at 8192 tokens;oklm chatwith the file's template thinks, then answers; decoding at 16, 8.3 and 5.2 tokens a second on 8 threads of a machine busy with other builds.Multi-token prediction: the 4B's
...-PQ2_0-MTP-Q8_0.gguf(SHA-2560acf517d...8406a4d6c0d; thePQ2_0model and a Q8_0 prediction block that reads the rotated embeddings and the tied head) withoklm run --mtp: 30 of 92 drafted tokens accepted, the text of the plain greedy run, 1.0 token a second against 5.2 (each rejected draft costs a hybrid model its prefix).The
dflashdraft model of the 4B and 9B was audited and is not read: six blocks that take the hidden states of eight of the target's blocks and draft 15 tokens in one pass through a lattice of candidates; what it would take is in the design note.
Left, on the roadmap under L5: the device backends (the context runs
on the CPU), the multi-token prediction files against llama.cpp's draft
pass, dflash, the vision projectors, the 4B and 9B
PTQ1_0 files.
The chats: replies read aloud in the dashboard (C8's dashboard part)
The dashboard's chat speaks on a server started with
--speak-model (server.md):
- A Speak button on each reply,
/speak(the last reply),/speak TEXT,/speak onand/speak offwith a Read aloud switch beside Clear (every new reply when it ends; off when the page opens, kept while the page is),/speak stopand a row above the message box that tells what the voice does and holds a Stop button. One thing is said at a time; the chat goes on meanwhile. - The command's row is
/speak [on|off|stop|TEXT], read as a text (none or one argument, the rest of the line as it is):on,offandstopare taken, in any case, when they are the whole argument, anything else is the text to say. Written for the page; the window's places stay "not yet". - Playback:
POST /v1/audio/speechasked forpcmwithwords12, read as it comes and played with WebAudio (anAudioContextat the server's 24 kHz, a buffer per piece read, each started where the one before ends); a stop closes the request and the server's synthesis ends. A text above the server'smax_inputgoes in parts cut at sentence ends. The page'sContent-Security-Policyis unchanged. - What is read: not the reasoning, "Code omitted." for a code block, a link as its text, no mark of Markdown.
- Refusals: no speech model (the buttons and the
switch not shown,
/speaksaying how to start the server), the server busy (503: said, the reply kept with its button), nothing to read, and a browser that plays nothing before a click (said, and played from its start at the first click or key). - The page grew from 60,821 to 69,672 bytes.
Proof: tools/check_dashboard.sh in a headless Chrome 157
with no sound card, 252 checks in all (CHECKS="voice" runs
the 50 of the voice alone), against SmolLM2-135M with OuteTTS 0.2 500M
(Q8_0) on the CPU, with a key and --speak-max 80: "The
harbour opens at six in the morning." asked by /speak is
one request read back (input, pcm,
words 12, the key), 2.39 s of sound scheduled at 24 kHz,
the seconds the page counts those of the server's
audio_seconds, every buffer played to its end; the samples
the page scheduled, written as a WAV file, are heard by
oklm transcribe -m models/whisper-base as "The harbour
opens at 6 in the morning.". The context made by the script and not by a
key is suspended: the page says so, nothing plays, and a click plays it
(no --autoplay-policy flag is given to the browser). Three
sentences (110 bytes) are two requests cut at a sentence's end;
/speak stop after the first sound raises the server's
cancelled and nothing more is scheduled; the reply's button
pressed twice cancels before any sound; another text takes the place of
the reply being said; /reset stops; /speak on
speaks the next reply by itself and the switch follows; with five texts
held at the server the page's request is answered 503 and noted, then
spoken when asked again. The first sound of the three sentences came 5.9
to 11.4 s after the command in these runs, on a CPU shared with other
work (5.2 to 27 s over the day's measures of the route). What is read
aloud is checked on 16 texts, the cutting on 6.
test_dashboard_chat holds the page's elements and its
table, tests/server/dashboard_cases.tsv nine lines of
/speak.
Graphical tools: servers kept across logins (T10, third part)
A server started from the server manager or from a terminal ended with the session. It can now be kept: a service of the user that the system starts at each login and starts again when it fails (server.md, gui.md).
- The module
(
include/oklm/harness/kept.h,src/harness/kept.c):oklm_kept_install()writesoklm-serve-NAME.servicein the directory systemd's user manager reads the user's units from (asked of the manager: the first of itsUnitPathending in/systemd/user), with the mode 0600, thendaemon-reloadandenable --now;oklm_kept_list()andoklm_kept_status()read the units written here and their state fromsystemctl --user show;oklm_kept_start(),_stop(),_restart(),_remove(). The unit:Type=exec, the working directory,Restart=on-failureafter five seconds, five starts a minute at most, no restart for the exit status 2, the output appended to~/.local/state/oklm/services/NAME.log. - The log is a file, not the journal: the unit names
it in
OKLM_PROC_LOGand the server records it in its registry entry, so the window follows it with the code that follows every other log, and it does not depend on what the journal keeps of a user's units. - Quoting by the rules of
ExecStart=: plain words as they are, the others between double quotes with\",\\,%%and$$; a control character is refused.Environment=the same without$$. - Refused: a name outside 1 to 48 lower case letters,
digits,
-and_; a secret given inline (--api-key,--key,--token,--password,--secret), since the unit stays on disk and its command line is shown bysystemctl statusand the list of processes (--api-keys FILEis kept); a name already kept. - The registry: the unit sets
OKLM_PROC_SERVICE, and a server that registers itself writes it in its entry (service, a new field; only with the log that goes with it, and never passed on to a command the registry starts). A kept server is listed once, marked. - No user manager (a container, another init,
Windows): every call answers
OKLM_ERR_UNSUPPORTEDwith the reason, the manager's own message included, and nothing is written.loginctl enable-lingeris never called: the docs give the command and what it changes. - The command line:
oklm serve --keep NAME ...(the same command kept and started, then a wait of twenty seconds at most for it to answer or fail),--kept(--json),--unkeep NAME;oklm serve --listmarks a kept server. - The server manager: a Keep across logins box with a name field, which a profile remembers; kept servers marked in the list, those that do not run listed too with the manager's reason; Stop (asks first: it stays kept), Start, Restart, Remove from kept (asks first), the log followed.
Proof, on 2026-11-13 with systemd 262's user manager, on the CPU:
tests/harness/test_kept.c, 267 checks: 24 names taken or refused; 21 arguments quoted forExecStart=andEnvironment=; a unit's whole text for fixed inputs, read back into the same arguments; what is refused;systemctl showoutput of seven units read; the module against a script in place ofsystemctl(no manager, a unit refused and removed again, modes 0600, a unit of another's left alone); and against the real manager: a command given 15 awkward arguments (two words,it's,say "hi",100%,%h %n %%,$HOME,${HOME},$$,C:\dir\,a\"b,;, an empty one,#x,café,*) and an environment value with$,%, quotes and a backslash printed each of them back unchanged; a command ending with the status 2 wasfailedwithits command ended with the status 2and no restart;sleep 600kept, listed running with its pid, stopped (the pid gone, still enabled), started, restarted (another pid each time), removed (no unit file, no process). 231 checks without the manager (OKLM_TEST_NO_SYSTEMD=1).tools/check_kept_servers.sh, 35 checks, the test model on a free port: a bad name and an inline key refused with nothing written;--keep, the unit 0600 with the command line andOKLM_VK_DEVICES=none, enabled and active;/v1/models200; one registry entry with the service, the unit'sMainPIDand the log; afterkill -KILLthe manager started another process, which answered (NRestarts=1); afteroklm serve --stopthe unit stayedinactiveandenabledsix seconds later;systemctl --user startbrought it back, registered once;--unkeepleft no unit file, no unit for the manager, no log, no process, an empty registry; lingering unchanged;OKLM_SYSTEMCTL=/bin/false:--keptand--keepsaid why and started nothing.- A model file that does not exist, kept:
startingwithits command ended with the status 1; it is started again in a moment, then, within 40 seconds,failedwithit failed five times within a minute and is no longer started again(Result=start-limit-hit,NRestarts=5), the server'scannot loadlines in its log. tools/check_gui_kept.sh, 10 checks, the window under Xvfb at 1100x720 with the real manager: the flow of gui.md, each step's screenshot looked at; then a window without a manager.tests/harness/test_procreg.c, 139 checks, three runs;make gui-testunder Xvfb. The MinGW build compiles the module and the window; under Wine 11test_kept.exepasses its 182 checks (the texts and the parsers, and every call that acts answering that kept servers are not available) andoklm.exe serve --keptprints the reason.
Not done: a reboot or a real logout and login; Windows and macOS; the kept servers' controls in a window narrower than the 1100 pixels the flow was driven at (the fifth button's label is cut below about 900).
Graphical tools: activations in the inspector (T10, second part)
"A graphical view of the models and their activation in the memory": the inspector drew a model's layout and a server's figures, nothing of what happens inside while it runs. It now does (gui.md, observe.md).
The runtime (
include/oklm/model/observe.h,src/model/observe.c). A context asked to record keeps, for the last token of each sequence a forward pass evaluates: per block, the norm of the residual stream after it and of what its attention and its feed-forward part added (the stream read at three points of the block, so whatever the block does around them is counted); for chosen blocks, the attention of that token over the last positions, a row a head, computed again from the queries and the key cache with the model's scale, soft-cap, window and sinks; for a mixture of experts, the experts each router picked and their weights, read where the server's expert map counts them. Bounded (256 positions, 64 heads, 64 MiB; the positions halved to fit). The evaluating thread publishes each record under a lock; any thread copies it. Hooks in the Llama family's pass (which LFM2, gpt-oss and the mixtures of experts go through) and in GPT-2's; the encoders' has none. CPU only: a block on a device is marked as not observed.The command and the route.
oklm runandoklm chat --observe FILE [--observe-attn all|L,L...]: a JSON line for the prompt's last token and each generated one.oklm serve --observeandGET /api/activations[?seq=N][&layer=L|&layers=L,L](src/server/observe_route.c): the last record of a sequence, the blocks named recorded from the next pass on (four at once); 404 with the way to start the server when it does not record, the key asked for as/api/statsdoes (server.md).The inspector. A prompt typed in the window (or
--prompt) is run through the file's model on the CPU by the worker thread, when the memory planner says it fits (the status line gives the figure, or the refusal). Over the map, a strip of what each block wrote for the token shown; under it, the attention of the block selected (a row a head over the prompt's tokens, their texts under it, the cell under the pointer read out) and its experts lit by weight; with no block selected, the map of the experts the routers picked (T7's leftover) or a dense model's writes as a chart. A slider and Left / Right step through the tokens. On a server started with--observethe same views follow the token being decoded, read four times a second off the drawing thread, with the experts from/api/experts.--prompt,--tokens,--step,--layerchoose what--screenshotshows.What the views mean is written beside them (observe.md): a large write is not an important one, attention is not an explanation, an expert picked is not an expert understood.
Proof. Against the float64 reference (
tools/infer_ref.py --observe, independent of the C code), on the committed tiny models,tests/model/test_observe.c:Model Records Attention rows Routings Worst norm (relative) Worst weight Worst expert weight tiny-llama 8 64 0 2.1e-7 1.1e-7 tiny-gpt2 8 64 0 6.1e-7 4.1e-6 tiny-granitemoe 8 64 16 9.0e-7 1.7e-6 6.5e-7 tiny-gptoss 8 96 24 2.9e-6 6.5e-6 1.7e-6 with an F32 cache, contiguous and paged; every expert the reference's, in its rank; every row adding up to 1 within 1e-5. With an F16 cache (the model then computes with rounded keys and values) within 2 % and 0.03. The logits with the recording on are bit for bit those with it off; two sequences in one batch give each the record of the sequence alone; some 19 000 records read by a second thread while 501 passes ran were all whole. The route:
test_activationsintests/server/test_server.c(401, 404, the fields, a block asked for and its rows adding up to 1, about 40 readings while a request of 300 tokens decoded). The window's data:tests/client/test_gui_inspect.c.make gui-test:oklm run --observeon the test model and the inspector with--prompt. The window was driven under Xvfb on the CPU with SmolLM2-135M and Granite 3.1 1B A400M, as files and as servers on a spare port, the screenshots looked at.Cost off. Decoding SmolLM2-135M Q4_K_M on the CPU, 300 tokens at temperature 0, the command built before the hooks and the one built after run in turn, twelve times each, on one thread: 27.6 tokens a second before (22.9 to 34.3; 11.1 s of CPU a run) and 29.9 after (22.0 to 36.7; 10.3 s), medians. The difference is inside the runs' spread, which is wide: the machine was busy with other work (a load of 17 on 16 cores). Idle, the command before gave 235 to 240 tokens a second on its default threads; the one after was not measured idle, nor was the cost of the recording when it is on.
Not done. See T10 in roadmap.md: device layers, latent attention's rows, the dashboard's page, the views on Windows.
Graphical tools: the registry and the listener on Windows (T10, first part)
The registry of background processes and
oklm agent --listen were POSIX only; they have a Windows
half, and the tools that start, list, follow and stop background work do
so there (server.md, agent-loop.md,
gui.md). Everything below was run with the MinGW
build under Wine 11 on Linux, not on Windows itself.
- The registry (
src/harness/procreg.c):%LOCALAPPDATA%\oklm\procs, its access list replaced by one entry for the user (Wine keeps it as the mode 0770 underneath: nobody else, the user's Unix group included); a command started byCreateProcessWwithout a console, in a process group of its own, its output appended to the log, no handle inherited but the log and an empty input; a pid used again told by the process's creation time; the log read without its carriage returns; the user'soklm.exebehind a port found in the system's table of listening sockets (GetExtendedTcpTable, which Wine 11 fills with the owning pids). - The command line: the Windows form quoted and split
by the C runtime's rules, beside the POSIX form, both on every system
(
oklm_procreg_command_windows(),oklm_procreg_split_windows()); the arguments a started program gets were compared with the ones given for a blank, a quote, a backslash before a quote and at the end, a UNC path, an empty argument,%PATH%anda&b|c^d. - Stopping: a process without a console cannot be
sent Ctrl-C or Ctrl-Break by another program, so the request is a named
event the process waits on (
oklm_procreg_on_stop(), whichoklm servecalls); it is terminated when it still runs after the delay, and at once when it waits on no event. - The listener (
src/harness/listen.c): one implementation for both systems now; on Windows the token comes fromBCryptGenRandomand the file is kept for the user by the same access list. - The tools: the server manager's test of a free port on Windows; the messages that named Windows as the reason are general.
Proof (Wine 11, 2026-11-13): test_procreg.exe 164 checks
and test_listen.exe 45 checks, each passed in five runs in
a row on the final build, as in every run before; on Linux
test_procreg 138 checks and test_listen 46,
five times each, and make gui-test passes.
oklm.exe serve on the test model started through the
registry, listed, answering /v1/models, stopped by
oklm.exe serve --stop 17932 in 0.3 s, its log ending with
server stopped. oklm.exe agent --listen 17941:
401 without the token and with a wrong one, its state, a message queued
and its events read back, nothing answering on the machine's other
address. oklm-gui.exe --tool servers under Wine and Xvfb: a
server started from the form, listed, its log followed, stopped, the
screenshots read. Left: see T10 in roadmap.md.
Graphical tools: the nine tools run under Wine (T10, second part)
Each tool of oklm-gui.exe was driven with the MinGW
build under Wine 11 and Xvfb on the CPU, with xdotool, its screenshots
read; what follows is Wine's, nothing was run on Windows itself. Each
tool's section of gui.md says what was run and
what was not.
- Launcher, from a staged folder (the executables,
their DLLs,
docs-html): the eight Open buttons each started a process and a window, which stayed when the launcher ended; Documentation founddocs-html\index.htmland Wine handed it toxdg-open. - Chat: the four unattended runs of
make gui-test; SmolLM2-135M streamed; the commands' list,/help; a file attached by its path, by--attachand by Wine's file dialog, read and answered from; the Models window; Connect... and--urlonoklm.exe serve;--agentonoklm.exe agent --listen; Record (Allegro's recorder under Wine: silence, said so; a sound file transcribed); a certificate nobody signed refused, accepted with--ca. - Models:
smollm2-135mdownloaded (144811360 bytes, TLS checked against the roots Wine's store holds), the three tools started on it, deleted. - Servers: started, context changed 256 to 8192,
restarted, stopped; a profile saved and loaded; a keys file (401, 200);
a server found by its port alone stopped
(
GetExtendedTcpTable). - Convert: SmolLM2-135M F16 to IQ3_M stopped on the
way (its
.tmpremoved), to Q4_K_M whole (105454144 bytes as announced), perplexity 10.139 against 10.417 (+2.74 %; 10.1391 for the source on Linux). - Inspect: a file and a running server.
- Train: a run of 200 steps to its end with its curve and four checkpoints; a run left by its window, found by another, stopped; the file dialog.
- Sessions: listed, searched, read, exported; a live agent marked and attached to; Resume in a console.
- Retrieval: an index built with
hash:384and withbge-small-en-v1.5, sources, a changed and a missing file, the update, a search, the chat window opened on the index.
Defects met and fixed, each in its own commit: a listening agent's
file named its process by the pid alone, so a file left behind made a
session look live once the pid was given again, which Wine does at once
(the file holds the process's start time now; test_listen,
2 checks that failed before); Resume started nothing
(cmd /c start with every word quoted, detached: the console
is started directly, on cmd.exe /k); the resume command and
the retrieval manager's oklm serve line were quoted for a
POSIX shell (test_gui_rag, both forms); the sessions
browser had no dialog on Windows (the system's now); the conversion
tool's file browser opened on the wrong folder and Up
emptied it; the training tool's checkpoint rows, the chat window's model
and the agent's model (apps/oklm.c) kept the whole path
when it was written with backslashes.
The agent's size overflow on the tiny test model, noted
with the first part, is not Windows': the same turn ends the same way on
Linux (the conversation no longer fits the context), the
agent's prompt of 11900 tokens being larger than that model's context,
and oklm.exe agent -p answers with SmolLM2-135M under Wine.
The harness was read for 32-bit long used for sizes or
times: none on a path the tools take; the offsets of a job's output are
long (fseek in
src/harness/jobs.c), which stops following an output above
2 GiB on Windows, left as it is and listed in T10.
Proof (Wine 11, 2026-11-13): sh tools/check_gui_wine.sh,
23 checks in 2 min 6 s, all passed, twice in a row on the final build;
test_listen.exe 46 checks, test_procreg.exe
164, test_gui_rag.exe 87, five runs each; on Linux
test_listen 47, test_procreg 138, five runs
each, test_gui_rag 87, and make gui-test
passes. Left: see T10 in roadmap.md.
L5. gpt-oss:
gpt-oss-20b
Audited against llama.cpp's graph for gpt-oss
("openai-moe"; inference.md). Three
things were asked about:
- The architecture. Attention wider than the residual
stream (64 heads of 64 over 8 key / value heads for a width of 2880)
with biases on every projection, the NeoX rotation with YaRN (factor 32
over 4096), a window of 128 positions on the even blocks and every
position on the odd ones, and attention sinks: one
score per head that joins every softmax of that head without a value.
Then, in every block, 32 experts of which 4 work on a token: a router
with a bias, weights that are the softmax of the 4 chosen scores,
experts with gate, up and down biases and the activation
min(g, 7) sigmoid(1.702 min(g, 7)) (clamp(u, 7) + 1). The attention is the llama family's with the sinks given to the CPU attention op (oklm_attention_params::sinks, on its three paths); the experts aremodel_gptoss_ffn()insrc/model/arch_gptoss.c. - MXFP4 (GGUF type 39, the experts of the published file): OKLM had the type already, blocks and values identical to ggml's (core.md); the audit only confirmed the layout (one E8M0 scale byte and 32 E2M1 codes per block, the low nibbles the first 16 values) and that the indexed expert products read it.
- Harmony. The template renders (system turn with the
date and the reasoning effort, developer turn with the tools as
TypeScript, the turns). A reply is several messages between special
tokens; OKLM ended a reply at its first control token, here
<|channel|>. The tokens inside a reply are now read as added tokens for a harmony vocabulary, so the reply runs to<|return|>or<|call|>; the reasoning splitter takes the analysis channel as reasoning and drops the final channel's header; the tool-call parser has aharmonyformat (a messageto=functions.NAME, its JSON arguments) with a call grammar after the address.
Proof: tiny-gptoss (3 blocks, sinks, a window of 4,
YaRN, 8 experts with biases and values past the clamp) within 1.6e-5 of
the float64 reference, batches equal to single tokens
(tests/model/test_model.c); the sinks on the prompt tiles,
the decode path and the scalar path, with a window and a quantized
cache, against a double reference
(tests/core/test_attention.c); the splitter on harmony text
cut at every length (tests/harness/test_think.c); the
harmony calls and their grammar
(tests/model/test_grammar.c). On Unsloth's
gpt-oss-20b-Q4_K_M.gguf (11.6 GB, CPU,
tests/model/test_infer_public.c): tokenization 5 / 5, 153 /
160 greedy choices on llama.cpp's tokens, logits within 3.96, two of
five 32-token continuations identical and the others parting where
llama.cpp's top two are within 0.11; 3.3 tokens/s. Through
oklm serve on that file: a question answered with its
reasoning as a thinking block and the answer as text
(/v1/messages), and, with a tool offered, a
tool_calls entry get_weather {"city":"Paris"}
with finish_reason tool_calls
(/v1/chat/completions, whose content keeps the
analysis markup). Catalogue: gpt-oss-20b (chat;
Apache-2.0). A device refuses the model at its first batch
(test_gptoss_device). Left: the devices, an unsplit view of
the channels, the speed, CyberPal 2.0.
L5. lfm2moe:
Tini-Cybersec-8B-A1B
lfm2moe (LFM2.5 8B-A1B) is the lfm2 stack
below (24 blocks, 18 of them short convolutions,
attention.head_count_kv 0 or 8, width 2048) whose
feed-forward becomes routed experts after
leading_dense_block_count (2) dense blocks. Audited against
llama.cpp's build_lfm_moe_ffn(): the router is DeepSeek
V3's form without groups or shared experts, which OKLM's MoE path
already computes: ffn_gate_inp times the normalized input,
a sigmoid (expert_gating_func 2, required by the file), the
4 experts of 32 chosen on the probabilities plus
exp_probs_b and weighted by the probabilities without it,
the weights renormalized to sum 1 (llama.cpp passes norm_w
true for this name; no expert_weights_scale), SwiGLU
experts of expert_feed_forward_length 1792 without biases,
summed in rank order. Of the three forms OKLM had, it is
glm4moe's routing (sigmoid, selection bias, renormalized)
without its scale and shared expert; qwen3moe and
granitemoe use a softmax and no bias, olmoe a
softmax without renormalization. The only code is the family's
renormalization default (src/io/hf_config.c) and the expert
tensors in model_bind_lfm2().
Proof: tiny-lfm2moe (8 experts, 2 a token, after two
dense blocks) within 1.8e-5 of the float64 reference on the CPU, OpenCL
and Vulkan, and through test_lfm2_states. On bartowski's
iselabvn_Tini-Cybersec-8B-A1B-Q4_K_M.gguf (5.17 GB, CPU,
tests/model/test_infer_public.c): tokenization 5 / 5, 156 /
160 greedy choices on llama.cpp's tokens, logits within 4.46, two of
five 32-token continuations identical and the others parting where
llama.cpp's top two are within 0.1. oklm chat answers (a
long <think> block, then the answer). Catalogue:
tini-cybersec-8b-a1b (security; the tune's card says
Apache-2.0, its base model is under the LFM Open License v1.0: asked,
with the base's licence).
L5. lfm2: Cyber-Prime
1.1 2.6B
Audited against llama.cpp's graph for lfm2 (inference.md): the stack mixes attention
blocks (the llama family's, with grouped keys and values, per-head Q/K
norms and the NeoX rotation) and short convolution blocks, told apart by
attention.head_count_kv, an array with 0 for a convolution
block. A convolution block projects its normalized input to a gate, a
scale and a value, runs gate x value through a causal depthwise
convolution of shortconv.l_cache (3) taps over the tokens,
multiplies by the scale and projects back; the SwiGLU feed-forward
follows either kind; the final norm is the tensor named
token_embd_norm. The loader binds the convolution tensors
in the linear block's fields and the block reuses the convolution op of
the Gated DeltaNet blocks (src/model/arch_lfm2.c,
gguf_lfm2() in src/io/hf_config.c).
The block's state is the last two gated inputs of each sequence. It
is a function of the last 2 x (convolution blocks) tokens and of the
cache, so OKLM rebuilds it whenever a sequence does not continue where
its states were left (a truncation, oklm_seq_copy(), a
cached prefix attached, a swap, a KV file): those tokens run again from
zero states with the key / value cache read and not written
(model_shortconv_sync(), model_replay()).
Before this the truncation of a hybrid stack left the states of the
sequence's former end in place.
Proof: tiny-lfm2 and tiny-lfm2moe (five
blocks, convolutions at 0, 1 and 3; the second with 8 experts after two
dense blocks) within 1.5e-5 of the float64 reference on the CPU, OpenCL
(all layers, one layer) and Vulkan; batches against single tokens,
truncations, copies between sequences, the paged cache with an attached
prefix and KV files read whole or in part all give the logits of the
tokens fed one by one, bit for bit on the CPU
(test_lfm2_states in tests/model/test_model.c;
with the rebuild disabled the same test is off by 17). On Liquid's
LFM2-350M-Q4_K_M.gguf (229 MB, CPU,
tests/model/test_infer_public.c): tokenization 5 / 5, 157 /
160 greedy choices on llama.cpp's tokens, logits within 1.2, and 24 / 24
choices of the float64 reference with logits within 1.3. On
mradermacher's Cyber-Prime-1.1-2.6B.Q4_K_M.gguf (1.67 GB,
CPU): tokenization 5 / 5, 154 / 160 greedy choices on llama.cpp's
tokens, logits within 2.73, two of five 32-token continuations identical
and the others parting where llama.cpp's top two are within 0.2; on a
chat turn opened on <think>, 31 / 32 choices and
logits within 1.26. Its template renders (85 prompt tokens with a tool,
llama.cpp's count) and oklm chat answers. Catalogue:
cyber-prime-1.1-2.6b (security; the LFM Open License v1.0,
asked). Left: the template's tool calls, the cache rows reserved for the
convolution blocks, the published files on OpenCL and Vulkan.
L1. Vision: LLaVA 1.5
oklm describe and oklm serve --mmproj run
LLaVA 1.5 (multimodal.md); the
catalogue offers llava-1.5-7b (the Q4_K model and the
projector of mys/ggml_llava-v1.5-7b, both run here; Llama 2
licence, asked).
The encoder's differences from SigLIP: a class token (the first row and position), patches without a bias and positions held in F16, a norm before the blocks, quick-GELU, no final norm, every block of the file (23 of CLIP's 24), the class token's row dropped; the projector
mm.0, exact GELU,mm.2; a file withoutclip.projector_typethat has a LLaVA projector ismlp; the picture fitted in 336 x 336 with a bicubic filter and CLIP's mean colour around it; Vicuna's turns when the file has no chat template.The audit's finding: the pinned llama.cpp does not compute this encoder. It puts the class token after the patches (each patch then takes its neighbour's position), runs 22 blocks and drops the first patch instead of the class token; and it refuses the published file, which names no projector. OKLM follows the model's definition.
Proof, on the published files (CPU):
Check Result the 336 x 336 pixels of two pictures llama.cpp's (on a copy of the file with the key added), byte for byte the 576 embeddings against a float64 numpy pass of the model's definition ( tools/vision_ref.py)relative difference 5.4e-6 and 6.9e-6 llama.cpp's embeddings against the same pass written its way ( --as-llama-cpp)6e-3 and 1.5e-2: the three differences are all of it llama.cpp's embeddings against the model's definition 0.99 and 0.79 the prompt's rows for "Describe the picture in detail." llama.cpp's 595 (576 of the image) llama.cpp's language model given OKLM's embeddings, its 48 greedy tokens fed one by one 47 choices equal, logits within 0.38; the two part at the fifth token, llama.cpp's two best 0.007 apart oklm describeandoklm serve --mmproj, 64 tokensthe same text; the headline read as "Men Walk on Moon" (llama.cpp with its own encoder: "Men Walked on Moon") the dashboard's scripted check with LLaVA 1.5 a pasted picture sent and answered Not reached: an answer identical to llama.cpp's own over 30 tokens; it cannot be while the encoders differ, and with the same embeddings the first near tie comes at the fifth token.
Tests: a tiny file in the layout of the published one (
tools/gen_vision_fixtures.py: embeddings from numpy, the resized picture from Pillow) intests/mm/test_vision.c, with the Vicuna turns; the published files intests/mm/test_mm_public.c.Left: LLaVA 1.6 and the rest, see L1 in the roadmap.
L1. Vision: Gemma 3's projector
oklm describe and oklm serve --mmproj run
Gemma 3's vision (multimodal.md);
the catalogue offers gemma3-4b (Gemma-3-4B-it Q4_K_M and
its projector, both run here).
Audited against llama.cpp's
tools/mtmdat the pinned revision: the picture fitted in 896 x 896 with a bilinear filter and black borders (llama.cpp's choice; the model's processor stretches it), every block and the final norm of the SigLIP encoder, a 4 x 4 mean over the 64 x 64 patches, an RMS norm, a projection stored with its sides exchanged;<start_of_image>and<end_of_image>around the 256 embeddings with no new line; and the image's rows evaluated attending to one another, which llama.cpp does for this projector and the audit of the encoder alone would have missed.Added for it: Pillow's fixed-point resampling in
oklm_image_resize()(the bytes llama.cpp gets) andoklm_image_fit(); themutualoption of the attention operation on the CPU (each query sees the keys up to the last query's),oklm_eval_embd_mutual()andoklm_eval_batch_mutual(), on the contiguous and the paged cache; the server's scheduler gives such an item in one piece.Proof, on the published files (CPU):
Check Result the 896 x 896 pixels of two pictures llama.cpp's, byte for byte the 256 embeddings against a float64 numpy pass ( tools/vision_ref.py)relative difference 8.9e-6 and 1.6e-5 the same against llama.cpp 3.2e-3 and 6.9e-3, the distance llama.cpp itself is from the float64 pass (it rounds activations to F16; numpy rounding them so moves 2.6e-3) the prompt's rows for a question about llama.cpp's test image llama.cpp's 274 (256 of the image) llama.cpp's 48 greedy tokens fed one by one 47 choices equal, the first 23 in a row, logits within 3.64; llama.cpp's two best 0.32 apart where they part oklm describeandoklm serve --mmproj, 64 tokensthe same text, with the headline and subheading llama.cpp writes The target was the per-tile figure SmolVLM had against llama.cpp (5e-4 of the largest value); against llama.cpp Gemma 3's encoder is at 7e-3 of it for the first picture and 2.6e-2 for the second, and the float64 pass shows on which side the rounding is. SmolVLM's own tile, recorded again with the new resampling, is at 9e-4.
Tests:
tests/mm/test_vision.con a tiny projector file oftools/gen_vision_fixtures.py(embeddings from numpy, the resized picture from Pillow);tests/core/test_attention.c(the option against the double reference, every thread count and kernel level);tests/model/test_model.con the cases oftools/gen_mutual_fixture.py(a numpy pass of the gemma3 architecture that first reproducestools/infer_ref.py's logits);tests/server/test_server.c(the server's answer is the one computed directly, on both caches, and differs from the causal one);tests/mm/test_mm_public.con the published files, withtools/mtmd_ref.cpp --imageand--answeras the reference.Left: see L1 in the roadmap.
The chats: speech to text in the chat window (C7's window part)
The chat window records the microphone and transcribes recordings (gui.md):
- Record, beside Attach, and
/listen [stop|send](written for the window on its own server, on a remote one and attached to an agent): Allegro's audio recorder at 16 kHz mono 16-bit, or the device's rate and channels mixed and resampled when the recording ends; a meter with the time and the level in the place of the path's field; ended by the button, Enter or the pause of--listen-silence(off by default), cancelled by Escape. The text lands in the message box after what it held and is not sent by itself (/listen send,--listen-send). The recording is in memory only. - The recorder's declarations. Allegro 5.2.11 exports
the recorder and its MinGW import library has it, but
allegro_audio.hdeclares it underALLEGRO_UNSTABLEonly, which ties a program to one exact version of the library: the five functions, the event's type (515) and its layout are declared intool_chat.c, as the drop events' display flag is ingui.h. A stopped recorder is destroyed two timer events later, since its last events in the queue point into its fragments. - Who transcribes. The window's own server is started
with
--transcribe-model NAME|DIR, else with the smallest installed speech recognizer of the catalogue (--no-transcribe: none), and transcribes by its route. A remote server does when/api/statssays"transcribe":true; otherwise the window says it has no transcription model and sends the sound nowhere. Attached to an agent, the window loads the model with the library at the first use. - Transcribe on an attached recording's box (by
itself with
--transcribe-attached): the text underTranscript of NAME:in the message; the box leaves the row when the model hears nothing, and stays when it hears. - What does not get lost. A recording whose
transcription fails joins the row of files as
recording-N.wav, in memory, with its Transcribe button. A recording ends at what the server'supload_maxholds (17 minutes for 32 MiB), 30 minutes at most. Silence (nothing above -55 dB) and a recording under a quarter of a second are not transcribed. - The client has
oklm_client_post_bytes(): a body with its length (the requests took a C string).
Proof, on the CPU:
tests/client/test_gui_listen.c, 171 checks: the meter, a recording filled by fragments of any size, the pause, the limit, 48 and 44.1 kHz brought to 16 kHz (a tone compared with itself, under 0.01 of error), 16-bit frames mixed, the WAV bytes read back by the library, the request received whole by a socket (path, headers, a body full of zero bytes), the answers, who transcribes for each link, the message.- Under Xvfb with
--mic-file models/samples/jfk.wav(the file fed in real time as the microphone's fragments) and Whisper base, the status line and the standard error readAnd so my fellow Americans, ask not what your country can do for you, ask what you can do for your country., whatoklm transcribeprints for the file, in each case: the window's own server (11.0 s of sound in 2.0 to 4.0 s), a remoteoklm serve --transcribe-modelon a free port (2.0 s; 7.7 s while other work held the CPU), an agent started with--listen(4.5 s, the model's loading included), an attachedjfk.wavwith a click on Transcribe and with--transcribe-attached, and/listen send(the message sent and answered). A remote server without the option:/listenanswers that it has no transcription model.--listen-silence 0.3ends the recording at the first pause (And so my fellow Americans, 3.2 s). Escape while recording keeps nothing, Escape while transcribing leaves the message as it was, Enter ends a recording: driven withxdotool. - The real recorder: with the sound server reachable, three seconds
were recorded from the machine's capture device (a USB codec) and the
time advanced; nothing was plugged into it, so the level stayed at zero
and the window answered
Only silence came in the 3.0 s recorded. With no sound server reachable:No microphone: this machine has no sound device the window can open, no crash, no wait. - The cost of the model at start, SmolLM2-135M as the chat model: 1002
MiB of peak memory against 749 with
--no-transcribe, 2.97 s against 2.86 s to the first screenshot; 1285 MiB after one transcription. make gui-testrecordstests/fixtures/mm/tone.oggwith the tiny Whisper fixture and looks for the transcription's line; the Windows build compiles (oklm-gui.exe, the test).
Left: see C7 in roadmap.md.
The chats: speech to text in the dashboard (C7's dashboard part)
The dashboard's chat records the microphone and transcribes
recordings on a server started with --transcribe-model (server.md):
- Record and
/listen [stop|send]: the browser's microphone (getUserMedia, given on a secure origin only: loopback or HTTPS), a row with the level, the time and the longest recording (the server's 30 minutes or whatupload_maxholds), ended by the button, the command or Enter, dropped by Escape. The text lands in the message box after what it held, and is not sent by itself (/listen sendsends it). - What is posted is a 16-bit PCM WAV file at 16 kHz
mono, built in the page: the samples come from a
ScriptProcessorNode(the page's policy loads no worklet module; the policy is unchanged) and are brought to 16 kHz by a windowed sinc. Chrome's own recorder was tried first: its WebM/Opus and MP4/Opus recordings of the sample are answered 400 by the route (the decoder reads no WebM), and it writes neither Ogg nor WAV. - Transcribe on the row of a dropped, pasted or chosen sound: the file posted as it is, its text in the message box, the sound then not sent. A model that hears nothing does not get the sound; a server that neither hears nor transcribes refuses it, naming both options.
- Refusals: no transcription model (the button
disabled,
/listensaying how to start the server), an origin that is not secure (the reason in the button's title, in/listen's answer and in the list of commands), the microphone refused, no microphone; a recording whose transcription fails is kept with the files. /listentakesstopandsendin the commands' table, and is written for the page; the page tells thelistenfeature fromGET /api/commandswhen the server names it there, else fromtranscribein/api/stats./speakstays "not available here yet".- The page grew from 52,426 to 60,821 bytes.
Proof: tools/check_dashboard.sh in a headless Chrome 157
whose microphone is whisper.cpp's jfk.wav
(--use-file-for-fake-audio-capture), against SmolLM2-135M
with Whisper base on the CPU, 192 checks in all, 46 of them on that
server: the Record button then a second press gives "And so my fellow
American, ask not what your country can do for you, ask what you can do
for your country." in the message box, nothing sent, the WAV posted read
back (PCM, mono, 16 kHz, 16-bit, the samples kept), the tracks ended and
the context closed; /listen ended by Enter after
"Dictated:"; Escape (nothing posted) and Escape on the list of commands
(the recording goes on); /listen send answered by the
model; a recording stopped at a limit of 2 s; the microphone refused,
missing, and given after the recording was dropped;
isSecureContext false, and a page opened as
http://plain.test (the loopback address under another name:
no mediaDevices there); the sample dropped, refused as a
sound, then transcribed, edited and sent as text. The resampler outside
the browser, on the sample brought to 48, 44.1, 32 and 22.05 kHz by
ffmpeg: 70.6 to 70.8 dB of signal over the error against the 16 kHz
original, and the reference's text from each. In Chrome the recording
differs from the file by Chrome's voice processing: with it turned off
(an experiment, not the page's setting) each half second of the
recording correlates at 0.95 to 0.998 with the sample but one that holds
a 10 ms gap of the fake microphone, and the text is
oklm transcribe's to the letter ("Americans"); with the
defaults the page uses, one "s" is lost.
test_dashboard_chat and test_chat_commands
hold the page's elements, the table and the shared lines (137).
The chats: the commands in the chat window (C2's window part) and the models that are not chat models (C6)
A message that starts with / in oklm-gui's
chat window is a command (gui.md), read
with the table of C1 for what the window is linked to: its own server
(window), a server at an address (remote), an
agent (agent). What the link can do comes from the window
for its own server (its context, its models, an index, a projector when
one is open), from GET /api/commands for a remote one, and
is nothing for an agent. Written in the window: /help,
/reset, /system (a state the window did not
have: sent first in each request, its start shown in the side panel),
/model and /models,
/context [N [force]] (the slider's and Apply's code, with
their refusals and the red line), /attach and
/detach, /rag INDEX|off, /save
and /export (a small writer: the session export takes an
agent's session, which a plain chat has not), /copy,
/retry, /rewind, /stop,
/stats, /temp and /max-tokens
(their sliders move), /top-k, /top-p,
/min-p, /seed and /thinking (kept
and sent in each request), /connect [URL], and
/decide through POST /v1/decide. A command's
answer is a note in the conversation, dimmed under the command's name,
that no request, copy or export carries. A list above the message shows
the matching commands with their help while one is typed, then the
values of its argument (on|off, markdown|json,
the installed chat models, the commands for /help): Up and
Down, Tab or Enter, Escape, a click. Attached to an agent, a line the
window does not take (unknown, or of the other chats) is sent to the
agent as a message.
Use this model (and /model NAME,
-m NAME, the model manager's Chat) takes
the catalogue's vision and audio-understanding models, served with their
projector. For an embedding or reranking model the button reads
Open in Retrieval and starts
oklm-gui --tool rag; for Whisper it says where it is used
(oklm transcribe,
oklm serve --transcribe-model). Whether a link takes images
and sounds is read in its figures (vision,
audio of /api/stats); the request made to be
refused stays for a server without these members. The window no longer
replaces a document's token count by an estimate when it looks too
small: with the server's fix of the same day SmolVLM counts 409 tokens
for a text of 1800 bytes, where it counted the template alone.
Proof (CPU, under Xvfb, driven with xdotool, the screenshots read,
2026-11-13). On its own server, SmolLM2-135M and Qwen2.5-0.5B:
/sys, Tab, a system prompt, and the model answered in
French to "You answer in French."; /temp 0.3,
/top-k 20, /seed 7 and
/thinking off shown in the side panel; /retry
wrote the reply again; /rewind put the message back in the
box; /copy 2; /context 16384 applied with the
red line of a size above the trained 8192; /max-tokens 8000
widened its slider; /decide Paris | London | Rome gave
74.67 %, 10.08 %, 15.25 % and the decision Paris; /save
wrote a file that oklm chat --continue --chat-file
continued (continued: 2 messages); /export and
/export json wrote through the system's file dialog.
/model smolvlm-256m started the window again on
SmolVLM-256M with its projector, which described the attached drawing
("A red circle is on the flat blue background."), refused a WAV
(this model's projector encodes images, not sounds, read
from the figures), and SmolLM2 refused the picture the same way.
/rag docs.gguf started a Qwen3-0.6B window again on an
index of three pages, which answered from a source;
/rag off and /rag docs.gguf switched it; a
file that is no index was refused. On a remote oklm serve
of SmolVLM-256M (a free port): /model and /rag
answered that they cannot be done there, /context 2048
resized the server, /attach of a Markdown file was counted
by the server (112 tokens, no estimate) and summarized,
/connect URL from another window kept the conversation,
picture included. Attached to oklm agent --listen:
/help listed the window's seven commands,
/save wrote the conversation, and /status
reached the agent as a message (read back in its events). In the Models
window: SmolVLM selected, Use this model enabled;
bge-small, Open in Retrieval opened the retrieval
manager; Whisper, the sentence.
tests/client/test_gui_commands.c (122 checks: the place and
the features per link, the list's rows, the help, the two exports and
the saved file parsed back, the request of /decide) and
tests/client/test_gui_attach.c (100 checks, the figures
read) cover what needs no window; make gui-test sends
/help. The Windows build compiles; it was not run. Not
done: /speak, /listen; a saved conversation
read back by the window; the retrieval manager opened with the model
selected; what an agent does of a command sent to it as a message is the
agent's.
The chats: speech synthesis with OuteTTS (C8's model)
oklm speak turns a text into a 24 kHz WAV file with
OuteTTS 0.2 and WavTokenizer's decoder (speech.md), on the CPU.
- Audit (the page's first half): the two GGUF files llama.cpp's tts example names, their metadata and tensors; the text's normalization, the prompt (a voice's words and audio codes, then the text's words), which tokens are codes, the sampling, the end; the vocoder block by block with its tensor names, and the inverse transform. Kokoro, Piper, Soprano and OuteTTS 0.3 / 1.0: what each would take.
- The language model runs through the
qwen2runtime unchanged. The vocoder is new (src/mm/vocoder.c): the published file read where it is mapped, convolutions as products over unfolded frames, group and layer normalizations, one attention block, ConvNeXt blocks, and a mixed-radix inverse transform.src/mm/speech.cnormalizes, cuts a long text into passes of whole sentences, builds the prompt, generates the codes (with llama.cpp's guide tokens) and writes WAV. - The reference: the pinned llama.cpp's tts tool no
longer runs OuteTTS, so llama.cpp at
1c3c9674(the last commit that does) was built for the CPU in a scratch directory. The prompt's token ids are identical on three sentences (906, 906, 913). Fed llama.cpp's tokens, OKLM's most likely token is llama.cpp's at 258 of 262, 252 of 254 and 289 of 300 steps, the others within 0.033 logit. On llama.cpp's codes the vocoder's rows are within 131.2 dB of a numpy double-precision implementation's (largest difference 1.1e-4) and its samples within 124.7 dB; against llama.cpp, 57.4 dB on the rows (71.1 dB when built with ggml's approximate GELU, which is what llama.cpp is to numpy) and 40.9 dB on the samples, the difference between the true inverse real transform and llama.cpp's one-sided sum (41.5 dB alone). - Heard by Whisper: six sentences spoken with the
defaults (temperature 1, top-k 4, guide tokens) and transcribed by
oklm transcribe: 53 of 57 words right with Whisper base (27 of 28 on the three sentences compared with llama.cpp), 50 of 57 with Whisper small, two of the differences being "seashells" for "sea shells". The model card's settings (temperature 0.1, repeat penalty 1.1) gave 42 of 57. - Library (
include/oklm/mm/speech.h):oklm_speech_open(),oklm_speech_synthesize(),oklm_speech_synthesize_chunks()(a callback per pass),oklm_speech_wav(), the vocoder alone. - Catalogue: a
speechcategory ("Speech synthesis" inoklm models, the model manager and the chat window's list) withoutetts-0.2-500m, its two files under the rolesmodelandvocoder, its licence (CC-BY-NC-4.0) asked. - Tests:
tests/mm/test_speech.cwithtools/gen_speech_fixtures.py(a tiny vocoder in the published layout, its output and the inverse transform against numpy, the normalization and the prompts against the script's own); the published files checked when installed. - Left: the server's route, playback,
/speak; a device path; other languages and voices made here; speed (0.43 second of sound per second at best on this CPU, 0.12 to 0.23 on the shared machine: the language model writes about 90 tokens per second of speech).
L5. starcoder2:
StarCoder2-3B
Audited against llama.cpp's graph for starcoder2:
GPT-2's block (LayerNorm with biases before attention and before the
MLP, an MLP of two products with biases and GELU, a final LayerNorm, the
head tied to the embeddings) with three differences: separate
attn_q / attn_k / attn_v
projections with biases instead of the fused attn_qkv, keys
and values of head_count_kv heads (2 for 24 query heads in
the 3B), and the NeoX rotation of whole heads
(rope.freq_base 999 999) instead of a position table. Of
OKLM's forward passes GPT-2's is the nearest (Phi-2 already runs it with
rotary positions): it gains the separate projections
(split_qkv() in src/model/arch_gpt2.c, the
buffers and the attention kernel of grouped keys being the llama
family's), and the model loader binds the tensors
(bind_starcoder2() in src/model/model.c).
Proof: tiny-starcoder2 (4 query heads over 2) within 8e-5
of the float64 reference on the CPU, OpenCL (every layer, and one of
two) and Vulkan, batches equal to single tokens
(tests/model/test_model.c). On second-state's
starcoder2-3b-Q4_K_M.gguf (1.85 GB, CPU,
tests/model/test_infer_public.c after
tools/gen_llama_ref.sh): tokenization 5 / 5, 159 / 160
greedy choices on llama.cpp's tokens, logits within 1.64, four of five
32-token continuations identical and the fifth parting after 134 of 160
tokens where llama.cpp's top two are 0.23 apart. 100 greedy tokens:
identical on def quicksort(arr):; on a C prompt ending in
{, which llama.cpp tokenizes as two tokens (the file names
no pre-tokenizer, see L4 above), 19 identical and then a choice where
llama.cpp's two candidates are 0.43 apart; over the 200 steps of both
recordings 199 choices equal, logits within 2.9, KL 0.0013 on average.
Catalogue: starcoder2-3b (code; bigcode-openrail-m, asked).
Left: the published file on OpenCL and Vulkan, the larger files, a
fill-in-the-middle option.
L5. mistral3:
Devstral Small 2 24B
The headers of Devstral-Small-2-24B-Instruct-2512
(mistral3) and Precog-24B (llama) were read
without their weights (the first 32 MiB by an HTTP range, then
truncate -s to the published size: a sparse file
oklm info -t reads). Same 363 tensors, names, shapes and
types. The metadata differs by: YaRN
(rope.scaling.type yarn, factor 48 over an original context
of 8 192, yarn_beta_fast 32, yarn_beta_slow 1,
yarn_log_multiplier 1),
attention.temperature_scale 0.1,
rope.freq_base 1e8, a trained context of 393 216, token
scores stored as integers, and Unsloth's chat template. llama.cpp's
mistral3 (the pinned reference, v0.8.0, has it) is its
llama graph with two additions: the queries multiplied after the
rotation by 1 + scale x ln(floor(position / original context) + 1), and
a YaRN magnitude correction divided by the log multiplier (so 1 here); a
time-conditioned norm exists for realtime speech models only. OKLM now
reads the name (src/io/hf_config.c,
src/io/model_check.c), runs it through the llama forward
pass with the temperature (attn_temperature() in
src/model/arch_llama.c) and refuses the speech variant and
YaRN boundaries other than 32 / 1. Proof: tiny-mistral3
(temperature steps at positions 12 and 24, batches crossing them) within
1e-6 of the float64 reference on the CPU, OpenCL and Vulkan
(tests/model/test_model.c), llama.cpp itself within 9e-4 of
that reference (F16 cache). On the published Q4_K_M file (14.3 GB, CPU,
tests/model/test_infer_public.c after
tools/gen_llama_ref.sh): tokenization 5 / 5, 158 / 160
greedy choices on llama.cpp's tokens, logits within 4.05, four of five
32-token continuations identical and the fifth parting after 130 of 160
tokens where llama.cpp's top two are 0.08 apart; oklm chat
renders the template and answers a code question. Catalogue:
devstral-small-2-24b (code; apache-2.0 by Mistral's card
and the file's own general.license, while Unsloth's card
says "other"). Left: beyond 8 192 tokens on the published file, OpenCL
and Vulkan on it, the HF conversion.
L4. DeepSeek-Coder 1.3B
TheBloke's deepseek-coder-1.3b-instruct.Q4_K_M.gguf
continued a prompt with white space. Compared with llama.cpp on the same
file: the token ids were already equal (no BOS on a plain prompt for
either: the file has no add_bos_token); the logits after
the prompt were not. The file, written in 2023, states its linear RoPE
scaling as llama.rope.scale_linear = 4 and names no scaling
type; llama.cpp reads that key when rope.scaling.factor is
absent and scales linearly by default, OKLM read neither and turned the
positions four times too fast (src/io/hf_config.c). The
same file has no chat template, so a chat was rendered in ChatML, which
the model was not trained on: a file without a template whose BOS and
EOS are DeepSeek-Coder instruct's now gets the template the model's
tokenizer_config.json publishes
(src/tok/tok_gguf.c). Proof, on the CPU:
tiny-rope-linear (a fixture with the old key) within 1e-4
of the float64 reference (tests/model/test_model.c); the
template rendered as jinja2 renders it
(tests/tok/test_tok_units.c); on the published file
(tests/model/test_infer_public.c, after
tools/gen_llama_ref.sh) tokenization 5 / 5, 159 / 160
greedy choices on llama.cpp's tokens, logits within 7.26 (a shift common
to a step's logits, between -2.5 and 6.4, on logits that span -76 to 87;
1.15 at most once removed, 0.41 on the log-probabilities of llama.cpp's
ten best tokens, KL 0.0014 on average), the free continuations equal for
143 / 160 tokens and parting where llama.cpp's top two are 0.16 apart;
on a prompt's first step OKLM and llama.cpp are equally far from the
float64 reference (0.5 to 1.8 against 0.6 to 1.3). 100 greedy tokens
identical to llama.cpp's on def quicksort(arr): and on a
chat turn in the model's template; oklm chat answers and
stops at <|EOT|>. Catalogue:
deepseek-coder-1.3b (code; DeepSeek's licence restricts
uses, so it is asked). Left: the 6.7B file, OpenCL and Vulkan.
The same audit, continued with StarCoder2: neither file names its
pre-tokenizer, OKLM split both with GPT-2's pattern and llama.cpp with
its default steps. Against Hugging Face's tokenizers on
DeepSeek-Coder's tokenizer.json, OKLM's ids differed on 21
of 400 lines of code ( = 0 as =, a space,
0). A file without the key now gets its family's
pre-tokenizer where the file tells the family
(deepseek-coder by the vocabulary, starcoder
by the architecture): 400 of 400 lines identical for both models; the
comparison with llama.cpp above is unchanged (5 / 5 prompts tokenize
alike, none has a space before punctuation) (tokenizer.md,
tests/tok/test_tok_units.c).
L4. DeepSeek-Coder 6.7B
Done 2026-11-13. TheBloke's
deepseek-coder-6.7b-instruct.Q4_K_M.gguf (4.08 GB) has the
1.3B's keys (llama.rope.scale_linear = 4, no chat template)
and needed no change to the runtime. Against the pinned llama.cpp on the
CPU: 5 of 5 prompts tokenized alike, 159 of 160 greedy choices on
llama.cpp's tokens, logits within 2.81, four of five continuations
identical and the fifth parting where llama.cpp's own gap is 0.08; 100
greedy tokens identical on a code prompt and on a chat turn;
oklm chat stops at <|EOT|>; 5.8 tokens/s
on six cores of a shared machine. In the catalogue as
deepseek-coder-6.7b; the file was run and removed, its
reference is recorded by tools/gen_llama_ref.sh. The same
day Qwen3.8-27B entered the catalogue as qwen3.8-27b,
marked run from the record of its run (160 of 160, the same SHA-256),
and the tunes Cyber-Ornith 1.5 9B and BugTraceAI-CORE-Ultra 27B as not
run.
The chats: files in the chat window (C3, C4's window half)
Files go with a message in oklm-gui's chat window (gui.md): dropped on the window
(Allegro's drop events), chosen with Attach (the
system's file dialog) or typed as a path, or named by
--attach FILE (repeatable); a row above the message lists
them (an image's thumbnail, the name, the kind, the size, a document's
tokens), each removable; the conversation then shows an image in small
and every file as a line, never a document's text. Images and sounds go
in the request as the server takes them (image_url,
input_audio); the window's own server opens the projector
the catalogue has for the model, or --mmproj FILE. Whether
a link takes them is asked of the server (a request holding an empty
image, refused at once for one reason or the other), and the window says
why before sending: its model has no projector, a remote server has
none, the projector is of the other kind, an agent takes text only. A
PDF, DOCX, HTML, JSON or text file is read by the window with the
library's reader, for its own server or a remote one, and goes under its
name in the form of --doc; its tokens are counted by the
server, and when the documents are longer than the context leaves room
for, the window says so and offers to send the beginning that fits, or
nothing.
Proof (CPU, under Xvfb, 2026-11-13). SmolVLM-256M (Q8_0) described a
drawing of a red disc ("A red circle on a pale blue background")
attached by --attach, by the typed path and Enter, with the
catalogue's projector (-m smolvlm-256m) and with
--mmproj, and through oklm serve --mmproj on
another port; a PNG and a PDF dragged from a GTK window were both
attached by a real XDND drop. Ultravox (Llama-3.2-1B) wrote down the JFK
sample attached as a WAV. SmolLM2-135M answered from a PDF made by
LibreOffice (179 tokens, "The ferry leaves at 9:15 am") on its own
server and on a remote one; a 26183-token text against a context of
4096: 3008 tokens fit, Send the beginning sent 15934 of
140037 bytes, a prompt of 2953 tokens. The four refusals were read on
screen (own model without projector, remote server without, a sound to a
vision projector, a binary file).
tests/client/test_gui_attach.c (83 checks) covers what
needs no window; make gui-test attaches a PDF fixture. Not
done: an image pasted from the clipboard (nilorea's clipboard carries
text only); drops on Windows compiled, not run.
The chats: the server's routes for documents and speech (C4, C7)
POST /api/ingest(server.md): a document sent as the body (PDF, DOCX, HTML, JSON, text) answered as the textoklm ingestgives, with its characters, its tokens by the served model's tokenizer, and cuts bymax_charsandmax_tokens; bodies of at most--upload-max(32 MiB). No path of the client is opened; a DOCX alone passes through a private temporary file.POST /v1/audio/transcriptions(server.md): on a server started with--transcribe-model(a Whisper checkpoint, orwhisper-base/whisper-smallby name), OpenAI's multipart form or the sound as the body;json,textandverbose_json; one request at a time beside the chat scheduler, four waiting, 503 beyond; the model listed by/v1/models./api/statssays what a server takes (vision,audio,transcribe,ingest,upload_max) and counts the documents and the sounds (ingest_stats,transcribe_stats).- Met on the way: every compressed sound (Ogg Vorbis, FLAC) crashed
the decoder, in
oklm transcribetoo (Allegro's audio addon was not installed); fixed, with a test.
Proof: tests/server/test_server.c
(test_ingest_route, test_transcriptions on a
tiny Whisper checkpoint the test writes,
test_transcriptions_real on Whisper base), six runs in a
row without a failure. By hand with curl on Whisper base (CPU):
whisper.cpp's jfk.wav as WAV, Ogg Vorbis and FLAC, by both
forms, answers the sentence oklm transcribe prints, 1.3 s a
request; simple.pdf and a DOCX answer the text of
oklm ingest; 40 MB answers 413; eight transcriptions at
once: five answered, three 503.
The chats: the server's route for speech synthesis (C8's route)
POST /v1/audio/speech(server.md) on a server started with--speak-model(outetts-0.2-500mby name, whose vocoder is found by itself, or a GGUF file with--speak-vocoder;--speak-voice,--speak-model-id,--speak-backend cpu,--speak-max): OpenAI's JSON request (input,voice,response_format,speed,model) and the synthesizer's own options (temperature,top_k,seed,guide,words).wav(the default) is the whole file with its lengths;pcm, andwavwith"stream": true(a header of unknown lengths), are sent pass by pass while the text is spoken. The voice is the server's (default, or its name): no file is opened for a request.speedis read and not applied.- One synthesis at a time in a thread of its own beside the chat scheduler, four waiting, 503 beyond; a client that leaves, or the server stopping, ends the synthesis between two tokens.
/v1/modelslists the model with"task": "speak";/api/statshasspeakandspeak_stats;GET /api/commandstellsspeakandlistenamong a server's features when it has the models, so/speakand/listenshow as available to the clients.- The synthesis' stop flag is read atomically
(
src/mm/speech.c), so another thread may set it.
Proof: tests/server/test_server.c
(test_speech_route on a tiny speech model the test writes
and the tiny vocoder: the three answer forms hold the bytes of
oklm_speech_synthesize() called directly; every refusal;
twelve long texts at once: five spoken, seven 503; a client that hangs
up; a server stopped while it speaks;
test_speech_route_real on OuteTTS 0.2 and Whisper base:
"The weather is nice today." spoken, heard "The weather is nice today"),
five runs in a row without a failure. By hand with curl on OuteTTS 0.2
500M Q8_0 (CPU, shared with other work): "The quick brown fox jumps over
the lazy dog." is 2.76 s of sound in 7.7 s, the WAV file byte for byte
the one oklm speak writes, the pcm answer the
same samples; oklm transcribe -m models/whisper-base hears
"The quick brown fox jumps over the lazy dobs." in both. Three sentences
(23 words): one pass by default, the first byte with the last (17 to 32
s); with "words": 8, three passes, the first byte after 3.7
to 12.7 s of 18.8 to 26.0 s. Chrome 157 (headless) plays the whole WAV
and the streamed one from a blob and decodes both with
decodeAudioData(), and reads pcm in as many
pieces as passes. Not done: Firefox and Safari not tried; the Windows
build compiled, not run.
4c. The chats: the commands in the terminal (C2)
oklm chat runs every command of the terminal's column of
the table (inference.md), in line
mode and in the full-screen UI: /model [NAME] and
/models, /context [N [force]],
/attach FILE and /detach [N],
/rag INDEX|off, /save FILE,
/export [markdown|json], /copy [N],
/retry, /rewind [N], /stats,
/temp, /top-k, /top-p,
/min-p, /seed, /max-tokens and
/thinking [on|off]; /system without a text
shows the system prompt and /system - removes it. A command
that takes a value shows the current one without it.
/model without a name is a selection (inference.md): the
catalogue's models under their groups' names, each with its size, its
state (current, installed, to download, with or without a licence to
agree to) and its description; the models no chat serves dimmed, with
what they are for. A panel in the full-screen UI (arrows, typing
searches, Enter, Esc), a numbered list and a question in line mode. A
model that is not installed is downloaded only once the selection asked,
naming its size, its licence and the address of the licence's text;
Ctrl-C stops the download and what arrived is kept. The model chosen is
loaded in place of the one in use, the conversation kept and read again
by it; a conversation longer than the new model's context keeps the
model in use, and nothing is dropped. oklm agent's
/model is the same selection, with the model files found
elsewhere listed after the catalogue; on a server (--url)
it offers the models the server lists and says so when the model cannot
be changed from there.
Shared code: the rows of the list
(include/oklm/client/model_select.h, which the model
manager's list takes its groups from); the selection and the question
before a download (model_select(),
model_fetch() in apps/oklm.c) for the two
chats; the memory check and the messages of a resized context
(context_fits(), context_resized()) for
/context and the agent's /context-size; the
reply without its reasoning for the two /copy; the document
block of --doc for /attach. The full-screen
UI's list panel takes headings and dimmed entries
(oklm_tui_marks(), oklm_tui_step()). Tab
completes a command's argument in the full-screen UI:
on|off, markdown|json, a model's name, a file,
force.
One change for every chat in the table: /system takes no
text ([TEXT|-], it shows the prompt) where it was a usage
error.
Proof: tests/client/test_model_select.c (the rows, the
numbers, the states, the list as text);
tests/client/test_chat_commands.c (every command of the
terminal written there, their arguments);
tests/harness/test_tui.c (headings stepped over and hidden
by a search, a dimmed entry chosen);
tests/harness/test_cli_chat.c, which types the commands
into oklm chat on SmolLM2-135M on the CPU with a model
directory of its own, 123 checks: the values shown and set, a document
attached (the model then answers "The document is about the harbour of
Brest opening on the Atlantic") and an image refused, the list and the
selection's question answered no for a model with a licence and one
without (nothing fetched, no licence recorded), an index opened by
/rag (the reply then reads "The lighthouse of Eckmuhl is 65
metres high" from its passage) and kept when a file that is no index is
named, a conversation saved, exported, written again, gone back in and
continued from its file, a second model loaded in its place (2 messages
kept, its prompt evaluated whole: 0 cached), a conversation of 1300
tokens refused by a context of 256 and taken once
/context 2048 made room, and the agent's
/model without a terminal;
tools/check_chat_tui.sh, 40 checks on a pseudo-terminal: 27
of the 28 it had (two changed with the table; the terminal has no
command left that is not written), then the selection's panel with its
groups, a model not installed asked about with its size and its licence
and answered no, a search for a model no chat serves, Esc, Tab on five
kinds of argument, /copy through the terminal's clipboard
in line mode and in the full-screen UI (where a reply longer than a line
did not reach the terminal as one sequence, for the agent's
/copy too: fixed). By hand:
/model smollm2-135m from a chat on another file downloaded
its 138 MiB into a scratch model directory after y, a first
try stopped by Ctrl-C and resumed by the second, and the chat went on
with the 4 messages kept; with Qwen3-0.6B loaded in place of SmolLM2 the
reply named the word asked two turns before, and
/thinking off then on removed and brought back
its reasoning.
4c. The chats: one command table (C1)
One static table of the chat's commands
(include/oklm/client/chat_commands.h,
src/client/chat_commands.c): 30 commands, each with its
name, aliases, arguments, a line of help, the chats it belongs to (the
terminal, the window on its own server, the window on a remote server,
the dashboard, the window on an agent), those where it is written, and
what it needs (a context that can be resized, a retrieval index, a model
that can be changed, speech). The parser reads a line with the agent's
rules (//text, names without case, /think left
to the model) and says whether it is a message, a command with its
arguments split and checked (quotes, numbers, choices, a file's name
with blanks), or why it does not run: unknown with the nearest names, a
wrong usage, a command of other chats, not written here yet, something
missing. Completion of a prefix and of a line, /help and
/help NAME, the table as JSON and as the Markdown table of
inference.md.
oklm chat reads its lines through it:
/system, /reset, /decide,
/tool, /tui and /exit as before,
with /help, the aliases (/clear,
/quit), Tab completing a command's name in the full-screen
UI, and a line such as /modle answered
(unknown command /modle: did you mean /model, /models?)
where it was sent to the model. The commands of C2 are in the table and
answer not available here yet. One change of behaviour:
/system without a text is a usage error (it was sent to the
model as a message).
GET /api/commands gives the table with what the server
can do (server.md):
/context is not available with
--fixed-context, /rag only with
--rag INDEX, and /model never from a remote
chat.
Proof: tests/client/test_chat_commands.c (the table well
formed, the parser's cases, the arguments of every kind, completion, the
help, the JSON and the features read back, the documentation's table
compared with the module's); the route in
tests/server/test_server.c (401, 405, the fields, a fixed
context, a retrieval index); tools/check_chat_tui.sh on
SmolLM2-135M on the CPU, 28 checks: the 15 it had, then in line mode
/help, /help context, an unknown command, one
not written yet, one of other chats, wrong arguments, /tool
without tools, and --stats counting one turn sent for ten
lines typed (the //help one); in the full-screen UI,
/he Tab completed and run, /mo Tab giving
/model and Tab again listing
/model /models.
Architectures:
qwen35 on Qwen3.5-9B (I4, L5)
The published hybrid of Gated DeltaNet and full-attention blocks, on
the CPU (the layout,
block by block, compatibility.md).
oklm info answered "architecture 'qwen35' is not known"; it
now says loadable, oklm plan gives 8.06 GiB at 8192 tokens
(weights 6.32 GiB, the recurrent state 67 MiB), and the model is
qwen3.5-9b in the catalogue.
- The audit:
unsloth/Qwen3.5-9B-GGUF,Qwen3.5-9B-Q4_K_M.gguf(SHA-25603b74727...52b7e8), read beside llama.cpp at the pinned commit8642541b(src/models/qwen35.cpp). Theqwen3nextlayout OKLM had from its own fixture differs from the published one in nine ways (the table of the design note): separate projections, a decay per token, key heads repeated over the value heads, a scaled query, the conv weights' order, gated queries in the full-attention blocks, the name of the norm before the FFN.qwen3nextis unchanged;qwen35reads the published layout on the same two kernels, the recurrence gaining a decay per token (oklm_gated_delta_tok()). - Against llama.cpp on the published file
(
tests/model/test_qwen35_public.c,tools/gen_llama_ref.sh): 5 / 5 prompts tokenized alike; 159 / 160 greedy choices equal on its tokens; the best three tokens after "The capital of France is" the same (17.63 / 15.39 / 14.51 against 17.68 / 15.40 / 14.47); the logit difference 0.116 on average, 0.4 to 0.7 at most a step, 13.8 on single low logits at four steps of a sentence the model repeats (the same file moves its own logits by 0.121 on average and up to 10 there between an F16 and an F32 cache). 100 greedy tokens after plain text: 100 / 100 the same; after a chat turn: 26, then a token llama.cpp held by 0.44, 97 / 100 choices equal on its tokens. A prompt of 296 tokens in batches of 64: 16 / 16 choices, 1.72 at most. Two sequences in one pass: 32 / 32 choices each. A sequence through a file, a copy and a cut: the same next 8 tokens.oklm chatwith the file's template, thinking off: the 38 first tokens of llama.cpp's greedy answer to the same rendered prompt, then a choice it held by 0.17; a second turn and--template-vars '{"enable_thinking": true}'answer as expected. - Qwen3.8-27B
(
Qwen3.8-27B-UD-Q4_K_M.gguf, SHA-2561bce504d...9d7dc529; 64 blocks at width 5120, 24 heads, 48 value heads over 16 key heads, IQ4_XS and IQ4_NL among its types), loaded, compared and removed: 5 / 5 prompts tokenized alike, 160 / 160 greedy choices, 3.22 at most, the five continuations identical (test_infer_public);oklm plangives 21.80 GiB at 8192 tokens. Its multi-token prediction block, which the 9B file does not have, runs (oklm run --mtp: 20 of 45 drafts accepted, the text of the plain greedy run). - The fixture:
tiny-qwen35(tools/gen_model_fixtures.py), a random model of the published layout with 2 key heads over 4 value heads, whose reference is llama.cpp's: every logit within 2e-4 (test_modeltest_qwen35, with the batches, the sequences and the state's rules). - A recurrent state only moves forward: cutting a
hybrid sequence back (
oklm_seq_truncate(), which the chat, the agent and the server use to keep a shared prefix) left the linear blocks' state at the old end, forqwen3nexttoo. The context now keeps each sequence's token ids and evaluates the kept ones again before the sequence's next use; a copy of a sequence takes its state; the prefix cache and the swap are off for a hybrid model.
Left, on the roadmap under L5: the device backends (the context runs
on the CPU whatever is asked), the 27B's catalogue entry, the
multi-token prediction block against llama.cpp's own, the vision
projector, kept states for cheap cuts, the trainer. The context also
still reserves key and value rows for the linear blocks, which do not
use them (oklm plan counts them: 768 MiB of the 1 GiB at
8192 tokens).
4c. The chats: commands and files in the dashboard (C2, C5)
The dashboard's chat (src/server/dashboard.html) reads
GET /api/commands and takes a / line as a
command, with a parser that is the library's written again in
JavaScript: a message, a command with its arguments, or why it does not
run (unknown with the nearest names, of other chats, not written yet,
missing something with the server's reason, a wrong usage). Written for
the page, their rows of the table marked so: /help,
/reset, /system, /context (a dry
run first; what does not fit and the warning above the trained context
in the warning's colour), /attach, /detach,
/rag on|off, /save, /export,
/copy, /retry, /stop,
/stats, /temp, /top-k,
/top-p, /min-p, /seed,
/max-tokens, /thinking, /rewind,
/decide, /models, and /model NAME
among the adapters of a server that has some. Their answers are notes in
the conversation; the commands matching a / word being
typed are listed (arrows, Tab or Enter, Escape). Files are dropped,
pasted or chosen and wait in a row, each removable: an image or a sound
goes in the request when the server has the projector, a document is
read by POST /api/ingest, cut to half the context, and its
text goes in the message under its name; a file above
upload_max is refused before any upload; the conversation
shows small pictures and names, not the documents' text. A conversation
written by /save is opened again when dropped (server.md). The page grew
from 29,090 to 52,436 bytes; nothing was added to the server.
Proof: tests/server/dashboard_cases.tsv, 130 lines with
what each is, answered the same by the library's parser
(test_dashboard_chat in
tests/server/test_server.c, which also compares the page's
table of how each command reads its arguments with the library's) and by
the page's parser in a headless Chrome 157
(tools/check_dashboard.sh, Node driving the browser over
the DevTools protocol, nothing installed). The same script then drives
the page on servers it starts on the CPU, 144 checks in all: on
SmolLM2-135M every command above with the requests' bodies read back
(the settings, the system prompt, the documents' text under their
names), the list under the keyboard, the file chooser opened by
/attach and by the button, a PDF, a DOCX, HTML, JSON and
text dropped, a document cut, a 1.7 MB file refused by a server started
with --upload-max 1 before any request, an image and a
sound refused without a projector, /stop ending a reply
with the server back to no active request, the downloads of
/save and /export read, the clipboard after
/copy; on a server with an adapter trained there in two
steps, an index of one file and --fixed-context:
/model brest naming the adapter in the requests,
/rag on answered from the file with its source noted,
/context refused with the server's reason; on a server with
--api-keys: no command before the key, then the commands, a
reply and a document read with it; on SmolVLM-256M with its projector: a
pasted PNG shown as a small picture, sent as a data URL before the text,
encoded once by the server and answered ("Bar chart." for a bar
chart).
Left: /listen, /speak and the microphone
(C7, C8); a sound for a server that only transcribes; a sound sent to an
audio projector was not driven (the path is the image's, with
input_audio); the page was driven in Chrome only (Firefox
is not driven by the script).
Graphical tools: the model manager as a tool
oklm-gui --tool models (gui.md), menu entry "OKLM Models",
a row of the launcher: the catalogue by category with what is installed
(a box keeps the installed ones only), the selected model's description,
size, context and licence, Download (cancelled and resumed, the licence
agreed to once when it asks), Delete (asked twice), and Chat, Serve...
and Inspect, which open that tool on the model. Until then the manager
was a window of the chat tool only. Proof: driven under Xvfb on a
scratch model directory, SmolLM2-135M downloaded then deleted;
make gui-test saves the window.
Speech recognition: Whisper large-v3 and large-v3-turbo (L3)
oklm transcribe runs the large-v3 family (multimodal.md), and the catalogue
offers whisper-large-v3-turbo and
whisper-large-v3.
The cause of the refusal: these two checkpoints are stored in F16 (base and small in F32), and the recognizer handed their biases, norms, positions and token embeddings to operations that take F32. They are now widened to F32 at load, and the matrices multiplied where the file is mapped. The three suspects were audited and were already right: the 128 mel bins and their filter bank (the bin count is read from
config.json; the 128-bin spectrogram was within 2.6e-5 of llama.cpp's since Ultravox), the layer counts (32 and 4 for turbo, fromconfig.json), the special and language tokens (found by name, so<|yue|>moving the others changes nothing). Added: shape checks of the embeddings and positions and a check that the special tokens are inside the embedding table, so mismatched files are refused at load.Reference: transformers 5.22.2 (PyTorch 2.15.0 on the CPU, in a scratch environment), greedy, no timestamps, each 30 seconds on its own, on the very
model.safetensorsand the same 16 kHz WAV. llama.cpp, the usual reference, has no Whisper decoder (its encoder half is the Ultravox check). Texts compared whole:Model jfk.wav (11 s) a 127 s radio address (5 chunks) base identical identical (375 words) small identical identical (365 words) large-v3-turbo identical 357 of 364 words: one sentence fragment before a cut left out large-v3 identical first 75 s: 214 of 215 words, the first word after a cut left out Both differences sit at a 30-second cut, and each of the two chunks transcribed alone gives transformers' text word for word: OKLM computes the spectrogram over the whole recording (its maximum sets the floor, and the frames at a cut see the samples across it, as in whisper.cpp), transformers over each chunk.
Tests:
tools/gen_whisper_fixtures.py(run bytools/gen_model_fixtures.py) writes two tiny random checkpoints, one in the layout of base (80 bins, F32), one in that of large-v3-turbo (128 bins, F16, fewer decoder layers, one more language token), with the text a double-precision numpy Whisper decodes from a 35-second test signal;tests/mm/test_audio.crequires the same text, for two languages each.tests/mm/test_mm_public.ctranscribes jfk.wav with each of the four checkpoints present.Left: speed and devices (see L3 in the roadmap).
Done (2026-11-12)
Graphical tools: the server manager (T4)
oklm-gui --tool servers (gui.md), menu entry "OKLM
Servers".
- The form: an installed model from a list or a file,
the backend, the context on a slider with the memory plan's line under
it (grey when it fits; red when it does not, or above the trained
context), the sequences, the port (empty: the first free one from 7430),
a keys file, a TLS certificate and key, adapters, a retrieval index, and
a field of more options split as a command line. Start
server runs
oklm servethrough the registry of background processes. - Profiles: the form kept under a name, loaded from a
list, deleted;
~/.config/oklm/gui-server-profiles.json, mode 600. - The servers: those of the registry and those
answering on the loopback ports 7430 to 7480, asked every two seconds
off the drawing thread (
/health,/api/stats,/api/context): model, port, load, tokens a second, context, pid. - The selected server: stop and restart (registered ones; the window says why not for the others), its log followed, its dashboard in the browser, a chat window on it, its context resized with the server's own answer for the size shown.
- In the registry (server.md):
oklm_procreg_split(), a command line back into its arguments, and a directory to start a command in (oklm_procreg_start()with the entry'scwd), both for Restart. - Proof:
tests/harness/test_procreg.c(the split, the start in a directory); the whole cycle driven under Xvfb with SmolLM2-135M on the CPU: the form filled, a profile saved, loaded and deleted, a server started, listed with its figures, its log followed, its context taken from 4096 to 1024, its dashboard and a chat window opened and a message answered there, restarted, stopped; a keys file; a server found by its port; a start that fails. - Left: on Windows the window lists and does not start (the registry is POSIX, T10); compiled for Windows, not run there. A server on TLS and adapters or an index in the form were not run, only their options written.
Graphical tools: the model inspector (T7)
oklm-gui --tool inspect (gui.md), one file,
apps/gui/tool_inspect.c, over what T3 made readable.
- Two sources. A GGUF file (an installed model, a
path, or
-m), read through the library: its tensor table (oklm_model_info_json()) and the memory planner (oklm_mem_plan_compute(),oklm_mem_plan_layout_json()), no weight loaded. Or a running server (found on loopback, or--urland--key):GET /api/model?tensors=1for the model and its real placement,/api/statsand/api/profileevery second for its figures. One worker thread does the reading; the window never waits. - The map. The embeddings, a bar a block, the head, sized by their bytes and split by role with a colour a role and a legend; under it where each part is (a layer whose routed experts stay in host memory cut in two) and each layer's cache. Pointing at a part shows its tensors by role, types and bytes; a click keeps it and lists its tensors in the table (name, type, shape, bytes, role).
- The placement. Backend, context, layers on the
device and
--n-cpu-moechosen with a list and three sliders, projected again at each change: host and device memory against what is available (red when it does not fit) and the planner's verdict.--ram MIBprojects against another machine's memory. - Proof. Driven under Xvfb with xdotool on the CPU,
the screenshots looked at: SmolLM2-135M and Granite 3.1 1B A400M as
files (a block pointed at and clicked, the context dragged,
--ram 1500red), theopenclbackend's projection with half the layers on the device and the experts of 18 layers in host memory, and aoklm serve --backend cpufound in the list with a request sent while the window read its figures.make gui-testsaves the inspector's window on the test model too. - Left out. The map of the experts the routers pick
(
/api/experts); a native file dialog (Allegro's dialog add-on is not among the libraries the window links).
Graphical tools: the retrieval manager (T5)
The retrieval manager of the roadmap's graphical tools (roadmap.md), over the commands' JSON of T2.
- The tool (gui.md):
oklm-gui --tool rag,apps/gui/tool_rag.c, with its menu entry (OKLM Retrieval). An index named, remembered or chosen with the system's file dialog, and five tabs. Build: folders and files, the embedding model among the installed ones, the main options;oklm rag index --progress-jsonstarted through the registry of background processes and followed (a bar, the current file, the counts, the result or the error), and stopped. Sources: the figures and each source with its state, the stale ones tinted, an update that keeps the index's own settings. Search: the mode, the passages, the reranker, each passage with its scores and its text, on a thread of the window. Evaluate:oklm rag eval --jsonon a judged set, its table. Attach: a chat window on the index, and theoklm serveline, copied to the clipboard. - The chat window on an index (gui.md):
oklm-gui -m MODEL --rag INDEX, new: the window's server opens the index and each message is answered from the passages found. - Without a window:
apps/gui/rag_cmd.hmakes the command lines and reads the progress, shared withtests/rag/test_gui_rag.c, which runs each of them. - Two faults met on the way, fixed: the string
buffer's
hb_free()had the name of a function HarfBuzz exports, so the window ended on a segmentation fault as soon as GTK (the file dialog) shaped a text: it is linked asoklm_hb_free. And GTK sets the program's locale: the numbers are kept in the C locale, or every JSON number written after a file dialog had a comma in a French session.
Proof: under Xvfb with xdotool on the CPU, an index of seven Markdown
files (672 chunks) built with bge-small-en-v1.5 in 62 s; a file edited,
one deleted, one added: one source changed, one missing, and the update
embedding 30 chunks in 1.4 s; searches in hybrid, sparse and reranked
dense mode; the evaluation's table on a six-document set; a rebuild
stopped, the index unchanged byte for byte; the chat window started from
the Attach tab, SmolLM2-135M answering from the indexed passage;
test_gui_rag (82 checks).
Not done: the tool was not run on Windows (it compiles there; a build and an evaluation run on the window's thread, without Stop, since the registry is POSIX); a window does not take back a build started by another; indexes with a graph or captions are searched but not built here; the chat window does not list an answer's sources.
Graphical tools: the groundwork (T0)
The first item of the roadmap's graphical tools (roadmap.md).
- A shell and a file a tool (gui.md): the window's code, one file until
then, is
apps/gui/:shell.c(the window, the faces, the event loop,--screenshot),http.c(the HTTP helpers),gui.h(what a tool is: a name, a title, a line about it, and callbacks),tool_chat.c(the chat window as it was) andtool_launcher.c.oklm-gui --tool NAMEchooses the tool,chatby default; a new tool is a file and a line ofGUI_TOOLS. Proof:make gui-test's run; under Xvfb with xdotool, a message typed to a server, the Connect window, Use this model starting the program again; the Windows build's window under Wine. - The launcher (gui.md):
oklm-gui --tool launcherlists the tools with a button each (the program again, a process of its own) and opens the bundled HTML documentation. Proof: driven under Xvfb (the opener started onindex.html, the chat window started and outliving the launcher). - A registry of background processes (server.md):
include/oklm/harness/procreg.h, a file a process for the user alone (pid, kind, command, log, port, address, model, start time): start detached with a log, register oneself, list (the dead removed), follow a log, stop (SIGTERM then SIGKILL).oklm serveregisters itself;oklm serve --list(--json) and--stop PORT|PID. Proof:tests/harness/test_procreg.c; a server listed and stopped by its port. - Menu entries (build.md): a desktop file a tool in the
.deband a Start Menu shortcut a tool in the Windows installer, both written frompackaging/gui-tools.txt; the AppImage opens the launcher. Proof: the installer run silently under Wine (the shortcuts OKLM and OKLM Chat,oklm-gui.exe --tool list, the launcher's window). - Left: T1 to T10, in the roadmap. POSIX only for the registry (T10).
Graphical tools: conversion and quantization (T8)
- The tool (gui.md):
oklm-gui --tool convert(apps/gui/tool_convert.c, menu entry OKLM Convert). A source (a GGUF file or a checkpoint folder, typed or chosen in the tool's file browser) described withoklm info --json's data or the checkpoint's configuration and tensors; the result's file, one of the 36 types ofoklm quantizewith the bits a weight and its use,--pure,--imatrix, the name; the exact command lines, copyable; Convert and Stop on background processes of the registry, their log and progress followed; the perplexity of the source and of the result (oklm pplon the CPU) with their sizes, side by side; the result opened in the chat tool. A checkpoint goes to the typesoklm convertdoes not write through an F16 file, in two commands. Proof: under Xvfb with xdotool on the CPU, SmolLM2-135M Q8_0 to Q4_K_M and its checkpoint to Q4_K_M, both measured; a run stopped; the failures shown;make gui-testruns the tool on the tiny model. - The size before writing (convert.md):
oklm convert --dry-runandoklm quantize --dry-run(oklm_gguf_write_opts,oklm_gguf_writer_file_size()), exact: the dry run of SmolLM2-135M to Q4_K_M gives the published file's 105 454 432 bytes. Proof:test_write_opts(tests/io/test_gguf.c),test_convert_commands(tests/harness/test_cli_progress.c). - Progress as JSON lines (training.md):
--progress-jsononoklm convert,oklm quantize(a line a tensor) andoklm ppl(a line a window, the figure in the end line). Proof:test_convert_commands. - Fixed on the way:
oklm pplon a text shorter than one window printed a perplexity of 1 over 0 tokens; it fails and says so. - Left: the window shows the command lines on Windows and does not run them (the registry, T10); the result of the two-step path is not estimated before its F16 file exists; no file dialog of the system (the program links none), the tool has its own browser.
Graphical tools: the training window (T6)
oklm-gui --tool train (gui.md), menu entry "OKLM
Training".
- A form a kind of run: from scratch (a size, the
text), fine-tune, LoRA (rank, alpha, targets), chat data, RL; the main
options of
oklm trainandoklm rlwith a tooltip each, the command's defaults named in the empty fields, More options as typed; the model a file, chosen in the system's file dialog or among the installed ones. - The command shown, then run in the background
through the registry of processes, with
--progress-jsononPREFIX-progress.jsonl; the window reads the file as it grows: the training and validation losses as a curve, the step, the rate, the time elapsed and left, the checkpoints, the samples, the log's tail; Stop. A run going on is found again when the window is opened:oklm_procreg_split()reads the progress file's path back from the registered command line. - After a LoRA run: Merge adapter
(
oklm lora-merge), Export adapter... (a copy: a LoRA checkpoint is already an adapter in llama.cpp's format, so there is no export command), Open a chat on the checkpoint or the merged model (oklm-gui --tool chat -m). - Advanced:
oklm refusals,oklm abliterate,oklm refusalson the model written, in a row or one alone; the two shares of prompts declined side by side, the KL divergence, each layer's separation as it arrives. - Met on the way: HarfBuzz exports
hb_free, the name of the harness's own; with the file dialog (GTK, Pango) in the program, HarfBuzz freed its memory through the harness's function and the window died. The harness's is nowoklm_hb_freein the object files (a define inharness_internal.h). GTK also sets the locale: with a decimal comma no progress line was read untilLC_NUMERICwas put back toC. - Proof: driven under Xvfb with xdotool on the CPU (a one-layer model
from scratch, LoRA on SmolLM2-135M with the merge and the chat, a long
run stopped, the window closed and reopened on a run going on, RL, the
Advanced tab on SmolLM2-135M);
tests/harness/test_procreg.cfor the splitting. Not on Windows: the registry is POSIX, the window shows the command there and says so (T10).
Graphical tools: the sessions browser (T9)
- The tool (gui.md):
oklm-gui --tool sessions,apps/gui/tool_sessions.c, with its menu entry (OKLM Sessions). A project folder typed, chosen with the system's dialog or taken from the folders opened before; its sessions listed (date, title, turns, model, size; sorted by a heading; those an agent started with--listenruns marked live); a text searched in their messages by a worker thread, with the lines found in each; a session read, the tools' calls and results folded to a line or in full; written as Markdown or JSON to a file chosen in the dialog; resumed in a terminal ($TERMINAL,x-terminal-emulator, the usual ones; the command to copy otherwise); the chat window attached to a live one; deleted after a confirmation. Proof: driven under Xvfb with xdotool on sessions made byoklm agenton SmolLM2-135M and by a scripted model, every flow (the list in gui.md). - In the harness (agent-sessions.md):
a session opened to be read only, its messages without an agent, a
search in them, the Markdown and JSON exports, the removal of one
session, the model, start, size and turns in the list's entries
(
oklm/harness/session.h); the agents listening now (oklm_listen_list(), the project written as an absolute path). Proof:tests/harness/test_session.c(test_read_only),tests/harness/test_listen.c. oklm agent sessions:[list] [--json],export ID [--format markdown|json] [-o FILE],rm ID, without a model. Proof:tests/harness/test_cli_sessions.c.oklm-gui --tool chat --agent PORT: the chat window attached at start to a listening agent. Proof: started by the browser's Attach under Xvfb, the agent's events shown.- Left: on Windows no dialog, no live agent, and Resume's console not
run (T10); the window's flows have no automatic test beyond
make gui-test's chat run.
Packages, HTML documentation and the model catalogue
- Packages (build.md):
make deb,make appimage,make windows,make installer-windows,make docker-build. Proof: the trixie.debinstalled in a cleandebian:trixie-slimcontainer and generated there; the installer run silently under Wine. - HTML documentation:
make docs-html(pandoc), with a check that fails on a dead link or a missing anchor. - Model catalogue (models.md): no package holds a model;
oklm modelslists those offered (general chat, coding, images, sounds, security, retrieval),oklm models pulldownloads them (pinned revisions, resumed, SHA-256 checked, licences asked once per model), and an installed model's name is accepted wherever a model file is. Proof:tests/client/test_models.c(a server of the test's own: redirects, a resumed and a cut download, a wrong sum refused); pulls from Hugging Face on Linux and with the Windows build under Wine; each catalogue file marked as run was run. - TLS in the Windows build:
make windows-depsbuilds OpenSSL; nilorea's client contexts trust the Windows root store. - Model manager in the window (gui.md):
oklm-guilists the catalogue, downloads (progress, cancel, the licence box), serves another model and deletes one. Proof: driven under Xvfb with xdotool (select, download, use, delete). - Context sizes (models.md): listed for every model,
changed while running in the window, the agent
(
/context-size) and a server (POST /api/context), up to 1 048 576 tokens, with the memory of the size shown and red warnings. - An agent other programs attach to (agent-loop.md):
oklm agent --listen, loopback and a token. Proof:tests/harness/test_listen.c, and a client that answered a permission question. - Connecting the window (gui.md): servers found on this machine or given by address and saved, listening agents, sessions read. Proof: driven under Xvfb.
- Left: vision projectors beyond idefics3 (L1), Whisper large-v3 (L3), DeepSeek-Coder (L4), in the roadmap.
1b. GPU backend for AMD, NVIDIA and Intel (V1 to V3)
ADR-0023: Vulkan as the main GPU backend, checked against the CPU on the RX 7900 XT (RADV) and the UHD 750 (ANV); vulkan.md describes each group. V4 (NVIDIA) stays in the roadmap.
- V1. Foundation (done 2026-11-11: the element-wise
ops on both local GPUs as the CPU, bit for bit where the CPU is exact;
every other op through the CPU fallback; vulkan.md): the loader (
dlopen, OKLM's own declarations, no SDK), instance, device choice (discrete GPUs first, as OpenCL:OKLM_VK_DEVICE), queue, memory (device-local buffers, staging copies, fake addresses mapped back at dispatch), pipelines built from embedded SPIR-V (glslcat build time; noglslc, no Vulkan backend), a dispatch with push constants and a descriptor scheme every kernel shares, synchronization, the backend registered asvulkan, the CPU fallback per op,test_vulkanwith the element-wise ops, and the file layout the parallel steps fill (a host file, a shader directory and a test file per group). The MinGW build too. - V2. Dense decoding, in parallel once V1 is in:
- V2a. Element-wise ops, RMS / layer norms, RoPE, softmax, get_rows, copies and set_rows, the KV cache's quantized rows. Done 2026-11-12 (vulkan.md).
- V2b. One-column products of every weight type (the dequantization of each), with accumulation into the destination. Done 2026-11-12 (vulkan.md).
- V2c. Attention over the cache (decode, then prompts), sliding window and soft-cap, F16 / F32 / quantized caches. Done 2026-11-12 (vulkan.md).
- V2d. The model on Vulkan: placement and buffers through the backend interface, then the decode step recorded once and replayed (the step marks, the varying values in device buffers), checked against the CPU on the tiny models and the public ones. Done 2026-11-12 (vulkan.md: the public models within OpenCL's bounds).
- V3. Coverage and speed, in parallel:
- V3a. Prompt products: tiles, then cooperative matrices where the device has them. Done 2026-11-12 (vulkan.md: every weight type; 38 to 46 TFLOPS on the RX 7900 XT's cooperative matrices, at or above OpenCL's).
- V3b. 8-bit products
(
VK_KHR_shader_integer_dot_product): the activations rounded as the CPU does, the CPU's results. Done 2026-11-12 (vulkan.md: Q4_K, Q5_K, Q6_K, Q8_0, Q4_0, Q5_0 bit for bit with the CPU, taken where measured faster). The K-quants' row layout for Intel GPUs (the activations rounded once, a weight row per invocation; the same bits), 2026-11-12: Q4_K 2048 x 16384 on the UHD 750 832 us against 3 780 us in floats, Llama-3.2-1B Q4_K_M decoding 6.4 -> 19.8 t/s (llama.cpp's Vulkan backend 15.6). Tried there and dropped:q8mv_<type>for the K-quants on Intel (4 152 us, slower than the floats); the row layout on the RX 7900 XT (Q4_K 2048 x 2048 9.9 against 5.8 us). - V3c. MoE: routing, indexed products (decode and grouped prompts), weighted sums, the fused gate / up / SwiGLU. Done 2026-11-12 (vulkan.md: every MoE op on the GPU, routing and gate / up / SwiGLU fused, bit for bit as unfused; Granite decoding 278 -> 295 t/s).
- V3d. The hybrid layers' ops, the Walsh-Hadamard rotation, latent attention (MLA). Done 2026-11-12 (vulkan.md).
- V3e. Several sequences and the server's scheduler,
several devices, the expert tier;
autopicks Vulkan where it is ahead. Done 2026-11-12 (vulkan.md,auto: integrated GPUs and non-AMD discrete ones).
2. CPU decoding of SmolLM2 and GLM-4.7-Flash against llama.cpp
The two CPU decode rows of the known speed gaps (SmolLM2-135M at four and eight threads, 0.96 to 1.00 of llama.cpp; GLM-4.7-Flash, 0.97 to 0.98), done 2026-11-12 (benchmarks.md).
- What the measure showed. The rows were listed as "fewer team loops a token, less fixed cost per loop" and "short K-quant products closer to the 37 GB/s a large one reaches". Neither was the cost: a worker starts its first chunk 0.15 microsecond after a loop is published and the loops' threads compute 99.8 % of their time (SmolLM2), and on weights out of the cache a product of any size reads at 29 to 30 GB/s, 96 % of the 31 GB/s this machine's memory gives a plain sum (37 GB/s was the fastest call on one matrix partly in cache). The cost was outside the loops: each product rounds its input row to bytes on the calling thread, in scalar code, 8 % of a SmolLM2 step and about 4 % of a GLM-4.7-Flash step.
- Done. The activation row rounded with AVX2, the
scalar bits (
oklm_cpu_quantize_act()); a token's rotation computing its cosines and sines once for all heads; the packed copies cut from chunks on huge pages (OKLM_CPU_PACK_HUGE). tg128, before / after / llama.cpp the same day: SmolLM2 Q8_0 184 / 208 / 190 at four threads, 182 / 200 / 187 at eight; Q4_K_M 250 / 280 / 256 and 250 / 279 / 257; GLM-4.7-Flash 11.50 / 11.99 / 11.78 and 11.47 / 11.86 / 11.59. The same logits byte for byte before and after. - Measured and left out. The mapped file collapsed into huge pages (GLM-4.7-Flash 11.94 / 12.00, within the noise); the loop fusions of the row (nothing to return, see above).
2. GPU decoding of the MoE models in fewer launches
The roadmap's first known speed gap (Granite 0.83 of llama.cpp, OLMoE
0.84, GLM-4.7-Flash 0.85 on the RX 7900 XT's OpenCL backend, a step
being mostly launches). Each candidate was first bounded by skipping its
launches (Granite: scale and add 317 to 370 t/s, norms 362, the keys'
rotation and cache writes 334, the router's product 329), then the ones
that paid were fused, each with its off switch and the bits of the
separate ops (test_opencl: test_residual,
test_rope_kv, test_route; the three models'
logits the same bits with and without).
- The end of a residual branch in one launch
(
res_norm,OKLM_CL_RES): expert sum, shared experts' add, residual scale, add into the stream, RMS norm of the new stream. Granite 424 launches a token to 304, 314 to 379 t/s. - The rotation with the cache writes in one launch
(
rope_kv,OKLM_CL_ROPE_KV). Granite 304 to 268 launches, 379 to 399 t/s. - A sigmoid router's routing in one launch
(
moe_route,OKLM_CL_ROUTE_SIGMOID). GLM-4.7-Flash 1168 to 1076 launches, 75.9 to 77.9 t/s. - Together, off / on: Granite 306 to 395 t/s (1.09 of llama.cpp's 364), OLMoE 296 to 318 (0.88), GLM-4.7-Flash 73.8 to 77.9 (0.89); the dense models gain too (SmolLM2-135M Q4_K_M 422 to 471, Llama-3.2-1B 391 to about 400) (benchmarks.md, opencl.md).
- Measured and left out: a norm work-group sum with three barriers instead of ten (within the noise). Left: the roadmap's row.
Done (2026-11-11)
J. Sub-agents sharing a working tree
From Anthropic's "Worse together" study (2026-11-11).
- J1. Irreversible steps wait for running background
sub-agents. Done (2026-11-11): a Bash call running a command of
agents.waitForBackground(git commit,git pushby default) while background sub-agents run is not run, its result naming them; the next turn waits for a report, so the model decides again with it read (agent-tools.md).test_agent'stest_background_wait:git -C . commitbehind acdand a variable held,echobeside it run, the report before the commit asked again. - J3. A behaviour eval. Done (2026-11-11): task 26, two sub-agents at once on one repository, judged on the outcome and on what they did (agent-loop.md): Qwen3-8B 1 of 3, its failures a fix described but not made and reported as made.
- J4. A sub-agent's report says what it wrote. Done (2026-11-11): its report line names the files it wrote with the file tools, or says it wrote none, from its tracked edits (agent-tools.md). The caller's report then matched the files written; it still said "both added a line" when one sub-agent had replaced the other's, which J2 took up.
- J2. Sub-agents running at once do not destroy each
other's work. Planned as a Bash guard (
git checkout --,reset --hard,stash,clean,rm); in seven runs of task 26 no such command came, but an Edit after a fresh Read replaced a sibling's line. Done (2026-11-11) for what was seen: the lines a sub-agent writes are kept from the sub-agents whose runs overlap its own (an Edit or Write removing one refused, a change beside it allowed);test_tools'test_sibling_lines. The Bash guard waits for evidence.
RAG (M17)
Retrieval-augmented generation, planned in rag_spec.md (the steps, what OKLM already has
to build on, the decisions, each step's proof). Before K2 OKLM had no
retrieval: no embedding model forward pass, no index, no reranker; it
has document loading (oklm ingest), the BERT / BGE
tokenizer, the agent's ToolSearch, memory files, System One scoring
(decide.c), the agent evals and the prefix cache to build
on. All in C, nothing from outside the project for the model or its
data, every feature off by default. In order:
- K0. Audit and design (done 2026-11-08): the "RAG" section of gap-analysis.md with file:line evidence and the capability matrix, rag-design.md (layout, stage contracts, API, surface, tests per step, evaluation), the index format adr/0022-rag-index-format.md (proposed: one GGUF file); bge-small-en-v1.5 and the MiniLM cross-encoder as defaults, exact search up to a million chunks. Six rows of the spec's seed table corrected.
- K1. Embeddings (done 2026-11-08, rag.md):
bertandnomic-bertencoders (GGUF and checkpoints), Qwen3-Embedding, pooling,oklm embed,/v1/embeddings,make rag-eval. Result: bge-small, nomic-embed and Qwen3-Embedding F32 / F16 within 1.0e-5 (1 - cosine) ofllama-embeddingover 1000 strings, the checkpoints within 5e-13 of a numpy float64 encoder; SciFact nDCG@10 71.27, the published MTEB figure (71.27); 286 texts/s on the CPU, 731 on the GPU (bge-small). - K2. Chunking and the index (P0, done 2026-11-08, rag.md): fixed, recursive, semantic and code-aware
chunkers over
ingest;oklm rag index|search|info, exact SIMD search, incremental updates. Search equal to a brute-force loop (same top 10, 1e-6); an update gives the bytes of a rebuild; a dense query over a million chunks (384 dimensions, F16) 27 ms at p50, 28 ms at p95, built in 11 s with the hash encoder: exact search kept, no approximate index. - K3. Hybrid retrieval (P0, done 2026-11-08): BM25 in
the same index, RRF or weighted fusion,
--mode dense|sparse|hybrid; BM25 equal to a hand-computed corpus. Proof (2026-11-09,oklm rag eval, embedding models inoklm rag, the 120-question documentation set): with bge-small-en-v1.5, nDCG@10 / recall@10 dense 71.27 / 83.6, sparse 67.41 / 79.5, hybrid RRF 71.29 / 83.7, weighted 73.04 / 86.1 on SciFact; dense 60.48 / 77.1, sparse 64.53 / 80.7, RRF 69.78 / 85.2, weighted 70.61 / 86.7 on the docs set: hybrid at least each alone on both sets (rag.md). - K4. Reranking (P0, done 2026-11-09, rag.md): cross-encoders (the
bertclassification head, token types for pairs), the decide.c fallback with a chat model,oklm rerank,oklm rag search --rerank,/v1/rerank. Result: ms-marco-MiniLM-L-6-v2 within 4.0e-6 (logits) of a numpy float64 reference over 990 pairs, 1.3e-3 ofllama-embedding(ggml's F16 GELU table); on SciFact over bge-small's hybrid top 50 nDCG@10 falls 71.64 to 69.50 (top 100: 68.95, the BEIR paper's 68.8 for that model), bge-reranker-base −3.3, for 0.6 s a query on the GPU and 1.9 s on the CPU: the proof's "above the hybrid" is not met on SciFact, reranking stays opt-in. - K5. Retrieval in generation and corrective RAG (P0,
done 2026-11-09, rag.md):
--rag INDEXon run, chat, serve (a request'sragfield) and the agent (aRetrievetool) with cited sources; chunks judged by decide.c, rewrite, WebSearch (agent) or "no reliable source"; every decision logged (--rag-log, the debug log,/api/stats). Result (Qwen3-8B Q4_K_M, 38 questions over the docs): accuracy 15.8 % without retrieval, 89.5 % naive, 92.1 % corrective, which abstains on 8 of 8 unanswerable questions (naive 7); faithfulness by the model's judge 85.3 % naive, 86.5 % corrective, no invented citation. - K6. Tool selection by retrieval (P0, done
2026-11-09, rag.md):
MCP tools, skills, agents and plugins in a hybrid tool index (Type,
Service, Tool; cached, re-embedded on change), ToolSearch ranked by it,
oklm agent --tool-rag Kgiving each message its deferred tools. Result over 105 tools and 152 requests: top-1 61.8 % against 27.0 % for ToolSearch's keyword score (recall@5 76.4 % against 46.6 %), 68.5 % of a request's tools defined from the first turn against 26.6 %, for 3902 prompt tokens against 3978; agent evals 12 of 17 against 13 and 12 without it (single runs, within their noise). - K7. Agentic, adaptive and conversational retrieval
(P1, done 2026-11-09, rag.md):
the built-in
Retrieveover the project's documentation, indexed and kept up to date by the agent (--rag-project), and over earlier sessions (--rag-memory); passages per message (--rag-inject always|route); a router through decide.c with multi-hop sub-questions (--rag-route, off); follow-ups rewritten from the conversation in chat, serve and the agent;--rag-rerank;ragon streams and/v1/messages. Result on eight new agent evals (Qwen3-8B, single runs): no retrieval 2 of 8, fixed retrieval 8 of 8 (3 turns a task, 137 k tokens), agentic Retrieve 7 of 8 (4 turns, 184 k), routed 8 of 8 (2 turns, 116 k); the router chose "one" for all 38 questions of the answer set (accuracy 92.1 % as without it); follow-up recall@5 32.8 % searched alone, 70.3 % rewritten, 79.2 % joined with the previous question. - K8. Graph, self-critique and multimodal retrieval
(P2, done 2026-11-09, rag.md):
entities and relations extracted by the generator into the index
(
--graph --gen-model), the graph walked and fused at query time (--rag-graph); self-critique of the answer with one retry (--rag-critique, run and serve); figures captioned by SmolVLM at index time (--caption-model), no joint text-image encoder (the captions are always found). Result: on 30 two-article questions both articles in the top 5 for 23 instead of 8, 23 right answers instead of 8 (Qwen3-8B); self-critique on the 38-question set: faithfulness 87.5 % → 96.1 % (judged by the same model), accuracy unchanged at 92.1 %; on 20 questions answered by charts, 9 right with captions against 0 text only (18 with SmolVLM2-2.2B captions, a follow-up). - K9. The evaluation report and docs (done
2026-11-09, rag.md): one summary
of every capability before and after with its model, set and date, the
commands that reproduce each row (
make rag-evalgives the retrieval table to the printed digits),docs/rag.mdreorganized as a user guide, the residual risks.
RAG follow-ups, each small and measured on the sets above:
- The condenser: search the previous question joined with the rewrite
(done 2026-11-09,
--rag-followup both, the default): recall@5 on the 30 follow-ups 83.3 % against 82.2 % joined and 71.1 % rewritten. - Weighted fusion as the default (done 2026-11-09): better than RRF on
both sets (73.04 against 71.29 nDCG@10 on SciFact, 70.61 against 69.78
on the docs);
--fusion rrfkeeps reciprocal rank. A dense-only option for tool ranking: done (2026-11-09),toolSearch.mode hybrid|dense|bm25, hybrid kept: dense defines more of a request's tools at the 0.62 floor (72.5 % against 68.5 %) only by giving more definitions; at the same number given, 67.9 % against 68.5 % (rag.md).Graph retrieval on real documents: done (2026-11-09) on an 82-section part of the docs set (the full extraction is 4.3 s a chunk, about 3 hours): two-section recall@5 93.6 % against 89.6 %, recall@10 100 % both, single-section checks 85 % against 95 %;--rag-graphstays off. Synthetic set under weighted fusion: 28 of 30 pairs in the top 5 against 7 (rag.md).A larger caption model than SmolVLM-256M: done (2026-11-09), SmolVLM2-2.2B with a mark-by-mark caption prompt answers 18 of 20 chart questions against 9 (rag.md).The router: another prompt or model: done (2026-11-09), negative: of three wordings only few-shot examples route Qwen3-8B (40 of 58 requests as expected), and they answer 32 of 38 against 34; GLM-4.7-Flash splits 46 of 58. Prompt unchanged, the router stays off (rag.md).Self-critique on streams and in the agent and(done 2026-11-09: streams flagged with the verdict on the last event, a retry on request;oklm chatoklm chat; the agent's final answers, one retry through the loop; rag.md).The judge, rewriter and condenser for an agent on(done 2026-11-09:--url/v1/decideon an OKLM server, chat completions' log-probabilities or text elsewhere,--rag-critiqueallowed; rag.md): the 8 retrieval tasks pass 7 of 8 with Retrieve and 8 of 8 routed, as in process, for fewer agent tokens (150,567 and 107,409 against 184,240 and 116,470).(done 2026-11-09):--caption-promptfor the small captionersoklm rag index --caption-prompt TEXT|@FILE, recorded in the index (rag.caption.prompt), a change captioning again; not re-measured.Precision@k and a second public set(done 2026-11-10): P@5 and P@10 inoklm rag eval(SciFact dense P@10 9.43, the model card's figure); BEIR NFCorpus inmake rag-eval(--sets scifact,nfcorpus,docs): dense nDCG@10 34.38 against the card's 34.31, BM25 32.09, hybrid 36.67 RRF and 36.37 weighted (rag.md).Retrieval on code(done 2026-11-10): 112 judged questions over the sources (tests/rag/code_set/,--sets code). The code-aware chunker is no better than the recursive one (hybrid nDCG@10 59.32 against 60.12, ahead only on two-file questions); Qwen3-Embedding-0.6B dense 63.63 against bge-small's 55.49;autokept (rag.md).Answer quality with reranking on(done 2026-11-10):oklm rag eval --rerank; MiniLM over the first 20 candidates lifts the docs' MRR@10 74.20 to 78.30 and recall@1 30.2 to 37.4 %, and answers 26 of 30 against 25 (corrective 89.5 % against 86.8 %), naive retrieval's faithfulness falling 88.7 % to 81.1 %; off kept (rag.md). The generation scorer no longer counts a subscript in code (qs[16]) as a citation.The CPU-only latency of the generation pipeline(done 2026-11-10): 20.9 s a question naive and 35.2 s corrective (22 s of judging) on the i7-11700K, against 1.5 s and 2.3 s on the GPU (rag.md).Error bars(done 2026-11-10):tools/rag_stats.py, bootstrap intervals and paired comparisons; four repeated runs of each generation configuration identical. Hybrid over dense (docs, NFCorpus), weighted over RRF (SciFact), Qwen3-Embedding on code and reranking's recall@1 hold; fusions elsewhere, chunkers and every answer difference on 38 questions do not (rag.md).Server determinism at temperature 0(done 2026-11-09): the cached rows are exact, but a cached prompt's last rows go as a small piece and the OpenCL products pick their kernel, and rounding, by a piece's rows.oklm serve --deterministicputs the pieces on a grid of--prefill-chunkpositions and cuts a cached prefix to it: 38 of 38 answers the same whole and streamed (19 before), a warm 906-token prompt 0.22 s against 0.03 s; off by default (server.md). Concurrent requests, and the grader's and critique's scoring requests sent together, still share steps by timing.
Done (2026-11-06 to 2026-11-08)
What the "Next" list of 2026-11-07 held and was finished by 2026-11-08, by block, each item with its result and its links (the proof). Items still open say so here (with their "Left" lines); their scoping is repeated in the roadmap, under "Next".
A. Agent and harness
compactThresholdlowered for small models, and an eval of Edit against a whole-file Write. Done (2026-11-08): without a setting the share follows the model's parameter count (0.60 under 2 billion, 0.75 under 8 billion, 0.85 above or unknown, as behind--url;oklm_agent_default_compact_share(), unit-tested); the rule is a choice, no eval measured it (no task here fills a window). Edit against Write on Qwen3-4B, tasks 01, 06 and 07, one sample (greedy: three runs agree): 1 of 3 each, but Write only wrote 9243 tokens against 1870 and took 172 s against 38 s on task 01 (agent-loop.md).
D. Processes and Windows
- Per-job memory and CPU limits on Windows (Job Object memory and
CPU-rate limits) [process 15]. Done (2026-11-07), cross-compiled,
untested on Windows:
JOB_OBJECT_LIMIT_JOB_MEMORY(the smaller of the sandbox's and the job's) and a hard CPU rate on the job's Job Object, a detached job with limits given one that does not kill it at close; the rate's arithmetic unit-tested on Linux. Left: a run on real Windows. - A port's holder named on macOS and Windows [process 2.6, 15]. Done
(2026-11-07), cross-compiled, untested on Windows:
GetExtendedTcpTablewith owning process ids, the image name read withQueryFullProcessImageNameA; the rows' parsing unit-tested on Linux with fake rows. macOS through libproc (proc_pidfdinfo), guarded code never compiled. Left: a run on real Windows, a macOS build. - A Windows toast in place of the bell. Done (2026-11-07),
cross-compiled, untested on Windows: with
preferredNotifChannel: system, Windows PowerShell shows a toast through Windows.UI.Notifications (the script passed encoded, nothing to install),MessageBeepwithout PowerShell; the command line unit-tested on Linux. Left: a run on real Windows. - The integrations run against a real Docker daemon (installed here),
not only behind flags [process 10.1]. Done (2026-11-08):
tools/check_docker.sh(a session's container and compose stack stopped at its end with their volumes kept,docker rmiasking and the image staying,/doctorquiet;SKIPwhen no daemon answers) passes all 11 checks against Docker 28.5.2 on this machine.
E. Training
--backend rocmcannot allocate the 5 GB of shared virtual memory the 8B base needs: find why, fix it or document the limit. Done (2026-11-08): not an SVM limit (17 GiB per allocation on the gfx1100) but the device full: at 6144 tokens the activations take 17.1 GB before the 5.0 GB packed base, and coarse-grained SVM is committed in device memory whereopencl's plain buffers overflow into host memory. A buffer that no longer fits now goes to fine-grained SVM, in host memory (a notice;OKLM_ROCM_DEVICE_MBcaps the device share,test_train_clruns with every buffer there): the Qwen3-8B LoRA step at 1 x 6144 (2 accumulated) 56.4 s onrocmagainst 91.2 s onopencl, the same losses (training.md).(Open, scoped.) A native on-device gated-delta kernel (the hybrid layers' recurrence runs on the host in inference and training; speed only). Assessed (2026-11-08), not started: more than the recurrence leaves the device. Training (
k_gated_delta_fwd/_bwd) downloads each linear block's input and output gradient and runs the whole block on the host, its projections included; inference (lin_blockinarch_llama.c) does the same, with the per-sequence state and conv ring in host memory (lin_state, whichoklm_kv_savewrites). On the device it takes: the projections through the existing products; kernels for the causal conv with SiLU, the per-head L2 norm, the recurrence (one work group per value head, the dk x dv state in device memory, the tokens in order) and the gated RMSNorm, forward and backward (the recurrence's backward walks the tokens in reverse with the states recomputed per chunk,oklm_gated_delta_recurrent_backward); the state and ring as device buffers per sequence, read back foroklm_kv_save; checks against the host ontiny-qwen3nextand the trainer's hybrid fixture.BF16 as a training storage type. BF16 works everywhere in inference without native hardware support (the CPU and the OpenCL kernels widen it to F32: weights, safetensors checkpoints run directly, KV caches,
oklm convert --type bf16), and training starts from a BF16 base (unpacked to F32, or kept and unpacked on the device with--quantized-base). What is F32 only: the trainer's master weights of a full run, activations, gradients, AdamW's moments and the checkpoints. Add, each opt-in and measured: activations (and their saved copies under--recompute) in BF16, AdamW's moments in BF16 (with stochastic rounding or a F32 master copy), checkpoints and merged models written in BF16 (--save-type bf16,lora-mergekeeping it). While OKLM's kernels compute BF16 in F32 (item 4 changes that) the gain is memory, not speed: the loss curves compared with F32 on SmolLM2-135M and OKLM-Nano, the memory table updated. Done (2026-11-07):oklm train --act-type bf16(the activations each block keeps, or its input under--recompute, rounded to nearest even and widened back into one F32 set),--opt-type bf16(AdamW's moments with stochastic rounding from a hash of seed, step, moment and element; the master weights F32),--save-type bf16(checkpoint matrices, andoklm lora-merge --save-type bf16; resuming one starts from the rounded weights). OKLM-Nano 300 steps: final validation 4.5278 (F32) against 4.5280 / 4.5269 / 4.5281 (activations / moments / all), optimizer 41 -> 20 MiB, kept activations 576 -> 288 MiB, peak RSS 813 -> 601 MiB; SmolLM2-135M 40 steps: 3.5236 against 3.5235, peak RSS 3.2 -> 2.5 GB (training.md). Not done: the OpenCL / ROCm trainer (--act-typeand--opt-typerefuse a device,autopicks the CPU; no kernel was changed or run), gradients in BF16. Done (2026-11-08): the OpenCL / ROCm trainer (tr_bf16_st/tr_bf16_ldkeep the BF16 copies on the device,tr_adamw16updates BF16 moments with the CPU's stochastic-rounding bits;autono longer falls back to the CPU). SmolLM2-135M full,rocm, 300 steps: final validation 3.1329 (F32) against 3.1335 (activations and moments BF16), 3.1356 with BF16 products too; optimizer 1026 -> 513 MiB, activations 1689 -> 1101 MiB, step 127 / 129 / 111 ms (training.md). Not done: gradients in BF16 (accumulated by every weight product, micro-batch and norm sum: each writer would need a 16-bit output with F32 accumulation kept elsewhere, for 2 bytes a trainable parameter). Done (2026-11-11):--grad-type bf16(grad_type, full training, any backend): the backward pass works region by region (a block's parameters, the final norm, the vocabulary matrices a tile of rows at a time), each computed into one F32 window the size of the largest block and added into the BF16 gradients with stochastic rounding (tr_grad16_add, the CPU's bits on a device); the update reads them widened. SmolLM2-135M full,rocm, 300 steps: final validation 3.1329 (F32) against 3.1329 (BF16 gradients) and 3.1334 (activations, moments and gradients BF16); gradients 513 -> 270 MiB, the three together 2799 -> 1413 MiB, step 127 / 132 / 135 ms (training.md).The matrix units (WMMA) of the gfx1100. RDNA3 multiplies BF16 and F16 tiles with F32 accumulation in hardware; nothing in OKLM uses it:
--backend rocmcalls rocBLAS's F32sgemmonly (src/train/gemm_rocblas.c), and the OpenCL kernels widen BF16 / F16 to F32. Cheapest first, each opt-in and measured against F32:- Training through rocBLAS
gemm_ex: BF16 (or F16) inputs, F32 accumulation and output, which rocBLAS runs on the matrix units of gfx11; activations converted by one kernel before each product (or kept in BF16, item 3), a BF16 base fed directly instead of unpacked to F32 (--matmul-precision bf16); the loss curves against F32. Done (2026-11-08):oklm train --matmul-precision bf16|f16(matmul_type,rocmonly): the products of at least 128 in every size on 16-bit inputs, F32 accumulation into a 16-bit product widened into the F32 output (rocBLAS's gfx1100 kernels with F32 outputs are slower thansgemm; with 16-bit outputs 2.5 to 4 times faster), a BF16 base read in place under--quantized-base. SmolLM2-135M 300 steps: validation 3.1329 (F32) / 3.1342 (BF16) / 3.2365 (F16, which underflows small gradients) full, 3.1573 / 3.1576 / 3.3573 LoRA; step 125 -> 114 ms full, Llama-3.2-1B LoRA 350 -> 227 ms (training.md). Left: loss scaling for F16, the attention's batched products and the thin LoRA products on 16 bits. Done (2026-11-08):--loss-scale dynamic|N(loss_scale, any backend: the gradients of the loss times S, divided back in the update; an update that overflows is skipped and halves S, 200 finite ones double it). SmolLM2-135M 300 steps with F16 products: validation 3.1347 full (F32 3.1329, F16 unscaled 3.2365), 3.1570 LoRA (3.1573, 3.3573), two updates skipped, the scale settling at 16384. Not done, measured not to pay: the thin LoRA products on 16 bits (a LoRA step 110 -> 164 ms when they were), the attention's batched products (all oftr_gemm, the OpenCL tiled products, is 9.9 % of a SmolLM2 step with BF16 products, while the roundings in and out of 16 bits of the large products already take 21 %). Done (2026-11-08): the weights' 16-bit copies kept between updates (tr_cl_set_weights16(): 2 bytes a parameter on the device, AdamW writing the rounding of what it updates, a LoRA run's frozen base rounded once; allocated when it fits with 512 MiB left,OKLM_ROCM_W16_CACHE=0turns it off), and the output gradient rounded once for a weight's two backward products. The same bits (checkpoints after 20 steps identical); step 102 -> 98 ms SmolLM2 full, 108 -> 104 LoRA, Llama-3.2-1B LoRA 219 -> 207 (BF16); the roundings 20.1 -> 16.3 % of a SmolLM2 step, 14.9 -> 8.2 % of a Llama-3.2-1B LoRA step; device memory +257 MiB / +2.3 GiB (training.md). Left: the activations' roundings (each product's input and output gradient in, its output out), new at every product. - WMMA from OpenCL: the ROCm OpenCL compiler is
clang, whose
__builtin_amdgcn_wmma_*builtins may be callable from OpenCL C; a probe kernel first, then BF16 / F16 tile products in OKLM's own kernels, no ROCm dependency, if it compiles and runs here. Done (2026-11-08):__builtin_amdgcn_wmma_f32_16x16x16_f16_w32and..._bf16_w32(half16 / short16 operands, float8 accumulator) build through clBuildProgram with-cl-std=CL1.2on the ROCm runtime (driver 3649.0) and run on the gfx1100 (test_opencl_wmma: one tile within 4e-8 of the CPU; the units' F32 sums are not always IEEE-rounded).OKLM_CL_WMMA=1|f16|bf16swaps the prompt products' tiled kernels forwm_<type>(every weight type, dequantized and rounded to F16 / BF16 per tile): Qwen3-8B Q4_K_M prompts 831 -> 1 320 t/s (F16) / 1 137 (BF16), Llama-3.2-1B 4 938 -> 8 713, the same 64 greedy tokens with F16, perplexity 19.9952 -> 19.9972 (opencl.md). Off by default (answers can part at near ties). Left: prefetching the next step asgm_<type>does, larger tiles, the attention's products. Done since: the larger tiles (w2_<type>, 2026-11-08); the attention (2026-11-11):attention_wm64h/attention_wm128htake the prompt attention of F16 caches withOKLM_CL_WMMA(OKLM_CL_WMMA_ATT=0keeps the half tiles), both products as 16 x 16 x 16 F16 blocks with F32 sums, the queries and probabilities rounded to F16: Qwen3-8B Q4_K_M prompts 2 478 -> 2 619 t/s (497 tokens), 1 992 -> 2 476 (2 059); Llama-3.2-1B 14 175 -> 15 370 and 10 840 -> 14 277; perplexity unchanged to the fourth decimal (opencl.md); the next step read while multiplying (2026-11-12), inwm_<type>and inw2_<type>: prompts of 512 tokens withOKLM_CL_WMMA=1Llama-3.2-1B 16 228 -> 17 553 t/s, Qwen3-8B Q4_K_M 2 825 -> 3 172 (llama.cpp 17 966 and 3 218 in the same session), of 48 tokens 5 759 -> 6 863 and 1 216 -> 1 523. Nothing is left of this item.
Scoped, then measured not to pay (the second of item 4's three steps):
- Inference prefill through the same path: a
quantized matrix unpacked to BF16 / F16 in a scratch, then
gemm_ex, over the OpenCL shared virtual memory the trainer already uses (llama.cpp's hipBLAS route); prompts only, decoding a token is bound by memory, not by arithmetic; the same greedy answers within the usual near ties. Measured (2026-11-11), not kept: rocBLAS reaches 36 to 71 TFLOPS on Qwen3-8B's shapes with an F16 product (12 with an F32 one), but the default 8-bit kernels (wq_<type>, added since) do a 500-token prompt's products in 150 ms against 161 for rocBLAS's products alone, before the F16 dequantization and conversions; Qwen3-8B Q4_K_M prompts 2385 -> 1595 t/s (500 tokens), 1994 -> 1665 (2000) (opencl.md).
- Training through rocBLAS
E2 (moved here 2026-11-11, done that day): the on-device gated-delta
block. Inference: oklm_ssm_conv() and
oklm_gated_delta() on the CPU and OpenCL, the device
layers' states in device memory (hybrid-layers-design.md).
Training: the tr_gd_* kernels, the recurrence's backward
from states kept every 16 tokens; a 4-block 512-channel hybrid step 2
626 -> 43.9 ms (training.md). Its scoping,
as the roadmap held it:
- E2. (Done 2026-11-11, inference and training; the
scoping kept for its history.) A native on-device gated-delta kernel
(the hybrid layers' recurrence runs on the host in inference and
training; speed only). Assessed (2026-11-08), not started: more than the
recurrence leaves the device. Training (
k_gated_delta_fwd/_bwd) downloads each linear block's input and output gradient and runs the whole block on the host, its projections included; inference (lin_blockinarch_llama.c) does the same, with the per-sequence state and conv ring in host memory (lin_state, whichoklm_kv_savewrites). On the device it takes: the projections through the existing products; kernels for the causal conv with SiLU, the per-head L2 norm, the recurrence (one work group per value head, the dk x dv state in device memory, the tokens in order) and the gated RMSNorm, forward and backward (the recurrence's backward walks the tokens in reverse with the states recomputed per chunk,oklm_gated_delta_recurrent_backward); the state and ring as device buffers per sequence, read back foroklm_kv_save; checks against the host ontiny-qwen3nextand the trainer's hybrid fixture.
F. Inference
--n-cpu-moe N(llama.cpp's flag of that name): the expert matrices (ffn_*_exps) of the first N layers kept in host memory and run on the CPU, while attention, the router and the shared experts stay on the device.--gpu-layerssplits whole layers only, so a MoE model larger than the GPU leaves its attention on the CPU too; experts are most of a MoE model's weights but each token reads a few, so they are what belongs in host memory (here 128 GB beside 20 GB of VRAM). Needs weights placed per tensor rather than per layer, and a layer whose graph crosses devices: the router on the device, the token rows and the chosen experts handed to the CPU expert kernels (moe_cpu.c), the sum handed back. Onrun,chat,pplandserve; checked equal to a device-only run on the tiny MoE models ofmake test, measured on a real one (Qwen3-30B-A3B). Larger than the items above: a day or more. Done (2026-11-07):--n-cpu-moe N|all(run, chat, ppl, serve, plan;oklm_context_params::n_cpu_moe,oklm_n_cpu_moe()): the device copy of such a layer leaves out its three expert stacks (their headers stay the mapped tensors) and the automatic layer fit counts the slimmer layers; inmoe_ffn()the router runs on the device, the chosen experts, weights and normalized rows are read back, the shared expert is queued and flushed (a new optionalflushbackend entry) to run while the CPU computes the routed experts by the grouped indexed products and their weighted sum, written back once and added there. The expert cache composes (on a device it covers the layers whose experts the CPU computes);oklm plancounts those experts on the host. Checked on the eight tiny MoE fixtures (references, device-only greedy tokens, two device instances, the cache bit for bit) inmake test. Granite-3.1-1B-A400M measured in moe.md: the crossing costs there (the model fits, its experts are small). Left: the measurement on a MoE larger than the 20 GB GPU (none inmodels/), one read for the routing and the rows instead of three, and streamed CPU layers (--gpu-stream) still carry their experts to the device. Done (2026-11-11): one round trip for the three reads (the backend's optionaldownload_many; Granite decoding 72.2 -> 72.8 t/s, within the spread on an idle bus), andcl_download_locked()reports its read's errors (it dropped them). Measured and not kept: streamed layers leaving their experts in host memory within--n-cpu-moe(slots 33.1 -> 1.9 MiB, but prompts 4 767 -> 2 855 t/s: copying a layer's experts for a large batch beats the CPU) (moe.md). Left: the measurement on a MoE larger than the GPU.- CPU decoding that holds up on a busy machine. Three
measures on 2026-11-07, the machine loaded by other jobs (load 30 to 35
on 16 threads): GLM-4.7-Flash decoded at 1.2 t/s with 8 threads where
llama.cpp kept 9.5 (OKLM 10.4 when the machine was quieter, 3.4 with 4
threads under the same load), and Granite-1B-A400M fell from about 40 to
1.6-10 t/s at 8 threads with or without the expert cache. llama.cpp's
threads degrade gracefully; OKLM's thread team looks to spin or sleep
badly when it does not own its cores (a step waits on the slowest
worker, descheduled). Find the cause (busy-wait barriers, the sleep /
wake protocol, static slicing against preempted workers), make a step's
work dynamic or the barrier yield, and measure against llama.cpp under a
synthetic load (
stress-ng --cpu 16beside both). Done (2026-11-08): the cause was nilorea's team, a fixed slice per thread and every thread checked in at the end of every loop: with one thread off its core, each of the few hundred loops of a token waited a time slice (Granite at 16 threads 2 to 11 t/s with no spinner, 72 at 8). The CPU backend now has its own team (team_cpu.c): chunks, four per thread, claimed from one atomic counter, a loop done when its chunks are, a short poll thensched_yield()rounds before sleeping, and decode matrix-vector products cut in four tiles per thread. Same bits (each element computed by the same code, whoever runs it); checked bytest_threadswith a participant asleep mid-loop and with four times more threads than cores beside spinning threads. Beside 8 spinning processes, 8 threads: Granite 1.9 to 54 t/s (llama.cpp 18), Llama-3.2-1B 3.1 to 24 (llama.cpp 12); every thread count now degrades gently (benchmarks.md). Left: llama.cpp stays ahead at 2 to 4 threads under load; OKLM still runs one loop, and so one wait, per op (three for decode attention), and fewer loops per token (fused ops) are the next lever; not re-measured on GLM-4.7-Flash and OLMoE; no thread cap from the cores actually free. Done (2026-11-11): a row loop's slices take at least 16384 values (a decode step's rotation and cache copies on the calling thread, about a third of a Llama-3.2-1B token's loops), and a loop takes no more participants than the cores other programs leave free (their runnable threads,/proc/stat'sprocs_runningless OKLM's running threads, every 100 ms;OKLM_CPU_FREE_CAP=0for none), the kernels' layout and splits going by the threads available (oklm_cpu_threads_now()). Beside 16 spinners: Llama-3.2-1B at 4 / 8 / 16 threads 10.7 / 17.0 / 13.1 -> 19.5 / 19.7 / 19.6 t/s (llama.cpp 14.7 / 1.5 / -), Granite 37.2 / 38.2 / 47.3 -> 43.7 / 41.1 / 49.2; quiet and 8 spinners unchanged; Granite at 2 threads 51.4 -> 44.7 (two threads beat the one the cap leaves there) (benchmarks.md). Left: GLM-4.7-Flash and OLMoE re-measured.
G. GLM models
(Zhipu AI / zai-org; run by OKLM itself, not through a remote API).
GLM-4.7-Flash runs (item 1, checked against llama.cpp on the published
file); glm4moe and glm4 run (items 2 and 3,
tiny fixtures only); the GLM-4 pre-tokenizer preset exists
(src/tok/pre_presets.inc). What llama.cpp (the pinned
v0.8.0) implements, in the order to work it:
- GLM-4.7-Flash (30B-A3B,
Glm4MoeLiteForCausalLM): llama.cpp writes it asdeepseek2(multi-head latent attention, sigmoid routing), which OKLM runs, with the GLM vocabulary and one MTP block. Likely little code: load a published GGUF, check its tokenizer (theglm4pre-tokenizer, the[gMASK]<sop>and role tokens) and greedy output against llama.cpp, its MTP block as the draft (--mtp). About 18 GB at Q4_K_M: fits the 20 GB GPU. Done (2026-11-07), on the CPU with unsloth's Q4_K_M (sha25629837ed2...b174): it loaded and ran with no model code change. The tokenizer (theglm4preset) gives llama.cpp's ids on 435 strings (code, Chinese and other scripts, emoji, the role,<think>and tool tokens), with and without special-token parsing and BOS (none is added:[gMASK]<sop>comes from the template); the tool tags (<tool_call>,<arg_key>, ...) are user-defined tokens, not control ones, so they never end a turn. The file's template is the fixturetests/fixtures/jinja/glm-4.7-flash.jinjabyte for byte. Greedy: 32 / 32 tokens identical to llama.cpp on four prompts (chat in English, code, Chinese; a raw continuation), a fifth parts at its first step where llama.cpp's own top two are 0.11 apart. Two fixes: ggml-org's file has noexpert_gating_func, and llama.cpp then picks sigmoid from the shape (47 or 48 layers, 154880 tokens), which OKLM now does too (softmax before: wrong routing);oklm pplstarted each window with the tokenizer's BOS even when it adds none, which for GLM put[gMASK]there and gave another perplexity thanllama-perplexity(26.66 against 43.28; now 44.42, chunks 1 to 5 within 0.7 %; the last chunk, ppl about 400, is one where the model, without[gMASK]<sop>at the start, writes logits near 150, and llama.cpp moves by 0.9 % with flash attention alone). Per token, OKLM's log-likelihoods differ from llama.cpp's by 0.10 on average, as llama.cpp's own differ between its two attention paths. A tool call withoklm chat --tools(formatglm45recognized from the template): one call with both arguments parsed, the result answered in the observation turn. No published GGUF (unsloth, bartowski, ggml-org) carries the MTP block (made before llama.cpp exported it), so--mtpis left to check: it says the file has none. 18.8 GB resident; speeds (CPU, 8 threads) in docs/compatibility.md: on a machine loaded by other jobs OKLM's decode fell to 1.2 t/s where llama.cpp kept 9.5 (10.4 for OKLM on a lighter load), which is left to look into. glm4moe(GLM-4.5, GLM-4.5-Air, GLM-4.6, GLM-4.7): DeepSeek-V3's MoE (sigmoid routing with the selection biasexp_probs_b, leading dense layers, shared experts, routed scaling, normalized top-k), all in OKLM already, on ordinary grouped-query attention with QKV biases, optional Q/K RMS norms and rotary on half of each head (NeoX order); one MTP (nextn) block, as the speculative draft. A tiny randomglm4moemodel against a float64 reference and llama.cpp, Hugging Face checkpoints loaded and converted (experts stacked). Real files: GLM-4.5-Air (106B-A12B, about 65 GB at Q4) needs--n-cpu-moe(F1) to run usefully here; GLM-4.6 and 4.7 (355B) do not fit this machine even at 2 bits. Context: 128K (4.5, 4.7), 200K (4.6). Done (2026-11-07):glm4moeGGUF files load with nothing new in the forward pass: the attention is the Llama path (QKV biases, Q/K norms when the file has them, NeoX rotation ofrope.dimension_countdimensions, half of each head), the MoE is DeepSeek V3's (sigmoid by default, theexp_probs_bselection bias,leading_dense_block_countdense layers,expert_shared_countshared experts as one FFN, the weights renormalized asexpert_weights_normsays and scaled byexpert_weights_scale); the file'spost_attention_norm, which is the norm before the FFN in this family (no norm after attention), is bound as the FFN norm. Thenextnblock past the main layers is the speculative draft as DeepSeek's (--mtp, also in a file of its own). Tiny random models (Q/K norms and a block; neither) within 3e-5 of the float64 reference on the CPU, the block's logits within 6e-6, its echo rewrite keeping every greedy proposal; llama.cpp (CPU, F32 cache) on the same files within 1.4e-5 of that reference, every greedy choice the same.Glm4MoeForCausalLMcheckpoints load directly and convert (per-expert matrices stacked,e_score_correction_bias,shared_experts, thenextnlayer kept when the checkpoint holds it); a checkpoint made from the tiny fixture gives its logits exactly, and its OKLM conversion runs in llama.cpp with the same logits. Left: a real file (GLM-4.5-Air with--n-cpu-moe, the smallest), and the OpenCL variants of the tests, written but not run here. Done (2026-11-11): the OpenCL variants run inmake test(tiny-glm4moeandtiny-glm4moe-noqk, every layer and one layer on the device, the experts on the CPU: every greedy choice the reference's). Left: a real file.glm4(dense GLM-4-0414 9B / 32B, GLM-Z1): sandwich norms (after attention and after the FFN as well as before), gate and up fused in oneffn_upof twice the width, QKV biases, partial rotary. A tiny fixture; GLM-4-9B-0414 or GLM-Z1-9B checked against llama.cpp. Done (2026-11-07):glm4runs on the existing paths: the norms after attention and after the FFN (post_attention_norm,post_ffw_norm) as Gemma 2's, the fusedffn_upsplit rows gate | up for SwiGLU (llama.cpp'sggml_swigluorder), QKV biases, normal-pair rotation of half of each head (llama.cpp's rope type for GLM4 without M-RoPE). A tiny random model within 6e-6 of the float64 reference on the CPU; llama.cpp (CPU, F32 cache) within 1e-5 of it, every greedy choice the same.Glm4ForCausalLMcheckpoints (post_self_attn_layernorm,post_mlp_layernorm,gate_up_proj, Q / K rows kept as stored, as llama.cpp does) load directly and convert, checked as forglm4moe. Left: a real file (GLM-4-9B-0414 or GLM-Z1-9B), and the OpenCL variants of the tests, written but not run here. Done (2026-11-11): GLM-4-9B-0414 Q4_K_M (bartowski) intest_infer_public: tokenization identical to llama.cpp's, 157 / 160 greedy choices on its tokens on the CPU and 158 / 160 on OpenCL (logits within 0.97 / 0.85), perplexity 23.78 against 23.81; 84 / 4.7 t/s prompt / decode on the CPU against llama.cpp's 60 / 4.8, 2 074 / 61 on OpenCL; the OpenCL variants of the tiny tests run inmake test(compatibility.md). Not done: its tool calls, which follow GLM-4's own convention (the function's name after<|assistant|>, JSON arguments, results in an<|observation|>turn the template reads from anobservationrole, calls from ametadatafield): a fifth format with its own message shaping, which llama.cpp does not have either. Done (2026-11-11): theglm4format (--tool-format glm4, recognized from a template with<|observation|>andmetadata): a turn opening with a tool's name, a newline and JSON arguments is a call, a turn opening with the newline text; the conversation handed to the template in its own form (each call an assistant turn with the name inmetadata, each result anobservationturn); its grammar a tool's name, a newline and the JSON of its parameters; the template rendered identical to jinja2 (its tools throughtojson(indent=4)). Throughoklm chat --tools, two calls and their results. The agent evals throughoklm serve(OpenCL, 16K context): 3 of 25, its 26 calls all parsed, none left as text at a turn's start; most turns end on a plan written in prose (once with the calls inside a code fence), so the model, not the format, is what fails.- The chat side. The GLM-4.5 template rendered by the
Jinja interpreter identical to jinja2; a
glm45tool-call format beside hermes / llama3 / mistral (<tool_call>NAMEthen<arg_key>K</arg_key><arg_value>V</arg_value>pairs, a value JSON when it parses, else a string; its grammar for constrained calls; results in the template's observation turn), parsed inoklm chat, the agent andoklm servealike. Thinking already followsenable_thinking(oklm chat,/effort); GLM-4.7's preserved thinking is the template variableclear_thinking: falseplus earlier turns' reasoning kept in the history: check whether the agent drops it. Unit tests: rendering, the parse of calls and reasoning, a constrained call end to end on the tiny model. Done (2026-11-07): theglm45format (--tool-format glm45, recognized from a template writing<arg_key>beside<tool_call>), checked against llama.cpp's GLM-4.6 and GLM-4.7-Flash templates and its parser tests:<tool_call>NAME, then<arg_key>K</arg_key><arg_value>V</arg_value>per argument and</tool_call>, a newline between the parts for GLM-4.5 / 4.6, none for GLM-4.7; several calls a turn, a value JSON when it parses as a number, boolean, null, object or array, else the raw string; calls cut short kept as the other formats keep them, an invented<|observation|>turn dropped. Its grammar (trigger<tool_call>): the tools' names, each tool's property names as keys, required ones first in schema order, strings raw up to</arg_value>, other values the JSON of their schema, enums as listed; a forced call on a tiny random model parses back, intest_grammarand throughoklm serve. The GLM-4.5 / 4.6 and GLM-4.7 templates render identically to jinja2 (system prompt, tools, calls, two results in one observation turn, thinking default, on, off andclear_thinking: false); no interpreter change was needed. Preserved thinking works with a local model: the agent keeps each turn's text with its reasoning (the</think>the template splits on included), and--template-vars '{"clear_thinking": false}'reaches the template (/effortsets or removes onlyenable_thinking, the other variables kept). Over an HTTP backend too (2026-11-07): the template variables go to the server aschat_template_kwargs(so/effortworks with--url), each turn's reasoning is kept apart from its answer and sent back asreasoning_contenton its assistant message whenclear_thinkingis false (orpreserve_thinkingtrue, or thereasoning.keepInHistorysetting says so), andoklm servegives that reasoning back to a template that reads reasoning as a<think>block before the content; checked against a recording server and throughoklm servewith a tiny model whose template renders earlier reasoning only underclear_thinking: false. Left: a real GLM model (items 1 to 3) to check the whole loop end to end (the type its GGUF gives<arg_key>and the other tags: as control tokens they would end a turn, and a grammar allows those only where it can end), the keys' order the model prefers against the grammar's schema order, and a string argument whose text is itself JSON ("42") read as JSON, the parser not knowing the schema. Done (2026-11-11), with GLM-4.7-Flash: its tool tags are user-defined tokens (<|user|>and<|observation|>the control ones), so they never end a turn; free, it writes the keys in the grammar's order (the required ones first,code,priority, thencount, though the schema listscountfirst); the parser now takes the tools (oklm_tool_calls_parse_tools(), used byoklm chat, the agent andoklm serve) and keeps a string property's value a string (<arg_value>42</arg_value>gives"42", free and constrained, throughoklm serve); the agent evals throughoklm serve: 21 of 25, its 163 calls all parsed (agent-loop.md). - (Open, scoped.) GLM-5 series, two architectures,
both far heavier than the above (sizes from their configs; experts are
most of them):
glm5-next, GLM-5.3-Flash (about 305B, 13B active: 45 layers, 288 experts of 2048, 8 a token, 1M context, a vision tower): hybrid Kimi Delta Attention linear layers (a short causal conv, a per-channel gated delta rule: a variant of the Gated DeltaNet kernels OKLM has) with multi-head latent attention without rotary on the full layers, a DeepSeek sparse-attention indexer with k-pool compression (top 2048), manifold hyper-connections (several residual streams mixed by Sinkhorn-normalized matrices) and DeepSeek MoE. Each is new to OKLM; together about the size of M14. About 175 GB at Q4: over 128 GB RAM and 20 GB VRAM together; at 2 to 3 bits it fits only with--n-cpu-moe(F1).glm-dsa, GLM-5 / 5.1 / 5.2 (about 745B, 78 layers, 256 experts):deepseek2plus the sparse-attention indexer (DeepSeek V3.2's), full on some layers, and an MTP block. Beyond this machine at any precision; the indexer is the shared piece, worth it only withglm5-next.
- (Open, scoped.) Later, optional:
chatglm(ChatGLM3, the first GLM-4-9B-chat), the GLM-4.5V / 4.6V vision projector (glm4v).
H. MoE experts streamed from disk
(after colibrì's "colibri mode", github.com/JustVugg/colibri): run a
MoE model larger than RAM on the CPU, the dense part resident and the
routed experts read from disk as the router picks them. Complements F1
(experts in RAM beside the device): H is for what does not fit RAM
either (GLM-4.6 / 4.7, GLM-5.3-Flash at Q4, Qwen3-235B). What exists:
weights are memory-mapped (src/io/file.c,
MADV_WILLNEED), so such a model already runs, paged blindly
by the kernel (the baseline to beat); quantized integer kernels with a
measured dispatch (oklm tune) cover colibrì's int8 / int4
containers; MTP and draft speculation exist; the server has paged KV
slots, prefix caching and swap; Windows builds; token-exact checks
against llama.cpp. What is missing, cheapest first, each behind a flag
and off by default:
- Batch-union expert products.
oklm_cpu_matmul_idcomputes row by row, so a batch reads an expert once per token routed to it: group the rows by expert, each expert's weights read once for all its rows (prefill and speculative verification). Faster in RAM too; checked bit-identical to the row-by-row product. Done (2026-11-07): rows sorted by expert, items of (expert, weight tile, up to 64 rows) claimed one at a time, four- and eight-row kernels; bit for bit against one dot product per element (6 types, skewed / single / every-expert routings, 1 and 6 threads). Qwen3-30B-A3B shape, Q4_K, 8 threads: 16 tokens 1.2x, 128 tokens 2.7-3.0x, 512 tokens 3.4-3.8x, decode as fast (benchmarks.md). Prefill on this path too (2026-11-07): on the CPU every MoE batch runs the indexed products (a device keeps them for n x k <= n_expert and per-expert products above), the same logits bit for bit; Granite-3.1-1B-A400M Q4_K_M prompts 1.38x (128 tokens), 1.33x (512), 1.16x (2048), the Qwen3-30B-A3B-shaped expert block 1.22-1.45x. - RAM-safe budgeting and a plan. Available memory
read (
MemAvailable,GlobalMemoryStatusEx,host_statistics64); the peak projected from the headers (resident weights, the expert cache, KV, draft rows, scratch) before anything loads;oklm plan -m MODEL(besideoklm info) says what stays resident, what streams and the largest safe cache, so the OOM killer never decides. Done (2026-11-07):src/io/sys.c(available / total RAM, free disk space; Linux, Windows, macOS);oklm_context_mem_estimate()sizes the cache, recurrent state, buffers, scratch and repacked copies with the functionsoklm_context_new()allocates with (equal on every tiny model,test_mem_plan);oklm plan(headers only, no read-ahead): weights by kind, host and device totals against free memory, verdict, the dense part with the routed experts streamed and the largest expert cache;--json;run/chat/servewarn before the context. - Expert residency. The dense tensors (attention,
shared experts, embeddings, norms) resident; routed experts read into a
per-layer LRU cache of aligned buffers, sized by 2;
--expert-cache GB. One read per expert where its matrices are adjacent: a GGUF stacks gate, up and down apart, so three reads, or a repacked file with each expert's three together (colibrì reads onepreadan expert).O_DIRECTas an option (colibrì: often a large win on fast NVMe, neutral or worse on others), and reads spread over a mirror on a second drive (colibrì: +37.5 % decode with two NVMe on independent controllers). Routing counts recorded per model (a usage file beside the cache) and the most used experts pinned (--pin-gb, an automatic pin from the counts);--expert-statsprints hit rates. Checked: the same tokens as the resident run on the tiny MoE models and on Qwen3-30B-A3B with a cache forced small. Done (2026-11-07):--expert-cache GB|auto(run, chat, ppl, serve;oklm_context_params::expert_cache, CPU): per MoE layer an LRU set of aligned slots, each one expert's three matrices read by threepreads; the routing known on the host after the router, the batch's experts run in groups that fit the slots (resident first, the missing read in parallel into evicted slots), each group's indexed products reading a table expert -> slot (oklm_matmul_id_ex()), so fewer slots than a batch needs cut the batch by expert. The model loads without read-ahead and the experts' mapped pages are dropped. Usage counts in~/.cache/oklm/usage/<hash>.json(--expert-usage),--pin-gbpins the most routed,--expert-direct(O_DIRECT),--expert-statsand/api/stats. Bit for bit against the resident run on the six tiny MoE fixtures (one slot per layer, half the experts, pins). Granite-3.1-1B-A400M measured in moe.md. Left: the repacked file with each expert's three matrices adjacent (one read), reads spread over a mirror drive, the check on Qwen3-30B-A3B, the cache counted byoklm_context_mem_estimate(); read-ahead and prediction are item 4, the device tier item 5. Done (2026-11-11):oklm expert-packwrites the routed experts with each expert's three matrices in one aligned record and--expert-pack FILEreads a miss in one read (Granite, direct reads, 10 % cache: decode 4.2 -> 9.6 t/s, 5.8 -> 12.9 with four loaders; 25 %: 6.3 -> 16.4);--expert-mirror FILEsends every other read to a copy on another drive (not measured: one NVMe here); the cache inoklm_context_mem_estimate()(expert_cache_bytes, equal to the context's with pins, direct reads and a pack) and inoklm plan's need. Left: the check on Qwen3-30B-A3B (moe.md). - Read-ahead. A pool of loader threads reading the
missing experts while the resident ones compute, and announcing the next
layer's (
posix_fadvise/WILLNEED, or reads into free slots); optionally the next layer's routing predicted from the current post-attention state (router look-ahead: colibrì measured GLM-5.2's routing 71.6 % predictable a layer ahead), kept only if measured faster. Done (2026-11-07):--expert-loaders N(default 4 with the cache, 0 the inline reads;oklm_context_params::expert_loaders): loader threads read the missing experts, one job per matrix, while the resident ones compute; a layer consumes its slots as they fill (the next round waits for the reads in flight unless experts wait for a slot), each output row still written once by the same kernel.--expert-predict last|router: the next MoE layer's predicted experts read ahead into its free or LRU slots, behind the layer's own reads (reads into slots rather thanposix_fadvise: the cache reads withpread,O_DIRECTbypassing the page cache).last(the experts the layer chose in the last batch) is right 41-45 % of the time on Granite-3.1-1B-A400M and reads little, the LRU slots holding them; the router look-ahead (the next router on this layer's FFN input, rescaled between the two norms) recalls 78-85 % at 50 / 25 % caches, 90 % of its reads used, hit rate 65 -> 87 % and 37 -> 75 %. Bit for bit against the resident run on the nine tiny MoE fixtures (one slot, half and every expert, 0 to 3 loaders, both predictions,test_model). Measured with--expert-directin moe.md: the forward pass waits 45 % (2 loaders) to 60-70 % (4) less for reads; decode with 4 loaders and the router look-ahead 1.5x (50 % cache) and 1.4x (25 %) the inline reads (2 loaders alone about equal to them), hence 4 loaders by default. OLMoE-1B-7B: the look-ahead recalls 74-79 %, hit rate 71 -> 85 % (50 %) and 47 -> 76 % (25 %), decode 10.8 -> 13.9 t/s at 50 % with 4 loaders. The read-ahead is off by default (--expert-predict). The same loaders and predictor serve the--n-cpu-moelayers of a device context (the routing read back to the host). Done (2026-11-08): the router look-ahead on those device layers (its scratchhb.probs, n_expert x the batch, now in the host set ofalloc_moe_host(), counted by the memory projection); on the gfx1100 with--n-cpu-moe alland half the experts cached, the tiny two-MoE-layer fixtures read 19-34 experts ahead, 40-74 % of them used, the logits bit for bit those of the cache without it (test_model). Left: a model larger than RAM; the next router applied on the device rather than on the host. Done (2026-11-11): on a device layer whose experts the CPU computes, the look-ahead's rescale (the ratio of the two norms' weights, uploaded once) and the next router run on the device and the logits come back in the routing's round trip (model_ecache_ahead_device(),oklm_expert_cache_stats::ahead_device); the same bits, 25 device look-aheads per fixture intest_model; Granite prompts 513 -> 526 t/s with--n-cpu-moe alland a 25 % cache, decode unchanged (bound by the reads) (moe.md). Left: a model larger than RAM. - A device expert tier. The hottest experts of every
layer (from the routing counts of 3) held in GPU memory, the rest in RAM
or on disk; in each step the device computes the experts it holds while
the CPU computes the misses, the sum joined. Generalizes F1 (a split by
measured heat, not by layer); refilled as the counts change. Done
(2026-11-07):
--expert-gpu GB|auto(run, chat, ppl, serve;oklm_context_params::expert_gpu,src/model/expert_gpu.c): per layer whose experts are in host memory (--n-cpu-moe, all of them when it is not given) device stacks of expert slots and a slot of zeros, filled from the usage file's counts (written back by the tier without an expert cache) or from the first steps' routing; the read-back routing mapped to slots, the device's indexed products over the held pairs (one product per slot for long batches) beside the CPU's grouped products over the others (a table without the held experts, or the expert cache told to skip them, its loaders and read-ahead included), the sums joined on the device; a layer holding every expert runs as a device layer; refills every 32 steps or when the hit rate falls, at most an eighth of the slots, queued after the layer's work;--expert-stats,/api/stats. The device-only greedy tokens on the nine tiny MoE fixtures (no expert, half, all, half from a usage file, half with the expert cache; logits within 1e-5). Simulated 8 GiB device, GLM-4.7-Flash: prompt 93.5 / decode 12.5 t/s against 60.8 / 4.9 for--n-cpu-moe alland 36.5 / 6.8 for--gpu-layers(63 % hits, 31 % of the experts held); OLMoE and Granite in moe.md. Left: the mapped ids computed on the device (one blocking write less per layer), uploads on the transfer queue, prompts with misses streamed to the device rather than run on the CPU, a per-layer share by heat rather than equal. Done (2026-11-11): batches of 32 tokens or more run every pair on the device, the missing experts copied through four staging slots (Granite with half the experts held: prompts 1 063 -> 2 060 t/s; OLMoE 466 -> 512), and refills copy on the transfer queue (decode +1 to 2 %). Measured and not done: the mapped ids on the device (their write is 12 us a layer, at most 3.5 % of a Granite decode step), and shares by heat (under a point of hit rate on Granite and OLMoE's routing counts) (moe.md). - Speculation under streaming. The MTP / draft path
and prompt lookup exist; add a guard that rests speculation while the
expert cache is cold (verification reads more experts; colibrì measured
MTP 32 % slower around an 85 % hit rate), keep the draft head in 8 bits
(4-bit heads collapsed acceptance to 0 to 4 %), and measure acceptance
against the hit rate. Grammar-forced drafts: when the grammar allows a
single continuation (JSON keys, a tool call's fixed text), its tokens
taken without sampling, verified in one batch. Done (2026-11-07): the
guard,
--spec-guard RATE(run, serve;oklm_speculative_set_guard(),oklm_server_params::spec_guard): no proposals while the expert cache's recent hit rate (oklm_expert_cache_recent_hit_rate(), about the last 8 forward passes) is below RATE. Granite-3.1-1B-A400M, direct reads, prompt lookup (--spec-ngram K, now inoklm runtoo,oklm_speculative_new_lookup()): at a 10 % cache lookup reads 20 % more and decodes 3.9 -> 3.5 t/s, the guard rests every round (4.1-4.3 t/s); warm (100 %) it rests only the first rounds (61.6 -> 72 t/s, lookup kept 26 % here). Grammar-forced drafts,--spec-grammar(oklm_grammar_forced_text()/oklm_grammar_forced_tokens(), with any draft or alone,oklm_speculative_new_grammar()): the text the grammar forces, tokenized, proposed and verified by the normal path, so the output is that of plain generation (checked greedy and sampled on a tiny model with a tokenizer, a schema and a tool call,test_model). Qwen2.5-0.5B: a tool call 41 -> 141 t/s (39 of 42 forced tokens kept); a JSON schema gains within the noise (its optional spaces cut the forced runs short). Left: the 8-bit draft head (no MTP model fits here), acceptance against the hit rate with an MTP / draft model on a large MoE. Forced drafts in the server (2026-11-08):oklm serve --spec-grammar(oklm_server_params::spec_grammar): while a request's grammar applies (a JSON schema, tools once the call has begun) the forced text's tokens are the step's drafts, verified by the server's own path, the guard resting only guessed drafts;spec_forced/spec_forced_acceptedin/api/stats. The answers equal plain generation's (a schema greedy and sampled, a tool call, with and without prompt lookup,test_server). Qwen2.5-0.5B, CPU, 4 threads, through the server (prompt lookup on, as by default; decode time of/api/profile, median of 5): a tool call (40 tokens) 33.3 -> 54.4 t/s (1.6x), a JSON schema (37 tokens) 33.5 -> 65.2 t/s (1.9x), 108 of 192 forced tokens kept (server.md). - KV persistence and parking. A conversation's KV
cache saved incrementally and crash-safely (written to a temporary file
and renamed, or appended per turn) so
oklm chat --continueand agent sessions reopen warm without prefilling again; the server's prefix cache optionally kept on disk. Checked: the next tokens identical with and without a reload. A short side request (an agent's title or classifier aside) parks the conversation's cache rather than evicting it (colibrì: an agent's second turn of 14,810 tokens, 86 s to 4.9 s); check what the server's prefix cache already keeps, and the in-process agent. Done (2026-11-07):oklm_kv_save()/oklm_kv_load()(src/model/kv_file.c): a header (model identity, cache types and row sizes per layer, adapter) and checksummed chunks of token ids and rows (MLA's latent rows, a hybrid stack's recurrent state), contiguous or paged, a device cache read back; whole files written topath.tmp, synced and renamed, appends of one chunk per turn from where the tokens differ, a torn append ignored, compaction when unused rows outweigh the used ones (inference.md). The next logits after a load equal those of the context that saved, bit for bit (test_kv_file).oklm chatkeeps its conversation for--continue,--kv-saveits cache: the continued turn evaluates only the new message.oklm agent --kv-savekeeps the cache beside the session's transcript, and a resumed session's first turn loads it. Parking: the server's prefix cache already kept a conversation's full blocks past a side request, evicting a prefix from its end under pressure; the contiguous cache lost the conversation's sequence to the side request (1 token of 897 found again), fixed by choosing the sequence that discards the fewest rows (server.md). The in-process agent's asides keep the main cache (with the conversation, or on the small model's context). Server's prefix cache on disk done (2026-11-08):oklm serve --prefix-dir DIR(--prefix-min,--prefix-disk-mb;oklm_server_params::prefix_dir,src/server/prefix_disk.c): a finished request's full blocks are written withoklm_kv_save()(one file per distinct head, a file a longer head extends deleted, the oldest deleted past the size budget) and loaded into the paged cache at start, oldest first, so a restarted server finds a repeated long prompt cached. The same greedy answer with the directory, after the reload and without it (test_prefix_disk, which also checks the size budget and that the contiguous cache is refused). Qwen2.5-0.5B Q4_K_M, CPU, a 2174-token prompt: time to first token 1.85 s cold, 0.026 s after a restart (2112 tokens cached, 26 MB file) (server.md). - The platform layer. The file layer already maps
files on POSIX and Windows; it gains
pread, aligned allocation, the advice calls, available RAM and an atomic rename, so the streaming code has no#ifdef; a macOS build checked. Done (2026-11-08): most of it existed (oklm_file_pread()with unbuffered reads,oklm_file_advise_dontneed(),oklm_file_replace(),oklm_sys_mem_available()); addedoklm_file_advise_willneed(),oklm_file_prefetch()(posix_fadvise),oklm_file_alloc_aligned()/oklm_file_free_aligned()andoklm_file_tmp_name(). The expert cache's usage file and the tuner's cache files now go throughoklm_file_make_parent_dirs(),oklm_file_tmp_name()andoklm_file_replace()(the tuner's Windows remove-then-rename gap is gone:MoveFileExreplaces in one call), soexpert_cache.c,kv_file.candtune.ccarry no platform branch for files. Unit test intest_sys(aligned allocation, short and direct positioned reads, the advice calls, a replace over an existing file); the MinGW build compiles the library andtest_sys.exe(make MINGW=1); not run under wine here. macOS: no machine, so its branches (F_NOCACHE,host_statistics64,sysctl) are written to the documented API only and the code stays POSIX-portable (aligned_alloc,pread,posix_fadviseguarded byPOSIX_FADV_WILLNEED). - Tuning, measures and docs.
oklm tuneextended to the streaming settings (cache size,O_DIRECT, loaders, read-ahead) and saved per machine. Tuning done (2026-11-08):oklm tune --expert-cache GB|auto(oklm_tune_stream()): the loaders (0, 2, 4, 8), the read-ahead (off, last, router) andO_DIRECT, one at a time from the defaults, timed over a 256-token prompt and 64 greedy tokens with that cache, kept only with every logit equal to the defaults' run; cached beside the kernel tuning per model, cache size, CPU and disk (device number and file system type), applied by contexts to the flags not given (oklm_tune_stream_resolve(); the cache size itself is the user's, the key), the winner timed against the defaults again with their runs interleaved. Granite-3.1-1B-A400M and OLMoE-1B-7B at 50 % and 25 % caches, 4 threads: the defaults kept on all four (the page cache holds the files here, soO_DIRECTis 1.3 to 3.6x slower and the loaders and read-ahead hide little; the interleaved check rejected two noise wins of the search on the loaded machine) (moe.md). Measures done (2026-11-08): the table of tokens/s, hit rate, bytes read and bottleneck for OLMoE-1B-7B and GLM-4.7-Flash under plain mapping, the cache (25 and 50 %), pinning, read-ahead and the device tier, on this machine's NVMe, a model larger than RAM emulated by direct reads with a cache of a quarter of the experts (no model above 124 GiB at hand, the page cache not droppable without root): OLMoE decode 33.9 t/s mapped, 5.5 cached (disk-bound), 6.1 pinned with read-ahead (hit rate 45 to 80 %); GLM 10.4 mapped, 6.3 cached, 17.1 with the device tier; the output identical in every mode (moe.md). Item 9 done.
The guardrail throughout is the one colibrì used: token-exactness (teacher-forced N of N and greedy N of N against the resident run and llama.cpp) before any speed is claimed. Left out: NUMA placement (one socket here) and colibrì's cluster mode (experts on other machines).
I. Beyond streaming: colibrì's other features
Missing here when the list was made:
- An Anthropic-compatible API in
oklm serve:/v1/messages(streamed, tool use, thinking blocks), so the Anthropic SDKs and the coding CLIs built on that API work against it, beside the OpenAI one. Done (2026-11-07):/v1/messagesand/v1/messages/count_tokenstranslated into the chat path (template, sampling, tool grammar, batching, prefix cache);x-api-key; thinking, text andtool_useblocks, whole and as Anthropic's named events; its error shape, 529 when full;/v1/modelsreadable by both. Not enforced: the thinking budget (server.md). Thinking budget done (2026-11-08):budget_tokenschecked as Anthropic's API checks it (required when enabled, at least 1024, belowmax_tokens); once that many tokens are written inside a<think>block the scheduler takes</think>and a blank line in place of the sampled tokens and the model answers after; exact (the reasoning the firstbudget_tokenstokens of an uncapped answer's), whole and streamed, the closing tokens inoutput_tokens; Qwen3-0.6B closes at 1024 and answers (server.md). - Decisions by scoring (colibrì's System One): a
request with a state and typed questions (a choice among labels, yes /
no, a score) answered with each allowed option's probability and a
confidence, nothing generated; several questions on one document reading
it once (the prefix cache). Logprobs exist; this is the endpoint, the
scoring rules and
/decideinoklm chat. Done (2026-11-07):POST /v1/systemone(and/v1/decide) with choice, yes / no and score questions, each option scored as the continuation of the question's prompt (the sum of its tokens' log-probabilities, then a softmax over the options), the state before the question so the prefix cache evaluates it once for every question and option;/decideinoklm chatthrough the same library function (server.md). - The dashboard: an expert map (which experts fire at
each step, the answer's) and a profiling page (each turn's time by
phase). Done (2026-11-07):
GET /api/experts(per layer and expert, the last decode step's choices, the answer's, the running total) counted in the forward pass where the chosen experts are, while someone looks (within noise on a CPU decode; a device layer reads them back);GET /api/profile(the last 30 requests' queue, prompt, decode and tail times, sampling and detokenizing within, token counts; the decode steps' distribution); the dashboard's tabs: the expert map as a grid, the profiles as stacked bars with their trend (server.md). - Architectures colibrì runs that OKLM does not, each
a model item like G, not audited here: OLMoE (7B, the
small MoE to develop H on), the Qwen3.6 / 3.8 hybrids (Gated DeltaNet
layers: OKLM has them against a synthetic fixture only), DeepSeek V4 /
V4.1 Flash, Kimi K3, MiMo-V2.6, Mistral-Large-4, Inkling. OLMoE done
(2026-11-07):
olmoethrough the Llama forward pass and the existing MoE path (indexed expert products, expert cache); new are only the Q/K norms over the whole projections (oklm_hparams::qk_norm_full) and the hyper-parameters (GGUF andOlmoeForCausalLMconfig.json, weights not renormalized). Tiny random model inmake test(float64 reference 7.8e-6, llama.cpp within its own F16-cache distance; the tiny checkpoint loaded directly and converted, 0 difference). OLMoE-1B-7B-0125-Instruct Q4_K_M: tokenizer identical to llama.cpp's on 400 strings, chat template identical to jinja2, 157 / 160 greedy choices on llama.cpp's tokens (parting only at near ties, margin 0.12), perplexity 2.196 against llama.cpp's 2.202 (oklm ppladding BOS only when the tokenizer does, as llama-perplexity), the expert cache at 50 % / 25 % giving the resident tokens with 68.5 % / 48.2 % hits (compatibility.md, moe.md). Not checked: OpenCL. OpenCL done (2026-11-08): nothing to fix. The tiny model inmake teston the device (all layers, one layer on the CPU, the experts of one or every layer on the CPU, batches against one by one) within 6.6e-6 of the float64 reference; OLMoE-1B-7B Q4_K_M on llama.cpp's tokens 157 / 160 (as the CPU), against the CPU 154 to 158 / 160 with all layers,--gpu-layers 8,--n-cpu-moe 8/all, a 50 % expert cache and--expert-gpu(differing only at near ties, margin at most 0.26), perplexity 2.1958 to 2.2015 against 2.1963 (compatibility.md, inference.md). Left: the Qwen3.6 / 3.8 hybrids on real weights, DeepSeek V4 / V4.1 Flash, Kimi K3, MiMo-V2.6, Mistral-Large-4, Inkling. - A model for this machine:
oklm plan(H2) extended to list models against the machine (resident part plus a minimum expert cache in RAM, the download on disk) and recommend the largest that fits. Done (2026-11-07):oklm plan --list [DIR] [--recommend]projects every GGUF model undermodels/from its headers (split sets once) at 8192 tokens, the need being the resident part plus, for MoE, the experts one token uses x MoE layers x 2; the verdict on the CPU, on the device's layers with--backend opencl, only with the expert cache, or none;--recommendnames the largest that fits resident and the largest that needs the cache;--json(memory-plan.md).
4. OKLM against llama.cpp
Closed on 2026-11-08 after round 9. OKLM is level with or ahead of llama.cpp on every CPU prompt row and on most CPU decode rows, and ahead on most GPU prompt rows; the gaps left (the table at the end of this block) are small or need larger work than they would return, so they are kept as known gaps (listed in "Next", section 2) and not pursued for now.
Once the follow-ups were in and no other job loaded the machine: find why llama.cpp is faster, then make OKLM as fast or faster. Every comparison of 2026-11-07 ran on a machine loaded by other jobs (load 20 to 38 on 16 threads), where OKLM got slower with more threads and llama.cpp did not:
| Decode, loaded machine | OKLM | llama.cpp |
|---|---|---|
| OLMoE-1B-7B Q4_K_M, 4 threads | 13.3 t/s | 20.4 t/s |
| OLMoE-1B-7B Q4_K_M, 8 threads | 5.5 t/s | 22.9 t/s |
| GLM-4.7-Flash Q4_K_M, 8 threads | 1.2 t/s | 9.5 t/s |
(GLM-4.7-Flash: OKLM 10.4 t/s when the machine was quieter.) Prompts showed a smaller gap that did not depend on the load: 28 against 43 t/s on GLM-4.7-Flash, 97 against 110 on OLMoE.
- The measure. OKLM against
llama-benchon the same files (SmolLM2, Qwen3-8B, OLMoE, GLM-4.7-Flash; Q4_K_M and Q8_0), the CPU at 1, 4, 8 and 16 threads, prompt (512) and decode (128) apart, then OpenCL against llama.cpp's Vulkan or HIP build where it has one, each the best of three, nothing else running. - Read it. Level at low thread counts and behind at 8 and more: the thread team (item F2 above). Behind at every count: the kernels (llama.cpp's repacked x86 quantized products, its MoE grouping, long attention), then a profile of the slowest op of a step on both.
- Fix and upgrade until OKLM matches or beats llama.cpp. The goal is not the diagnosis but the result: in every row of the measure (each model, thread count, prompt and decode, CPU and GPU), OKLM at least as fast as llama.cpp, faster where it can be. Work gap by gap, largest first: find its cause with a profile of a step on both, fix or upgrade it (thread team, kernels, repacking, MoE grouping, attention, overlap, whatever the profile shows), measure every row again the same way, keep a change only when the outputs stay as they were (bit-identical where the code promises it, the same greedy tokens otherwise) and no row got slower. Repeat until no row is behind; record each round in benchmarks.md, and say plainly any row where llama.cpp stays ahead and why.
Done so far (2026-11-08, quiet machine; benchmarks.md):
tools/bench_vs_llama.sh and
tests/model/bench_llama.c give the table (llama.cpp's CPU
build and a HIP build made for it); four rounds kept. Decoding with one
or two threads on the packed weights (dense models now 1.07 to 1.18
times llama.cpp at one thread, from 0.64 to 0.86); OpenCL prompts on
larger matrix-unit tiles with the activations rounded once, tiled
attention from 128 keys, one-column product kernels (Qwen3-8B Q4_K_M
prompts 1386 to 2645 t/s with OKLM_CL_WMMA, Q8_0 1504 to
2791, 1.71 times llama.cpp); MoE prompts grouped by expert on the device
(Granite 5585 to 15401, 1.07 times; OLMoE 4175 to 8740). Round 4 (GPU):
quantized prompt products on the 8-bit matrix units by default, the
CPU's products bit for bit (dense prompts 0.76 to 1.68 times llama.cpp
by default, from 0.28 to 0.92; Granite 1.07, OLMoE 0.86), a MoE layer's
routing and Qwen3's query and key norms with their rotation fused into
one launch each (the same bits; Granite decoding 247 to 285 t/s, OLMoE
260 to 271), and a tiled prompt attention for latent attention's heads
(GLM-4.7-Flash prompts 820 to 2156 t/s, 1.05 times llama.cpp). Measured
and left out: a four-row file-layout decode kernel, and the packed copy
for every one-row product. Round 4 (CPU): packed copies of the routed
experts, made when they fit beside the mapping and counted by the memory
plan (OKLM_CPU_PACK_EXPERTS): MoE prompts 1.4 to 1.5 times
faster (OLMoE 56 to 84 t/s at one thread, 1.51 times llama.cpp; Granite
1.37 to 1.40 times), OLMoE decoding at one thread 21.1 to 25.5 (1.10
times); the packed copy for decode steps from three threads measured 2
to 6 % slower and is left out. Round 5 (CPU): one team loop for the
products sharing an input (Q, K, V; gate, up;
oklm_matmul_n()): SmolLM2 decoding 4 to 8 % faster from
four threads (Q4_K_M 204 to 219 t/s at four, llama.cpp 254); the rest of
its gap is the Q5_0 matrices' one-row products and the head, both near
the memory's speed. Round 6 (CPU): the packed Q4_K kernel scales each 16
columns' int16-sized sums with one multiply-add instead of a 32-bit
multiply, unrolled per row count: every CPU prompt row now ahead of
llama.cpp (Llama-3.2-1B 1.02 to 1.31, Qwen3-8B Q4_K_M 1.13 to 1.20,
GLM-4.7-Flash 1.19 to 1.32, the MoE models 1.48 to 1.75), decoding 1.09
to 1.21 at one thread and 0.96 to 1.02 from four, SmolLM2 0.88 to 0.93
(benchmarks.md).
Round 7 (GPU): one-column products of the K-quants (Q4_K, Q5_K, Q6_K,
rows of 2048 values or more) on 8-bit dot products, the activation
column rounded in each work-group as the CPU rounds it: the CPU's
products bit for bit, decoding Qwen3-8B Q4_K_M 84.7 to 92.0 t/s (0.80 to
0.87 of llama.cpp), Llama-3.2-1B 359 to 368 (0.89); measured and left
out: the same for Q8_0 / Q4_0 / Q5_0 and for short rows (slower), and
two ways of loading the next step of the Q4_K prompt tiles while
multiplying (one halved the prompts) (benchmarks.md).
Round 7 (CPU): the packed Q8_0 and Q5_0 copies are the file's size (the
scales as halves, Q5_0's nibbles beside a word of fifth bits per row),
so one-row products read them at every thread count; one-row products
run on half the cores on a packed copy and one thread per core on the
file layout (OKLM_MATVEC_THREADS; fewer readers get more
from the memory); decode attention with few keys runs as one loop:
SmolLM2 Q4_K_M decoding at four threads 219 to 250 t/s (0.99 of
llama.cpp), Qwen3-8B Q8_0 from four threads 1.00 to 1.08 (from 0.96 to
1.01). Round 8 (CPU): packed copies of latent attention's per-head
matrices (GLM-4.7-Flash prompts at four threads 71.9 to 79.2). Every CPU
row is now level or ahead but SmolLM2 decoding from eight threads (0.91
to 0.95) and GLM-4.7-Flash's (0.97 to 0.98); Qwen3-8B Q8_0 prompts lost
1 % to the half-precision scales (still 1.71 to 1.80 times llama.cpp)
(benchmarks.md).
Round 9 (GPU): a decode step's RMS norm folded into the 8-bit product
that reads it (one launch, the same bits: the norm kernel sums the
foldable widths in the MQ kernels' layout), the key and value cache
writes in one launch, and the 8-bit decode products under
OKLM_CL_WMMA too: GPU decoding Llama-3.2-1B 368 to 392 t/s
(0.95 of llama.cpp), Qwen3-8B Q4_K_M 92.0 to 94.7 (0.90), Q8_0 66.3 to
68.1 (0.92), OLMoE 274 to 291 (0.81), GLM-4.7-Flash 71.5 to 73.1 (0.83),
SmolLM2 Q4_K_M 378 to 400 (1.04); Granite unchanged (279, 0.77) (benchmarks.md).
Round 9 (CPU): a loop limited to n participants takes the workers of the
lowest indices instead of the first to win a seat, the workers poll the
loop generation on a line of their own, those no loop of a decoding step
takes sleep, and small file-layout products run on the packed copies'
thread count: SmolLM2 decodes at 4, 8 and 16 threads alike (Q4_K_M 249
to 251 t/s, where 16 gave 220), 0.98 to 1.07 of llama.cpp from eight
threads; measured and left out: Q8_0 scales stored apart from the values
(prompts unchanged), expert and one-row tile sizes, and the waiting
workers prefetching the next product's weights (SmolLM2 3 % slower,
GLM-4.7-Flash no faster) (benchmarks.md).
Round 10 (CPU, 2026-11-12, after the block closed): the activation row
of a product rounded with AVX2, a token's rotation computing its cosines
once, the packed copies on huge pages: SmolLM2 decoding at four and
eight threads 1.07 to 1.09 of llama.cpp (from 0.96 to 1.00),
GLM-4.7-Flash 1.02 (from 0.97 to 0.98) (benchmarks.md).
Done (the 2026-11-05 list)
The list of 2026-11-05, kept with each item's proof. Item 13 (the security fine-tune) is deferred, under "Next".
Features, upgrades and fixes (1–11)
Hybrid Gated DeltaNet — inference and trainer (hybrid-layers-design.md; the large base of M15, split out). Mostly done against a synthetic
qwen3nextfixture: the kernels (recurrent forward, chunked scan, short causal conv, per-head L2 norm) and the recurrence's reverse-mode gradient (src/model/gated_delta.c,test_gated_delta.c); the hyper-parameters and GGUF round-trip; tensor binding and the fixture; the CPU forward with a per-sequence recurrent state cache (prefill equals decode) and the same under OpenCL (the linear block computed on the host); the CPU and OpenCL trainer (gradients against finite differences, OpenCL equal to the CPU) —test_model.ctest_qwen3next_,test_train.c/test_train_cl.ctest_qwen3next. Left: the real-weightssm_*parity against a real Qwen3-Next (hardware-blocked — the only release is 80B-A3B, which does not fit this 20 GB GPU), and a native on-device gated-delta kernel (a performance step; the block is computed on the host in both inference and training for now).Command classifier (agent spec §6.4). Done: with a
smallModelset andcommandClassifieron, the small model rates each Bash command for injection / exfiltration risk before it runs (agent.cagent_classify_bash, a/no_thinkYES/NO aside on the aux model); a flag makes the call ask even when a rule allows it, and it fails open so the dangerous-pattern list still decides otherwise. test_tools.ctest_command_classifier, test_agent.ctest_classify_bash.Secrets found in tool output (agent spec §6.4). Done: the model-, transcript- and spill-facing masking (
oklm_mask_secrets, agent.cclip); a scanner (oklm_scan_secrets) over the shapes (sk-,ghp_,github_pat_,AKIA,AIza,xox?-,glpat-,Bearer,token=,password=...) called fromoklm_tool_run(oklm_audit_scan) that registers each secret found in a tool's output with the audit log (masked in recorded arguments) and tells the user. test_telemetry.ctest_scan, test_tools.ctest_secret_scan.Permissions and safety polish. Done:
/add-dir --remember(saves the directory topermissions.additionalDirectoriesof the local settings,oklm_perms_save_dir); read-only pipeline filters (a read piped into a pure stdin filter —… | head/wc/grepwith no file argument and no write redirection — stays read-only, needing no rule; permissions.cis_read_filter); broader "always" rules (the approval offers a prefix choiceBash(cmd:*)beside the exact rule —OKLM_ANSWER_ALWAYS_PREFIX,bash_prefix_rules); memory imports outside the project allowed through the trust question rather than refused (optstrustcallback, oklm.cmem_trustover the persistent trust store); and the bypass mode without the sandbox behind a flag and a louder banner (--dangerously-skip-permissions,oklm_perms_allow_unsafe_bypass, ADR-0010 — a deliberate, dangerous opt-in). test_permissions.c, test_tools.c, test_memory.c.Hooks expansion (M11). Done: regex matchers (a
re:prefix on a matcher component,oklm_hook_matchesoveroklm_regex, beside the existing*globs);prompthooks that ask the model (type:"prompt", run through aprompt_fnwired to the aux model, their answer used as the hook's output); theCommandRun(commands.c, before a slash command) andDirectoryAdded(oklm.c,/add-dir) events, beside the existingUserPromptSubmit; adisableAllHookssetting that turns hooks off (settable by a managed admin); and a hook's otherwise-swallowed output shown in verbose mode (oklm_hooks_set_verbose, tied to/verbose). test_hooks.c.Sub-agents expansion (M12). Done: a definition's
modelis acted on (model: small/fastruns the sub-agent on the aux model with its own context, agent.csub_model); worktree isolation (isolation: "worktree"gives the sub-agent its own worktree and a tool context rooted in it); sub-agents listed in/tasks(oklm_agent_background_list); and/agents edit NAMEopens the definition in$EDITORand reloads it in place (oklm.creload_agent_defs); and parallel agents through a backend (2026-11-06): a backend may give each sub-agent a copy of its own (oklm_agent_backend.clone/release), the HTTP backend of--urla connection of its own per copy (oklm_client_clone), so background sub-agents and Agent calls started together run in parallel threads over it as with the in-process model; a backend without copies keeps them one at a time (test_agent.ctest_backend_parallel, test_server.ctest_agent_http_retrythrough a copy).Compaction and rewind. Done. The PreCompact hook (called from
agent_compact_now()); snapshots carried into forks (oklm_session_copy_snapshots(), turn-matched so a fork rewinds the files its parent changed); rewind past a compaction (the turns from before a compaction offered from the transcript, their files restored; the conversation is not rewound past a compaction); and rewind of Bash effects (oklm_session_checkpoint_tree()before each command, once per turn: in a git work tree agit stash createsnapshot kept underrefs/oklm/, its tracked files restored on rewind; outside git a bounded copy of the project's files, so modified and deleted files come back but files the command creates are left).Full-screen terminal UI (M9). Done (
/tui, POSIX; the scrolling view stays the default and the Windows path). Atuiengine (src/harness/tui.c) composes the whole screen with a pureoklm_tui_render()(tested without a terminal): a scrolling transcript that follows the end and wraps, a status bar kept at the bottom with the context indicator always on, an input area, a task list kept on screen, and overlay panels. The driver inoklm.ccaptures stdout and stderr through a pipe into the transcript (a drain-and-render thread), so everyprintflands in the view;view_pick()dialogs (the permission confirmation with "always" = don't-ask-again,/mode,/style,/theme, rewind) draw as panels, and/resumeis a session picker filtered as you type. Later: the task list as a strip that follows each TodoWrite (Ctrl-T hides it, and prints it in the line-mode UI) and/tasksas an interactive panel (follow a job's output, stop, restart, keep it running at exit)./skill-doctorusage counts and the skills' warnings in/doctorwere already present. Then (2026-11-07)/configas a settings panel (the settings in force with their sources, edited in place, checked and saved to a scope) and a progress row above the status bar (writing with tokens and speed, a tool running with its time, jobs, context);oklm chatis left as a follow-up (commands.legacyDirsis read since).Windows parity. Done. Wide-character column widths (CJK, emoji) across the UIs (
oklm_wcwidth()); desktop (OS) notifications (preferredNotifChannel: system--notify-sendon Linux,osascripton macOS; the bell on Windows for now); graceful job stop on Windows (GenerateConsoleCtrlEvent(CTRL_BREAK)to the process group, the grace, then terminate, in place of terminating at once); and a persistentcmd.exeshell (a per-turn.batrestoring and saving the directory and the environment, like the POSIX shell). The Windows shell and graceful stop are correct per the Windows docs but unverified here: wine'scmd /swill not run a quoted.batand itscalldoes not propagate a script's environment, sotest_shell_winskips under wine. A Windows toast (in place of the bell) is a follow-up.Editor: vim mode (M9). Done. The line editor gains vi key bindings (
oklm_line_edit_set_vi()): each line starts in insert mode, Esc enters normal mode where h/l/0/$/^/w/b/e move, i/a/A/I/o start insert, x/s/D/C and the d/c operators (dw,dd,cw,d$...) edit, and j/k walk the history; arrows still move in normal mode. Chosen with theeditorModesetting (vi/vim) or toggled with/vim; the full-screen prompt shows[N]/[I]. Emacs stays the default. (Images pasted into the prompt stay under "Later", with multimodal.)CPU prompt speed. Concluded as a no-op on this hardware. The kernels are instruction-bound (CPU generation at 80–90 % of llama.cpp), the SIMD level is detected per machine and
oklm tunesplits the work, and the 512-bit path is kept for CPUs whose AVX-512 units run at full rate — but on this i7-11700K AVX-512 downclocks and every idea tried (512-bit registers, a broadcast-light activation layout, Q4_K in byte form, eight activation rows for the K-quants) measured slower (see "Closed"). There is no beneficial change to make here; left for a CPU that reports otherwise.
Training runs and evals (12–16)
RL trainer upgrade. Done (2026-11-06): several updates per sampled batch with PPO's clipped ratio (
oklm_rl_params.updates/clip,--updates,--clip) and a KL penalty against a reference trainer (kl/ref,--kl,--ref, the k3 estimate per sampled token), onoklm_rl_step();test_traintest_rl_kl(KL from the start 3.34 to 1.12 nats) andtest_rl_updates(26 sampling steps against 81) (training.md).Security fine-tune ("Default models" step 4, security-training-design.md). The data tools (
tools/make_security_data.sh, the seed corpus), the chat-data trainer (oklm train --chat-data) and the measures (tools/eval_security.sh) are done; left is the LoRA run itself on the quantized frozen base and its evaluation (the base finds 3 of 4 seed vulnerabilities — the headroom the run targets).MCP tool-use fine-tune (M16, mcp_training_spec.md). Done (2026-11-07), T1 to T3 (mcp-training-design.md): a tool zoo (
tools/mcp_zoo.py, training and held-out tools), tasks of the five categories captured on Qwen3-8B and judged (117 kept, 151 after round 2, every call schema-valid), two QLoRA rounds (rank 16, 46 minutes each), measured with the grammar off (oklm agent --no-tool-grammar): trained tools 8 to 11 of 15 tasks and 78 to 94 % schema-valid calls after round 1; the held-out tools no better (7 of 9, then 5), round 2 no gain.tools/mcp_loop.shruns the loop again.Longer runs of the released models (M5, M15). Done (2026-11-07): the released Micro and Small conversation tunes continued 6 000 steps on SmolTalk (held-out loss Micro 2.993 to 2.844, Small 2.262 to 2.217; Small's answers alike), with
tools/fetch_datasets.sh --releaseadded (the sources at pinned revisions) (training.md).The eval gaps. Done (2026-11-07): the seventeen agent evals on Qwen3-8B, the default, 13 of 17 (the git evals 05, 11 to 14 pass, 15 is the model's; the process evals 10 and 17 pass, 16 is the model's), after a fix the rebase eval found (the agent's own
.oklm/state showed in the repository's status; now kept out,oklm_git_ignore_dir()); the review commands' seeded-bug measure on Qwen3-8B 6 of 6 and 5 of 6, above the 80 % asked; per-job memory and CPU limits in a cgroup of their own on Linux (oklm_job_spec.mem_limit,cpu_limit, a service'sresources) (agent-loop.md, processes.md).
Default models: the steps
The decision itself is in the roadmap.
The work this implies:
- Load and check the model. Done (2026-11-04), the
default context length included (32768: a 27,500-token prompt at 339
tokens/s on the GPU, beside the weights within 20 GB):
tools/make_qwen3_8b.shfetches the pinned revision, checks the shards and makes BF16, Q8_0 and Q4_K_M (importance matrix) GGUFs; greedy output and logits agree with llama.cpp within the bounds of the other models, perplexity within 0.4 % on the CPU (inference.md); OKLM's converter now pads the vocabulary to the embedding rows as llama.cpp's does (it wrote 151,669 tokens for 151,936 rows, which llama.cpp refused). Qwen3-8B from its safetensors repository, loaded directly (OKLM reads Hugging Face directories, sharded BF16 included, inference.md): greedy output and perplexity compared with llama.cpp's on the same weights, as for the models of M3. Then quantized (Q4_K_M, Q8_0, with an importance matrix) and measured on the CPU and OpenCL. It is the largest dense model run so far (16.4 GB in BF16, about 5 GB in Q4_K_M): the measurements set the default quantization and context length. - Make it the default. Done (2026-11-04):
oklm agentlooks for Qwen3-8B-Q4_K_M.gguf in$OKLM_MODELS, the user's models directory,./modelsand the source tree's when nothing names a model, says where to get it when it is missing, and the evals run on it. It becomes the agent's model when neither-m,--urlnor themodelsetting names one, found in the user's models directory, with a clear message (where to get it, how to quantize it) when it is missing: OKLM ships no weights. It is also the model the agent's evaluations (spec §25) run on, and the reference for the timeouts and context defaults. - Abliteration, done by OKLM. Done (2026-11-04), in
OKLM rather than with Heretic (a Python/PyTorch tool that would hold the
BF16 model in most of the GPU's 20 GB):
oklm abliterateruns a declined and an answered prompt set through the model in its chat template, reads the residual stream entering every layer at each prompt's last token (the importance-matrix observer ofoklm_imatrix_compute(), the residual divided back by the norm's weights), takes the mean difference per layer as the candidate direction, chooses the layer that separates the two sets best within a band of the depth, and makes every matrix that writes the residual stream (and the token embeddings, when the head is its own) orthogonal to it (W - s r (r^T W)). Pure C on the CPU, no new dependency, the same loaders and writers asoklm convert; the result is a trainable F16/BF16 GGUF whose making OKLM controls.oklm refusalsmeasures the refusal rate (greedy answers judged by the usual markers) and the first-token KL divergence from the base on answered prompts;tools/fetch_abliterate_data.shfetches AdvBench (Zou et al.) and Alpaca at pinned commits and splits each into a part for the direction and a held-out part for the measures (training.md). On Qwen3-8B (36 layers, the direction found from 416 declined and 416 answered prompts) the picker's layer (24, separation 11.2) takes the held-out refusal rate from 97.5 % to 7.5 % greedy, near the published figures; the first-token KL from the base on answered prompts is 1.29 there, so the removal also shifts answered replies — a shallower layer trades refusal reduction for a much smaller KL (layer 16: 62 % refusals, KL 0.18), the operating point left to--layer/--scaleand the security training (step 4). On Qwen3-0.6B the held-out refusal rate goes from a near-half to none with the KL on answered prompts staying small (test_infer_publictest_abliterate). - Security training. Scoped defensive (2026-11-05):
the fine-tune teaches secure code review — finding and explaining
vulnerabilities — not offence, so it trains on the normal Qwen3-8B
(defensive review does not trigger refusals; the abliterated model stays
a separate capability). The data, recipe and evaluation are defined in
security-training-design.md:
data captured by running the reviewer on a seed corpus of
known-vulnerable and clean snippets with
oklm agent --save-training, keeping the correct runs (tools/make_security_data.sh); LoRA on the quantized frozen base (--quantized-base); the detection and false-positive measures oftools/eval_security.sh(the base finds 3 of 4 seed vulnerabilities and flags the clean one — the headroom the run targets). The trainer reads the captured chat-data with the loss on the assistant turns only (oklm train --chat-data, done 2026-11-05, shared with the MCP plan's T2); the LoRA run itself is the compute step. - The small model (spec §2.5). Done (2026-11-05):
smallModel(Qwen3-0.6B, the same tokenizer and template) loads beside the main model and answers the asides — the titles and the compaction summaries — in its own context, leaving the main model's cache untouched; a sub-agent definition'smodel: smallruns that sub-agent on it (agent-tools.md,oklm_agent_set_aux_model). The command classifier (spec §6.4) still uses it as a next step, not built.
The repositories and figures above were confirmed in step 1 (the model loads, matches llama.cpp and trains on the GPU).
The chats: speech synthesis leftovers (C8)
Done 2026-11-13. OuteTTS's prompt is its header, the voice's words,
the text's words, then the voice's 658 codes: the codes follow the text,
so a cache keeps only the 62 to 70 tokens before it. What makes the
first sound wait is the first pass's own codes (the model writes 25 to
40 tokens a second, a second of speech is about 90). So: the passes that
hold in 64 words are written under one prompt (2 prompts of 1 913 tokens
for 4 of 3 687 on a 64-word text), each pass sent to the vocoder as its
last word is written, with a cross-fade; first_words cuts
the first pass (first sound after 6.0 to 8.0 s against 21 to 65 s; on
the route the first byte after 4.1 s against 32.0 s); the same text
asked again keeps its whole prompt. speed 0.25 to 4 with
the pitch kept (1.5: 13.31 s for 19.96 s, the same words heard by
Whisper). Voices by name: GET /v1/audio/voices,
--speak-voices, the route's voice.
response_format: flac from an encoder of OKLM's own
(oklm/mm/flac.h): 49.9 % of the WAV answer, accepted by
flac -t, decoded to the same samples by
flac -d and ffmpeg. OuteTTS 0.3 (CC-BY-SA-4.0) runs and is
in the catalogue as outetts-0.3-500m (heard 54 and 52 words
of 57 with two seeds, as 0.2). Measured with four threads on a loaded
machine; synthesis stays slower than it is heard (0.11 to 0.38 s of
sound a second).
L5. Gemma 4's small models, tool calls and thinking channel
Done 2026-11-13. E2B and E4B run on the CPU: an
embedding per layer computed once a batch and added at each block's end,
the last 20 / 18 blocks projecting queries only and attending over an
earlier block's cache (15 caches for 35 blocks, 24 for 42), a
feed-forward width per block. The tiny model against the float64
reference: 36 of 36 choices, logits within 7.27e-6. E2B Q4_K_M against
the pinned llama.cpp: 5 of 5 prompts tokenized alike, 160 of 160 greedy
choices, logits within 8.08, five of five continuations identical. E4B:
156 of 160, within 3.51, one of five continuations identical, parting
where llama.cpp's margin is 0.34. In the catalogue as
gemma4-e2b and gemma4-e4b. Tool calls
and thinking: the gemma4 format
(<|tool_call>call:NAME{...}), its parser and grammar,
the channel <|channel>thought ... <channel|> as
reasoning in oklm chat, the agent and both chat routes, a
channel the prompt opens itself, <|tool_response> and
<eos> ending a reply (a reply never ended before).
Through oklm serve on the E2B: a call with its arguments,
the answer from the tool's result, tool_choice: required,
the reasoning split whole and streamed. Also
tokenizer.ggml.suppress_tokens read and applied (unit
tests: no file at hand carries it), and the memory plan counting each
layer's cache at its own width (E2B at 8192 tokens: 144 MiB of cache
instead of 560).
Kept states of the recurrent blocks
Done 2026-11-13 (src/model/lin_keep.c, hybrid-layers-design.md).
A context of a qwen35, qwen35moe or
qwen3next model keeps eight states shared by its sequences
and found by a hash of their tokens, a journal of each sequence's last
96 positions, and the state before each draft. A cut starts from the
nearest state, through the recurrence alone when a journal covers the
gap, else by a forward pass over the rows the cut left; the paged prefix
cache serves these models. Logits after a cut equal a fresh evaluation
bit for bit on the tiny models, both caches
(test_state_keep). Ternary Bonsai 2 4B's MTP file, 96
greedy tokens, four threads on a loaded machine: --mtp 4.0
/ 4.4 / 1.9 / 2.2 tokens/s against 3.5 / 4.8 / 2.0 / 3.9 plain, 0.5
without kept states; 64 of 156 drafts accepted, the plain text. Prompt
lookup on the 2B: 4.3 against 0.4. A server's shared 700-token system
prompt is found cached from the second request on. /retry
in the chat: 7 tokens through the journal. Memory: 640 MiB at most for
the 9B and the 4B, counted by oklm plan;
OKLM_STATE_KEEP=0 turns it off.
Graphical tools: the inspector on devices, the lens and ablation (T10)
Done 2026-11-13 (observe.md, gui.md).
Devices: the observation hooks read back from OpenCL
and Vulkan; nine tiny models beside the CPU, norms within 2.4e-5,
attention weights within 3.3e-5, routings identical, logits the same
bits with recording on and off. Recording costs when on (Llama 3.2 1B on
Vulkan: 315 to 326 tokens/s off, 105 to 112 on) and nothing when off.
Lens and ablation:
oklm run --observe-lens K, --ablate,
--ablate-scan, --ablate-mean; the lens within
2.2e-6 of the float64 reference; on Llama 3.2 1B the lens reads " Paris"
from block 11 of 16 on. The inspector's Lens and
Effects views, a Chat box for the
model's template, a server's last 512 records stepped through
(GET /api/activations?serial=S), and the rows of latent
attention.
Graphical tools: the Windows leftovers (T10, fourth part)
Done 2026-11-13, under Wine 11 and on Linux; nothing was run on
Windows itself. UTF-8 paths:
src/core/win_utf8.[ch] included in every source of the
MinGW build, the C runtime's and Win32's wide functions behind the usual
names, the arguments read from the wide command line. A model, a PDF, an
index, a session and screenshots in a folder named
Modèles Ελληνικά 日本語 are read and written by
oklm.exe and oklm-gui.exe, where the build
before answered cannot open ... (error 123);
test_sys 84 checks on both systems. 64-bit
offsets: _FILE_OFFSET_BITS=64 and
oklm_file_seek(); test_big_output (a sparse
log of 3 GiB). Stop requests: oklm train
and rl write the checkpoint of the step reached,
convert and quantize remove their partial
file, a listening agent closes its listener
(oklm agent --stop-listening PORT), on a signal too;
tools/check_stop_requests.sh, 16 checks. The flows
left: files dropped on a window (written for Windows: Allegro's
library has no drop there), the save dialog with its name, a picture and
a sound to a projector, the server manager's Dashboard and Chat, a
conversion from a Hugging Face folder, the training tool's tabs, the
sessions browser's Delete, the retrieval manager's Evaluate:
tools/check_gui_wine.sh, 55 checks. Kept
servers: a value in the user's Run key on Windows
(test_kept.exe 217 checks),
oklm serve --rekeep NAME, the user manager reached at
/run/user/UID (tools/check_kept_servers.sh 42
checks), a launchd agent written for macOS and never built.
L1. Vision: LLaVA 1.6
Done 2026-11-13 (multimodal.md). A file
with clip.vision.image_grid_pinpoints is no longer refused:
the best resolution chosen, the image fitted in it, the overview and the
336-pixel tiles encoded, the tiles' features put back in one grid,
un-padded, image_newline after each row, as transformers'
LLaVA-NeXT defines it. Held to that processor and model code rewritten
in numpy (tools/llava_next_ref.py; no PyTorch here): on two
pictures of 2 x 2 tiles the tiles' pixels are equal, each of the ten
tiles' embeddings within 2.1e-6 to 7.6e-6 of the float64 pass, the 1 948
and 2 340 rows placed with no value different, and 48 of 48 greedy
choices are those of llama.cpp's language model given the same rows
(logits within 0.323). oklm describe reads the headline of
llama.cpp's test image; a prompt with the full grid takes 105 s on five
threads of a shared CPU, 26 s with the overview alone. In the catalogue
as llava-1.6-mistral-7b. The question LLaVA 1.5 left is
closed: the model's reading of the CLIP encoder is kept (class token
first, the layer before the last, 23 blocks in the file).
The chats: the Soprano speech model (C8)
Done 2026-11-13 (speech.md). Soprano 1.1
80M (Apache-2.0, English, one fixed voice, 32 kHz) runs behind
oklm speak, the library and
POST /v1/audio/speech, in the catalogue as
soprano-1.1-80m (a Q8_0 language model and an F32 decoder,
197 MiB). Its text is cleaned as its package cleans it, each sentence is
a prompt of three tokens beside its text, and the decoder reads the
language model's hidden states. Against llama-tts built
from the pinned llama.cpp (F16 file, three sentences): the same prompt
ids and the same 142 tokens written, hidden states within 64 dB. Against
a double-precision numpy decoder: samples within 107 dB. The published
decoder file is the author's checkpoint byte for byte. Whisper base and
small hear every word of six sentences but "vocoder". Four threads on a
loaded machine, a 64-word text: 2.7 to 4.0 s of sound a second and a
first sound after 0.6 to 1.1 s, where OuteTTS 0.2 makes 0.08 to 0.23 and
waits 33 to 104 s; the route's first pcm byte after 0.45 to
1.1 s; 397 MiB at most. The cleaner's reference is a transcription of
the package's rules (the package itself was not run), equal on 386
chosen and 80 000 random texts.
L5. Cyber-Ornith 1.5 9B
Done 2026-11-13.
DuoNeural/Cyber-Ornith-1.5-9B-OBLITERATED-GGUF (Q4_K_M, 5.6
GB; a security tune of Qwen3.5-9B with its 427 tensors), run on the CPU
and removed, with no change to the runtime. Against the pinned
llama.cpp: 5 of 5 prompts tokenized alike, 158 of 160 greedy choices on
its tokens, logits within 3.58 (one low logit; 0.089 on average), three
of five continuations identical and the others parting where llama.cpp's
two logits are at most 0.254 apart; 100 greedy tokens identical after a
plain prompt and after a chat turn. oklm chat with the
file's template, which opens a thought by default, gives llama.cpp's
greedy answer to the same prompt. In the catalogue as
cyber-ornith-1.5-9b, marked run; the test's bounds are 6.0
/ 0.95 / 0.6.
Speech recognition: speed, timestamps, languages (L3)
Done 2026-11-13 (multimodal.md).
Speed on the CPU: the encoder's products through a
blocked kernel (oklm_matmul_dense()), the spectrogram on
the worker team, the decoder's queries, keys and values as one loop, an
F32 checkpoint's decoder matrices read as F16 copies when that loses
nothing. One 30-second window at eight threads on the shared i7-11700K
(load 9 to 11, the two builds in turn): base 1.45 to 1.97 s -> 0.78
to 1.42, small 4.61 to 7.17 -> 2.48 to 3.98, large-v3-turbo 11.48 to
17.45 -> 7.78 to 11.94, large-v3 17.92 to 21.19 -> 13.58 to 15.71;
one thread, large-v3: 132.6 -> 73.6 CPU seconds; the transcripts
unchanged. Timestamps: the reference's rules, timed
segments, SRT and VTT, windows that start where the last closed segment
ended: on an address of 127 s large-v3 writes 375 words against 364 with
a cut every 30 s, large-v3-turbo 375 against 357, the two texts the
same. Languages: detection (0.985 to 1.000 on French,
German, Spanish and English samples with the four models),
--language auto|CODE, the translation task. Held token for
token to a second numpy reference on the tiny checkpoints.
oklm transcribe --timestamps / --format / --translate / --detect-language;
the route's verbose_json segments,
language_probability, srt, vtt;
POST /v1/audio/translations;
transcribe_stats.languages. A resampler defect met on the
way: a 52 s Ogg file at 44.1 kHz took 17 s to read, about 1 s now.
The chats: the window's and the dashboard's leftovers
Done 2026-11-13 (gui.md, server.md).
The chat window. A document longer than the context
is indexed by the window with an installed embedding model
(--embed-model) and each message goes with the passages
found for it, for its own server and a remote one: on a text of 26 872
tokens with qwen3-0.6b at 4096 and bge-small-en-v1.5, 100 passages, 11
sent, the right answer. A Speech... window: a voice by
name on any server, a speed, Pause and Resume (/pause),
first_words in one request, a reply's sound kept and heard
again without a synthesis, a reading played once whole, the rate of the
model that speaks (Soprano at 32 kHz: 3.6 s of sound in 0.5 s); a
microphone among the system's sources (--mic), the language
found or chosen and kept for the next window, the pause, the level and
the recording settings as controls. /model NAME on a model
that is not installed asks in the window (size, folder, licence) and
downloads; oklm-gui --tool rag --model NAME. A picture
pasted with Ctrl+V, on n_clipboard_get_data() added to
nilorea (X11 with incremental transfers, Windows written): SmolVLM-256M
answered "Red circle on blue background." make gui-test: 16
flows.
The dashboard. A document too long is kept by the
server for the conversation
(POST /api/ingest?conversation=ID,
/api/ingest/search, /api/ingest/forget;
bounded, in memory, forgotten after --ingest-keep seconds)
and each question goes with its passages: a document of 3 510 tokens in
24 passages, 3 sent, the answer right. The voice's controls (a chooser,
a speed, pause and resume, first_words, a reply's sound
kept, the voice loaded ahead), a level that follows the room, turns kept
while one is transcribed, the settings kept across a reload. A language
for transcription ("auto" and the model's codes, the language found
shown, a "to English" box on the translations route).
/rag NAME among the indexes of --rag-dir,
/models and /model telling what is installed
(GET /api/models/installed). The activations page
(GET /activations): the writes, the lens, the attention,
the experts, the kept records stepped through. The page's check run in
Chrome (405 checks), in Firefox through a WebDriver BiDi driver
(tools/check_dashboard_firefox.mjs) and, on a shorter walk,
in WebKitGTK (tools/check_dashboard_webkit.mjs): nothing
had to change in the page.
The chats: a full context (C10)
Done 2026-11-16 (inference.md, gui.md, server.md, agent-sessions.md).
Compaction in the chats.
/compact [FOCUS] in oklm chat and in the chat
window, on the agent's code moved to one place
(src/client/compaction.c: the threshold rule, the summary
asked after the conversation or from a transcript in parts, and for the
chats which messages stay). The older messages become a note the model
reads after the system prompt; the last exchange, the recent ones within
a quarter of the context and a message that carries a file stay as they
are. The messages replaced stay in /export and in the
conversation's file, and in the window on screen.
Continuing. A reply the context or the token limit
cut is sent again as the open start of the model's turn.
/continue in the three programs, a
Continue on a cut reply in the window. Room is made
first when the context is full: a larger context (the terminal chat up
to the trained one when the memory plan holds it, the window where the
line under its slider has nothing to warn of), else a compaction; a note
says which. A model cut inside its <think> block goes
on reasoning.
Automatic, both, everywhere, and each manual.
--auto-compact on|off, --auto-continue on|off,
--continue-max N in oklm chat,
oklm agent and oklm-gui;
/autocompact and /autocontinue in the three;
two boxes in the window's model settings; kept in the agent's settings
(autoCompact, autoContinue,
continueMax) and in the window's settings file. Manual, a
full context says so and names /compact,
/continue and the larger context.
Settled while building. "Nearly full" is the agent's
threshold (a share of the context less the reply's reserve, by the
model's size). A reply is continued 8 times in a row at most, and not
while it repeats itself. The server takes
continue_final_message: true, or prefix: true
in the last message, and nothing changes for a request with neither. A
limit the person gave ends a reply for good; the program's own is the
size of a piece.
Measured.
tests/server/test_server.c,test_continue_reply: on SmolLM2-135M Q8_0, greedy, a reply of 48 tokens cut at 17 and one of 60 cut at 23, continued, are the replies written in one go, whole and streamed, the cut reply's rows read from the cache.tests/harness/test_cli_chat.c,test_full_context(SmolLM2-135M, the CPU): a reply cut by a context of 128 tokens ends at 138 tokens in a context made 256, the same text as in a context of 2048; manual, a limit given, the bound,/compact(271 tokens to 131) then/continue, and compaction by itself (163 tokens to 42, 169 to 40).tests/harness/test_agent.c,test_continue_answer, andtest_agent_continuein the terminal test: the agent's cases on the scripted backend.make gui-test: seven flows on the tiny model at temperature 0 (gui.md).- Qwen3-0.6B Q8_0 on the CPU behind
oklm serve -c 2048 --fixed-context, in the chat window: an essay asked after three exchanges was cut by the context after 709 tokens, six messages became a note of 288 bytes, and the reply went on for 521 tokens to its own end. Read as exported it is one essay, with no restart nor repeated heading where it was cut; its repetitions are the model's, before the cut as after.
The chats: a full context, its leftovers (C10)
Done 2026-11-16 (inference.md, server.md, server.md, agent-sessions.md).
--resume. The option of
oklm chat and oklm agent that goes on with the
conversation or the session kept is --resume in both;
--continue, which shared its word with
/continue, is refused with a line that names
--resume (exit status 2). oklm agent --resume
alone reopens the latest session; /resume lists them.
One setting for the terminal programs.
oklm chat reads autoCompact,
autoContinue and continueMax where the agent
keeps them, and /autocompact and /autocontinue
write the user's file; the start options are over them for the run.
Channel formats through a server. An open last
message is no longer refused for harmony, Gemma 4 and Muse Glimmer: the
turn is written again in the model's channels
(oklm_think_open_turn()), left open in the reasoning
channel when content is empty and in the answer after the
closed reasoning otherwise. oklm chat keeps the reasoning
block of such a reply open on the screen across a cut.
A reply cut inside a tool call. It is never parsed
nor run half written: the server answers length with the
call's text so far (tool_call_open) and no
tool_calls; the text goes back in the open last message,
the calls' grammar reads it (oklm_tool_call_open()) and
generation goes on under it. The in-process agent and
oklm chat do the same on the reply's own text; an agent on
a server sends tool_call_open.
The dashboard's chat. /compact,
/continue, /autocompact,
/autocontinue, a Continue on a cut reply,
both by themselves by default; POST /api/compact runs the
chats' plan and summary on the served model for the page.
Measured.
- Gemma-4-E2B-it Q4_K_M on the CPU, temperature 0, thinking on,
through
oklm serve: a reply of 228 tokens (618 bytes of reasoning, 270 of answer) cut at 84 tokens inside its reasoning and at 198 inside its answer, continued whole and streamed, is the reply in one go; 112 of 114 and 224 of 228 prompt tokens came from the cache. Inoklm chatwith a context of 128 tokens the same reply is cut twice (the context made 256, then 512) and what is printed is byte for byte what a context of 4096 prints (919 bytes). test_continue_channels(tests/server/test_server.c): a tiny model for each of the three formats, a reply cut after every count of tokens from the first that holds reasoning, 3 cuts inside the reasoning and 2 inside the answer each, whole and streamed.- A call of
get_weatherthroughoklm serve, cut after each of its tokens and continued whole and streamed: Qwen3-0.6B Q8_0, 33 tokens, 32 cuts, 64 answers that are the call in one go (the same with thinking on, a reply of 135 tokens); Gemma-4-E2B-it, 27 tokens, 26 cuts, 52 answers.test_continue_call: the tiny model, a required call of 64 tokens, 63 cuts. test_continue_call_model(tests/harness/test_agent.c): Qwen3-0.6B in process with turns written in pieces of 6 tokens callsRead {"file_path":"p1.txt"}once, the call of the run in one go; the answer after the tool's result differs by two backquotes.test_continue_answer: the scripted backend, a call cut twice, run once.test_call_cut(tests/harness/test_cli_chat.c): SmolLM2-135M with a call required and replies of 8 tokens, the one[tool calls]line that of the reply in one go.test_compact_route: SmolLM2-135M Q8_0 in a context of 256 tokens, a conversation of four exchanges goes from 170 tokens to 103, six of eight messages replaced by a summary of 164 bytes.CHECKS=full sh tools/check_dashboard.shin Chrome: 35 checks on SmolLM2-135M in a context of 256 tokens with replies of 24.
The chats: a full context outside the chat window (C10)
Done 2026-11-16 (server.md, server.md, server.md, agent-sessions.md, inference.md).
Cut inside a channel's header. A reply of harmony,
Gemma 4 or Muse Glimmer cut before the first header of its turn is
complete is answered with no text, never the header's markup; sent back,
it opens the turn afresh (the template writes the header). An open last
message that is such markup is read as empty; oklm chat and
the agent drop it before they go on. The header of a later message of
the turn is left out of a cut reply the same way
(oklm_think_header_cut(),
oklm_think_cut_held(),
oklm_think_split_cut()).
/v1/messages. An answer cut inside a
tool call has no tool_use block for it:
stop_reason is max_tokens and the call's text
so far is the message's tool_call_open (a member of
message_delta when streamed). A last assistant
message that carries it back, the prefill of that API, is read as the
chat completions' open last message, and the call goes on under the
tools' grammar.
An agent on a server. The reasoning block a server
leaves in the content (<think>, closed, empty or
still open) goes back as the server wrote it, before the answer's text:
an answer continued after an empty block no longer starts to reason
again, and one cut inside its reasoning goes on.
A larger context before a compaction. The
dashboard's chat asks the server for a context twice as large after the
resize's dry run, judged as the chat window judges a size, never on
--fixed-context. The agent does the same by the chats' rule
(oklm_agent_params.grow_context): in process a context
twice the size when the memory plan holds it, on a server through its
route; with --url it counts against the context the server
tells. A notice says which.
One block. oklm chat's full-screen UI
writes the pieces of a continued reply in one block, the note about the
room made after it.
Measured.
test_continue_channels: the three formats on a tiny model each, cut after every count of tokens from 1: 3 cuts inside the first header for harmony and Gemma 4 and 6 for the recipient format, then 3 inside the reasoning and 2 inside the answer each, whole and streamed, all the reply in one go; the markup sent back as content gives the same.test_header_cut(tests/harness/test_think.c): every prefix of the five headers, and the header of a later message.test_continue_call_messages: the tiny model, a required call of 64 tokens through/v1/messages, cut after each of 63 counts, continued whole and streamed. Qwen3-0.6B Q8_0 on the CPU, temperature 0, one tool: a call of 33 tokens, 32 cuts, 64 answers that are the call in one go and no cut answer with atool_useblock; with thinking on (135 tokens) the same 32 cuts and 64 answers.test_agent_call_joint(tests/server/test_server.c):oklm agent's loop on anoklm serveof Qwen3-0.6B with pieces of 6 tokens:Read {"file_path":"p1.txt"}as in the run in one go, run once, and the same final answer.test_continue_grow(the scripted backend) andtest_agent_grow(tests/harness/test_cli_chat.c): Qwen3-0.6B in process, a prompt of 4637 tokens in a context of 4701 made 9402, the answer (1468 bytes) the one of a context of 16384. Throughoklm serve -c 4692: made 9384 by the server's route, the answer the one of a server started with-c 16384; on--fixed-contextwith compaction manual, the full context is said.CHECKS="chat full" sh tools/check_dashboard.sh: 206 checks in Chrome and in Firefox, 69 in WebKitGTK's MiniBrowser (its shorter walk ofchat, and the 50 offull), none failed and nothing differing in the page. Infull: the context of 256 tokens made 512 for a reply it cut, which is the reply in one go; Qwen3-0.6B with thinking on, a reply of 408 tokens in 9 pieces, 6 cut inside its reasoning and 2 inside its answer, the reply in one go in the three browsers.tools/check_chat_tui.sh(56 checks): a reply cut by a context of 128 tokens, in line mode and in the full-screen UI: one block there, the same 678 bytes copied, the same conversation file and export.
The agent: room for a turn in a small context (C10)
Done 2026-11-16 (agent-sessions.md, inference.md). What C10 left of
the agent: with oklm agent --url on an
oklm serve --fixed-context -c 4692, where the agent's
system prompt and tools alone are about 4600 tokens of Qwen3-0.6B's, the
compaction before the first turn made the prompt longer, twice, the
server answered 400, and the program ended on a double free; and a
larger context was asked for a cut answer only.
- No summary that frees nothing. The chats' rule
(
oklm_compact_gains(),src/client/compaction.c: the messages replaced hold an eighth of the context more than the note may take) was inoklm_compact_plan(), which the agent does not use: its own compactions (the threshold, a refused prompt, a full context under a cut answer) now ask it first./compactis still always made.oklm chatandPOST /api/compactalready held to it: a system prompt of 535 tokens in a context of 512 is compacted by neither ("compacted":0;The context is full (535 of 512 tokens): the message was not answered). - A larger context before a turn. Once the model
counted a prompt, a conversation that leaves less than a turn's room
(512 tokens, an eighth of a window under 4096, the turn's token limit
when smaller) is given a larger context by the program that owns it
(
oklm_agent_params.grow_context, in process and on a server that resizes), and so is a prompt the model refuses for its length (in process; anoklm serve'sthe prompt holds N tokens, the context M, which the agent now reads as that refusal and as the model's count). Only then is a summary made, when it frees room. - No room said. When the system prompt and the tools
alone leave less than a turn's room and the context stays, the message
is not sent:
(no room for a conversation in the context: the agent's system prompt and tools take 4653 tokens (about 1855 and 2797) of 4692, and a turn needs 512 more. ...), with the ways out (oklm serve -c N,/context-size N,contextWindow,--tool-rag K).oklm_agent_send()returnsOKLM_ERR_OVERFLOW,oklm_agent_no_room()gives the text,oklm_agent_context_infohasfixedandroom_min. - The double free: the summary's turn freed the
reasoning of the answer being gone on with;
agent_ask_private()sets it aside.
Proof: test_agent 636 checks (test_room
among them, on the scripted backend; it ends on the double free with the
fix taken out); test_compact 122,
test_context_full 84, test_subagents 41. On
Qwen3-0.6B, CPU,
oklm agent --url ... -p "Say hello in one word.":
--fixed-context -c 4692, one request (4666 tokens in, 26
out, cut), no summary, the message;
--fixed-context -c 4400, the server's 400, the message with
of 4400; -c 4692 without
--fixed-context, the context made 9384 tokens and the
answer hello; in process with -c 4096, the
context made 8192 tokens and the turn answered.
A chat template for a file that carries none
Done 2026-11-13. --chat-template FILE|NAME in
oklm chat, run, serve,
agent and describe: a Jinja file or a built-in
format (vicuna, chatml, llama2,
llama3, mistral, mistral-v3,
gemma, phi3, alpaca,
deepseek-coder, foundation-sec), one table of
Jinja templates for every command (oklm_chat_builtin(),
oklm_model_set_chat_template()). A catalogue entry names
its format ("chat_template"): LLaVA 1.5
(vicuna), DeepSeek-Coder, Codestral 22B
(mistral-v3), Foundation-Sec-1.1-8B-Instruct, so
-m llava-1.5-7b and the chat window's own server chat in
the model's turns without an option. A file with no template and none
given is said once on stderr. oklm agent renders an LFM2
file whose template drops the tools with the replacement
oklm serve and oklm chat use. Each format is
held to its definition on a two-turn conversation
(test_chat_builtins); LLaVA 1.5 7B on the CPU answers "The
capital of France is Paris." from a 16-token Vicuna prompt and stops;
LFM2-350M calls Read in the agent, where it wrote prose
before.
The terminal chat on a server (C11)
Done 2026-11-17 (points 1 and 3 of C11, the terminal chat's part).
oklm chat --url URL [--api-key KEY] [--remote-model NAME]
is the terminal chat, line mode and full-screen UI, on the model of an
OpenAI-compatible server (inference.md):
- The transport is the agent's HTTP backend, which
now takes a chat's request (no
tools,oklm_agent_http_set_temperature(),oklm_agent_http_set_extra()fortop_p,top_k,min_pandseed,oklm_agent_http_last_error()for the status of a failure). The key is--api-key, elseOKLM_API_KEY, elseOPENAI_API_KEY; with neither-mnor--urlthebaseUrlsetting is the server.-mwith--url, and the options that need the model in this process, end with status 2 and a line. - The model
(
oklm_client_remote_models(),src/client/remote_models.c:GET /v1/models, OKLM's answer, llama.cpp's, an OpenAI-style one): the one a server lists; among several a picker on a terminal, and for input that is not typed status 2 with the list and--remote-model;/modeland/modelslist them,/model NAME|Nchanges it. - The window: an
oklm serve's own (GET /api/context,/context Nasks it for another); elsewhere-c,contextWindow, or 32768. The use is an estimate (the server's count of the last turn, four bytes a token after it); compaction goes by it. - A cut reply goes back as an open last message
(
"prefix": true); after a server's 400 or 422, closed with a request to go on. One the server's context cut has anoklm servemake its context larger. Reasoning deltas are shown within<think>tags (folded in the full-screen UI) and not kept. - The commands: those of the conversation unchanged,
the sampling and
/thinkingsent in each request,/statsby the server's counts;/rag,/decide,/tooland a picture say in a line that they need the model in this process.
Proof: test_cli_chat_url 118 checks against a server of
the test's own that records each request (no model);
test_remote_models 55.
sh tools/check_chat_url.sh, 31 checks, Qwen3-0.6B Q8_0 on
the CPU, on loopback, against llama.cpp's server 0.8.0-dev (build
8642541) and oklm serve. On llama-server:
, model .../Qwen3-0.6B-Q8_0.gguf, a context of 32768 tokens (the default for a server that does not tell its own);
two turns, Lena. answered to "What is my name?"
(a prompt of 44 tokens, 22 of them in the server's cache; a reply of 142 tokens in 3.43 s);
(continued: 4 messages from a.json); the numbers 1 to 40
with -n 8 and three /continue (pieces of 32,
32 and 15 tokens, each after the open reply), one reply in
the file with the numbers in a row; /compact:
(conversation compacted: about 882 tokens to 226; 2 messages kept as they are...);
in the full-screen UI thought, 172 words and
context 0% (261/32.8k) · 42.8 tok/s. With
--api-key sk-local-check on the server: no key
the server asks for a key (status 401: Invalid API Key), a
wrong one refuses the key, both status 1; the right one
from the option and from OKLM_API_KEY answered. Its router
over two models:
lists 2 models and none was named: --remote-model NAME chooses one
(status 2) for piped input, the picker on a pseudo-terminal,
/model 1 then an answer from the other model. On
oklm serve -c 1024:
(oklm serve), model Qwen3-0.6B-Q8_0, a context of 1024 tokens (the server's);
/context 256, then counting to 150:
(the server's context was full: it is now 512 tokens, and the reply goes on),
again to 1024, the numbers 1 to 150 in one reply; --resume;
six turns with -n 40 in 1024 tokens:
(compacting the conversation: 8 of its 11 messages become a note, the context being nearly full...),
about 1051 tokens to 265.