Server
The HTTP API: a model behind OpenAI-compatible endpoints (and an
Anthropic-compatible /v1/messages beside them), a
dashboard, access rules, and the scheduler that shares one context
between concurrent requests. The decisions are those of adr/0018-server-api.md.
Status: implemented: oklm serve, chat and raw
completions (whole and streamed), tool calls, JSON output, images and
sounds in chat messages (with --mmproj), the Anthropic
Messages API, the dashboard, API keys and TLS. The native GUI is still
to come.
Running it
oklm serve -m models/qwen2.5-1.5b/qwen2.5-1.5b-instruct-q4_k_m.gguf
serving qwen2.5-1.5b-instruct-q4_k_m on http://127.0.0.1:7430/ (64 requests at once, 4096 tokens each)
Open http://127.0.0.1:7430/ for the dashboard (the
figures, and a chat whose replies are rendered as Markdown: code blocks
with a copy button, inline code, headings, lists, quotes, tables, links;
a <think> block folded away, opened on a click; for a
MoE model the expert map, and the profiles of the last requests, each on
its tab, see The dashboard), or point any
OpenAI client at http://127.0.0.1:7430/v1:
from openai import OpenAI
c = OpenAI(base_url="http://127.0.0.1:7430/v1", api_key="unused")
r = c.chat.completions.create(model="x", messages=[{"role": "user", "content": "Hello"}])Options, besides the model and context options of
oklm run (-c, --backend,
--gpu-layers, -ctk; --temp,
--top-k, --top-p, --min-p,
--repeat-penalty, --seed and -n,
the defaults of a request;
--tool-format, and --chat-template FILE|NAME:
a Jinja file or a built-in format's name in place of the file's chat
template, for a file that carries none, inference.md; a
catalogue model whose file has none gets its own without the option, a
server started on a file with no template at all says so once, and a
template asked for is never replaced by the one an LFM2 file's tools are
otherwise written with):
| Option | Default | Meaning |
|---|---|---|
--host ADDR |
127.0.0.1 | address to listen on |
--port N |
7430 | port |
--tls-cert FILE --tls-key FILE |
none | serve HTTPS (PEM files) |
--api-keys FILE |
none | accepted keys, one per line (# comments); the file must
not be readable by others |
--insecure |
off | allow a non-loopback address without TLS or keys |
--fixed-context |
off | refuse POST /api/context: the context stays the one
given at start (Resizing the
context) |
--fixed-model |
off | refuse POST /api/model: the model stays the one given
at start (Switching the model) |
--parallel N |
the batch's rows (512) or the paged cache's blocks, the fewer (4
with --kv-mem -1) |
requests generating at once (the context's sequences); with the paged cache a sequence costs only its table, so by default the memory, not this count, bounds the requests |
--queue N |
256 (16 with --kv-mem -1) |
requests waiting for a sequence (a burst that free sequences will take does not count); beyond, 503 |
--prefill-chunk N |
the batch (512) | prompt tokens of one request evaluated in a step at most |
--prefill-share F |
1 | while requests decode, a step's work on long prompts (more than one
--prefill-chunk left) is sized to take about F decode
steps' time: their time per token stays within about 1 + F times its
value while a long prompt is read, short prompts going whole; a smaller
F reads the prompt more slowly, 0 turns the sizing off. The
cost of a piece is a fitted model of its tokens and of its attention
over the keys before it (pieces of 16 tokens until it is fitted), so a
piece deep in a long prompt is sized at its depth. A piece's attention
over the keys before it is a floor: with a 30k-token prompt the
requests' time per token stays near twice its value whatever F |
--deterministic |
off | prompt pieces on a grid of --prefill-chunk positions, a
cached prefix used up to its last line: a request alone gets the same
tokens at temperature 0 whether its prompt was cached or not, streamed
or not; costs re-evaluating up to --prefill-chunk - 1
cached tokens. See determinism
at temperature 0 |
--observe |
off | record what happens inside the model for the token each sequence
decoded last, read with GET /api/activations (Activations), the last 512 records kept to be
read again; off, the route answers 404 |
--observe-lens K |
off | --observe, and with each record the K (1
to 16) tokens the output head would give for the stream after each
block; each block then costs one more product with the output head a
token (observe.md) |
--swap-mem MB |
0 | a request preempted for memory with 64 positions or more moves its rows to host memory while the swapped requests hold under MB, and brings them back when it runs again, instead of computing them again (not for a quantized cache on a device, nor several devices) |
--prefix-dir DIR |
off | keep the prefix cache between runs: as a request ends, its full
blocks are written to DIR (oklm_kv_save_media(), one KV
file per distinct head, a head holding images or sounds too, a file a
longer head extends deleted), and at start the files are loaded into the
paged cache, so a restarted server answers a repeated long prompt
without prefilling it. Needs the paged cache (--kv-mem -1
is refused). Requests with images or sounds, or with a LoRA module, are
not kept. Files of another model or cache layout are left alone and not
loaded. See the prefix cache on
disk |
--prefix-min N |
256 | with --prefix-dir: heads shorter than N tokens (rounded
down to whole blocks) are not written |
--prefix-disk-mb MB |
1024 | with --prefix-dir: the directory's total size; the
oldest files are deleted beyond it |
--kv-mem MB |
0 | the paged KV cache (ADR-0019) the requests share, that many MiB;
0 what fits (the device's free memory, four contexts' worth
on the CPU: the rule is oklm_kv_cache_rule(), which
oklm plan --serve and the graphical tools' memory line
apply too, memory-plan.md); a
model larger than its device has the cache of the layers that go there,
the rest in host memory, instead of one block, and so has a model the
device holds whole with less than one context of cache beside it, when
--gpu-layers is not given: fewer layers go there so that a
conversation filling the context is held, slower than the whole model on
the device (memory-plan.md);
-1 the contiguous cache, -c cells reserved per
sequence (what --gpu-stream uses unless
--kv-mem is given, which it refuses). On a device shared
with an encoder or a sound model, a cache that leaves them no room is
refused at start with the figures (sharing
the device) |
--spec-ngram K |
4 | speculative decoding: up to K tokens that followed the request's last tokens earlier in its own text (prompt lookup), verified in the same step; the answers are those of plain decoding, greedy or sampled (a draw a token, from the same logits). Only while at most 4 requests decode, and while a request gains half a token a step |
--spec-grammar |
off | grammar-forced drafts: while a request's grammar applies (a JSON
schema, json_object, tools once the call has begun) and
allows a single continuation (a key, the fixed text of a tool call),
that text's tokens (oklm_grammar_forced_tokens()) are the
step's drafts, verified like the others, so the answers are those of
plain generation, greedy or sampled. They take the place of the prompt
lookup's or the draft model's drafts in that step, whatever the number
of requests decoding, and do not count in a request's trial of guessed
drafts; --spec-guard does not rest them
(spec_forced, spec_forced_accepted in
/api/stats) |
--spec-guard RATE |
0 | with --expert-cache: no guessed drafts (prompt lookup,
draft model) in a step where the expert cache's recent hit rate is below
RATE (spec_rested in /api/stats); forced
drafts go on |
--draft MODEL |
none | a smaller model of the same vocabulary drafts instead (requests with
images or sounds keep --spec-ngram): it proposes up to
--draft-n tokens for each decoding request, a batch a
round, verified in the step; the answers are those of plain decoding.
Its device memory comes out of the paged cache's default size. Its speed
is measured against plain decoding over 16 decode steps, and it rests
256 steps when it was slower (draft_trials,
draft_kept in /api/stats) |
--draft-n N |
5 | tokens the draft model proposes a step at most (fewer for a request that accepts few: about twice its mean run) |
--workers N |
--parallel + --queue + 8, at most
1024 |
connections served at once; a request waits on its connection, so fewer workers than that keeps requests from reaching the scheduler |
--mmproj FILE |
none | a projector GGUF (a vision encoder, SmolVLM's, Gemma 3's, LLaVA 1.5's or 1.6's, Qwen2-VL's or Qwen2.5-VL's; or an audio one, Ultravox's): chat messages may then hold images, videos or sounds (below) |
--transcribe-model DIR |
none | a Whisper checkpoint directory (or the catalogue's
whisper-base, whisper-small):
POST /v1/audio/transcriptions turns a sound into text and
POST /v1/audio/translations into English text (transcriptions);
--transcribe-model-id NAME names it in the API (the
directory's name by default),
--transcribe-backend cpu|opencl|vulkan|auto is where it
computes (cpu by default; on a device its passes take their
turn with the model's: sharing the
device) |
--speak-model NAME|FILE |
none | a speech model (the catalogue's outetts-0.2-500m,
outetts-0.3-500m or soprano-1.1-80m, or the
language model's GGUF file with --speak-vocoder FILE):
POST /v1/audio/speech speaks a text (speech); --speak-voice FILE replaces the
built-in voice by a speaker file, --speak-voices DIRS names
the directories whose speaker files a request may ask for by name (the
user's ~/.config/oklm/voices by default; the model's folder
is always one; GET /v1/audio/voices lists them),
--speak-model-id NAME names the model in the API (the
catalogue's name, or the file's, by default),
--speak-backend cpu|opencl|vulkan|auto is where its
language model computes (cpu by default; sharing the device),
--speak-max BYTES is the longest text of a request (4096;
at most 65536) |
--upload-max MB |
32 | the largest file sent as a request's body: a document to
/api/ingest, a sound to
/v1/audio/transcriptions (413 above; at most 1024) |
--ingest-keep SECONDS |
3600 | on a server with an embedding model (--embed-model, or
--rag's): how long the documents /api/ingest
keeps for a conversation stay after its last question (a long document kept for
retrieval) |
--tiles SIDE |
0 | images: the longest side before slicing into tiles, as
oklm describe --tiles (0: the encoder's default); for
Qwen2-VL and Qwen2.5-VL, the largest area of an image, SIDE x SIDE
pixels (0: up to 4 096 embeddings, and at most three quarters of a
sequence's context: 3 072 with -c 4096) |
--ffmpeg |
off | videos: containers other than Ogg Theora are decoded through an
ffmpeg subprocess, as oklm describe --ffmpeg
does (OKLM_FFMPEG=1, or
OKLM_FFMPEG=/path/to/ffmpeg, allows it too); without it
such a video is refused (videos) |
--model-id NAME |
the file name | the model's name in the API |
--lora FILE [--lora-scale S] |
none | merge one LoRA adapter into the model as it loads (ADR-0020): every request runs with it, at the model's speed |
--lora-module NAME=FILE |
none | serve a LoRA adapter beside the model (repeatable, ADR-0021): a
request whose model is NAME runs with it, any other with
the model alone, all in the same batches; the model's weights are
untouched (a quantized model stays quantized) |
-c N |
4096 | context length of each sequence |
--rag INDEX |
none | a retrieval index (oklm rag index) for the chat
completions that opt in with rag (retrieval) |
--rag-dir DIR |
none | a directory of retrieval indexes (NAME.gguf): a request
may name one of them, "rag": {"index": "NAME"}, opened at
its first use with the --rag-* settings (several indexes) |
--rag-k N, --rag-mode dense|sparse|hybrid,
--rag-crag on|off, --rag-log FILE |
5, hybrid (weighted fusion), on, none | with --rag: a request's defaults, and every retrieval
appended to FILE as a JSON line |
The sampling options become the defaults of what a request leaves
out. The scheduler serves the requests first come, first served. Each
step gives every generating request its next token, then admits waiting
requests while sequences and the blocks of their prompts are free (the
head of the queue waits for its blocks, nothing passes it), then feeds
prompt pieces, oldest request first. A request that needs a block when
none is free takes it from the newest one, which goes back to the head
of the queue with its cache dropped (preemption): it later evaluates its
prompt and what it generated again, or with --swap-mem
brings them back from host memory, keeps its sampler (its grammar and
penalties) and sends nothing twice. A request too long for the whole
cache ends with length.
Full blocks are kept as a prefix cache shared by every request: a
request starts from the longest prefix any request computed (a system
prompt, a conversation so far), while the pool does not need those
blocks; a request whose prefix another is computing waits for it a few
steps rather than compute it again.
usage.prompt_tokens_details.cached_tokens says how many
prompt tokens were found.
Determinism at temperature 0
A greedy answer depends on how its prompt's rows were evaluated, not
only on the tokens. The cached rows are the bytes the first request
computed, so a cached prefix is exact; what differs is the rest. A first
request of 906 tokens evaluates them in pieces of 512 and 394 rows; the
same request again finds 896 cached (the full blocks) and evaluates the
last 10 rows as a piece of 10. The OpenCL products choose their kernel
by a piece's rows (the 8-bit matrix units from q8_cols
rows, the F16 matrix-unit tiles from 32, tiles of 64 with K split by the
tile count, the row kernels below), each with its own rounding, so those
10 rows' keys, values and logits differ in the last bits and a near tie
can go the other way. Streaming plays no part: a stream and a whole
answer are the same code path. The CPU's products give the same rows
whatever the piece (Qwen3-0.6B Q8_0: the same answer cold and
cached).
--deterministic makes the pieces independent of the
cache: every piece ends on a multiple of --prefill-chunk
positions, a cached prefix is used up to the last such multiple only
(the rows after it evaluated again, in the piece a cold cache gives
them), and pieces are not sized by time (--prefill-share is
off). A request alone then gets the same tokens with a cold cache, its
prompt cached, streamed or not. It does not cover requests that share a
step with others: their rows join the same products, whose kernel and
rounding follow the step's total rows (two identical requests at once
answered alike, but not as one alone). A cached prefix that came from
generated tokens (a conversation's earlier answer) was evaluated a row
at a time, so a later turn can still answer otherwise than on a cold
server. The option is off by default: only the GPU needs it, and it
costs every request that finds its prompt cached.
Measured on 2026-11-09, Qwen3-8B Q4_K_M on OpenCL, a 906-token prompt, 200 tokens greedy: without the option the first answer differs from the three after it (cached 896) from the first sentence on ("in a way that reduces" against "in a coordinated way"); with it the four are the same (cached 512) and equal the cold answer. Two of them sent at once differ from it in the first sentence (each step's rows doubled).
The cost is the cached tokens past the last grid line, evaluated
again (same server, max_tokens 1, each the median of three
warm requests):
| Prompt | Cold | Warm | Warm, --deterministic |
Warm, --deterministic --prefill-chunk 64 |
|---|---|---|---|---|
| 906 tokens | 0.43 s | 0.030 s (896 cached) | 0.218 s (512 cached) | 0.030 s (896 cached); cold 0.89 s |
| 816 tokens | 0.39 s | 0.038 s (800 cached) | 0.166 s (512 cached) |
A smaller --prefill-chunk re-evaluates fewer tokens but
reads every prompt in smaller pieces: pieces of 64 halve the cold
prefill's speed. The 38 questions of the docs set through
--rag (corrective retrieval, the critique on,
tools/rag_stream_critique_eval.py), whole then streamed:
without the option 19 of 38 answers were the same text; with it 38 of
38, the verdicts equal and their support probabilities equal to 4
decimals, the critique's median 816 ms against 261 ms (its scoring
requests find the answer's prompt cached and evaluate its tail again).
With --prefill-chunk 64: 37 of 38, the critique 331 ms. The
corrective grader's and the critique's scoring requests go to the
scheduler together, so how their rows share steps is timing; one of them
deciding otherwise changes the context or the verdict (the one answer of
38 that differed, and support probabilities up to 0.15 apart at 64).
llama.cpp documents the same effect: its server's
cache_prompt (default on) reuses the slot's cache, and
"because (depending on the backend) the logits are not guaranteed to be
bit-for-bit identical for different batch sizes (prompt processing vs.
token generation) enabling this option can cause nondeterministic
results"; its only remedy is cache_prompt: false, which
evaluates the whole prompt every time.
Prefix cache on disk
With --prefix-dir DIR the prefix cache outlives the
process. When a request ends with stop or
length, its cache's full blocks (at least
--prefix-min tokens) are written by
oklm_kv_save_media() to
DIR/<hash of the tokens>.kv (written to a temporary
file, synced and renamed, so a crash leaves the previous files), unless
a file already holds them; the files whose tokens they extend are
deleted, so a conversation keeps its latest turn's file only, and the
oldest files go when the directory passes --prefix-disk-mb.
At start, oldest first, each file is loaded into a sequence
(oklm_kv_load_media()) that is then emptied: its full
blocks stay in the pool as cached prefixes, found by the usual
longest-prefix lookup, so the first request that repeats a head reports
it in cached_tokens and evaluates only the rest. A pool
smaller than the files keeps the newest. The rows are the cache's own
bytes, so the answers are those of a server that computed the prompt: a
greedy answer is the same with and without the reload
(test_prefix_disk). Writing happens on the scheduler thread
as the request ends (a write is one sequential file, a few milliseconds
for a short head); Measured with Qwen2.5-0.5B Q4_K_M on the CPU, a
2174-token prompt (max_tokens 1): the first request of a
server takes 1.85 s (nothing cached); the same request to a server
restarted on the directory takes 0.026 s with 2112 tokens cached (the
full blocks; the rest is evaluated), the same as a second request to a
running server, and answers the same token. The file is 26 MB (12 KB a
token). /api/stats has a prefix_disk object
(files, bytes, saved,
loaded, loaded_tokens, save_s)
when the option is on.
Ctrl-C (SIGINT) or SIGTERM stops the server: requests still generating or waiting end, the connections close. A client that hangs up frees what its request held at the next step: a stream's failed write marks it, and a request answered whole looks at its connection every 100 ms while it waits (a sequence freed, or its place in the queue).
A request holding an image or a sound is kept like any other: the
file has each row's token or media hash and, for Qwen2-VL and
Qwen2.5-VL, the positions of the image's rows (inference.md, version 2 of the
format), and the file's name hashes the media hashes with the tokens. A
restarted server finds the head when a request brings the same media
(the encoder still runs, the language model does not read the rows
again). Run once on Qwen2-VL 2B on the CPU: a request with a picture,
416 prompt rows of which 391 the image's, wrote a file of 12.4 MB; the
same request to a server started again on the directory reported 400
cached tokens and gave the same answer. test_media_describe
runs it on the tiny encoder of Qwen2-VL with the tests' language model:
a picture's request of 115 rows, the server started again on the
directory (112 rows cached, the same answer), and another picture, which
finds the one block of text before it. The tests' language model gives
each row one position: a served model with three-part positions has no
tiny file with a tokenizer, so its files are tested below the server
(test_kv_file), and a video's the same way
(test_video_describe).
The servers that run
A server writes a file for the user alone at its start and removes it
when it stops: its pid, its port and address, its model, its command
line and its working directory, in
$XDG_RUNTIME_DIR/oklm/procs (else
~/.cache/oklm/procs), one PID.json each. It is
the registry of background processes of
include/oklm/harness/procreg.h, which the graphical tools
read and start servers through (gui.md); a server
started from a terminal is in it like one a tool started. A command line
of the registry is split back into its arguments by
oklm_procreg_split(), and oklm_procreg_start()
starts a command in the directory an entry names (its cwd):
how a server is started again as it was. The server manager
(oklm-gui --tool servers, gui.md) is the window on the
registry: it starts a server from a form, lists them, stops one and
restarts one. A server that is not in the registry (one started before
servers registered themselves, a chat window serving its model) can
still be stopped there when it is the user's own:
oklm_procreg_port_owner() finds, from /proc
(Linux) or the system's table of listening sockets (Windows), the user's
oklm or oklm-gui process that holds the socket
listening on a port, and oklm_procreg_stop_listener() stops
it after finding it again, so a pid that no longer holds the port is
never signalled.
oklm serve --list
PID PORT STARTED MODEL ADDRESS
3957463 7455 2026-11-12 21:59 smol test http://127.0.0.1:7455
oklm serve --list --json
oklm serve --stop 7455
stopped the server of smol test on port 7455 (pid 3957463)
--listgives the servers of this user that run, the oldest first, with the log of those started in the background;--jsongives{"servers": [{"pid", "port", "url", "model", "started", "command", "cwd", "log"}]}. The file of a server that is gone (killed, a crash) is removed by the listing: a pid is checked against the time its process started, so one the system gave to another program does not pass.--stop PORT|PIDstops one of them, by the port it listens on or by its pid: SIGTERM, as Ctrl-C does, then SIGKILL when it still runs ten seconds later. Only a registered server is signalled.
Checked on 2026-11-12: a server of SmolLM2-135M on port 7455 listed (text and JSON), stopped by its port, its file removed and its exit status 0.
On Windows the registry is the same, with that system's means (checked on 2026-11-13 with the MinGW build under Wine 11, not on Windows itself):
- The files are in
%LOCALAPPDATA%\oklm\procs. Where POSIX gives the directory the mode 0700 and each file 0600, the directory's access list is replaced by a single entry, full control for the user, that the files made in it get too, and nothing is inherited from the folder above any more; the default list of%LOCALAPPDATA%already keeps other accounts out, and names the system and the administrators, who can read there as root can on POSIX. Wine keeps such a list as the mode 0770 of the file underneath: nobody else, but the user's Unix group too, which Windows itself does not have. - A command is started with
CreateProcess: no console (DETACHED_PROCESS), a process group of its own, standard output and error appended to the log, an empty input, and no other handle of the starting program. Its single command line is quoted by the rules of the C runtime there (an argument with a blank or a quote between double quotes, a quote after a backslash, the backslashes before a quote doubled), andoklm_procreg_split()reads those rules back, where a backslash is part of a path and not an escape. Both forms are there on every system (oklm_procreg_command_windows(),oklm_procreg_split_windows(), and the_posixpair). - A pid used again is told by the time its process was created
(
GetProcessTimes), where Linux has the kernel's start time. --stopand the tools' Stop ask the server to end through an event it waits on (Local\oklm-stop-PID,oklm_procreg_on_stop()): the server stops as on Ctrl-C, saysserver stoppedand removes its file; it is terminated when it still runs ten seconds later. A process started without a console cannot be sent the console's Ctrl-C or Ctrl-Break by another program, which is why the request is an event; Ctrl-Break in the server's own console stops it the same way. The commands that end by themselves when asked wait on it too, and take SIGINT and SIGTERM the same way on the other systems:oklm trainandoklm rlfinish the step they are in and write the checkpoint of that step (stopped on request at step N, the status 0,--resumegoes on from it);oklm convertandoklm quantizestop before the next tensor and remove their partial file (the status 1, the result not written, the file there before left as it was);oklm agent --listencancels the turn in progress, closes its listener, removes its file and keeps its session (oklm agent --stop-listening PORTasks it, the status 0). A second signal ends any of them at once. A background process that waits on no such event (an index being built, an evaluation) is terminated at once, as SIGTERM ends it on POSIX.sh tools/check_stop_requests.shruns the four with SIGTERM (16 checks),tools/check_gui_wine.shthe training run and the agent through the event under Wine.- A log is read without the carriage returns that end its lines there.
Under Wine: tests/harness/test_procreg.c (164 checks: a
command started, listed, followed and stopped, one that ends when asked
and one terminated after the delay, the directory it starts in, the
arguments a started program really gets back from the awkward ones, no
handle inherited, the access lists, a pid used again, a process that
registers itself, an oklm.exe found behind its port and
stopped), and oklm.exe serve on the test model started
through the registry, listed (text and JSON), answering
/v1/models, stopped by
oklm.exe serve --stop PORT in 0.3 s with
server stopped as the last line of its log.
Servers kept across logins
A server started as above ends with the session, or with the machine. One that should be there again after the next login is kept: its command becomes a service of the user, which the system's service manager starts at each login and starts again when it fails.
oklm serve --keep smol -m smollm2-135m --backend cpu -c 4096 --port 7440
kept as smol: /home/me/.config/systemd/user/oklm-serve-smol.service
log: /home/me/.local/state/oklm/services/smol.log
running: SmolLM2-135M-Instruct-Q8_0 on http://127.0.0.1:7440 (pid 41873)
it starts at each login of yours, and again when it fails; oklm serve --unkeep smol removes it
oklm serve --kept
NAME STATE AT LOGIN PID PORT ADDRESS
smol running yes 41873 7440 http://127.0.0.1:7440
unit: /home/me/.config/systemd/user/oklm-serve-smol.service
log: /home/me/.local/state/oklm/services/smol.log
oklm serve --kept --json
oklm serve --unkeep smol
smol is no longer kept: stopped, its unit and its log deleted
--keep NAMEtakes the rest of the command line as it is: every option ofoklm serve, without--keep NAME. The program is thisoklm, by its absolute path, and the working directory the current one, so a model named by a relative path is found again. Give--port: a kept server should be where its clients expect it. The command waits up to twenty seconds for the server to answer and says whether it runs, is still loading its model, or failed. A name is 1 to 48 lower case letters, digits,-and_, starting with a letter or a digit; anything else is refused, since the name goes into a unit's name and a file's. A name already kept is refused by--keep.--rekeep NAMEwith a new command line changes the command kept under a name without removing it: the unit is written again in place of its own, the manager reads it, and the server is started again when it ran (the command kept as smol is changed: ..., thenrunning: ...); a stopped one stays stopped and has the new command at its next start. It stays enabled and keeps its log. A name that is not kept, a key given inline: refused, the unit as it was; when the manager refuses the new unit, the one that was there is put back. The server manager's Re-keep with form does it for the kept server selected there, with its form's settings (gui.md).--keptlists the kept servers by name with their state (running,starting,stopping,stopped,failed), whether they start at login, and, for one that runs, its pid, port and address from the registry; under a server that is not running, the manager's reason (its command ended with the status 1,it failed five times within a minute and is no longer started again).--jsongives{"kept": [{"name", "state", "enabled", "pid", "port", "url", "reason", "active", "sub", "result", "exit_status", "restarts", "unit", "path", "command", "cwd", "log"}]},active,subandresultbeing the manager's own words.--unkeep NAMEstops the server, disables and deletes its unit, deletes its log and has the manager forget it. Only a unitoklmwrote is touched.- A kept server that runs is in
oklm serve --listlike any other, once, withkept across logins as NAMEunder it ("service"in the JSON).oklm serve --stop PORTstops it as it stops any server: it ends with success, so the manager leaves it stopped, and it starts again at the next login.systemctl --user start|stop|restart oklm-serve-NAMEdo what they say.
The unit. On Linux a kept server is a unit of
systemd's user manager, written in the directory that manager reads the
user's units from (the first of its UnitPath ending in
/systemd/user: ~/.config/systemd/user), with
the mode 0600, then systemctl --user daemon-reload and
enable --now:
[Unit]
Description=OKLM server smol
StartLimitIntervalSec=60
StartLimitBurst=5
[Service]
Type=exec
WorkingDirectory=/home/me/work
Environment=OKLM_PROC_LOG=/home/me/.local/state/oklm/services/smol.log OKLM_PROC_SERVICE=smol
ExecStart=/usr/bin/oklm serve -m smollm2-135m --backend cpu -c 4096 --port 7440
Restart=on-failure
RestartSec=5
RestartPreventExitStatus=2
TimeoutStopSec=15
StandardOutput=append:/home/me/.local/state/oklm/services/smol.log
StandardError=append:/home/me/.local/state/oklm/services/smol.log
[Install]
WantedBy=default.target
[X-OKLM]
Name=smol
Command=/usr/bin/oklm serve -m smollm2-135m --backend cpu -c 4096 --port 7440
Log=/home/me/.local/state/oklm/services/smol.log
- Quoting.
ExecStart=is not a shell's line. An argument made of letters, digits and_ @ + = : , . / -is written as it is; any other goes between double quotes, with\"and\\for a quote and a backslash,%%for a%(the manager expands%h,%n...) and$$for a$(it expands$NAMEand${NAME}). An argument holding a line break or another control character is refused: a unit is made of lines.[X-OKLM]is a section the manager skips; it holds the command as a shell takes it, which is what the listings show. - Restarts. A server that ends with a failure (a
crash, a kill) is started again five seconds later; five failures within
a minute and the manager gives up (
failed, until Start orsystemctl --user start). The exit status 2, a command lineoklm serverefuses, is not tried again. A server that ends by itself with success (Ctrl-C's SIGTERM,oklm serve --stop) is left stopped. - The log is a file,
$XDG_STATE_HOME/oklm/services/NAME.log(else~/.local/state/oklm/services), for the user alone, appended to across restarts, and not the journal: the unit names it inOKLM_PROC_LOG, the server writes it in its registry entry, and the log is then followed exactly as the log of a server started in the background is (oklm_procreg_log_read(), the server manager's Log), whether or not the journal keeps the user's units' output.tail -freads it;journalctl --user -u oklm-serve-NAMEstill has the manager's own lines (started, failed, the exit status). - The environment. The manager does not start the
server from your shell. The unit carries the variables named
OKLM_*that were set when the server was kept (OKLM_MODELS_DIR,OKLM_VK_DEVICES...) and theXDG_*_HOMEandXDG_RUNTIME_DIRdirectories, so the models, the settings and the registry are the same; a variable whose name holdsKEY,TOKEN,SECRETorPASSWORDis left out. - Keys.
--api-keys FILEis kept: it is a path. A secret given on the command line (--api-key,--key,--token,--password,--secret, alone or with=value) is refused for a kept server: the unit is a file that stays, and its command line is shown bysystemctl statusand, while the server runs, by the list of processes to every user of the machine.
At boot, without a login. The user manager runs
while you have a session: a kept server starts when you log in (a
desktop session, an SSH connection) and stops when your last session
ends. To have your services started at boot and kept running without any
login, the system has to be told to keep your user manager: that is
lingering, a setting of the machine's that oklm never
changes for you.
loginctl enable-linger # your services start at boot and outlive your sessions
loginctl disable-linger # back to: only while you are logged in
Removing by hand. What --unkeep does,
when oklm is no longer there:
systemctl --user disable --now oklm-serve-smol.service
rm ~/.config/systemd/user/oklm-serve-smol.service
systemctl --user daemon-reload
rm ~/.local/state/oklm/services/smol.log
Where it is not available. On Linux kept servers
need systemd's user manager. Where systemctl --user does
not answer (a container, a system with another init, a user without a
session), --keep, --kept and
--unkeep say so with the manager's own message, write
nothing and start nothing; oklm serve in the foreground or
in the background is what there is. A session whose
XDG_RUNTIME_DIR is not the manager's (a sandbox,
su, a test with a directory of its own) reaches it all the
same: when /run/user/UID/bus is there,
systemctl is run with
XDG_RUNTIME_DIR=/run/user/UID and that bus (not a program
given in OKLM_SYSTEMCTL, which decides for itself). The
kept server registers itself in the registry of the directory it was
kept from, which the unit carries: --kept and
--list of that session show it.
On Windows a kept server is a value of the user's
Run key
(HKCU\Software\Microsoft\Windows\CurrentVersion\Run, named
oklm-serve-NAME), which the system runs at each login of
the user and which needs no elevation, and a key that holds its command,
its directory and its log (HKCU\Software\OKLM\kept\NAME).
The value runs "...\oklm.exe" serve --start-kept NAME,
which starts the command in the background through the registry of
processes, as the tools start a server (no console, its output appended
to %LOCALAPPDATA%\oklm\services\NAME.log), and ends. The
state is the registry's: running with its pid while a
process registered under the service's name lives, stopped
otherwise. --keep, --rekeep,
--kept, --unkeep, --stop and the
server manager's box are the same commands. What Windows does not give:
a kept server is not started again when it fails (a
Run key starts a program once, at login), no
failed state and no reason, and a console window may show
for an instant at login while the starter runs. A task run at logon
(schtasks) would restart no more and is refused to a
standard user by some policies, which is why the key was chosen. Run on
2026-11-13 with the MinGW build under Wine 11, never on Windows itself
and through no real login: test_kept.exe (217 checks: a
command kept, its key and its Run value read back, listed
running, started twice, stopped, started, restarted, its command changed
while running and while stopped, a secret and a second use of the name
refused, removed with no value and no key left) and the
kept group of tools/check_gui_wine.sh (8
checks: oklm.exe serve --keep on the test model answering,
the Run value
"...\oklm.exe" serve --start-kept NAME, --kept
and --list, --stop, --start-kept
as the system runs it at login, --rekeep,
--unkeep with nothing left).
On macOS a kept server is an agent of the user for
launchd:
~/Library/LaunchAgents/net.nilorea.oklm.serve.NAME.plist
(RunAtLoad, KeepAlive on a failure, the log in
~/Library/Logs/oklm/NAME.log), loaded with
launchctl bootstrap gui/UID, stopped with
bootout, its state read from launchctl print.
This half is written to launchd.plist(5) and
launchctl(1) as documented and was never compiled
or run: OKLM's authors have no such machine. It is a hundred
and fifty lines of src/harness/kept.c under
__APPLE__; a first build there will have to be read against
them.
The functions are in include/oklm/harness/kept.h:
oklm_kept_install() (a command, a directory, an
environment), oklm_kept_replace() (the command of a name
kept already), oklm_kept_list() and
oklm_kept_status() (the state, from
systemctl --user show), oklm_kept_start(),
oklm_kept_stop(), oklm_kept_restart(),
oklm_kept_remove(), and the texts on every system:
oklm_kept_unit_text(), oklm_kept_quote(),
oklm_kept_name_ok().
Checked on 2026-11-13 with systemd 262's user manager
(tests/harness/test_kept.c,
tools/check_kept_servers.sh): a command given fifteen
awkward arguments (blanks, both quotes, %h %n %%,
$HOME, ${HOME}, $$, backslashes,
;, an empty one, #x, *,
non-ASCII) and an environment value with $, %,
quotes and a backslash got each of them back unchanged; a command ending
with the status 2 was failed with that reason and not
started again; a server of the test model kept on a free port answered,
was in the registry once with its service and its log, was started again
by the manager after a SIGKILL, stayed stopped after
oklm serve --stop, started again with
systemctl --user start, and --unkeep left no
unit file, no unit known to the manager, no log and no process.
Lingering was not touched.
Access
- On a loopback address (
127.0.0.0/8,::1,localhost) no key is needed, unless--api-keysis given. - Any other address needs TLS (a certificate and a key) and at least
one API key. Without them the server refuses to start, unless
--insecureis given; the dashboard then says so. - Keys come as
Authorization: Bearer <key>orx-api-key: <key>(Anthropic's clients), compared in constant time. A missing or wrong key gets 401 withWWW-Authenticate: Bearer, and the answer does not say which. GET /healthand the dashboard page itself need no key; the page asks for one when the API answers 401 and keeps it in the browser.- The Windows build has no TLS (no OpenSSL for MinGW here): it serves
plain HTTP, so loopback, or
--insecureelsewhere.
Endpoints
Every route oklm serve answers, and no other:
| Method | Path | What it does | Described in |
|---|---|---|---|
POST |
/v1/chat/completions |
a chat completion | Chat completions |
POST |
/v1/completions |
a raw completion of prompt; with --mmproj,
<__media__> markers in it and as many base64 items in
multimodal_data (llama.cpp's form), of the projector's
kind |
Images and sounds |
POST |
/v1/messages |
an Anthropic Messages API message | Anthropic Messages API |
POST |
/v1/messages/count_tokens |
the prompt tokens of a Messages request,
{"input_tokens": N} |
Anthropic Messages API |
POST |
/v1/embeddings |
with --embed-model, OpenAI's embeddings of
input by the embedding model |
Embeddings |
POST |
/v1/rerank, /rerank |
with --rerank-model, passages scored against a query
and sorted, best first (Cohere / Jina shape); the two paths are one
route |
Reranking |
POST |
/v1/audio/transcriptions |
with --transcribe-model, OpenAI's transcription of a
sound by a Whisper model: the language found or given, timed segments,
subtitles |
Transcriptions |
POST |
/v1/audio/translations |
the same sound written in English (Whisper's translation task) | Transcriptions |
POST |
/v1/audio/speech |
with --speak-model, OpenAI's speech of a text by a
speech model, as a WAV file, a FLAC file or raw samples sent while they
are made, with a voice and a speed |
Speech |
GET |
/v1/audio/voices |
with --speak-model, the voices a speech request may
name |
Speech |
POST |
/v1/systemone, /v1/decide |
closed questions about a state answered with each option's
probability, nothing generated (the two paths are one route);
oklm agent --url judges its retrieval and its answers with
it (rag.md) |
Decisions by scoring |
GET |
/v1/models |
the loaded model, then each --lora-module adapter with
the model as its parent, then the
--transcribe-model model with
"task": "transcribe" and the --speak-model
model with "task": "speak"; each entry also carries
Anthropic's type, display_name and
created_at, the list has_more,
first_id and last_id |
this table |
POST |
/api/compact |
a conversation compacted for a client that cannot run the compaction: which messages stay, and the summary of the others written by the served model | Compacting a conversation |
POST |
/api/ingest |
a document sent as the body (PDF, DOCX, HTML, JSON, text) turned into text, for a client that cannot read it | Documents |
POST |
/api/ingest/search |
the passages of the documents kept for a conversation that answer a question, within a budget of tokens | A long document kept for retrieval |
POST |
/api/ingest/forget |
the documents kept for a conversation dropped | A long document kept for retrieval |
GET |
/api/stats |
the dashboard's figures, JSON: requests running and waiting, tokens,
the KV cache's blocks and use (kv_usage, %), the prefix hit
rate (%), the mean wait for a sequence, preemptions; images and sounds:
their rows evaluated (media_rows) and found cached
(media_cached), items encoded (media_encoded)
and whose embeddings were kept from before (media_reused),
media parts decoded (media_decoded) and known by their
bytes, not decoded (media_known); speculative decoding:
drafts proposed and accepted (spec_proposed,
spec_accepted), of them the grammar-forced ones
(spec_forced, spec_forced_accepted), steps the
guard rested (spec_rested), the draft model's speed trials
and those it won (draft_trials, draft_kept);
swapping: preempted requests swapped out and brought back
(swaps_out, swaps_in), the host bytes they
hold (swap_bytes); kv_check_failures with
OKLM_KV_CHECK=1 (steps after which the cache's bookkeeping
was broken, for tests); what the server takes, for a client to know
before it sends (What a server takes): vision,
audio, transcribe, ingest,
upload_max; where its encoders compute:
vision_device (the image encoder's blocks are on the
model's device), backend in transcribe_stats
and in speak_stats |
What a server takes, Scheduler |
GET, POST |
/api/context |
the context's length: read (n_ctx, the model's
trained context, resizable), or changed with
{"n_ctx": N, "force": false} |
Resizing the context |
GET |
/api/model |
the layout of the model the server holds, JSON: the metadata and the
tensors by block and role, where each layer is, the context's memory by
part, the adapters loaded; ?tensors=1 adds every
tensor |
The figures as JSON |
POST |
/api/model |
another model loaded in place of the one served,
{"model": NAME} |
Switching the model |
GET |
/api/models/installed |
the catalogue's models installed on the server's machine, which
POST /api/model can be given, and whether the server may
switch (switch) |
The chat's commands |
GET |
/api/commands |
the commands of a chat on this server, JSON: each
/command with its arguments, its help and whether this
server can do it |
The chat's commands |
GET |
/api/experts |
the expert map, JSON; a dense model answers
{"moe":false,"n_layer":N,"n_expert":0,"n_expert_used":0} |
The figures as JSON |
GET |
/api/profile |
the last 30 requests' time by phase and the decode steps' times, JSON | The figures as JSON |
GET |
/api/activations |
with --observe, what each block wrote, the experts
picked and the attention of the blocks asked for, for the token a
sequence decoded last; 404 without it |
Activations |
GET |
/health |
{"status":"ok"}, without a key |
Access |
GET |
/ |
the dashboard, a page served without a key | The dashboard |
GET |
/activations |
the page of the activations, a second page served without a key | The page of the activations |
A GET route answers HEAD too. A route asked
with another method answers 405 with an Allow header, any
other path 404. Every route but /,
/activations and /health needs a key where the
server asks for one (Access).
oklm serve answers nothing under /agent/:
the routes /agent/state, /agent/events,
/agent/message, /agent/permission and
/agent/interrupt are those of an agent started with
oklm agent --listen, a server of its own on another port,
with its own token (agent-loop.md).
/v1/logs and /v1/metrics are not routes of
OKLM either: they are the paths of an OpenTelemetry collector the agent
posts its telemetry to when that is turned on (agent-settings.md).
Chat completions
Chat completions read:
model: an adapter's NAME runs the request with that adapter (the answer'smodelis then the NAME); any other value is ignored.messages, rendered by the model's chat template.contentmay be a string or an array of{"type": "text", "text": ...}parts, and with--mmprojof images, videos and sounds (below). An assistant message'stool_calls(arguments as a JSON string) and tool messages (tool_call_id) are passed to the template. Adeveloperrole is read assystem.- Sampling:
temperature,top_p,top_k,min_p,seed,frequency_penalty,presence_penalty,repeat_penalty;max_tokensormax_completion_tokens;stop(a string or up to four). toolsandtool_choice:- With tools, the template gets them, and calls are constrained by the
tool-call grammar from their opening marker
(
<tool_call>for Hermes and GLM-4.5,[TOOL_CALLS]for Mistral,<|tool_call_start|>for LFM2,<|tool_call>for Gemma 4); the wire format is the template's (--tool-format hermes|llama3|mistral|glm45|glm4|harmony|lfm2|gemma4to set it). LFM2's calls, a Python list ([get_weather(city="Paris")]), come back astool_callswith JSON arguments like the others', and so do Gemma 4's (call:get_weather{city:<|"|>Paris<|"|>}); an LFM2 file whose template says nothing of tools is recognized from its vocabulary and rendered with a template that writes them, as inoklm chat --toolsandoklm agent(inference.md). Llama 3 without<|python_tag|>and GLM-4-0414 (glm4) open a call with no marker: with"auto"the answer runs free (a Llama 3 call is constrained from<|python_tag|>when the vocabulary has it), its text held back and parsed when it ends; with"required"the grammar applies from the first token. - The answer's calls come back as
message.tool_calls, withfinish_reason: "tool_calls"andcontent: nullwhen nothing else was written. "required"(or a named function) writes the marker after the prompt, so the answer is a call; a named function's grammar admits that function only (a name not among the tools is a 400)."none"leaves the tools out.
- With tools, the template gets them, and calls are constrained by the
tool-call grammar from their opening marker
(
response_format:{"type": "json_object"}: any JSON value.{"type": "json_schema", "json_schema": {"schema": ...}}: values of the schema.- Both are constrained by a grammar, and the answer ends when the value is complete.
stream: server-sent events, onechat.completion.chunkper piece, thendata: [DONE].- The first chunk carries the role.
- With tools, only the text before a call's marker is streamed, and the calls come in one chunk at the end.
stream_options.include_usageadds a last chunk with the usage.
chat_template_kwargs: an object of more chat template variables (vLLM's and llama.cpp's field), e.g.{"enable_thinking": false}.- The answer's
contentis the reply as the model wrote it, a<think>...</think>block included. A model that writes harmony messages (gpt-oss: recognized from its template's tool-call format or from the vocabulary's<|channel|>and<|message|>tokens) is split as/v1/messagessplits it: the message of the analysis channel isreasoning_contenton the answer's message (absent when there is none), the final channel's message iscontent, and no channel markup is left in either. Gemma 4 (its tool-call format, or the vocabulary's<|channel>and<channel|>tokens) is split the same way: its thinking channel (<|channel>thought...<channel|>, written whenchat_template_kwargshasenable_thinkingtrue) isreasoning_content, what follows iscontent; after a tool's result its template opens the channel in the prompt, and the reply's start is then reasoning. Muse Glimmer (the vocabulary's<|eom|>and<|message|>tokens) likewise: the message the model addresses to itself (to=self<|message|>...<|eom|>) isreasoning_content, the message to the user iscontent. Streamed, the reasoning comes as{"delta": {"reasoning_content": "..."}}chunks before thecontentones; a reply the token limit cuts inside its analysis has reasoning and an empty content./v1/completionsis not split. reasoning_content(orreasoning) on an assistant message of the history: its reasoning, given back to a template that reads reasoning (one that splits on</think>or namesreasoning_content) as a<think>block before the content, unless the content holds one already. The template decides whether it renders it: GLM-4.7's withchat_template_kwargs{"clear_thinking": false}(preserved thinking), most others only after the last user message.continue_final_message:trueleaves the last message open, to continue a reply that was cut (see continuing a reply)."prefix": truein the last message says the same.nmust be 1.rag: retrieval from the server's--ragindex for the last user message (default off; retrieval):true, or an object ofk,crag,max_tokens,route,rewrite,followup,graph,critique.
Every whole answer carries usage, with
prompt_tokens_details.cached_tokens (the prompt tokens
found in the sequence's cache). Text that is not valid UTF-8 (a
byte-fallback token, a model writing bytes) is sent with U+FFFD in its
place, and a NUL byte is left out. A streamed piece never ends inside a
UTF-8 sequence or inside what may begin a stop string, so the pieces add
up to the whole answer.
Errors of the OpenAI routes are OpenAI-shaped:
{"error": {"message": ..., "type": ...}}. They are 400
(invalid request, a prompt longer than the context), 401, 404, 405 (with
Allow), 413 (body over 4 MiB, 32 MiB with
--mmproj; over --upload-max for a document or
a sound sent as the body) and 503 (queue full, with
Retry-After: 1).
Not served: the agent harness (ADR-0018).
Continuing a reply
A reply ends with finish_reason: "length" when its token
limit (max_tokens) or the context cut it. A client goes on
with it by sending the conversation again with the cut reply as its last
message, of role assistant, and one of two words, each as
another server has it:
"continue_final_message": truebeside the messages (vLLM's and Text Generation Inference's field), or"prefix": truein that last message (Mistral's and DeepSeek's field).
curl -s localhost:7430/v1/chat/completions -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "Write three sentences about the sea."},
{"role": "assistant", "content": "The sea is vast and"}],
"continue_final_message": true, "max_tokens": 64}'What is rendered. The chat template renders the messages before the last one and opens the model's turn, as for any request; the last message's text is written after that, with no end of turn. The model goes on from the text's last token.
What is answered. The text the model adds, and nothing of the message sent: the client appends it to the reply it holds.
usagecounts the open message inprompt_tokens; a reply cut on this server is found in the sequence's cache (cached_tokens), so continuing costs the tokens that follow.The same reply. The open text is tokenized apart from the prompt, as the model wrote it after the prompt's last token. At temperature 0 a reply cut and continued is then the reply written in one go: measured with SmolLM2-135M Q8_0 on the CPU, a reply of 48 tokens cut at 17 and one of 60 cut at 23 (
test_continue_reply). It holds where the reply's text gives back the tokens the model wrote; a reply that is not valid UTF-8 (sent with U+FFFD in place of its bytes) does not.Reasoning. The
contentof a model that writes<think>blocks holds the block, so a reply cut inside its reasoning is sent back with the block open and the model goes on reasoning, then answers. Areasoning_contentbeside an emptycontentis the same open block (<think>, the reasoning, no</think>); beside a content it is a closed block before it. A<think>the template opens itself is not written twice.A model that writes its reasoning in a channel (harmony's analysis channel, Gemma 4's thinking channel, Muse Glimmer's message to itself) is answered with
reasoning_contentapart fromcontent, and the open message says in the same two fields where the reply was cut. The turn is written again in the model's channels, with no end of channel nor of turn after the text:The open message harmony (gpt-oss) Gemma 4 Muse Glimmer reasoning_contentR, emptycontent: cut inside the reasoning<|channel|>analysis<|message|>R<|channel>thought, a newline, Rto=self<|message|>RR and a contentC: cut inside the answer<|channel|>analysis<|message|>R<|end|><|start|>assistant<|channel|>final<|message|>C<|channel>thought, a newline, R<channel|>Cto=self<|message|>R<|eom|><|start|>assistant to=user<|message|>CC alone <|channel|>final<|message|>CC to=user<|message|>CCut inside the reasoning, what the model adds is answered as
reasoning_contentuntil it closes the channel, then ascontent; cut inside the answer, ascontentalone. A channel Gemma 4's template opened itself is not written twice. Measured on the CPU with Gemma-4-E2B-it Q4_K_M at temperature 0, thinking on ("Name three colors and write one sentence about each. Think briefly first."): the reply in one go is 228 tokens, 618 bytes of reasoning and 270 of answer; cut at 84 tokens (inside the reasoning) and continued it adds the other 309 bytes of reasoning and the 270 of answer in 144 tokens, 112 of the prompt's 114 read from the cache; cut at 198 (inside the answer) it adds the answer's other 146 bytes in 30 tokens, 224 of 228 from the cache. Both are the reply in one go, whole and streamed.test_continue_channelsholds the three formats to it on a tiny model each, cut after every count of tokens from the first.Cut inside a channel's header. A reply cut before the first header of its turn is complete (
<|channel|>or<|channel|>analysis, Gemma 4's<|channel>,to=self) has written markup and nothing else: it is answeredlengthwith an emptycontentand noreasoning_content, never with the piece of markup as text. Sent back as the open last message (emptycontent), it opens the turn afresh: the prompt is the one the template renders, the model writes the header again, andusage.prompt_tokensis the cut request's. An open last message whosecontentis such a piece of markup, with noreasoning_content(a client that got it from a server that answered it), is read as empty. The header of a later message of the turn (harmony's<|start|>assistant<|channel|>finalafter the reasoning) is left out of a cut reply the same way: the reply is the reasoning, and going on with it the model closes the reasoning and writes the header again. On the tiny models: 3 cuts inside the first header for harmony and Gemma 4, 6 for the recipient format, each continued whole and streamed into the reply in one go.Cut inside a tool call. With
tools, a reply the token limit or the context cuts after a call's marker and before the call's end is answered withfinish_reason: "length"and notool_calls: a call that is half written is never given as a call (a parser that repairs JSON would make one of the half). Its text so far is in the message, as the model wrote it from the first call marker on, beside thecontentthat holds the text before it:{"role": "assistant", "content": "", "tool_call_open": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Par"}A stream sends it as one
{"tool_call_open": "..."}delta before the chunk that carries the finish reason. The client sends the message back as the open last one,tool_call_openas it got it, with the sametools(for a model whose reply is one text, the call's text at the end of thecontentis read the same way: a call marker in an open message's text opens a call). The call's text is written after the content, the tools' grammar reads it, and generation goes on under the grammar from there, so the call is completed as one written in one go. The answer then holds the whole call intool_calls,finish_reason: "tool_calls", and ascontentwhat the model wrote outside the calls. Cut again inside the call, it islengthagain andtool_call_openholds the piece just written, which the client adds to the text it holds. Withtool_choice: "required"the marker is not written a second time. A model that writes its reasoning in a channel gets its reasoning as a closed channel before the call's text. This covers the formats whose calls open with a marker (Hermes and Qwen, Mistral, GLM-4.5, harmony, LFM2, Gemma 4); Llama 3's bare JSON and GLM-4's name line have none, and a reply of theirs cut inside a call is answered as before (without a marker a half call cannot be told from text)./v1/messageshas the same in its own shape (a reply cut inside a tool call).Measured on the CPU at temperature 0 with Qwen3-0.6B Q8_0 and one tool (
get_weather, three required arguments): the reply in one go is a call of 33 tokens; cut after each of its first 32 tokens and continued, whole and streamed, the 64 answers are that call (with thinking on, a reply of 135 tokens: the same 32 cuts inside its call, the same 64 answers). Gemma-4-E2B-it Q4_K_M, whose calls are<|tool_call>call:NAME{...}<tool_call|>: a call of 27 tokens, cut after each of its first 26 and continued, 52 answers that are that call.test_continue_calldoes it on the tiny model with a required call of 64 tokens, cut after each of 63 counts.Refused, with a 400: either word when the last message is not an
assistantone; a last message withtool_calls(parsed calls are a closed turn: a call the reply was cut inside goes back as text, intool_call_open).Nothing else changes. A request with neither word is read as it always was: a last
assistantmessage is a closed turn and the model answers after it./v1/messageskeeps its own rule (a lastassistantmessage of text is a prefill; with atool_call_openit is an open message as here).With
tools, a call the model opens in what it adds is constrained from its marker as in any reply.
oklm chat, oklm agent, the chat window and
the dashboard's chat continue their replies this way (inference.md, the dashboard's chat).
Compacting a conversation
POST /api/compact does for a client what the chats do
themselves when a context is nearly full (inference.md): it says which
messages of a conversation stay as they are and writes the summary of
the others, by the code oklm chat, the chat window and the
agent share (src/client/compaction.c), with the served
model's tokenizer for the counts and the served model for the summary.
The dashboard's chat is a page and cannot run that code: it asks the
server. Nothing is kept by the server; the client replaces the messages
itself.
curl -s localhost:7430/api/compact -H 'Content-Type: application/json' -d '{
"messages": [{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is a tide?"},
{"role": "assistant", "content": "A tide is the regular rise and fall of the sea..."},
{"role": "user", "content": "Name a fish."},
{"role": "assistant", "content": "A cod."}]}'| Field of the request | Meaning |
|---|---|
messages |
the conversation, as a chat completion takes it. A message whose
content has a part that is not text (a picture, a sound, a video), or
that carries "pinned": true (an attached document), stays,
with its reply |
focus |
what the summary is asked to dwell on
(/compact FOCUS) |
summary |
an earlier compaction's summary: it is summarized with the messages that go |
reply_tokens |
the tokens of a cut reply about to be gone on with, which is not among the messages: the plan leaves it room, and keeps the last exchange alone |
auto |
true: compact only when the conversation is past the
chats' threshold for this context (threshold in the
answer), as a chat does by itself before a message;
max_tokens is then the reply's token limit the threshold
reserves |
model |
an adapter's name, as in a chat completion |
count |
in place of messages: an array of texts to count,
nothing compacted (below) |
The answer:
{"compacted":6,"kept":2,"window":256,"used":170,"threshold":115,"used_after":103,
"keep":[false,false,false,false,false,false,true,true],
"summary":"...","note":"This session continues an earlier conversation that grew too long for the context; it was summarized:\n\n"}keep has one value for each message that is not a
system one, in order; summary is the model's
text, asked with reasoning off, after the conversation when the context
has room for the request and else from a transcript read in parts;
note is the sentence the chats put before a summary, so the
client's system message is its system prompt, a blank line,
note and summary. used and
used_after are the conversation rendered by the model's
template and counted by its tokenizer, before, and with the note in the
place of what it replaces. {"compacted":0, ...}, with no
keep nor summary, when nothing is compacted:
the messages that would go free too little, or, with auto,
the conversation is under the threshold. Errors are OpenAI-shaped: 400
(no messages, a message without a role), 500 (the model
wrote no summary), 503 (the queue is full).
Measured on the CPU with SmolLM2-135M Q8_0 in a context of 256 tokens
(test_compact_route): a conversation of four exchanges, 170
tokens, has six of its eight messages replaced by a summary of 164 bytes
and takes 103 tokens after.
The counts alone
A chat that runs the compaction itself (the chat window, attached to
a server at an address) needs only what it cannot have: the served
model's tokenizer. count, an array of strings, asks for it;
nothing is compacted, the model is not asked anything, and
messages is not needed:
curl -s localhost:7430/api/compact -H 'Content-Type: application/json' -d '{"count": ["What is a tide?", "A cod."]}'
{"window":4096,"counts":[5,3]}counts has the tokens of each text, in order, as the
tokenizer writes the text alone (no template around it, special tokens
written in it read as text); window is the context. An
entry that is not a string is a 400
(count[1] is not a string). A server from before the member
answers 400 for the missing messages: the window then
counts by an estimate (gui.md).
Resizing the context
POST /api/context with {"n_ctx": N} gives
every sequence a context of N tokens without restarting the server or
loading the model again; GET /api/context reads the current
one:
curl http://127.0.0.1:7430/api/context
{"n_ctx":4096,"trained":40960,"min":16,"resizable":true,"token_bytes":114688,"cache_tokens":16384,"cache_bytes":1879048192,"host":true,
"backend":"cpu","n_gpu_layers":0,"n_layer":28,"n_devices":0,"n_cpu_moe":0,"expert_gpu":0,"cache_type_k":"f16","cache_type_v":"f16",
"sampling":{"temperature":0.8,"top_k":40,"top_p":0.95,"min_p":0.05,"repeat_penalty":1}}
curl -X POST http://127.0.0.1:7430/api/context -d '{"n_ctx":16384}'
{"n_ctx":16384,"trained":40960,"min":16,"resizable":true,"token_bytes":114688,"cache_tokens":65536,"cache_bytes":7516192768,"host":true,
"backend":"cpu","n_gpu_layers":0,"n_layer":28,"n_devices":0,"n_cpu_moe":0,"expert_gpu":0,"cache_type_k":"f16","cache_type_v":"f16",
"sampling":{"temperature":0.8,"top_k":40,"top_p":0.95,"min_p":0.05,"repeat_penalty":1}}
cache_tokens and cache_bytes are the cache
the server has: the tokens its requests share, whatever the context of
each (on a device, what was left beside the model when the cache was
made). With "dry_run": true they are the cache another
context would get, the cache in place counted as freed first.
The answer also tells what the server was started with, for a client
that shows it: backend (cpu, or the name of
the device that holds the output head), n_gpu_layers (the
layers on devices) over n_layer (the model's),
n_devices (the devices those layers are spread over,
--devices; 0 on the CPU), n_cpu_moe (of the
layers on devices, those whose routed experts stay in host memory and
run on the CPU, --n-cpu-moe), expert_gpu (the
bytes of the device tier of experts, --expert-gpu; 0
without one), cache_type_k and cache_type_v
(the types of the cache's keys and values, as -ctk and
-ctv name them), and sampling, the defaults of
what a request leaves out: temperature, top_k,
top_p, min_p, repeat_penalty
(--temp, --top-k, --top-p,
--min-p, --repeat-penalty), then
seed and max_tokens when the server was
started with --seed and -n (the defaults of a request);
without them the two members are left out, since a request then draws
its own seed and has no length but its sequence's. The chat window takes
these sampling defaults as its own when it attaches to the server and
shows the rest in its model settings (gui.md); an older server answers
without these members, and a client then keeps its own values.
The defaults of a request
What a generation request leaves out is the server's, on the three
routes that generate (/v1/chat/completions,
/v1/completions, /v1/messages); a request's
own value always wins.
| Option | Member of a request | Without the option |
|---|---|---|
--temp T |
temperature |
0.8 |
--top-k K |
top_k |
40 |
--top-p P |
top_p |
0.95 |
--min-p P |
min_p |
0.05 |
--repeat-penalty R |
repeat_penalty |
1 |
--seed S |
seed |
a new seed for each request, from the clock and the request's number |
-n N |
max_tokens, max_completion_tokens |
no length: the reply ends with the model's turn, or when the sequence is full |
--seed S is the seed of each request
that names none, not of the first one alone: the sampler of every such
request starts from S, so the same request sent twice gives the same
text, and two different requests are each reproducible. (The text is the
same as far as the logits are: requests decoded together or a prompt
found in the cache can move a logit's last bits, which --deterministic
rules out.) It is what oklm run --seed does for one prompt.
A server that should vary from a reply to the next is started without
it.
-n N bounds a request that gives no length; a request
that gives one, larger or smaller, has its own.
/v1/messages requires max_tokens as
Anthropic's API does, unless the server was started with
-n: the member may then be left out, and a reasoning budget
is checked against N. GET /api/context tells both in its
sampling object (Resizing
the context).
oklm serve -m qwen3-0.6b --seed 7 -n 64
curl http://127.0.0.1:7430/api/context
{... "sampling":{"temperature":0.8,"top_k":40,"top_p":0.95,"min_p":0.05,"repeat_penalty":1,"seed":7,"max_tokens":64}}
token_bytes is what one cached token takes over every
layer, cache_tokens and cache_bytes what the
cache holds over every sequence, and host whether it is in
host memory. To know what another length would take without changing
anything, add "dry_run": true
(oklm_server_context_plan()):
curl -X POST http://127.0.0.1:7430/api/context -d '{"n_ctx":1048576,"dry_run":true}'
{"dry_run":true,"n_ctx":1048576,"trained":40960,"token_bytes":114688,"fill_bytes":120259084288,
"cache_tokens":4194304,"cache_bytes":481036337152,"available":127912747008,"host":true,"fits":false}
fill_bytes is one conversation filling the context. On
the CPU the paged cache is four conversations of the context's length,
in host memory: it grows with the context, and fits
compares it with the memory available (what the cache in place holds
counted as free). On a GPU the cache is what fits on the device whatever
the context, less what the encoders and the sound models are still to
take there (Sharing the device):
fits is true, and a conversation longer than
cache_tokens stops where the cache ends. With
--kv-mem -1 the whole context is reserved for every
sequence.
The scheduler and its cache are made again for the loaded model
(oklm_server_resize()):
- Between requests. No new request enters while it
runs, and it waits up to three seconds for those being answered; if they
go on,
409and nothing changes. Every cached prompt is lost, the prefix cache in memory included, so the next request of a conversation is evaluated whole: a cache is laid out for one length and cannot be carried into another. - Above the trained context. Accepted, with a
warningmember in the answer (and in everyGETwhile it holds): a model reads badly beyond what it was trained with. A model with a table of positions is capped at it by the context. - Memory. A length whose cache does not fit in the
available host memory (
"fits": falsein the dry run) is refused with507and the two figures;"force": trueasks for it anyway. A cache on a device keeps its size, so nothing is refused there: the layers stay where the start placed them, so a context above what the device has beside them is made with the cache that is left (cache_tokensbelown_ctxin the dry run), where a server started at that context would place fewer layers on the device (memory-plan.md). When the new context cannot be created the one before is made again and the answer is500; if that fails too the server answers503until it is started again. - Access. The API key like every other route.
oklm serve --fixed-contextrefuses the change with403("resizable": false), for a server whose clients must not be able to drop each other's cache. - Bad requests are
400(a body that is not a JSON object,n_ctxbelow 16).
oklm-gui --url uses it for its Context slider and its
model settings, and oklm agent --url for
/context-size N. Tested by test_context_resize
in tests/server/test_server.c, the settings and the
sampling defaults included.
Switching the model
POST /api/model with {"model": NAME} loads
another model in place of the one served, the server running:
curl -s http://127.0.0.1:7430/api/model -d '{"model": "qwen3-4b"}'
# {"model":"Qwen3-4B-Q4_K_M","n_ctx":4096,"vision":false,"audio":false,"switches":1,"seconds":7.962,"loaded_first":false}| Member | Meaning |
|---|---|
model |
the name of an installed model of the catalogue that a chat is
served with (chat, code, security, vision, audio with a model file), or
a file of the model directory of the server's user named from that
directory (qwen3-8b/Qwen3-8B-Q8_0.gguf, the
.gguf optional). Never a path outside it: no leading
/, no .. |
mmproj |
optional: the projector to open with it, named the same way. Without it a catalogue model comes with the catalogue's projector when it has one, a file with none |
wait |
optional: seconds the server waits for the requests being answered, 600 at most (3 by default) |
The answer gives the name the model now has in the API (its file's,
without .gguf, as at start), the context's length, whether
images or sounds are read, how many switches the server made, how long
this one took, and loaded_first: true when the model was
loaded before the one before it was freed (step 3 below).
GET /api/models/installed lists what can be named, with
switch true on a server that takes the request.
A reload in place, not a new process. The other way
to change a server's model is the one the server manager has for every
option: stop the process and start its command again with another
-m (The servers that
run). That loses the port for a moment, the clients' connections,
the registry entry, the transcription and speech models already loaded,
and it cannot fall back when the new model does not start. The server
already had what a reload needs: one gate every request enters under and
a resize closes (POST /api/context), a scheduler that is
the only holder of the context and its cache, and figures
(/api/stats, /api/model,
/api/context) that read the model through the server at
each call. So the switch is a resize that also replaces the model
(oklm_server_switch()):
- The new file is opened first, nothing closed: it
loads, it has a tokenizer, its chat template reads, a
--draftmodel shares its vocabulary, its projector is its width. A file refused here costs the running server nothing (400,404for a name that is no model, the model served named in the message). Its weights are mapped, not yet on a device. - The server waits until no request is being
answered: new ones wait at the gate, requests that generate without a
stream are waited for to their end, streams for
waitseconds; if they go on,409and nothing changes. - The new model's memory is planned, before anything
is freed (
oklm_mem_plan_loaded(): the plan ofoklm plan --servefor a model that is loaded, with the server's own settings, its cache sized byoklm_kv_cache_rule()as the scheduler will size it). First with the model served freed: what it holds on its device and in host memory is counted as free, and a model the host's or the device's memory does not hold then is refused here, nothing touched (507,NAME does not fit: it takes 20.4 GiB on vulkan, which has 18.3 GiB for it with the model served freed ...; OLD is served as before). A model that does not take the server's settings (a quantized cache its heads are too narrow for) is refused here too (400). A cache that would hold less than one context is not a refusal: a server starts with one. Then beside the model served, what that one holds on the device taken out of the device's budget: when the memory holds both and the new model gets the same cache and the same layers on the device either way, it can be loaded first. - Loaded first, when the memory holds both: the
scheduler is made for the new model while the one served stays whole,
then the old scheduler, its cache and the old model are freed
(
"loaded_first": true). A failure here costs nothing: the model served was never touched, and when the reason is room (the device gave less than the plan counted) the switch goes on as in step 5. This is the case on the CPU while the memory lasts, and on a device with a cache of a given size (--kv-mem); a device whose cache is what is left beside the model holds no second model, since that cache took the room. A server with--observeor--prefix-diralways frees first. - Else the old model goes first: the scheduler, its cache, the prefix cache in memory, the model with its weights on the device, its projector and the embeddings kept of pictures and sounds; then the scheduler is made for the new model.
- The scheduler is made for the new model with the
server's settings either way: the backend and
--gpu-layers, the context's length (the onePOST /api/contextgave, else-c; the new model's own default when neither was given),--kv-mem, the share the encoders and the sound models take of the device (Sharing the device) with its refusals,--draft,--spec-ngram,--prefix-dir, the sampling defaults. The projector is placed on the device again. The embedding, reranking, transcription and speech models and the retrieval indexes are not touched. - A switch never ends with nothing loaded while a
model can be. When the new model cannot be served after the old one was
freed, the model before is loaded again from its file with its settings
and serves as it did:
507with the reason andthe previous model, NAME, was restored and serves as before. Only when that fails too does the server saythe previous model could not be restored either: no model is loadedand answer503, until a switch succeeds or it is started again.
--chat-template and --tool-format belong to
the file the server was started with: another file gets its own template
(or the catalogue's) and its tool format detected, and they come back
with the first file. The clients' conversations are theirs: a chat sends
its whole conversation with each message, so it goes on with the new
model, which reads it from its start.
What does not switch: a server with --lora-module
adapters or a --lora merged at load (they belong to their
model), and one started with --fixed-model; they answer
403, and GET /api/commands leaves
models out of their features. The number of connections
served at once stays the one made at start.
Run on 2026-11-15 on the RX 7900 XT (OpenCL, -c 4096),
the server started on Qwen3 0.6B: Qwen3 4B Q4_K_M loaded in 7.96 s and
answered (7 030 blocks of cache beside it), back to Qwen3 0.6B in 2.81
s, SmolVLM 256M with its projector in 0.31 s
("vision": true, the encoder on the device). With a stream
of 400 tokens running, "wait": 0.2 answered
409 and the stream went on to its end;
"wait": 120 waited for it and loaded. With
--kv-mem 12000 and a Whisper large-v3 still to be loaded on
the device, Qwen3 8B Q8_0 was refused by the memory plan
(507:
--kv-mem 12000 does not fit on opencl with what shares it: the device has room for 10550 MiB beside the model, of which 3725 MiB are still to be taken by the encoders ...; Qwen3-0.6B-Q8_0 is served as before)
and the first model answered the next request.
Run again on 2026-11-16 on the same card, on Vulkan and on OpenCL,
the server started on Qwen3 0.6B Q8_0 at 4096 tokens. With
--kv-mem 512, Llama 3.2 1B Q4_K_M was loaded before Qwen3
0.6B was freed ("loaded_first": true, 0.31 s on Vulkan,
0.33 s on OpenCL) and answered. With the cache left to what fits (161216
tokens), the old model was freed first
("loaded_first": false, 0.80 s on Vulkan, 0.34 s on OpenCL)
and the new one had the cache its plan gives it (559680 tokens,
oklm plan --serve), on both. On Vulkan the driver gives
memory back a moment after the buffers are released (20 GiB over 150 ms
on this card): the server waits for the device's own figure of what is
free to show what was freed before it sizes the new cache (two seconds
at most), after a switch and after a resize; without the wait that cache
was one block (16 tokens) and the reply empty. With
--kv-mem 16000 --gpu-layers 99, Qwen3 8B Q4_K_M was refused
by the plan with nothing touched (507:
Qwen3-8B-Q4_K_M does not fit: it takes 20.4 GiB on vulkan, which has 18.3 GiB for it with the model served freed (fewer --gpu-layers, a smaller --kv-mem); Qwen3-0.6B-Q8_0 is served as before,
the same figures on OpenCL), and Qwen3 0.6B answered the next request
with its cache of 146272 tokens. A load that fails after the model
served was freed was not produced on the card: the plan refuses what
does not fit before anything is freed, and a card told to be larger than
it is (OKLM_GPU_MEM_MB=40000) did not fail the load, its
driver giving memory beyond the card's. That path is run on the counting
device of test_switch_memory.
Tested by test_model_switch in
tests/server/test_server.c (two tiny models of different
widths and templates, the second loaded before the first is freed) and
by test_switch_memory there, on a counting device whose
size and whose allocations the test holds: a device with room for both
(loaded first), one without (freed first), a model the plan refuses with
nothing touched, a load that fails after the model served was freed
(restored, and answering), a device that gives nothing any more
(503, no model loaded, then the next switch serves); the
page's switch group
(CHECKS=switch sh tools/check_dashboard.sh) and
make gui-test.
Images and sounds
With --mmproj, a message's content parts may be images,
videos and sounds, in OpenAI's shapes (a video in the shape the
OpenAI-compatible servers that take one use, vLLM's and Qwen's own API),
as base64 data only (the server fetches no URL and opens no file):
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
{"type": "video_url", "video_url": {"url": "data:video/ogg;base64,..."}, "fps": 2, "frames": 8}
{"type": "input_audio", "input_audio": {"data": "<base64>", "format": "wav"}}oklm serve -m models/smolvlm-256m/SmolVLM-256M-Instruct-Q8_0.gguf \
--mmproj models/smolvlm-256m/mmproj-SmolVLM-256M-Instruct-f16.gguf
- Decoding. Images as
oklm describereads them (PPM here; PNG, JPEG, BMP, TGA through Allegro), sounds likewise (PCM WAV here; FLAC, Ogg Vorbis, Opus, MP3 through Allegro), told by their first bytes; at most 16 in a request. The projector takes one kind: the other is refused, as is a request with media to a server without--mmproj. - Videos. A
video_urlpart is decoded by the codeoklm describereads a video with (multimodal.md): Ogg Theora by the server itself, any other container throughffmpegon a server started with--ffmpeg(a subprocess given a request's bytes, hence the operator's choice; without it the refusal names both ways out). The frames are sampled asoklm describesamples them, one a second and 8 at most;fps(above 0) andframes(1 to 32) say otherwise, as numbers of the part or of itsvideo_urlobject, and--tiles SIDEbounds a frame's area as it bounds an image's. What the model gets is whatoklm describegives it:- Qwen2-VL and Qwen2.5-VL read the video as one item, as their own
code defines one: the frames two by two through the two patch kernels,
the grids of all the pairs between one pair of vision tokens, the time
position advancing with the seconds between the frames kept (multimodal.md).
The scheduler gives each piece of it to the model with the grid's
columns, rows and time step (
oklm_eval_batch_video()). - The other image projectors get the frames as still images, each
announced by its time: the part becomes the lines
Video (4.0 s):, thenFrame at 0:01:and the frame's marker, the next part right after them. This isoklm describe's text without the file's name (a request has none). - An audio projector (Ultravox) gets the video's sound track as a
sound, where the part stands; a video without sound is refused. A
request cannot make the server decode without bound: the frames asked
for times a frame's width and height are at most 66 355 200 pixels (8
frames of 3840 x 2160, or 32 of 1920 x 1080;
oklm_server_params.video_pixels), checked from the stream's headers before a frame is decoded, and refused with the figures (the video is too large (8 frames of 7680 x 4320 are 265.42 megapixels, above the limit of 66.36: fewer frames or a smaller video)). While a video is read up to twice the frames asked for are held, about 400 MB at the limit. A request holds at most 64 media items, a video one for Qwen's projectors and one a frame for the others, and its body 32 MiB as with images. The other refusals:this server takes no videos (no --mmproj),video_url must be a data: URL holding base64 (the server fetches nothing),the video is not base64,the video cannot be decoded (...)with the decoder's reason,a video's frames must be an integer from 1 to 32 ...,a video's fps must be a number above 0 .... A file cut short gives the frames read before the cut, or is refused when none was.
- Qwen2-VL and Qwen2.5-VL read the video as one item, as their own
code defines one: the frames two by two through the two patch kernels,
the grids of all the pairs between one pair of vision tokens, the time
position advancing with the seconds between the frames kept (multimodal.md).
The scheduler gives each piece of it to the model with the grid's
columns, rows and time step (
- The prompt. Each part becomes a marker where it
stands, written as
oklm describewrites it (oklm_mm_content(), the one function both use), by the projector: the next part right after a marker for SmolVLM, Gemma 3, Qwen2-VL and Qwen2.5-VL, whose templates write content in parts so; a new line after a marker that another part follows for LLaVA 1.5 and 1.6, the form of the model's own conversations (multimodal.md has the table and the references); nothing before a marker, nothing between two texts. For Ultravox a sound right after a text gets a space before it, the model's own form. On a server with the projector of Qwen2-VL or Qwen2.5-VL,"chat_template_kwargs": {"add_vision_id": true}(or"add_vision_id": truebeside the messages) numbers the conversation's pictures and videos as the models' template does with that variable:Picture 1:,Picture 2:,Video 1:right before each, counted through all the messages (multimodal.md); no label without it, which is the template's default and llama.cpp's, and no label for another projector./v1/messageshas no such member and writes none. An image part and a question are then the tokensoklm describegives that picture and that question (-p), and the same answer under greedy decoding; a message without media is unchanged. With a sound the order of the parts matters: Ultravox's own prompt has the sound after the question (a text part, then theinput_audiopart, which is whatoklm describewrites). Asked to transcribe jfk.wav, Ultravox v0.5 1B writes the sentence in that order and quotes a hymn when the question comes after the sound. The chat template renders the conversation, or, when it drops the markers (SmolVLM's reads structured content only), the projector's own format does. On a server with SmolVLM's projector a conversation without any image is rendered by that format too: the template drops a message's text as it drops the markers. Each marker becomes its item's rows: an image's tiles between their separators, a sound's embeddings, asoklm describelays them out. Gemma 3 reads an image's 256 rows attending to one another: the scheduler gives them to the model in one piece, waiting a step when the batch has no room left rather than cutting them (multimodal.md). LLaVA 1.5's files have no chat template: a server with that projector writes Vicuna's turns, and so does one started on the catalogue's model, with or without its projector, from the format the catalogue names (multimodal.md). LLaVA 1.6's image is 577 to 2 928 rows (an overview and the tiles of the resolution that suits it, multimodal.md): give the context room for the images of a conversation (-c), or--tiles 336for the overview alone. Qwen2-VL and Qwen2.5-VL read an image at its own size, up to 4 096 embeddings (3.2 megapixels) and three quarters of a sequence's context: a larger picture is reduced to that rather than refused for its length, and the bound follows a context resized through/api/context(--tiles SIDEsets the largest area instead), its rows placed by their row and column in the image, in as many pieces as the steps cut them into (multimodal.md). - Encoding. The encoders run in the request's own
thread when it arrives, one request at a time, not in the scheduler's
steps. When the model has layers on a device, the blocks of the image
encoder run on that device too, as
oklm describedoes (vision_devicein/api/stats): each image is one pass on the device, which the scheduler's steps wait for (sharing the device). So do the blocks of the audio encoder (audio_device), each 30-second chunk of a sound a pass (multimodal.md). A video's pairs of frames are a pass each. The embeddings of recent items (256 MiB) are kept by content hash, so a picture sent again skips its encoder. - Decoded once. The content hash is of the decoded
pixels or samples, so finding an item's embeddings took decoding it
first, at every request, and a chat window sends every earlier part
again with each message. The server therefore remembers the last 128
media parts it decoded by what was sent: a hash of the part's encoded
bytes and their count, its kind, a video's
fpsandframesand whether ffmpeg was allowed, the slicing side (--tiles) and the bound on an image's embeddings. Each maps to the content hashes of the items the part became and to what the prompt needs of it beside their embeddings (an image's size, a video's frame sizes, times and length). A part found there whose items' embeddings are all still kept is not decoded: its items are named by their hashes (oklm_mm_item.hash) and built from the embeddings, which the request copies when it finds the part, so that nothing the cache drops meanwhile can miss. The rows, their keys in the paged cache and the answer are those of the decoded part. A part whose embeddings were dropped, another sampling of the same video, or the same file after/api/contextchanged the bound, is decoded again; the least recently asked part is forgotten first. The same picture in another encoding is another part here: it is decoded, and then found by its pixels as before.media_decodedandmedia_knownin/api/statscount both. Measured on the CPU, a request sent a second time (max_tokens1, every row but the last in the prefix cache): an 8-frame 1920 x 1080 Theora video 0.153 s -> 0.0075 s, a 4000 x 3000 JPEG 0.202 s -> 0.009 s, ten minutes of Ogg Vorbis 1.76 s -> 0.0125 s (the tiny test model, where little else is left of the request); Qwen2-VL 2B with a four-frame 448 x 336 clip 0.070 to 0.084 s -> 0.044 to 0.055 s, the rest being the model's two rows and one token. For the encoders that cut an image into tiles (SmolVLM, LLaVA 1.6) the grid is worked out again by cutting an empty picture of the same size, which costs the cutting and not the decoding. - Cached by content. The paged cache keys a media row
by its item's content hash (of the decoded pixels and the slicing side,
or of the samples) and the row's index within it (ADR-0019), never by
its floats: a conversation resent with the same picture, under any file
name or encoding, finds its blocks; another picture does not. A video
read by Qwen's projectors is one item, hashed over its frames' pixels,
their count, the seconds between them and the slicing side: the same
video with the same sampling finds its embeddings and its blocks again
(without being decoded, when the same bytes are sent: Decoded
once, above), and another
fpsorframesis another item, encoded anew. Given as still images, each frame is hashed alone, so the frames two samplings share are found again. The prefix cache on disk keeps a head holding a video as it keeps one holding an image. - The scheduler. A prompt piece is tokens or one
item's rows, the item's embeddings given in place of the tokens'
(
oklm_seq_tokens.embdand.media). Preemption replays them from the embeddings; speculative drafts never come from media rows.
Anthropic Messages API
POST /v1/messages speaks Anthropic's wire format, so the
Anthropic SDKs and the coding CLIs built on that API work against the
server (ANTHROPIC_BASE_URL pointed at it, the key as
x-api-key). A request is translated into the chat
completion request above (src/server/messages.c) and takes
the same path: the chat template, sampling, the tool-call grammar, the
scheduler, the prefix cache; the answer is written in Anthropic's shape.
anthropic-version and anthropic-beta headers
are accepted, whatever their value, and ignored.
| Anthropic | Served as |
|---|---|
model |
as for chat completions: an adapter's NAME selects it, any other value runs the model; the answer names the served model |
max_tokens (required, a positive integer; not required
by a server started with -n) |
max_tokens: reasoning and text together |
system (a string, or text blocks joined by a blank
line) |
a first system message |
messages[].role user /
assistant |
the same roles |
text block |
a text part |
image block, source.type: "base64" |
an image part (a data: URL): with --mmproj
only, else 400; url and file sources are
refused (the server fetches nothing) |
video block, source.type: "base64", with
fps and frames if wanted |
a video part (video_url, a data: URL),
sampled and given to the model as above. Anthropic's format has no video
block: this one is shaped as its image block, in user
messages |
tool_use block (assistant) |
the message's tool_calls, input as the
arguments |
tool_result block (user) |
a tool message before the rest of the user message:
tool_use_id as tool_call_id, the tool's name
found in the earlier tool_use, content a
string or text and image blocks; is_error: true prefixes
Error: |
thinking, redacted_thinking blocks
(assistant) |
left out, as chat templates leave out earlier turns' reasoning |
a last assistant message of text |
a prefill: written after the rendered prompt, the answer continues it |
tool_call_open (a string) in that last
assistant message |
the text of a tool call the answer was cut inside: written after the
message's text, the call goes on under the tools' grammar (below). An OKLM member; a 400
when it is not a string or the message holds a tool_use
block |
tools[]
{name, description, input_schema} |
OpenAI tools, input_schema as
parameters |
tool_choice auto / any /
tool + name / none |
auto / required / a named function /
none |
temperature, top_p,
top_k |
the same |
stop_sequences |
stop (four at most) |
stream |
named server-sent events (below) |
thinking {type: "enabled", budget_tokens}
/ "adaptive" / "disabled" |
the chat template's enable_thinking true / true /
false; budget_tokens caps the reasoning (below) |
output_config.format or output_format
{type: "json_schema", schema} |
response_format json_schema: the text
constrained by the schema's grammar |
metadata, service_tier,
cache_control on any block,
disable_parallel_tool_use |
accepted and ignored |
The answer is
{"id": "msg_...", "type": "message", "role": "assistant", "model", "content": [...], "stop_reason", "stop_sequence", "usage"}:
- Content. Text the model wrote inside
<think>...</think>(or after a prompt the template ends with<think>; for gpt-oss, the message of its analysis channel) comes first as athinkingblock with an emptysignature; then the text as atextblock (the blanks after the reasoning dropped); then each tool call as atool_useblock,inputa JSON object,idthe model's ortoolu_.... stop_reason.tool_usewhen calls were made,max_tokensat the limit (or the context's end),stop_sequencewithstop_sequenceset to the string found, elseend_turn.tool_call_open. Only in an answer cut inside a tool call: the call's text so far (below).usage.input_tokens(the prompt tokens evaluated),cache_read_input_tokens(those found in the prefix cache, so the two add up to the prompt),cache_creation_input_tokens(0),output_tokens.
Thinking budget. With
"type": "enabled", budget_tokens is required,
at least 1024 and below max_tokens, as Anthropic's API asks
(a 400 otherwise; count_tokens does not compare it with
max_tokens). Once the model has written that many tokens
inside a <think> block (the one a template opens
after the prompt included), the server writes
</think> and a blank line in its place, one token
after another instead of the sampled ones (after a newline, as the
templates do, when the reasoning did not end with one), and the model
goes on with the answer. The cap is exact: the thinking
block holds the first budget_tokens tokens of the
reasoning, the same text as an uncapped answer's. The closing tokens
count in output_tokens and against max_tokens,
which still bounds reasoning and answer together. A block the model
opens again is closed after its first token (the budget covers all the
reasoning). "adaptive" sets no cap. Not capped: a reasoning
block under a grammar (a call being written), and models whose reasoning
is not marked with <think>. Measured with Qwen3-0.6B
Q8_0 (budget 1024, max_tokens 1300, a proof asked): the
reasoning closed at 1024 tokens, the answer 887 characters after it.
Streamed, the events come in Anthropic's order, each
event: NAME with data of the same
type: message_start (the message with empty
content; its usage counts the whole prompt as input_tokens,
the cache being known later), one ping, then for each block
content_block_start, its deltas
(thinking_delta then an empty signature_delta,
text_delta, or one input_json_delta holding a
call's whole input as partial_json) and
content_block_stop; then message_delta with
stop_reason, stop_sequence and the final
usage, and message_stop. Reasoning and text stream as they
come; with tools, text stops before a call's marker and the calls come
at the end, as for chat completions. A generation failure ends the
stream with an error event. A client that goes away,
streamed or not, cancels the request.
A reply cut inside a tool call
With tools, an answer the token limit or the context
cuts after a call's marker and before the call's end has
stop_reason: "max_tokens" and no
tool_use block for that call: a call that is half
written is never given as a block (its input would be a
JSON object made of half a text). The call's text so far, as the model
wrote it from the first call marker on, is a member of the message,
beside content, which holds the reasoning and the text
written before the call:
{"id": "msg_...", "type": "message", "role": "assistant", "model": "...",
"content": [],
"stop_reason": "max_tokens", "stop_sequence": null,
"usage": {"input_tokens": 12, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 0, "output_tokens": 9},
"tool_call_open": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Par"}Streamed, no tool_use block is opened for it and the
text is a member of the message_delta event, beside
delta and usage:
event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"max_tokens","stop_sequence":null},"usage":{...},"tool_call_open":"<tool_call>\n{\"name\": \"get_"}
The client goes on the way this API continues an answer, with a last
assistant message (the prefill): the content
it got (a string, or the answer's blocks: text, and
thinking, whose text is written back as a closed reasoning
block before the rest), and tool_call_open as it got it,
with the same tools and tool_choice:
curl -s localhost:7430/v1/messages -H 'Content-Type: application/json' -d '{
"max_tokens": 300,
"tools": [{"name": "get_weather", "input_schema": {"type": "object",
"properties": {"city": {"type": "string"}}, "required": ["city"]}}],
"messages": [{"role": "user", "content": "What is the weather in Paris?"},
{"role": "assistant", "content": [],
"tool_call_open": "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Par"}]}'The message is then read as the chat completions' open last message
(continuing a reply): the call's text
follows the message's text, the tools' grammar reads it and generation
goes on under it. The answer holds the whole call as a
tool_use block, stop_reason: "tool_use", and
as text what the model wrote outside the calls; nothing of
the call's text is given as text. Cut again inside the call, it is
max_tokens again and tool_call_open holds the
piece just written, which the client adds to the text it holds. With
tool_choice any or tool the
marker is not written a second time. A last assistant
message without the member stays a prefill, and an answer that was not
cut inside a call has no such member: nothing changes for a client that
does not know it. The formats covered are those of the chat completions
(a marker opens the call).
Measured on the CPU at temperature 0 with Qwen3-0.6B Q8_0 and one
tool (get_weather, three required arguments), thinking
disabled: the answer in one go is a call of 33 tokens; cut after each of
its first 32 tokens (no cut answer holds a tool_use block)
and continued, whole and streamed, the 64 answers are that call. With
thinking on (adaptive) the answer is 135 tokens, a
thinking block then the call: the same 32 cuts inside the
call, sent back with the thinking block, the same 64
answers. test_continue_call_messages does it on the tiny
model with a required call of 64 tokens, cut after each of 63
counts.
POST /v1/messages/count_tokens takes the same request
(max_tokens not needed) and answers
{"input_tokens": N}: the prompt a message would be given,
tools, forced call marker and images included, equal to that message's
input_tokens + cache_read_input_tokens.
Errors are Anthropic-shaped,
{"type": "error", "error": {"type": ..., "message": ...}},
for every path under /v1/messages (a missing key included):
400 invalid_request_error, 401
authentication_error, 404 not_found_error,
405, 413 request_too_large, 500 api_error, and
529 overloaded_error (with Retry-After: 1)
when the queue is full, which Anthropic's clients retry.
Not supported: real thinking signatures (an empty one is sent, and
earlier turns' thinking blocks are dropped rather than checked); prompt
caching control (cache_control is ignored: the prefix cache
works on its own, cache_creation_input_tokens is always 0);
documents and PDFs (document blocks), citations, search
results, server tools (type other than custom:
web search, code execution, bash, text editor, computer use), MCP
connectors, container uploads, the Files API, image URLs, Message
Batches and GET /v1/models/{id};
stop_sequences beyond four; pause_turn and
refusal stop reasons are never produced.
Embeddings
oklm serve -m MODEL --embed-model EMB [--embed-model-id NAME] [--embed-backend cpu|opencl|vulkan|auto]
loads a second model, a text encoder (bert,
nomic-bert) or a decoder embedding model (Qwen3-Embedding),
and answers POST /v1/embeddings with it as OpenAI does (rag.md):
curl -s localhost:7430/v1/embeddings -H 'Content-Type: application/json' \
-d '{"model": "bge", "input": ["first text", "second text"]}'
{"object":"list","data":[{"object":"embedding","index":0,"embedding":[-0.0147,...]},
{"object":"embedding","index":1,"embedding":[...]}],"model":"oklm-embed",
"usage":{"prompt_tokens":10,"total_tokens":10}}
input: a string, an array of strings, an array of token ids, or an array of such arrays (at most 2048 inputs).encoding_format:float(default) orbase64(the little-endian F32 bytes, as OpenAI's).dimensions: accepted only when it is the model's width (no truncated vectors).modelis not read; the answer'smodelis--embed-model-id(oklm-embedby default).- The vectors are pooled as the model says and L2-normalized, the same
as
oklm embedgives;usage.prompt_tokenscounts every token run, special tokens included.
Embedding requests run on the embedding model's own context, beside
the scheduler's, one request at a time (the texts of one request share
its forward passes). Without --embed-model the route
answers 404; invalid requests 400.
Reranking
oklm serve -m MODEL --rerank-model R [--rerank-model-id NAME] [--rerank-backend cpu|opencl|vulkan|auto]
loads a reranker beside the generative model, a cross-encoder (a text
encoder with a classification head, such as
cross-encoder/ms-marco-MiniLM-L-6-v2) or a chat model that scores
through decide.c, and answers POST /v1/rerank (also
/rerank) in Cohere's and Jina's shape (rag.md):
curl -s localhost:7430/v1/rerank -H 'Content-Type: application/json' -d '{"query": "How many people live in Berlin?",
"documents": ["Berlin has 3.5 million inhabitants.", {"text": "Paris is in France."}], "top_n": 2, "return_documents": true}'
{"object":"list","model":"oklm-rerank","results":[{"index":0,"relevance_score":0.99985,"document":{"text":"Berlin has 3.5 million inhabitants."}},
{"index":1,"relevance_score":0.0000163,"document":{"text":"Paris is in France."}}],"usage":{"prompt_tokens":30,"total_tokens":30}}
query: a string.documents: an array of strings or of objects with atextstring (Jina's form), 1 to 1000 of them.top_n: the results returned, a positive integer (default: all; more than the documents gives all).return_documents:trueto put each document back as{"text": ...}(defaultfalse, as Cohere's).results: best first, each with the document'sindexin the request and itsrelevance_score, the sigmoid of the reranker's logit (a cross-encoder's head, or the generator's log p(yes) − log p(no)).modelis not read; the answer'smodelis--rerank-model-id(oklm-rerankby default).usage.prompt_tokenscounts every token run (each pair's, special tokens included; the generator's prompts in full).
Rerank requests run on the reranker's own context, one at a time; the
pairs of one request share its forward passes (a cross-encoder). Without
--rerank-model the route answers 404; invalid requests 400
with the field at fault.
Documents
POST /api/ingest turns a document into the text a model
reads, with the reader of oklm ingest and
--doc (multimodal.md). It is
there for a client that cannot read a PDF or a Word file itself (the
dashboard's chat): the text comes back and the client puts it in its
message. Every server has the route.
curl -s --data-binary @report.pdf 'localhost:7430/api/ingest?name=report.pdf&max_tokens=3000'
{"name":"report.pdf","type":"pdf","chars":79,"total_chars":79,"tokens":29,"truncated":false,
"text":"Hello, World!\nKerning stays one gap\n..."}
The request:
- The body is the file, as it is (no multipart, no base64).
name(query), or anX-Filenameheader: the file's name. It is read for its extension and given back without its directories; the server opens no path a client names.max_chars,max_tokens(query): cut the text to that many characters, then to the longest head of at most that many tokens (at a character, never inside one). 0 or absent: no cut.- The kind is told by the first bytes (
%PDF-: PDF; a ZIP archive: DOCX), then the name's extension (.pdf,.docx,.json,.html,.htm,.xhtml), then, for a name without extension, theContent-Type(application/pdf, the DOCX type,application/json,text/html,application/xhtml+xml), then an HTML start in the content; anything else is plain text (Markdown, CSV, code).
The answer:
nameas given,type(pdf,docx,html,json,text).text: PDF pages in order, DOCX headings as#lines and tables as| a | b |lines, HTML rendered, JSON indented, text with its line ends cleaned.charsandtokensof that text, the tokens counted with the served model's tokenizer (special tokens written in the document are text);total_charsof the whole document;truncatedwhen a cut applied.
Errors and limits:
- 400: an empty body, a binary file of no kind read, a PDF that is
malformed or encrypted with a password, a ZIP archive that is no Word
document, JSON that does not parse, a bad
name,max_charsormax_tokens. - 413: a body above
--upload-max(32 MiB by default). - 401 without a key on a server that asks for one, 405 for another method.
- A scanned PDF holds images, not text: its
textis empty. - One document is read at a time. Nothing is written to disk but a
DOCX, which goes through a temporary file (its ZIP reader opens a path):
one created for this user only under
TMPDIRand removed before the answer.
/api/stats counts them in ingest_stats:
requests (documents read), failed,
bytes received and chars answered.
A long document kept for retrieval
A document longer than max_tokens is cut. A server that
holds an embedding model can keep it whole instead and give, for each
question, the passages nearest to it: the model of
--embed-model when the server has one, else the one its
--rag index was built with (/api/stats says
"ingest_retrieval": true). The client asks for it by naming
a conversation:
curl -s --data-binary @manual.pdf 'localhost:7430/api/ingest?name=manual.pdf&max_tokens=1024&conversation=3f9c2a7e51d04b8896aa0c1d2e3f4a5b'
{"name":"manual.pdf","type":"pdf","chars":182340,"total_chars":182340,"tokens":41877,"truncated":false,
"text":"","indexed":true,"chunks":214,"conversation":"3f9c2a7e51d04b8896aa0c1d2e3f4a5b"}
curl -s localhost:7430/api/ingest/search -H 'Content-Type: application/json' \
-d '{"conversation":"3f9c2a7e51d04b8896aa0c1d2e3f4a5b","query":"How is the pump primed?","k":6,"max_tokens":512}'
{"conversation":"3f9c...","documents":["manual.pdf"],"chunks":214,"passages":[
{"document":"manual.pdf","chunk":57,"lines":[812,829],"score":0.7431,"tokens":188,"text":"..."}],"spread":false,"reranked":false,"tokens":188}
curl -s localhost:7430/api/ingest/forget -H 'Content-Type: application/json' -d '{"conversation":"3f9c2a7e51d04b8896aa0c1d2e3f4a5b"}'
{"removed":true}
conversation(query of/api/ingest): an identifier the client chooses, 16 to 64 letters, digits,-or_; the dashboard draws 32 hexadecimal digits at random. It names what that client had indexed and nothing else: whoever knows it (and has a key, on a server with keys) can search those documents, so it should not be guessable.- What
/api/ingestthen does. A text that holds inmax_tokensis answered as before, with"indexed": false. A longer one is cut into passages (the recursive chunker ofoklm rag index: headings, paragraphs, sentences; about 256 tokens each, under the embedding model's longest input), each passage is embedded, and the answer has"indexed": true,chunks, theconversation, thetokensandcharsof the whole document,"truncated": falseand an emptytext. When it cannot be kept (no embedding model, a bound below reached,max_charscut it first) the text is cut as before andindex_errorsays why. Withoutconversationnothing changes. POST /api/ingest/search:conversation,query, and optionallyk(1 to 50 passages; 6),max_tokens(a budget counted with the served model's tokenizer: the passages are given best first while they hold in it, the first one always),documents(the names of the documents to search; all of them without it),spreadandrerank. The score is the cosine of the question's vector and the passage's.- A reranker, when the server has one. On a server
started with
--rerank-modelthe nearest passages (four timesk, twenty at least, 48 at most) are read by the reranker with the question and given in its order, each with itsrerank_scorebeside the cosine, and the answer says"reranked": true: the cosine finds what resembles the question, a cross-encoder what answers it. Without a reranker, with"rerank": falsein the request, for a question about the whole, or when the reranker fails, the order is the cosine's and"reranked"is false. The reranker is held under its own lock, taken while the documents' lock is held. - A question about the whole. The passages nearest
"Summarize this" tell little of a document, so a
querythat asks about the whole (a summary, an outline, an overview, the main or key points, what the document is about,tl;dr: English words, andrésum) is answered withkpassages spread over the documents searched: the first, the last and those at equal steps between them, in the documents' order, fewer when they do not hold inmax_tokens. The answer says"spread": true."spread": truein the request asks for them whatever the question,falsefor the nearest ones. It is a sample of the document, not all of it: a summary of what was not sent cannot be had this way. - The refusals. 404 when nothing is kept under that identifier (never sent, forgotten, or removed since); 400 for a missing or malformed member.
POST /api/ingest/forget:conversation; the documents are removed andremovedsays whether there were any.- Bounds. Everything is kept in memory, never on
disk: at most 64 conversations, 16 documents a conversation, 4096
passages a document, 16 MiB a conversation and 64 MiB in all (texts and
vectors). A conversation nobody added to or searched for
--ingest-keepseconds (an hour) is removed; when room is missing, the conversations asked least recently are removed first. A client whose documents are gone gets the 404 and sends them again. - Cost. The passages are embedded while the request
waits, 16 at a time, taking turns with
/v1/embeddingsor the index's searches: with bge-small-en-v1.5 (Q8_0) on the CPU a text of 3510 tokens is 24 passages. A search is one embedding of the question and a dot product with every passage of the conversation. - Both routes ask for a key like the others; 404 with the way to start the server when it has no embedding model.
/api/stats has ingest_retrieval_stats on
such a server: model (the embedding model), what is kept
now (conversations, documents,
chunks, bytes), the bounds
(max_bytes, max_conversation_bytes,
max_conversations, max_documents,
max_chunks, keep_seconds) and, since the
start, indexed (documents kept), searches,
refused (documents that could not be kept),
expired (conversations removed for their age),
evicted (removed to make room) and forgotten
(removed by their client).
Transcriptions
oklm serve -m MODEL --transcribe-model whisper-base [--transcribe-model-id NAME] [--transcribe-backend cpu|opencl|vulkan|auto] [--preload]
holds a Whisper checkpoint beside the model (the directory
oklm transcribe -m takes, or the catalogue name of an
installed one: oklm models pull whisper-base) and answers
POST /v1/audio/transcriptions and
POST /v1/audio/translations as OpenAI does:
curl -s localhost:7430/v1/audio/transcriptions -F file=@speech.wav -F model=whisper-1
{"text":"And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country."}
curl -s --data-binary @fable.ogg 'localhost:7430/v1/audio/transcriptions?response_format=verbose_json'
{"task":"transcribe","language":"fr","language_probability":0.9967,"duration":52.056,"text":"Maître Corbeau, sur un arbre perché, ...",
"segments":[{"id":0,"seek":0,"start":0.00,"end":6.00,"text":"Maître Corbeau, sur un arbre perché, ...","avg_logprob":-0.3974,"no_speech_prob":0.1022},
{"id":1,"seek":0,"start":6.00,"end":11.36,"text":"Maître Renard, par l'odeur alléché, ...","avg_logprob":-0.3974,"no_speech_prob":0.1022},
...
{"id":10,"seek":2516,"start":43.74,"end":50.62,"text":"Le Corbeau, ...","avg_logprob":-0.3866,"no_speech_prob":0.0004}]}
curl -s --data-binary @fable.ogg 'localhost:7430/v1/audio/transcriptions?language=fr&response_format=srt'
1
00:00:00,000 --> 00:00:06,000
Maître Corbeau, sur un arbre perché, ...
curl -s --data-binary @fable.ogg localhost:7430/v1/audio/translations
{"text":"Master Corbeau, on a perched tree, ..."}
(A 52-second fable read in French, Whisper base for the transcription: eleven segments from two windows, the second starting at 25.16 s where the last whole segment of the first ended. The translation is Whisper small's: base writes English words, not the fable.)
Two forms of request:
- OpenAI's,
multipart/form-data: afilepart and text fields, what the OpenAI libraries send. The parser is small and strict: onefile, at most 16 parts, a part's head at most 2048 bytes, a text field at most 4096 bytes. - The sound as the body (any other
Content-Type), the parameters in the query string: what the dashboard and the chat window send.
| Field | |
|---|---|
file |
the sound: PCM WAV (8 to 32-bit integer, 32-bit float) always; FLAC,
Ogg Vorbis, Opus and MP3 on a build with Allegro 5 (the packaged ones).
Any rate and channel count, mixed to mono at 16 kHz. A WAV written to a
stream, whose lengths say 0xFFFFFFFF (what
/v1/audio/speech sends with "stream": true,
and ffmpeg to a pipe), is read to the end of the file |
language |
a code the model knows (en, fr, ...:
transcribe_stats.languages of /api/stats lists
them), or auto. Absent or auto: the
model finds it from the first 30 seconds (the most probable
language token after the start token), and verbose_json
says which |
response_format |
json (default): {"text": ...};
text: the text and a line end, as text/plain;
verbose_json: task (transcribe or
translate), language (the code decoded with),
language_probability (only when the model found it),
duration (seconds), text and
segments; srt: SubRip subtitles, as
text/plain; vtt: WebVTT, as
text/vtt |
model |
accepted, not read: the server has one transcription model |
temperature, prompt,
timestamp_granularities[] |
accepted, not read: the decoding is greedy, without a prompt, and the times are the segments' (no word times) |
POST /v1/audio/translations takes the same request and
writes English whatever the language spoken (Whisper's translation task;
400 from a model of English only).
The recognizer decodes with Whisper's timestamps (multimodal.md): the text
comes as segments, each with id,
seek (where the 30-second window it was decoded from
starts, in hundredths of a second), start and
end (seconds from the start of the sound),
text, and the two figures of its window,
avg_logprob (the mean log-probability of its tokens) and
no_speech_prob. A window starts where the last whole
segment of the one before ended, so a long sound is cut at its pauses.
The text is what oklm transcribe prints for the same file,
srt and vtt what its --format srt
and --format vtt print.
Limits and errors:
- The recognizer runs on the CPU by default, one request at a time on
the request's own connection: neither the chat scheduler nor the other
routes wait for it. Four more requests wait their turn; one beyond is
answered 503 at once (
Retry-After: 1).--transcribe-backend opencl|vulkan|autoputs the encoder's blocks and the decoder on a device, asoklm transcribe --backenddoes (multimodal.md): jfk.wav with large-v3-turbo is then answered in 0.26 s on Vulkan and 0.76 s on OpenCL (RX 7900 XT, the model on the same device). On the model's device each of its passes (a window's encoder, a token of the decoder) takes its turn with the scheduler's steps (sharing the device); a backend this machine does not have stops the server at start, a device without room for the weights leaves the recognizer on the CPU with a line in the log (backendintranscribe_statssays where it is). - A sound of at most 30 minutes and at least a tenth of a second (413,
400); a body of at most
--upload-max(32 MiB by default; 413). - 400: a sound that cannot be decoded, a language the model does not
know, another
response_format, a translation asked of a model of English only, a multipart body without afile, without a boundary or cut short. - 404 without
--transcribe-model, the message saying how to start the server; 401 and 405 as the other routes.
Measured on the CPU of the measures below (Whisper base, the
11-second jfk.wav of whisper.cpp's samples): 1.3 s a
request once the weights are in memory, the text that of
oklm transcribe; the same sound as Ogg Vorbis and as FLAC
gives the same text.
/api/stats has a transcribe_stats object:
model, requests (sounds transcribed),
failed, busy (answered 503),
waiting (the requests that wait their turn: 0 to 4),
audio_seconds and the seconds spent on them,
active (1 while a sound is being transcribed, which
waiting does not count), loaded (whether the
weights are in memory) and load_seconds (what loading them
took), and what the model offers, read from its tokenizer when the
server starts (so before the weights are loaded):
languages, the array of the codes language
takes, in the model's order (["en","zh","de",...]: 99 for
base and small, 100 for the large-v3 family; empty for a model of
English only), translate (whether
/v1/audio/translations answers) and timestamps
(whether the segments are timed). A client builds its language list from
languages, with "auto" before them.
The words while speaking
The recognizer reads a whole sound: Whisper decodes 30-second windows, not a stream. The text of a recording that is still coming is therefore the recording so far transcribed again, and the route takes one more parameter for it, in the query or as a form field:
| Field | |
|---|---|
partial |
true (or 1): the sound is the start of a
recording still coming. The answer is the same as without it, with
"partial": true (json,
verbose_json). Such a request never waits:
when the recognizer is at work, on a whole recording or on another
partial text, it is answered 503 at once
(the transcription model is at work: a partial text does not wait, ask again with more sound),
since it will be asked again with more sound and must not hold up a
whole recording. It is counted apart: transcribe_stats has
partial (true: this server takes the parameter),
partials, partial_busy,
partial_audio_seconds and partial_seconds, and
requests, audio_seconds and
seconds stay those of the whole recordings |
What a client does with it is the same in the dashboard, in the chat
window and in tools/bench_partial.py (the page's
part_tick(), listen_partial in
apps/gui/chat_listen.h):
- A partial text every two seconds of new sound, the first after one second, one at a time; when an answer took more than a second the next one waits twice that, so that a slow recognizer keeps half of its time.
- The stable prefix shown, the rest replaced. The
words two partial texts in a row start with alike are shown as said;
what the last one heard after them is shown apart, and the next one
replaces it. Whisper rewrites the end of what it heard as more sound
comes (
my fellow Americans.becomesmy fellow Americans, ask not), rarely its start. - A long recording costs its last sentences, not its
length. Once 20 seconds of sound were sent and the answer has
two timed segments or more, all but the last are kept as said and the
next requests send the sound after them: the segments'
endtimes say where. - The final text is unchanged. When the recording
ends it is sent whole, once, without
partial, as before: the message gets that text, never the partial one, and a partial text on its way then is finished first (the whole recording waits one partial text's time at most).
What it costs, measured with tools/bench_partial.py on
2026-11-15 (Core i7-11700K, RX 7900 XT; another job used the CPU: the
load average is given), a recording played at its own pace, a server
holding the recognizer alone on the backend named: jfk.wav
(11 s, English) and a fable read in French (82.9 s, a public-domain
LibriVox recording), the recognizer's seconds per second of recording,
for the partial texts and for the final text alone:
| Model | Backend | Load | 11 s: partial texts | per second of sound | final | 82.9 s: partial texts | per second of sound | final | Longest wait for a word |
|---|---|---|---|---|---|---|---|---|---|
| Whisper base | CPU | 1.3 | 5 | 0.136 s | 0.034 s | 41 | 0.273 s | 0.047 s | 3.0 s |
| Whisper base | OpenCL | 5.1 | 5 | 0.030 s | 0.010 s | 41 | 0.061 s | 0.008 s | 2.3 s |
| Whisper base | Vulkan | 4.8 | 5 | 0.024 s | 0.007 s | 41 | 0.050 s | 0.006 s | 2.2 s |
| large-v3-turbo | CPU | 4.0 to 4.9 | 2 | 0.920 s | 0.475 s | 6 | 0.522 s | 0.273 s | 32.7 s |
| large-v3-turbo | OpenCL | 11 | 5 | 0.145 s | 0.033 s | 41 | 0.187 s | 0.019 s | 2.5 s |
| large-v3-turbo | Vulkan | 7.9 | 5 | 0.115 s | 0.026 s | 41 | 0.150 s | 0.015 s | 2.4 s |
The partial texts cost four to ten times the final text: each one
reads up to 20 or 30 seconds of sound again (463 s of sound read for the
82.9 s with Whisper base, 562 s with large-v3-turbo, whose segments are
longer). With Whisper base, or with either model on a device, that is a
tenth to a quarter of the recognizer's time and a word is shown two to
three seconds after it was said. large-v3-turbo on the CPU cannot follow
a voice: asked as soon as each answer came it took 1.0 to 1.6 s of the
recognizer per second of sound, more than there is; with the rule above
it is asked 6 times in 82.9 s and a word waits up to half a minute, the
final text being the same either way.
"the transcription model is at work" was answered in 0.5 ms
to a partial request sent one second into a whole recording's
transcription (large-v3-turbo on the CPU, 5.7 s), which was not
delayed.
When the sound models are loaded
The transcription model and the speech model are checked when the server starts and loaded by the first request that needs them. A server that nobody speaks to or asks a voice of then never pays for them: with SmolLM2-135M as the chat model on the CPU, 874 MiB at start with both named, against 2604 MiB with both loaded.
- At start, the files are read as far as it is cheap:
for Whisper,
config.jsonis a Whisper configuration,model.safetensorsopens and its header names the first tensors of the encoder and the decoder, the tokenizer reads and has Whisper's special tokens; for the speech model, the vocoder's GGUF header and metadata, the language model's GGUF header and its vocabulary (the tokens of OuteTTS 0.2), the speaker file and its codes. A wrong path, another kind of file or a voice the models do not know stops the server there with the same message as before, not at the first request. - The first request of each route loads its model, then is answered as any other; the requests that arrive meanwhile wait their turn behind it. Measured on the CPU: 0.18 s for Whisper base (536 MiB more once a sound was transcribed), 0.53 s for OuteTTS 0.2 500M with its vocoder (1.6 GiB more), small beside the request itself (2.6 s for a 3-second sound, 6.9 s for two words spoken). A request refused before its turn (400, 413, 503) loads nothing.
- A model that passed the check and cannot be loaded
(a file removed or cut after the start, not enough memory): the request
is answered 500,
the transcription model could not be loaded: ...orthe speech model could not be loaded: ..., the model stays not loaded, the other routes go on, and the next request tries again. --preloadloads both when the server starts, as before:loadedis true from the first/api/stats, and a model that cannot be loaded stops the server. For a server whose first caller must not wait, or that should fail at once on a machine without the memory.
Speech
oklm serve -m MODEL --speak-model outetts-0.2-500m [--speak-vocoder FILE] [--speak-voice FILE] [--speak-voices DIRS] [--speak-model-id NAME] [--speak-backend cpu|opencl|vulkan|auto] [--speak-max BYTES] [--preload]
holds a speech model beside the model (what oklm speak -m
takes: the catalogue name of an installed one,
oklm models pull outetts-0.2-500m, or
outetts-0.3-500m, whose vocoder is found by itself; or the
language model's GGUF file with --speak-vocoder) and
answers POST /v1/audio/speech in OpenAI's shape (speech.md for the models and the voice). With
--speak-model soprano-1.1-80m the route is the same and the
sound is Soprano's (below): 32 kHz,
one voice, made faster than it is heard.
curl -s localhost:7430/v1/audio/speech -H 'Content-Type: application/json' \
-d '{"model":"outetts-0.2-500m","input":"The weather is nice today."}' -o speech.wav
curl -sN localhost:7430/v1/audio/speech -H 'Content-Type: application/json' \
-d '{"input":"A longer text. Spoken as it is made.","response_format":"pcm","first_words":5}' \
| ffplay -f s16le -ar 24000 -ch_layout mono -nodisp -autoexit -
curl -s localhost:7430/v1/audio/voices
curl -s localhost:7430/v1/audio/speech -H 'Content-Type: application/json' \
-d '{"input":"Another voice, a little faster.","voice":"anna","speed":1.25,"response_format":"flac"}' -o speech.flac
(aplay -f S16_LE -r 24000 -c 1 plays the same
stream.)
The request is a JSON object:
| Member | |
|---|---|
input |
the text, UTF-8: at most --speak-max bytes (4096 by
default, OpenAI's limit in characters; at most 65536). The synthesizer
speaks English: it reads the letters a to z and the numbers, the rest is
dropped |
response_format |
wav (default here: OpenAI's default is
mp3, which OKLM does not write): a mono 16-bit PCM WAV file
at 24 kHz (32 kHz with Soprano: X-Sample-Rate says which).
pcm: the samples alone, 16-bit signed little-endian mono at
24 kHz, as OpenAI's pcm (32 kHz with Soprano).
flac: the same samples compressed without loss, about half
the WAV's size (speech.md).
mp3, opus and aac are refused
(400, naming the three written): their encoders are whole codecs, which
OKLM does not carry |
stream |
with wav or flac: true sends
the file while it is made (below). pcm is always sent
so |
voice |
default (the voice the server was started with), or the
name of a voice the server offers (GET /v1/audio/voices,
below): the --speak-voice file's name,
en_male_1 (the model's built-in voice), or a speaker file
NAME.json of the voice directories and of the model's
folder. Another name is refused (400, listing those offered): OpenAI's
voices (alloy, ...) do not exist here. A request gives a
name, never a path: the server opens speaker files of
its own directories only |
model |
accepted, not read: the server has one speech model |
speed |
a number from 0.25 to 4 (1 by default): how fast the text is spoken, the pitch kept (speech.md). A sound at speed 2 lasts half as long |
stream_format |
audio or absent; sse is refused: the sound
is sent as its bytes, not as events |
temperature, top_k, seed,
guide, words |
the synthesizer's own options, as oklm speak's
--temp (0 to 2; 1), --top-k (4),
--seed (1234), --no-guide
("guide": false) and --words (1 to 64; 24):
the words one pass of the model speaks |
first_words |
0 to 64 (0 by default): the first pass is cut in two when it holds
more words, its first words sent as a pass of their own, so that the
first sound comes earlier; as --first-words (speech.md) |
prompt_words |
1 to 64: the words the model speaks from one prompt, as
--prompt-words: the passes that hold in it share the
prompt |
The answer, with X-Sample-Rate: 24000
(32000 with Soprano), X-Channels: 1 and
X-Voice: NAME (the voice spoken with) in every form:
| Asked | Answer |
|---|---|
wav |
Content-Type: audio/wav, Content-Length:
the whole file once the text is spoken, its header holding its lengths.
The bytes are those oklm speak writes for the same text and
options |
wav with "stream": true |
audio/wav, Transfer-Encoding: chunked: a
44-byte header whose two lengths are 0xFFFFFFFF (not known
yet), then each pass's samples as a chunk. Saved as it is, the file is
one oklm transcribe and
/v1/audio/transcriptions read |
pcm |
audio/pcm, Transfer-Encoding: chunked:
each pass's samples as a chunk, no header |
flac |
audio/flac, Content-Length: the whole
file, its count of samples in its header. flac -t and
ffmpeg read it; the samples are the WAV file's |
flac with "stream": true |
audio/flac, Transfer-Encoding: chunked:
the header (its count of samples 0, "not known"), then the frames of
4096 samples each pass completes; the samples left over go with the next
pass, the last ones in a shorter frame |
The text is spoken pass by pass: whole sentences
while they hold at most words words together. A streamed
answer begins when the first pass is made and carries each further pass
as soon as it is, so a client plays the first sentences while the next
are computed; a pass is not cut further, and a text of one pass arrives
at once at its end. first_words gives an earlier first
sound: the first pass's first words are sent as soon as they are spoken,
and joined to the rest without a seam, since the passes that hold
together in 64 words are written under one prompt (speech.md). On
a CPU a pass is still made slower than it is heard: an early first sound
is followed by a wait. The status is decided with the first pass: a
synthesis that cannot run answers an error, not a stream; one that fails
later ends the chunked stream without its last chunk.
Playing it:
- A browser. An
<audio>element sends aGET: it cannot name this route as itssrc.fetch()the route, then either take the whole answer (await response.blob(),audio.src = URL.createObjectURL(blob), ordecodeAudioData()on its bytes), or readpcmas it comes (response.body.getReader()): each piece of 16-bit samples (an odd byte kept for the next piece) is divided by 32768 into anAudioBufferof one channel at 24000 Hz, played by anAudioBufferSourceNodestarted where the one before ends. Chrome 157 plays both WAV forms from a blob and decodes both withdecodeAudioData()(checked); a player that refuses a header without lengths wants the whole file ("stream": false). OKLM's own sound decoder (oklm transcribe,/v1/audio/transcriptions) reads the whole file, not the streamed one. - A program.
pcmneeds no parsing: every two bytes are a sample, low byte first, 24000 a second, one channel; the chunks of the HTTP transfer are the passes.
The voices. GET /v1/audio/voices lists
what voice may name:
{"object":"list","default":"en_male_1","data":[
{"id":"en_male_1","builtin":true,"default":true},
{"id":"anna","builtin":false,"default":false}]}
First the voice the server was started with (the built-in one, or the
--speak-voice file's name: what default and a
request without voice get), then the built-in voice when it
is not that one, then the speaker files NAME.json of the
directories of --speak-voices (a list separated by
:, ; on Windows; oklm serve takes
the user's voices folder by default,
$XDG_CONFIG_HOME/oklm/voices or
~/.config/oklm/voices, the one the chat window reads;
--speak-voices "" for none) and of the speech model's
folder, by name; a name met twice is the first one's. A file is listed
when it is a speaker file of OuteTTS (an object with words,
each with word, duration and
codes) of at most 8 MiB; at most 64 voices. The directories
are read again when the list is older than two seconds, and at once for
a name it does not hold: a file dropped in the folder is a voice without
a restart. The same list is the voices array of
speak_stats. A voice whose codes the model does not know is
refused when it is asked for (400). Changing voice costs nothing but the
prompt's evaluation, which every text pays anyway: the synthesizer keeps
its models and takes the other speaker file.
Limits and errors:
- The synthesizer runs on the CPU by default, one text at a time in a
thread of its own: neither the chat scheduler nor the other routes wait
for it. Four more requests wait their turn; one beyond is answered 503
at once (
Retry-After: 1).--speak-backend opencl|vulkan|autoputs its language model on a device, asoklm speak --backenddoes (speech.md; its vocoder's network with it): each token it writes is a pass that takes its turn with the scheduler's steps, and each call of the vocoder (sharing the device). A backend this machine does not have stops the server at start. - A client that closes its connection stops its synthesis, between two tokens of the language model, whether it runs or waits its turn; a server that stops ends them the same way. A client that reads slower than the sound is made holds the synthesis back once 8 MiB wait for it.
- 413: an
inputabove--speak-max. 400: a body that is no JSON object, noinput, aninputwithout any word to speak, aresponse_formator avoicethe server does not have, a voice the model cannot use, an option out of its range. 500: the model wrote no sound for the text. - 404 without
--speak-model, the message saying how to start the server; 401 and 405 as the other routes.
Measured on the CPU of the measures below, shared with other work at
the time (OuteTTS 0.2 500M Q8_0): "The quick brown fox jumps over the
lazy dog." is 2.76 s of sound in 7.7 s, as a WAV file and as
pcm (the same samples), the file byte for byte
oklm speak's; Whisper base hears "The quick brown fox jumps
over the lazy dobs." in both. Three sentences of 23 words are one pass
by default: the first byte arrives with the last (17 to 32 s). With
"words": 8 they are three passes: the first byte after 3.7
to 12.7 s of 18.8 to 26.0 s.
With the passes under one prompt (four threads, the machine's 16 held
by other jobs: load 13 to 17), three sentences of 32 words, 11.05 s of
sound: as pcm the first byte after 32.0 s and the last
after 44.6 s; with "first_words": 5 after 4.1 s and 39.0 s.
As flac 264,866 bytes for the 530,284 of the WAV answer
(49.9%), accepted by flac -t and decoded by
flac -d to the WAV answer byte for byte; streamed with a
first pass of 5 words, the first byte after 9.0 s, accepted by
flac -t too. With "speed": 1.5 the WAV is
353,538 bytes: 7.36 s of sound, in which Whisper base hears the words it
hears at speed 1. A speaker file of ten words put in the voices folder
while the server ran was listed by GET /v1/audio/voices and
spoken with by name (X-Voice: short). The same text asked
nine times in other forms had 5790 of its 8054 prompt tokens kept in the
cache (cached_tokens): a prompt that does not change is not
evaluated again but for its last token.
Speech with Soprano
--speak-model soprano-1.1-80m (or Soprano's language
model as a GGUF file with --speak-vocoder naming its
decoder) answers the same route with the same members. What differs, all
of it read from the model's files when the server starts:
- The sample rate is 32000: in
X-Sample-Rate, in the WAV and FLAC headers, inspeak_stats.sample_rate. A client that playspcmreads the rate from the header, not from a constant. - One voice.
GET /v1/audio/voiceslistssopranoalone, whatever the voice directories hold;"voice": "default"and"voice": "soprano"are that voice; another name is refused (400: "... Soprano speaks with one fixed voice: speaker files do not apply"); a server given--speak-voicewith this model does not start, saying the same. - The text keeps more: punctuation, and the numbers, dates, times, amounts and abbreviations Soprano's cleaner reads (speech.md).
- The sampling is the model's own when the request
names none: the most likely token with a repeat penalty of 1.2
(
temperatureandtop_kare read when given;guideandprompt_wordsare accepted and not read). - Every sentence is a pass (one of fewer than 30
characters goes with the one before it): a streamed answer carries each
sentence when it is made.
first_wordscuts nothing: the sound of about that many words (0.38 s a word) is sent while the rest of the first sentence is written, and the bytes are those of an answer without it.
Measured (the catalogue's Q8_0 file, four threads, the machine's 16
held by other jobs: load 15 to 20), two sentences of 26 words, 7.94 s of
sound: as a WAV file after 2.1 s, 2.6 s and 3.2 s; as pcm
the first byte after 0.75 s, 0.77 s and 1.10 s and the last after 1.9 s
to 2.9 s; with "first_words": 5 the first byte after 0.45
s, 0.49 s and 0.69 s, the same 507,904 bytes. The WAV answer is
oklm speak's file byte for byte, flac -t
accepts the FLAC answer (289,661 bytes for 507,948), and Whisper base
hears every word at speed 1 and at 1.5.
/api/stats has a speak_stats object:
model, voice (the server's own),
voices (the array GET /v1/audio/voices gives
as data), formats
(["wav","pcm","flac"]), speed_min and
speed_max (0.25 and 4), first_seconds (how
long the last text spoken waited for its first sound),
prompt_tokens, cached_tokens and
written_tokens (the tokens of the prompts evaluated since
the start, those the context's cache already held, those the model
wrote), sample_rate, max_input,
requests (texts spoken to their end), failed,
busy (answered 503), cancelled (stopped: the
client left, or the server stopped), waiting (the texts
that wait their turn: 0 to 4), audio_seconds made and the
seconds spent on them, active (1 while a text
is being spoken, which waiting does not count: with one
spoken and four waiting the figures read "active":1 and
"waiting":4, and the next request is answered 503; a client
that leaves while its text waits gives its place back at once and counts
as cancelled), loaded and
load_seconds (when the sound models are
loaded).
Sharing the device
A server's model, its image or audio encoder (--mmproj),
its recognizer (--transcribe-backend), its synthesizer's
language model (--speak-backend) and its embedding and
reranking models (--embed-backend,
--rerank-backend) can be on the same device, each computing
in a thread of its own: the scheduler's, and the threads of the
requests.
One pass at a time. A backend's calls are each safe from any thread, and what belongs together is a pass (
oklm_backend_pass_begin(),oklm_backend_pass_end()): a forward pass of a context (a scheduler step, a token of the synthesizer), the blocks of the image encoder over one tile or one image, those of the audio encoder over a 30-second chunk, a window of the recognizer's encoder, the keys and values its decoder's cross-attention reads (every layer's), a token of its decoder, a call of the vocoder. A pass holds the device's pass lock, so another thread's pass comes before or after it, never between its ops: Vulkan's recorded decode steps (vulkan.md) and the ops a backend holds back to fuse stay those of one computation. Nothing is locked on the CPU backend, and two devices have a lock each.A model spread over several devices (
OKLM_CL_DEVICES,OKLM_VK_DEVICES: opencl.md, vulkan.md) holds the lock of each device its layers are on for a forward pass, the head's and those of the earlier layers (oklm_backend_pass_begin_all()). The locks are taken in the order the devices were registered in, whatever the order of the layers, and given back in the reverse order: two contexts spread over the same devices cannot wait for each other. An encoder on one of those devices takes that device's lock alone, and lets the model's passes in between its blocks as on one device. The model's own pass is cut where its stream goes from one device's layers to the next's: the stream is read into host memory there, the layers before it are whole, and the pass gives every device back to the passes that wait and takes them again (oklm_backend_pass_yield_all()). A pass on one device is not cut: its steps are what Vulkan records and replays, and nothing in it is in host memory between two ops. Run on 2026-11-14 with Qwen2-VL 2B spread over the RX 7900 XT and the Intel UHD 750 (OKLM_VK_DEVICES=all): nine new images answered, three of them while another request streamed. A stream's gaps there are those of the language model on the slower device: 6.4 to 7.2 s for the step that reads an image's 416 rows through the UHD 750's layers (the whole request alone is 7.4 to 9.2 s), 0.3 to 0.7 s with--prefill-chunk 16. The cut between the devices does not shorten them, and cannot: the stream and the image's rows are steps of the same context, so the stream waits for its own pass. Measured before and after it (2026-11-14, the same server, a 400-token stream while three new images arrive, tokens 31 ms apart): the largest gap 5.71 s and 5.71 s; a request with a new image 7.77 to 7.78 s and 7.20 to 7.23 s among the stream (its encoder's blocks now come in between the two devices' layers), 6.56 to 6.57 s alone either way.In the order they ask. The passes of a device run in the order they asked for it: each draws a ticket and waits for its turn, so a thread that ends a pass and starts its next one goes behind the pass that was waiting (a mutex released and taken again by the same thread would leave the other waiting for as long as the first has work).
An encoder's pass is cut between its blocks. An encoder's pass is the longest: 0.04 to 0.7 s for an image, 0.12 to 0.2 s for a window of sound, where a decode step is a few milliseconds. Between two blocks (and after the patch embedding or the convolutions, before the projector, after a vocoder's first stack, after each layer's keys and values of a recognizer's cross-attention) the encoder's stream is whole and in its own buffer, so another pass may run there (
oklm_backend_pass_yield()): the work queued so far is waited for, which leaves the backend with no op held back, no command buffer open and nothing queued, and the device goes to the pass that waits, then back to the encoder. The wait matters as much as the hand-over: an encoder queues its blocks much faster than the device runs them, and a step let in behind twenty queued blocks would wait for them all. It is only made on a device another thread is using: a pass of its waits, or it has a stream of passes open there, or it ended a pass within the last quarter of a second. The scheduler opens a stream on its model's devices when it has a request to decode and closes it when it has none (oklm_context_stream(),oklm_backend_pass_stream_all()), so a model that decodes slower than four steps a second is still seen between two of its steps, and an encoder that starts in such a pause does not queue its first blocks in one piece; the quarter of a second is what marks a device shared for the threads that say nothing (two encoders, a recognizer and a synthesizer). An encoder alone on its device queues its whole pass as before and takes the same time.OKLM_PASS_CUT=0keeps every pass whole (build.md).What a decoding request sees. The scheduler's next step waits for one block of an encoder, not for its pass. Measured on 2026-11-14 (RX 7900 XT, load average 1.1 to 2.8, one server at a time; a request streaming 400 tokens while three others arrive one after another, each with a picture or a sound the server has not seen; the largest gap between two of the stream's tokens; the image models stream from their own language model, tokens 4 to 12 ms apart, the sound models beside Llama-3.2-1B Q8_0, 3 to 7 ms apart). "Before" is the day's first measurement with every pass whole; "now" was measured again at the day's end, the whole encoders on the device, Vulkan's wide products, the vocoder's network there and the scheduler's stream of passes declared:
Arriving Vulkan: before, now OpenCL: before, now an image, Gemma 3 4B (4 096 rows) 0.70 s, 0.076 s 0.75 s, 0.093 s an image, Qwen2-VL 2B (391 embeddings, 1 564 rows) 0.25 s, 0.042 s 0.29 s, 0.049 s a picture of 2 600 x 1 982 pixels, Qwen2-VL 2B (3 024 embeddings, 12 096 rows) not measured, 0.22 s not measured, 0.19 s (0.30 s once) an image, Qwen2.5-VL 7B (391 embeddings) not measured, 0.16 s not measured, 0.17 s an image, LLaVA 1.5 7B (576 rows) 0.20 s, 0.19 s 0.19 s, 0.19 s an image, LLaVA 1.6 Mistral 7B (five tiles, 2 340 embeddings) not measured, 0.047 s not measured, 0.055 s (0.30 s once) an image, SmolVLM2-2.2B (five tiles) not measured, 0.017 s not measured, 0.024 s jfk.wav (11 s), Whisper large-v3-turbo 0.20 s, 0.014 s 0.28 s, 0.018 s jfk.wav, Whisper large-v3 (32 decoder blocks) not measured, 0.013 s not measured, 0.018 s jfk.wav in a chat message, Ultravox v0.5 0.20 s, 0.020 s 0.27 s, 0.022 s a sentence to speak, OuteTTS 0.2 (its vocoder's call included) 0.04 s, 0.025 s 0.07 s, 0.030 s a sentence to speak, Soprano 0.02 s, 0.016 s 0.02 s, 0.017 s What is left is a step of the language model, or one block: LLaVA 1.5's encoder is 0.04 s, and its gap is the step that reads the image's 576 rows with the question (0.06 s on Vulkan and 0.09 s on OpenCL with
--prefill-chunk 64, which cuts a prompt into pieces), as Qwen2.5-VL 7B's 0.16 to 0.17 s is the step of its 391 image rows (one gap an image; a block of its encoder is 5 ms); Gemma 3's 256 image rows see each other and are read in one step, 0.08 to 0.09 s with a block of its encoder (0.025 s); a block of a picture of 3 024 embeddings is 0.19 to 0.22 s, and that is the gap (the request is 6.7 to 8.1 s alone, 7.8 to 10.4 s among streams); LLaVA 1.6 and SmolVLM2 are a pass a tile, and a tile's blocks are short (LLaVA 1.6's stream runs at 23 to 28 ms a token while the image's 2 340 rows are read beside it, a piece a step); the synthesizer's passes are single tokens and its vocoder's network 14 ms for a sentence of 7 s. Whisper large-v3's decoder computes the keys and values of its 32 layers' cross-attention in one pass a window: whole, it left 0.024 s (Vulkan) and 0.031 s (OpenCL) in the stream; cut after each layer's, 0.013 s and 0.018 s.What the encoder pays. Nothing alone: the whole request (the encoder, the prompt, 8 tokens) took 0.89 -> 0.87 s for Gemma 3, 0.39 -> 0.39 s for Qwen2-VL, 0.45 -> 0.45 s for LLaVA, 0.29 -> 0.29 s for Whisper, 0.30 -> 0.30 s for Ultravox on Vulkan, and 0.96 -> 0.96, 0.43 -> 0.43, 0.47 -> 0.46, 0.36 -> 0.37 and 0.38 -> 0.38 s on OpenCL (medians of five, a request every 0.6 s; at the day's end, with the table's "now": 0.865, 0.382, 0.432, 0.275 and 0.279 s on Vulkan, 0.94, 0.42, 0.45, 0.355 and 0.359 s on OpenCL, and among a stream 1.13, 0.39 to 0.51, 0.89 to 1.01, 0.40 to 0.48 and 0.40 s on Vulkan, 1.27 to 1.34, 0.62 to 0.64, 0.87 to 1.04, 0.59 to 0.61 and 0.50 s on OpenCL). With a stream decoding it waits for a step after each block, where the stream used to wait for it: 0.92 to 1.10 -> 1.13 s for Gemma 3, 0.39 -> 0.51 s for Qwen2-VL, 0.30 -> 0.40 to 0.44 s for Ultravox, 0.43 to 0.48 -> 0.40 to 0.50 s for Whisper on Vulkan; 1.01 to 1.08 -> 1.30 to 1.35 s, 0.44 -> 0.62 to 0.64 s, 0.39 to 0.42 -> 0.50 to 0.53 s and 0.54 to 0.61 -> 0.60 to 0.63 s on OpenCL. Cutting the passes is the default for that reason: it costs a lone request nothing and a request among streams a quarter to a half more, for streams that no longer stop. Waiting for the queued work at every block whether the device was shared or not, the first form tried, cost a lone request 1 to 5 % (Qwen2-VL 0.386 -> 0.40 s, Whisper 0.278 -> 0.29 s on Vulkan).
Sounds for the model. The audio encoder of
--mmproj(Ultravox) follows the model's device as the image encoder does: a chat request with a new sound and 48 tokens went from 5.80 to 5.88 s (the encoder on the CPU) to 0.57 to 0.60 s on OpenCL and from 5.65 to 5.79 s to 0.42 to 0.43 s on Vulkan (Ultravox v0.5 with Llama-3.2-1B, jfk.wav and its first 8 and 5 seconds, RX 7900 XT, 2026-11-14), each answer the text it was with the encoder on the CPU. Its pass is the 32 blocks of Whisper large-v3-turbo's encoder, the recognizer's in the table above, cut between its blocks as the recognizer's is.Device memory. The model's paged cache takes what the device has free (resizing the context), so what shares the device is counted first, each part as it allocates, and the part not taken yet comes out of the cache:
- the image encoder (
oklm_vision_device_bytes_at()): its weights and the activations of the largest picture the server accepts, a tile for the encoders of one size, three quarters of a sequence's context for Qwen2-VL and Qwen2.5-VL (3 072 embeddings at-c 4096, or the area--tilesasks for); - the audio encoder of
--mmproj(oklm_audio_device_bytes()): its weights and a chunk's activations; - the recognizer (
oklm_whisper_device_bytes()): its weights and the one buffer of its passes, a window's stream with the keys and values of every decoder layer; - the synthesizer (
oklm_speech_device_bytes()): its language model's weights, cache and activations, and its vocoder's network with the activations of its longest call; - the backend's own buffers, when one of the four above is on the
device: the scratch of its split attention and products and its staging,
which grow with the largest pass, an encoder's. A backend says what it
holds of them and what they are left
(
oklm_backend_iface::own_memory): 64 MiB on OpenCL, 352 MiB on Vulkan, whose scratch for a pass of 4 096 rows is about 300 MiB. What it does not hold yet of that reserve comes out of the cache.
The embedding and the reranking model of
--embed-backendand--rerank-backendare loaded before the cache is sized, so the device's free memory is already less by them; they are counted (oklm_embedder_device_bytes(),oklm_reranker_device_bytes(): a context's weights, activations and cache on the device) and shown with the others. Their share comes out of the device's budget too (oklm_kv_blocks_fit_beside()): where the budget, not the free memory, bounds the cache, the cache is smaller by what they take, and the whole stays within the budget (Qwen3-0.6B at 8192 tokens with bge-small-en-v1.5 Q8_0 on the device: 160352 tokens of cache where the model alone has 161216). The retrieval's own encoder (--rag) computes on the CPU in a server and takes nothing there.The two sound models are loaded by the first request that needs them (
--preload: at the start), after the cache is made: their weights are read once at the start to be counted (0.1 s more for Whisper large-v3-turbo and OuteTTS 0.2 together). Without that room the device pages memory in and out: a transcription of jfk.wav then took 42 to 97 s on OpenCL instead of 0.76 s.GET /api/statsgives the counts and the device's own figures indevice_memory(bytes:total,free,vision,audio,transcribe,speak,vocoder,embed,rerank,backend_own(the reserve of the backend's own buffers, 0 when nothing shares the device),backend_held(what the backend holds of them now), andpending, what of them no request has made the device give yet).The same figures are available before anything is loaded:
oklm_server_beside_plan()sizes each part from its file with the functions above, where the server would place it (on the model's device, else in host memory: its weights and what it works with there, the embedding and the reranking models' contexts asoklm_embed_context_params()andoklm_rerank_context_params()make them, the recognizer's arenas, the speech model's context and a call of its vocoder), and a memory plan given them (oklm plan --serve, the graphical tools' memory line) takes them out of the cache's room through the rule the server sizes its cache with,oklm_kv_cache_rule(): the plan's cache is the server's (memory-plan.md, gui.md). The draft model's context is made withoklm_serve_draft_params()by both. A part on a device that is not the model's takes nothing of the model's cache: it is counted on its own device and that device judged for it (oklm_server_beside::other), and the server tells what it holds there inside_devicesofGET /api/stats.- the image encoder (
--kv-membeside them. A cache given with--kv-memleaves the same share alone: the server refuses at start a cache that does not fit with it, with the figures ("--kv-mem 15000 does not fit on vulkan with what shares it: the device has room for 17279 MiB beside the model, of which 5363 MiB are still to be taken by the encoders (the image encoder 2492 MiB, the transcription model 1726 MiB, the speech model 903 MiB, its vocoder 243 MiB): --kv-mem 11916 at most, or 0 for what is left"). Without--kv-mem, a device that cannot hold the model, what shares it and one context of-ctokens is refused at start too ("... cannot hold what was asked: ..., that leaves N MiB to the cache, and a context of 4096 tokens takes M MiB"), where the first request used to fail; so is a recognizer asked for on the device with--preloadthat does not fit there ("the transcription model ... does not fit on vulkan"), which used to move to the CPU with a warning. A device that held less than one context before their share is served as before, with the cache that is left.The counts against the device (RX 7900 XT,
--kv-mem 512, the device's free memory in/api/statsbefore and after the first request of each kind, MiB; the rows "at the start" are of 2026-11-14, the others were measured again on 2026-11-15 with the backends' reserve in the count, which the first column leaves out: 288 MiB more pending on Vulkan, 352 less the 64 of its staging buffer, and 64 on OpenCL):Counted Vulkan OpenCL Qwen2-VL 2B's encoder at the start (1 280 embeddings) 1 782 1 782 1 784 its first picture (391 embeddings) 0 50 38 a picture of 3 024 embeddings (the buffer grows) 709 998 692 Gemma 3 4B's encoder at the start 1 148 1 148 1 150 its first picture (4 096 rows) 0 297 38 LLaVA 1.5 7B's encoder at the start 658 658 658 its first picture 0 38 42 Ultravox's encoder at the start 1 410 1 418 1 414 its first sound 0 39 32 Whisper large-v3-turbo, the first transcription 1 726 1 763 1 752 OuteTTS 0.2 and its vocoder, the first sentence 1 161 1 078 1 070 Soprano and its decoder, the first sentence 348 270 274 bge-small-en-v1.5 F32 with --embed-backend, at the start154 161 164 MiniLM-L6 F32 with --rerank-backend, at the start113 121 124 The weights and the buffers the encoders ask for are counted to 0.6 %. What a backend makes for itself at a first pass, 32 to 50 MiB, and Vulkan's scratch for a pass of 4 096 rows or more, 297 to 301 MiB, were in no count before the reserve: they came out of the cache's margin (3 % of the device, 256 MiB at least). With it, Gemma 3's first picture leaves nothing pending on Vulkan (288 MiB pending at the start, 297 taken), and Qwen2-VL's two pictures take 1 048 MiB of the 997 that were pending, 51 MiB over where it was 351. The reserve is one figure a backend, so an encoder whose passes are short leaves most of Vulkan's unused: 243 to 244 MiB still pending after the first request of LLaVA 1.5, Ultravox and Whisper (55 to 64 MiB of OpenCL's 64, whose query sees 5 MiB of the 38 a first pass takes: the rest is the driver's). Counted and not reached by one sentence: a vocoder's activations for its longest call (2 048 frames, what the synthesizer's context of codes allows; a sentence of 7 s is 527), about 110 MiB of each speech row with the rows of WavTokenizer's attention; they were counted as fifteen activations of the stacks' width where a call makes twelve, 19 MiB less.
GET /api/contextand a dry run plan the default cache with the share left out, as the server makes it (they read the device's free memory with the cache in place, as before, and do not count a draft model's).Answers. An image's answer is the same with the encoder on the device or on the CPU on Vulkan (SmolVLM-500M, Gemma 3 4B, LLaVA 1.5 7B, 48 tokens of llama.cpp's test image,
--deterministic), and on OpenCL but for Gemma 3 4B, whose answer changes one word at the 41st token (a near tie: OpenCL's embeddings are 1e-5 from the CPU's). SmolVLM-500M's wasoklm describe's on both backends; Gemma 3's and LLaVA's differed fromoklm describe's by a word in these runs, with the encoder on the CPU as on the device: the chat route then wrote the question right after the image whereoklm describeputs a new line (273 tokens against 274, 592 against 594). Both write it through one function since: on the CPU, Gemma 3 4B's prompt is 274 tokens and LLaVA 1.5 7B's 594 by either way, and the answers about the test image are the same tokens (24 asked of Gemma 3; LLaVA's 11, to its end of sequence),--deterministic, greedy. Not run again on a device. The form both write was then set by projector: the question right after the image for Gemma 3 and SmolVLM, as their templates write it (Gemma 3 4B's prompt is 273 tokens, SmolVLM-500M's 348), LLaVA's new line kept. The image went from 22.5 s (encoder on the CPU) to 1.26 s for Gemma 3 4B on Vulkan (22.7 to 3.68 s on OpenCL), 1.97 to 0.91 s for LLaVA 1.5 (2.07 to 1.05 s), 3.48 to 0.34 s for SmolVLM-500M (4.14 to 0.59 s). Twenty requests at once (five each: an image, jfk.wav, a streamed completion, a sentence to speak) gave the transcriptions, the sounds (byte for byte) and the images' answers of the same requests alone; streamed completions, and an image's answer on a server without--deterministic, move by a word with what shares their batches, as they do without any encoder (determinism).
What a server takes
GET /api/stats says what can be sent, so a client (the
chat window, the dashboard) knows before it sends:
| Member | True when |
|---|---|
vision |
chat messages take images (--mmproj with a vision
encoder) |
audio |
chat messages take sounds (--mmproj with an audio
encoder) |
transcribe |
POST /v1/audio/transcriptions answers
(--transcribe-model); transcribe_stats then
holds languages (the codes its language
parameter takes), translate and timestamps (transcriptions) |
speak |
POST /v1/audio/speech answers
(--speak-model) |
ingest |
POST /api/ingest answers (always, on a server that has
the route: an older one has no such member) |
ingest_retrieval |
a long document sent with a conversation is kept for
retrieval instead of cut (--embed-model, or
--rag) |
host_memory |
what the server holds of the host's memory, bytes:
context (the model's context: its cache's rows in host
memory, its recurrent state, its activation buffers and scratch, an
expert cache), repacked (the copies of the matrices the
CPU's matrix product repacked), embed and
rerank (what the contexts of the embedding and the
reranking models hold beside their mapped files,
oklm_context_host_held()), transcribe (the
recognizer's arenas and copies once a request loaded it, 0 before),
speak (the synthesizer once a request loaded it, 0 before:
its language model's context in host memory and its vocoder's copies,
oklm_speech_host_held()), speak_call (what the
longest call of its vocoder adds on the CPU while it runs, as many codes
as the language model's context holds tokens: allocated and freed by the
call, and not in held), vision and
audio (the arena the projector's encoder computes its
activations in on the CPU, oklm_vision_host_held() and
oklm_audio_host_bytes()), and held, the sum of
what is held: what stopping the server gives back, the mapped weights
being the system's to drop. A client that plans another setting for the
same model counts held as available, as it counts
device_memory.held |
side_devices |
present when a model beside the model is on a device that is not the
model's (--embed-backend vulkan1 beside a model on
vulkan, or on the CPU): an array, a device each, of
backend, total and free (the
device's own figures, free -1 when it gives none) and the
bytes the recognizer (transcribe), the synthesizer's
language model (speak), its vocoder (vocoder),
the embedding model (embed) and the reranking model
(rerank) are counted to take there |
device_memory |
present when the model computes on a device: an object of bytes,
total and free (the device's own figures,
free -1 when it gives none), what the image encoder
(vision), the audio encoder (audio), the
recognizer (transcribe), the synthesizer's language model
(speak), its vocoder (vocoder), the embedding
model (embed) and the reranking model (rerank)
are counted to take there, the reserve of the backend's own buffers
(backend_own) and what it holds of them
(backend_held), pending, the part of these no
request has made the device give yet (sharing the device), then
model, what the model's own context holds there (the
weights of its layers on the device with the output head, its
activations, its cache as allocated), and held, what
stopping the server gives back to the device: model and
what the others took already. A client that plans another setting for
the same model counts held as free: the chat window's
memory line does (gui.md) |
upload_max is the largest document or sound, in
bytes.
Retrieval
oklm serve -m MODEL --rag INDEX [--rag-k 5] [--rag-mode dense|sparse|hybrid] [--rag-crag on|off] [--rag-route on|off] [--rag-rewrite on|off] [--rag-followup rewrite|join|both] [--rag-rerank MODEL [--rag-rerank-top 20]] [--rag-graph on|off] [--rag-critique on|flag|off] [--rag-log FILE]
opens a retrieval index (built by oklm rag index, its
embedding model loaded on the CPU to embed the questions). A chat
completion, or an Anthropic message, asks for retrieval with its
rag field; without it, or with false or
null, nothing changes:
curl -s localhost:7430/v1/chat/completions -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "How do I keep the prefix cache across restarts?"}],
"rag": {"k": 5, "crag": true}, "chat_template_kwargs": {"enable_thinking": false}}'
rag |
|
|---|---|
true |
the server's settings (--rag-k,
--rag-crag, --rag-route,
--rag-rewrite, --rag-followup) |
{"index": "NAME"} |
the index searched, by its name: the server's own, or one of the
--rag-dir directory (below);
without it, the server's own |
{"k": N} |
passages retrieved, 1 to 50 |
{"crag": false} |
naive retrieval: the top k passages, not judged |
{"max_tokens": N} |
the sources' token budget in the prompt, 64 to 65536 (2048) |
{"route": true} |
the model decides first: no retrieval, one, or several (sub-questions retrieved apart, merged) |
{"rewrite": false} |
a multi-turn request's last message searched as it is, not condensed from the messages before it |
{"followup": "both"} |
what such a message searches: the previous user message and the
standalone rewrite (both, the default), the rewrite alone
(rewrite), or the previous user message and the message as
written (join, no generation) |
{"critique": true} |
the answer checked against its sources (self-critique): a whole answer generated once more when a cited statement is unsupported, a stream flagged only |
{"critique": {"retry": true}} |
the same, and a stream's answer generated again too, sent as one
more event when kept; {"retry": false}: never a retry |
The last user message is searched; its text is replaced, for this
request, by the instructions, the numbered sources and the question (rag.md). With
crag (the default) each passage is judged by the model (two
scoring requests of the scheduler, as decisions by scoring are run), the
question is rewritten by the model and searched again when no passage is
relevant, and when still none is, the prompt asks for an answer saying
that no reliable source was found. In a multi-turn request the last user
message is first rewritten as a standalone query from the messages
before it (a short greedy generation of the scheduler; rag.md), and the previous
user message joined with that query is what is searched. With
route, three scoring requests decide whether the request
needs no retrieval, one or several (rag.md); with
--rag-rerank, the candidates are reordered by the reranker
first. The answer carries the run as rag: a member of the
whole answer (chat completion or Anthropic message); streamed, a member
of the last chunk of a chat completion, the one carrying the
finish_reason (before data: [DONE]), and of
the message_delta event of an Anthropic stream:
"rag": {"time": 1762650000, "surface": "serve", "query": "How do I keep ...", "standalone": null, "route": null, "hops": [], "rewrites": [],
"rounds": [{"query": "How do I keep ...", "hop": 0, "web": false, "ms_search": 4.1, "ms_judge": 890.2,
"passages": [{"chunk": 412, "source": "docs/server.md#prefix-cache-on-disk", "lines": [1, 30], "heading": "",
"score": 0.0325, "from": "dense+sparse", "rerank": 0, "p_relevant": 0.9871, "verdict": "relevant", "cite": 1}]}],
"fallback": "none", "corrective": true, "context": ["docs/server.md#prefix-cache-on-disk"],
"context_tokens": 1210, "ms": 902.5, "error": "",
"sources": [{"cite": 1, "source": "docs/server.md#prefix-cache-on-disk", "lines": [1, 30], "heading": "", "text": "..."}]}fallback is none (the first search),
rewrite, or no_source; the server has no web
search. standalone is the condensed follow-up (null when
nothing was rewritten), route the router's choice with
route_p its three probabilities (null when not routed),
hops the sub-questions. /api/stats adds the
counters rag (requests, rounds,
passages, judged, relevant,
ambiguous, irrelevant, rewrites,
web, no_source, context_chunks,
context_tokens, errors,
condensed, routed, route_none,
route_one, route_multi, hops),
and --rag-log FILE gets each run as a line (without the
texts).
Self-critique (critique, or
--rag-critique on|flag as the default; rag.md): each statement of the answer
that cites a source is asked against it, and the answer against the
question, as scoring requests of the scheduler. The answer gains a
critique member, the verdict on the answer kept, and the
run's rag member the whole record as
rag.critique:
"critique": {"flagged": true, "retried": false, "chosen": 0, "supported": 0.75, "pairs": 4,
"unsupported": [{"statement": "...", "cite": 3}], "useful": 0.97, "ms": 412.3}A whole answer with an unsupported statement is generated once more
(greedily, the critique after the user message) and the better one is
returned, unless the request says "retry": false or the
server runs with --rag-critique flag. A
stream is sent as generated: once its last token is
out, the whole text is critiqued, and the verdict rides on the last
chunk (the one with the finish_reason, beside
rag) or on the Anthropic stream's
message_delta, so the last chunk comes after the critique's
scoring requests. It is flagged, not retried, unless the request asks
with "critique": {"retry": true}; then a second answer,
when kept as better, follows as one more event before
data: [DONE]:
data: {"id":"chatcmpl-7","object":"chat.completion.critique_retry","created":...,"model":"...",
"choices":[{"index":0,"message":{"role":"assistant","content":"..."},"finish_reason":"stop"}],"critique":{...}}
or, in an Anthropic stream, before message_stop:
event: critique_retry
data: {"type":"critique_retry","content":[{"type":"text","text":"..."}],"critique":{...}}
Answers that may be tool calls (tools offered) are not critiqued.
/api/stats counts critiqued,
critique_retries and critique_flagged in
rag, whose last member, reranker, names what
reorders the candidates (cross-encoder or
decide, the kind of the model given as
--rag-rerank; null without one). A request
asking for rag from a server without --rag
(and that names no index of a --rag-dir) gets 400, and so
does a field of the wrong type. Retrieval applies to
/v1/chat/completions and /v1/messages (raw
completions do not read the field). Searches of the index and reranking
run one at a time (the encoders keep buffers); judging, routing and
rewriting are ordinary requests of the scheduler, so a busy server
judges more slowly, and a full queue makes the run naive (noted in
error).
Several indexes
oklm serve -m MODEL [--rag INDEX] --rag-dir DIR offers,
beside the index of --rag, every index file
NAME.gguf of a directory. A request names the one to
search:
curl -s localhost:7430/v1/chat/completions -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "How long is the quay?"}], "rag": {"index": "quay-index", "k": 4}}'
- A name, never a path. The name is the file's
without
.gguf: letters, digits,.,-and_, not starting with a dot. The server looks it up asDIR/NAME.ggufand nowhere else; anything with a/or a..is refused (400), as is a name the directory does not hold. The server's own index answers to its file's name too. - Opened at the first use, with its embedding model
(the one the index names) and the server's retrieval settings
(
--rag-k,--rag-mode,--rag-crag,--rag-rerank...), and kept open. At most 8 indexes of the directory are open at once; a ninth is refused. An index whose embedding model is a file loads that model: several such indexes are several models in memory. A file that is no index is refused when it is first named, with the reason. - Listed in
/api/statsasrag_indexes: the server's own first ("default": true), then the directory's files by name (64 at most), read again at each request for the figures, so that an index written into the directory is offered without a restart. Each hasopen, and once open its own counters asstats(the members ofrag, which stays the server's own index's). - Without
--rag,--rag-diralone serves the directory's indexes:"rag": trueis then refused (there is no index to default to) and a request names one. The--rag-*settings go with--rag-diras they go with--rag: the directory's indexes are opened with them (--rag-dir DIR --rag-k 2 --rag-mode sparse), and with the defaults when none is given.
"rag_indexes":[{"name":"harbour-index","default":true,"open":true},
{"name":"quay-index","default":false,"open":true,"stats":{"requests":1,"rounds":1,...}}]Decisions by scoring
POST /v1/systemone (also /v1/decide)
answers closed questions about a state with probabilities instead of
generated text. The wire format is colibrì's System One endpoint, itself
TypeSafe's Jev API:
{
"state": "The order arrived two weeks late and the box was crushed.",
"questions": {
"topic": {"type": "choice", "instructions": "What is the message about?",
"criteria": {"delivery": "shipping and delays", "billing": null, "product": "the item itself"}},
"complaint": {"type": "noul", "instructions": "Is the customer complaining?"},
"anger": {"type": "score", "instructions": "How angry is the customer?", "min": 1, "max": 5}
},
"system": "You classify customer messages.",
"model": "optional: an adapter's NAME, as for chat completions"
}state: a string, or any JSON value (given to the model as its compact JSON text). Required.questions: an object of named questions, 1 to 32, answered under the same names in the same order. Each has atypeand optionalinstructions:choice:criteria, an object whose keys are the labels and whose values describe them (a string, ornullfor none); an array of label strings is read too. 1 to 64 labels.noul: yes or no.score:levels, an array of numbers, or integersminandmax(the levels min, min + 1, ..., max). 1 to 64 levels.
system: more system text (optional).
The answer:
{"id": "decide-7", "object": "systemone.result", "model": "oklm",
"answers": {
"topic": {"type": "choice", "label": "delivery", "probabilities": {"delivery": 0.91, "billing": 0.02, "product": 0.07}, "confidence": 0.68},
"complaint": {"type": "noul", "label": "yes", "probabilities": {"yes": 0.97, "no": 0.03}, "confidence": 0.81, "p_yes": 0.97},
"anger": {"type": "score", "label": "4", "probabilities": {"1": 0.01, "2": 0.04, "3": 0.2, "4": 0.5, "5": 0.25}, "confidence": 0.26, "expected": 3.94}},
"usage": {"prompt_tokens": 1727, "cached_tokens": 1024, "scored_tokens": 21}}label: the most probable option (the first on ties);yesornofor anoul; a score's level as written inprobabilities.probabilities: each allowed option's, normalized over the options (they sum to 1); for a score, the distribution over its levels.confidence: 1 minus the distribution's entropy over that of a flat one: 0 when every option is as likely, 1 when one is certain.p_yes(noul): the probability of yes.expected(score): the sum of probability times level.usage:prompt_tokens, the tokens of every sequence scored (one per option, below), of whichcached_tokenswere found in the prefix cache and not evaluated;scored_tokens, the options' tokens.
How a question is scored
(include/oklm/model/decide.h,
src/model/decide.c, shared with oklm chat's
/decide). The question is rendered through the model's chat
template: a system turn ("Answer with exactly one of the allowed
options.", then system), a user turn holding the state,
then the instructions, then "Allowed options:" and one line per option
(- label: description), then the assistant turn opened,
with enable_thinking false for the templates that read it
(a template that still opens the turn on <think> has
it closed empty). Each option's tokens (tokenized apart, without special
tokens) are then scored as the continuation of that prompt,
teacher-forced. Nothing is generated.
- The rule: the sum of the option's tokens'
log-probabilities, the log-probability of the whole option,
then a softmax over the options. Not the mean: the options are listed in
the prompt, so once an option's first token is written its rest is
nearly certain and the first token decides; dividing by the length would
lift a long option for being long (an unlikely first token of four sure
ones nearly ties a likely one-token option by the mean, and is seen as
unlikely by the sum:
tests/model/test_decide.c). - Yes / no are scored as
Yes/No, oryes/nowhen the model's tokenizer writes the lower case in fewer tokens (the capitalized ones on a tie, as a reply's first word is written); the answer always names themyes/no. - A score's levels are scored as their numbers
written out (
%.15g:3,0.5).
The shared prefix. Each option is a scoring request
of the scheduler: the question's prompt followed by the option's tokens,
the logits of the rows from the prompt's last token on kept and turned
into log-probabilities (srv_req.score_from,
score_lp), the request ending once its prompt is in,
without a token. The state comes before the question in the prompt, so
every question on one state shares the system turn and the state, and
every option of a question the whole prompt. The first option's request
runs alone and leaves those full blocks in the prefix cache; the others
(as many at a time as the queue takes) attach them and evaluate only the
rest: the question, the part block, the option. A scoring request is
never given the rows it scores from the cache, so its scores are always
computed. With the contiguous cache (--kv-mem -1) a
sequence's own previous tokens serve instead.
Limits. The probabilities are the model's, not
calibrated: they rank the options and say how sure the model is relative
to the others, they are not frequencies. They are relative to the
options offered: an answer that is none of them is not seen. Options
should be short and distinct, best with different first tokens; two
options of the same tokens are refused (400). The prompt and the longest
option must fit the context. Errors are OpenAI-shaped (400 for a missing
state, no question, an empty criteria, an unknown
type, a score without levels; 401, 405, 503); keys as for
the other routes, Authorization: Bearer or
x-api-key.
The chat's commands
GET /api/commands gives the commands a chat takes (inference.md) as JSON, for
the dashboard's chat and for a chat window connected from another
machine, which then offer what this server does and say why for the
rest. The table is the library's
(oklm_chat_commands_json(),
include/oklm/client/chat_commands.h), the same one
oklm chat reads.
{"places":["remote","page"],
"features":["context"],
"commands":[
{"name":"context","aliases":["ctx","context-size"],"args":"[TOKENS [force]]",
"help":"Show the context's size and its use, or change the size (the conversation is kept)",
"values":[],"where":["terminal","window","remote","page"],"implemented":[],
"needs":["context"],"available":true,"reason":""},
{"name":"rag","aliases":[],"args":"INDEX|off","help":"Answer from a retrieval index's passages, or stop doing so",
"values":[],"where":["terminal","window","remote","page"],"implemented":[],
"needs":["rag"],"available":false,"reason":"no retrieval index here"}]}| Field | Meaning |
|---|---|
places |
the chats this answer is for: remote (a chat window
connected to this server) and page (the dashboard) |
features |
what this server can do: context (the context can be
resized: absent with --fixed-context), rag
(started with --rag INDEX or --rag-dir DIR),
media (started with --mmproj),
listen (started with --transcribe-model),
speak (started with --speak-model),
models (another model is loaded in place by
POST /api/model: absent with --fixed-model,
with --lora and with --lora-module, whose page
tells it itself for the adapters) |
commands[].name, aliases,
args, help, values |
the command without its slash, its other names, its arguments as the help shows them, a line of help, the values its first argument takes when they are a fixed set |
commands[].where |
the chats the command belongs to: terminal,
window (the chat window on a server of its own),
remote, page, agent (the chat
window attached to an agent) |
commands[].implemented |
of those, the chats whose client has the command written in this version; a client checks its own |
commands[].needs |
the features the command needs |
commands[].available |
whether a remote or page chat can run it
on this server: it belongs to one of them and every feature it needs is
there |
commands[].reason |
when it cannot: not a command of this chat,
the context's size is fixed here,
no retrieval index here,
the model cannot be changed from here... |
The route is read only (GET, HEAD; 405
otherwise) and asks for a key like the others.
oklm_chat_commands_features() reads features
back for a client written in C.
GET /api/models/installed lists the
models of the catalogue that are installed in the model directory of the
server's user (OKLM_MODELS_DIR, else
~/.local/share/oklm/models), for a chat that is asked for
another model and offers what could be served instead:
{"serving":"SmolLM2-135M-Instruct-Q8_0","switch":true,"models":[
{"name":"smollm2-135m","title":"SmolLM2 135M Instruct (Q8_0)","categories":"chat",
"about":"the smallest chat model: a first try on any machine","size":144811360,"context":8192,
"serves":true,"served":true}]}serving is the model the server holds and
switch says whether it loads another in its place while it
runs (POST /api/model, Switching the model): true unless the
server was started with --fixed-model, --lora
or --lora-module, for which another model is another server
(oklm serve -m NAME --port N, or the server manager,
oklm-gui --tool servers, which starts, stops and restarts
them: above). serves
tells the models a chat is served with (not a Whisper checkpoint, an
embedding or a speech model), served the one whose file the
server holds now. The route gives names, sizes and descriptions, no
path. Read only, a key like the others.
Activations
A server started with --observe records, for the token
each sequence evaluated last, how much each block wrote to the residual
stream, the experts each router picked and, for a few blocks asked for,
where the attention of that token looked (observe.md, with what the figures mean and do
not mean). It is what the model inspector draws while a request is
decoded (gui.md).
oklm serve -m models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf --backend cpu --observe
curl -s http://127.0.0.1:7430/api/activations # norms and experts of the token decoded last
curl -s "http://127.0.0.1:7430/api/activations?layer=5" # and block 5's attention rows from the next token on
curl -s "http://127.0.0.1:7430/api/activations?seq=2&layers=0,5,23"
curl -s "http://127.0.0.1:7430/api/activations?serial=41" # the record the sequence's 41st pass left, when it is still kept
| Parameter | What |
|---|---|
seq=N |
the sequence (the scheduler's slot of a request); default: the one a forward pass recorded last |
serial=S |
one of the records kept, by the serial it was published
with, in place of the last one. It carries the attention rows of the
blocks that were chosen when it was made. 404 when it is not kept (never
made, or dropped since) |
layer=L, layers=L,L... |
the blocks whose attention rows are wanted, at most
recording.attn_slots (4, or the model's blocks when it has
fewer). The choice holds for the forward passes that follow, for every
sequence, until another request changes it: the first answer after a new
choice may carry no row yet, or those of the choice before. Without the
parameter the answer has no row and the choice is left as it is |
The answer is the record of observe.md (seq,
pos, token, piece,
serial, n_layer, n_observed,
embd, resid, attn,
ffn, attention, experts, and
lens with --observe-lens), with two more
members:
{"seq":0,"pos":9,"token":253,"piece":" a","serial":6,"n_layer":30,"n_observed":30,
"embd":2.89421,"resid":[37.9678,...],"attn":[3.70575,...],"ffn":[...],
"attention":{"n_head":9,"n_head_model":9,"pos0":0,"n_pos":10,"tokens":[...],
"pieces":["The"," capital"," of"," France"," is"," Paris","."," Paris"," is"," a"],
"layers":[{"layer":5,"heads":[[...],...]}]},
"recording":{"attn_slots":4,"attn_positions":256,"attn_heads":9,"lens":0,"bytes":39910608},
"kept":{"count":6,"first":1,"last":6}}recording gives the bounds as applied: blocks recorded
at once, the positions of a row (the last ones; halved until the
recording fits 64 MiB, which a server of many sequences and a model of
many heads needs), heads a block, the tokens of a block's lens (0
without --observe-lens), and the bytes the recording holds.
Before anything was decoded the answer is
{"serial":0,"recording":{...}}. A client that polls
compares serial: it changes with each record of the
sequence.
kept gives the records of the sequence that can be read
again with serial=S: their count and the serials of the
oldest and the newest. The server keeps the last 512 records of all its
sequences together, in 16 MiB at most (a record with attention rows of
256 positions for four blocks of 32 heads is 131 KiB: about 120 of those
fit; without rows a record is a few KiB), the oldest dropped first. A
sequence is a scheduler's slot: a new request in the slot goes on with
the slot's serials, and its positions start again from where its prompt
was not cached. This is what lets a client step back through the tokens
of an answer after it was given (the inspector does: gui.md).
- Access: as
/api/stats(a key when the server has keys: 401 without). 404 with the way to start the server when it does not record, or for aserialthat is not kept; 400 for a block or a sequence that does not exist, or more blocks than slots. - What a record is: the last token of the sequence's
last forward pass: a prompt piece's last token while the prompt is
evaluated, then each decoded token. A pass verifying drafts
(
--spec-ngram,--draft,--spec-grammar) evaluates several tokens of a sequence: the record is that of the last, which may be a draft rejected afterwards. - On a device: the blocks a device runs are recorded as the CPU's are, read back from its memory once a pass, which costs it some speed while the server records (observe.md).
- Cost: off, one pointer tested per block. On, the
norms and experts of every pass (two reads of one row of the stream per
block and sequence) and, for each block asked for, one dot product per
head and position of each sequence, on the scheduler's thread. The route
reads the published record under the recording's own lock and never
waits for the scheduler. A context made again
(
POST /api/context) starts a new recording.
The dashboard
src/server/dashboard.html, embedded at build time and
served on / with no external resource. The figures of
/api/stats (every 2 s) head the page, n_expert
among them; under them, tabs:
- Chat: a conversation with the model, streamed, with the chat's commands and files (below).
- Expert map (a MoE model only): one row per layer,
one cell per expert. A cell is white when the layer's router chose that
expert in the last decode step, amber by how often it was chosen in the
answer being generated (the brighter, the more; the square root of its
share of the most used cell), dark when not chosen; a toggle shows the
cumulative heat instead. Hovering a cell gives its three counts. The tab
asks
/api/expertsevery 0.4 s while requests are generating, every 2 s otherwise, and only while it is shown. - Activations, a link at the right of the tabs: a page of its own (below).
- Profile: the last requests as stacked bars (queue, prompt, decode, after the last token; hovering gives each figure, the tokens, the decode rate, the sampling and detokenizing time), the trend of their total time and their time to the first token, and the distribution of the recent decode steps' times with their mean, median, 90th and 99th percentiles.
The page of the activations
GET /activations is a second embedded page
(src/server/activations.html, generated and served as the
dashboard is, with the same policy and no external resource): the model
inspector's views (gui.md) in
a browser, on what GET /api/activations answers. On a
server started with --observe it shows, for one token:

- The token: its text, its id, its position, its
sequence, and where its record stands among those kept. With
follow the last token ticked the page asks for the last
record twice a second and draws each new one. Older,
newer, the slider and the left and right arrows step
through the kept records (
serial=S), which stops the following: an answer is read again token by token after it was given. New tokens then do not move the view; the slider's range grows. A record dropped since (the server keeps the last 512) is said and the last one shown. sequence picks a scheduler's slot; empty, the one a forward pass recorded last. - What each block wrote: a row a block, the length of the attention's write and of the feed-forward's on one scale, the length of the stream after the block on its own; a row under the pointer tells its three figures.
- The lens (a server started with
--observe-lens K): a row a block, the K tokens the output head finds likeliest after it, shaded by their probability. Without the option the page says how to get it. - Attention: the blocks named in the field (at most
attn_slots, 4) are recorded from the next token on; each is a grid, a row a head and a column a position, darker for more weight, with the positions most looked at named above it and the cell under the pointer told (head, position, the token there, the weight). A block the model does not have is refused with the server's reason. A record made before the blocks were named has no row. - Experts (a mixture of experts): a row a routed block, a cell an expert, darker for more of the router's weight; a cell tells its expert, its weight and its rank.
Each view carries the caution of observe.md in a line: a
large write is not an important one, attention is not explanation, the
early lenses are noise, the experts picked say what must be in memory.
On a server without --observe the page says so, with the
command; on one with keys it asks for the key, the one the dashboard
keeps. It does not take parts out (--ablate is the
command's and the window's) and draws one sequence at a time.
tools/check_dashboard.sh
(CHECKS=activations) drives it in Chrome and in Firefox on
Granite 3.1 1B-A400M with --observe --observe-lens 3: the
link from the dashboard, the record of the last token, the 24 rows of
the writes drawn and one told, the lens's three tokens a block, the
experts' grid and a cell, a block out of range refused, the rows of
blocks 0 and 5 drawn after the next request and a cell told, the kept
records stepped by the buttons, the left arrow and the slider while new
tokens arrive, following again, and a server that records nothing.
The dashboard's chat
The page reads GET /api/commands when it opens and takes
a line that starts with / as a command, by the rules every
chat follows (inference.md): a command
only at the line's start, //text for a text that starts
with /, an unknown name answered with the nearest ones and
never sent to the model. What a command answers appears in the
conversation as a note, in a fixed-width grey type with a bar on its
left, which no reply of the model has; a refusal or a warning of the
server is a note in the warning's colour. While a / word is
typed the matching commands are listed under the conversation, each with
its arguments and its help, greyed with the reason when it cannot run
here: the arrows choose, Tab or Enter completes, Escape closes the
list.
| Command | In the page |
|---|---|
/help [NAME] |
the commands of the page, each with why it does not run when it does not; one command in full |
/reset (/clear, /new), the
Clear button |
a new conversation; what is attached stays |
/system [TEXT|-] |
the system prompt, heading every request, and a new conversation;
/system alone shows it, /system - removes
it |
/context [TOKENS [force]] |
GET /api/context, or a resize (above): a dry run first, whose
fits false is told in the warning's colour with the two
sizes and nothing changed unless force follows; the
answer's warning (above the trained context) in that colour
too. The conversation is kept in the page; the server's cache is
lost |
/attach |
opens the file chooser, as the Attach button does (a page cannot open a file by its name: a name given is ignored, and said so) |
/detach [N] |
removes what is attached, or the Nth file |
/rag on, /rag off, /rag NAME,
/rag |
on a server started with --rag or
--rag-dir: /rag on answers from the server's
own index (the requests carry "rag": true),
/rag NAME from one of those it offers
(rag_indexes in /api/stats; the requests carry
"rag": {"index": "NAME"}), and the sources used are noted
under the reply (Retrieval). /rag
alone tells whether retrieval is on, with which index, and lists the
indexes. A name the server does not offer, or a path, is refused with
the list: a page names an index and cannot open a file |
/save FILE |
the conversation downloaded as a JSON file (the system prompt, the messages as sent, what was typed and the attached files' names); dropped on the chat later, it is opened again and continued |
/export [markdown|json] |
chat.md (a heading per message, the files' names) or
chat.json (the request's messages)
downloaded |
/copy [N] |
the Nth latest reply to the clipboard |
/retry (/regenerate) |
the last reply dropped and asked again; after an error or a stop before the first word, the message asked again |
/rewind [N] (/undo) |
the last N exchanges removed from the conversation and from its view, the last message taken back put in the message box |
/stop |
ends the reply being written (the request is dropped, the server stops generating); the Send button reads Stop meanwhile and does the same. What was written is kept as the reply |
/stats (/usage) |
the messages, the context, the prompt and generated tokens of the last exchange, their share of the context, the reply's speed; the note that replaced earlier messages, and the two settings of a full context |
/continue, the Continue button of a
cut reply |
goes on with the last reply where the context or the token limit cut
it: the conversation is sent again with the reply as the open last
message ("prefix": true, its reasoning apart in
reasoning_content; continuing
a reply), and what comes is added to the same reply. After a message
the full context left unanswered, it answers the message |
/compact [FOCUS] |
the older messages become a note the model reads after the system
prompt (POST /api/compact, above); they stay on the page,
dimmed, and in /export and /save; the last
exchange and a message with a file stay as they are |
/autocompact [on|off] |
shows or sets whether the conversation is compacted by itself: before a message when the server finds it past the threshold, and when the context cut a reply or refused a message and could not be made larger (room in a full context). On by default, kept in the browser |
/autocontinue [on|off|N] |
shows or sets whether a cut reply is gone on with by itself, and how many times in a row at most (8). On by default, kept in the browser |
/temp, /top-k, /top-p,
/min-p, /seed, /max-tokens |
shown, or set for the requests that follow
(temperature, top_k, top_p,
min_p, seed, max_tokens); unset,
the server's default |
/thinking [on|off] |
"chat_template_kwargs": {"enable_thinking": true|false}
in the requests that follow; unset, the model's default |
/decide A | B | ... |
POST /v1/decide with the conversation as the state and
the options as a choice: the option chosen, its confidence, each
option's probability. The conversation is kept |
/models |
the entries of GET /v1/models, the one in use marked;
then the catalogue's models installed on the server's machine
(GET /api/models/installed), each with its size, its
categories and what it is for, under a line that says how another is
served: /model NAME loads one in place of the model served,
or, on a server that keeps its model, the command that starts another
server |
/model [NAME], the model control |
the model in use; with a name, the server loads that model
in place of its own (POST /api/model with thirty
seconds of wait: Switching
the model): a note says the load started, then
model: NAME, loaded in 2.8 s, 4096 tokens of context; the conversation goes on with it
(with images or sounds when it came with a
projector), and the page's title, features and commands follow. The
conversation is kept: the next message sends it whole to the new model.
A refusal is the server's reason, in the warning's colour, the model
unchanged. The model control beside the chat's buttons
lists the installed models a chat is served with, the one served
selected; choosing another does the same. On a server with
--lora-module adapters a name chooses the adapter (or the
model itself again) as the model of the requests that
follow, and on that server, as on one started with
--fixed-model or --lora, the control is hidden
and the name of an installed model is answered with the command that
serves it (oklm serve -m NAME --port PORT) and the server
manager (oklm-gui --tool servers -m NAME) |
/listen [stop|send], the Record button |
on a server started with --transcribe-model: records
the microphone, and writes what was said in the message box (below). /listen again,
/listen stop, the button again or Enter ends the recording;
Escape drops it; /listen send sends the text once it is
written |
/talk [on|off], the Talk switch |
on a server started with --transcribe-model and
--speak-model: a spoken conversation (below): the
microphone listened to, a pause ends the turn, which is transcribed and
sent, the reply is spoken while it is written, and speaking interrupts
it. /talk alone changes it; Escape leaves |
/speak [on|off|stop|pause|resume|TEXT],
/pause, the Speak button of a reply, the Read aloud
switch |
on a server started with --speak-model:
/speak reads the last reply aloud, /speak TEXT
a text, /speak on every new reply while it is written,
until /speak off; /speak stop ends what is
being said and the request behind it; /speak pause and
/speak resume hold the voice and let it go on, and
/pause does the one or the other, as in the chat window (below) |
/connect, /tool, /tui and
/exit belong to other chats and are answered so. While a
reply is written a command still runs (those that change the
conversation stop the reply first); a message waits, and the page says
/stop ends the reply.
Files are dropped on the chat, pasted from the
clipboard (an image), or chosen with the Attach button or
/attach. They wait in a row above the message box, each
with its name, its size and a cross that removes it, and leave with the
next message:
An image or a sound goes in the request as a content part before the text (Images and sounds), when
/api/statssays the server takes it (vision,audio). On a server without the projector an image is refused with that reason. An image shows a small picture, drawn from the file by the browser (a TGA or a PPM, which a browser does not draw, shows its name only). A sound for a model that hears was run on 2026-11-15 with Ultravox on Llama 3.2 1B (CHECKS=sound sh tools/check_dashboard.sh, on the CPU): the 11 s recordingjfk.wavdropped on the chat went as aninput_audiopart of base64 before the text, the server encoded it once for two questions, and the model repeated the sentence ("And so, my fellow Americans ask not what your country can do for you, ask what you can do for your country.").A video (
.ogv,.mp4,.mkv,.webm,.mov,.avi,.m4v,.mpg, or a file the browser callsvideo/...) goes the same way as avideo_urlpart, on a server with a projector (visionoraudio), sampled by the server's defaults (one frame a second, 8 frames); its row shows its name. The file must fit the request (32 MiB with its base64), and a container other than Ogg Theora needs the server's--ffmpeg: the server's refusal is shown as it is.A sound on a server that transcribes (
transcribein/api/stats) has a Transcribe button on its row: the file is sent as it is toPOST /v1/audio/transcriptions(Transcriptions), its text is added to the message box, where it can be edited, and the sound leaves the row: it is not sent. When the model hears nothing itself the row saysto transcribeand a message is not sent while the sound waits there; when the model hears, the sound is sent as a sound unless it was transcribed. A server that neither hears nor transcribes refuses the file, naming--mmprojand--transcribe-model.A document (PDF, DOCX, HTML, JSON, text: anything that is no image or sound by its type or its extension) is sent to
POST /api/ingestwhen it is attached (Documents), and may take half the context's tokens. Its row tells the kind the server read and its tokens. Its text goes in the message under its name,[name]on a line of its own, before what was typed. A document the server cannot read (a password, a binary file) or that holds no text is refused with the server's reason.A document longer than that is, on a server with an embedding model (
ingest_retrievalin/api/stats:--embed-model, or--rag), kept by the server for the conversation (above): its row readskept for retrieval in N passages, a note says so, and from the message it leaves with, each question is first sent toPOST /api/ingest/searchand goes to the model with the passages nearest to it (in the reranker's order on a server started with--rerank-model: the message's row then readspassages, in the reranker's order:; passages spread over the documents for a question about the whole, such as "summarize this": the line above them and the message's row say which), a quarter of the context at most, under a line that names the documents, each passage under its document's name and its lines. The message shows which passages it was sent with. Only the last question keeps its passages in the requests: the ones before go back to what was typed (their replies stay), so that the context does not fill with them./statslists what is kept;/reset,/system, a conversation opened from a file and leaving the page tell the server to drop it (POST /api/ingest/forget), and the server drops it by itself after--ingest-keepseconds without a question: the page then says the documents are gone and to attach them again. On a server without an embedding model the document is cut to what holds, its row sayscut to N of M characters, and a warning in the conversation says so and what would keep it whole.
A file larger than
upload_maxis refused before anything is sent.The conversation shows a message's files as its small pictures and their names and sizes, not the documents' text; the text is in the request, in
/exportand in/save.
What the page cannot do, and why: open a file by its name or write
one where it chooses (a browser gives a page the files its user picks,
and downloads go where the browser puts them); open a retrieval index by
its path (it names one of those the server offers) or load another model
(a server serves the one it was started with: /model says
how another is served); record the microphone on an origin that is not
secure (below); keep a
conversation across a reload (the browser stores the API key and the
settings of the voice and of the spoken conversation, nothing of what
was said: /save first).
Speech in the dashboard
On a server started with --transcribe-model (Transcriptions) the chat has a
Record button, and /listen does the same.
The browser asks once whether the page may use the microphone; then a
row above the message box shows the microphone's level, the time
recorded and the longest recording. The button (which reads Stop
meanwhile), /listen, /listen stop or Enter in
the message box ends the recording: it is sent to
POST /v1/audio/transcriptions and the text is added to the
message box after what the box held. It is not sent to
the model: it is read again, corrected, and sent like any message.
/listen send sends it once written (typed while a recording
goes on, or before: the recording it starts is sent when it ends).
Escape drops a recording, and nothing is posted. A recording whose
transcription fails (the server busy, an error) is kept in the row of
files as recording.wav, with its Transcribe button.
While the recording goes on, a line under the meter reads
heard so far: with the words while
speaking: what the recognizer heard of the recording so far,
asked every two seconds of new sound (The words while speaking). The
words two readings agree on are in the text's colour; what the last one
heard after them is in italics, and the next reading replaces it. The
line goes when the recording ends, and the message box gets the text of
the whole recording, as before. A turn of a spoken conversation shows
the same line while it is said. The words while
speaking box beside the language turns it off (kept for the
next visit); it is not offered on a server that does not say it gives
partial texts.
- Where the microphone is given. A browser gives a
page the microphone on a secure origin only:
http://localhostandhttp://127.0.0.1(the server's own machine) count as secure, any other address needs HTTPS (--tls-cert,--tls-key). On another origin (http://192.168.1.20:7430, a host name over plain HTTP) the button is disabled with that reason in its title,/listenanswers it, and the list of commands shows it. Dropped recordings are transcribed on any origin. Without--transcribe-modelthe button is disabled and/listensays how to start the server. - What is sent. The browser's own recorder writes
WebM with Opus (Chrome;
audio/mp4with Opus as its other choice) or Ogg with Opus (Firefox, by its documentation), and the server's decoder does not read WebM: Chrome 157's recordings, posted to the route, are answered 400. The page therefore takes the samples itself (WebAudio: aScriptProcessorNode, since an audio worklet needs a module and the page'sContent-Security-Policyloads none), brings them to 16 kHz mono as they come (a windowed sinc 8 zero crossings wide, from whatever rate the browser's audio runs at) and builds a 16-bit PCM WAV file in memory when the recording stops: 32 kB a second. The policy is unchanged:getUserMediaand WebAudio are not under it. - The longest recording is the server's 30 minutes,
or what
upload_maxholds if that is less: 17 min 28 s with the default 32 MiB. A recording that reaches it is stopped there and transcribed, and a note says so; a dropped file aboveupload_maxis refused before any upload. - Refusals. The microphone refused in the browser, or no microphone: a note says which, and nothing stays recording (the tracks are stopped and the audio context closed, also when the recording is dropped while the browser still asks).
- The language. Beside the switches, a
language chooser lists
autoand the codes the server's model takes (languagesoftranscribe_stats: 99 for Whisper base; a model of English only hasenalone, and for a server that does not tell them the page offers sixteen common codes, English first). Withautothe request names no language and the model finds it: the page asks forresponse_format=verbose_jsonand shows what was found beside the chooser ("found: fr (99 %)"). A code chosen is sent aslanguage=, and kept in the browser across a reload. Where the model translates (translateintranscribe_stats), a to English box sends the sound toPOST /v1/audio/translationsinstead: the text comes in English whatever was spoken. The chooser and the box apply to the Record button, to/listen, to the Transcribe button of a dropped recording and to a spoken conversation. - Limits. The microphone is asked for as
{audio: true}, so the browser's voice processing (echo cancellation, noise suppression, automatic gain) is applied before the page sees the samples. The text comes when the recording has ended, not while speaking. A short or noisy dictation left onautomay be taken for another language: choose the language then.
On a server started with --speak-model (Speech) the page reads replies aloud:
The controls. Each reply ends with a small Speak button: it reads that reply, and reads Stop meanwhile.
/speakreads the last reply,/speak TEXTthe text given./speak on, or the Read aloud switch beside Clear, reads every new reply while it is written (below; one that was stopped is read as far as its last whole sentence);/speak offor the switch ends that. The switch is off when the page opens and lasts as long as the page:/resetkeeps it, a reload does not.on,off,stop,pauseandresumeare taken in any case when they are the whole argument:/speak on the quayis a text. While something is said a row above the message box tells it ("making the sound of the first words", then "speaking", with "part 2 of 3" for a long text) and holds a Pause and a Stop speaking button;/speak stop, the reply's own button,/reset, and a/retryor a/rewindthat removes the reply do the same as the second. One thing is said at a time: speaking another reply or text stops the first. The chat is not held meanwhile: messages and commands go on.The voice and its speed. Beside the switches, a voice chooser lists what the server offers (
voicesofspeak_stats, the list ofGET /v1/audio/voices: the voice the server was started with, marked as the server's, the model's built-in one, and the speaker files of its voice folders, read again every two seconds, so that a file dropped in the folder appears without a reload), and a speed field takes 0.25 to 4 (1: as the model speaks; the pitch is kept). Both are sent with each text (voice;speedwhen it is not 1), apply from the next text asked for, and are kept in the browser: the page opens again with them (a voice the server no longer offers gives way to the server's own). The route still takes a name and never a path, and a voice cannot be made from a recording here.Pause and resume. The Pause button of the row,
/speak pauseor/pause, holds the voice where it is (theAudioContextis suspended): the row reads "paused", the button Resume. The request is not stopped: the sound made meanwhile waits behind what was not played. Resume,/speak resumeor/pauseagain, goes on from there; a stop while paused ends the voice. A click elsewhere in the page does not wake a paused voice.
A reply heard again. The samples of a reply spoken to its end (by its button, by
/speak, or while it was written) are kept with it in the page. Its Speak button and/speakthen play them at once and ask nothing of the server: the row reads "speaking again (the sound was kept: nothing is asked of the server)". They are used only for the voice and the speed they were made with: with another choice the server is asked again, and the new sound takes the place of the old one. A reply that was stopped, a text of/speak TEXTand a conversation opened from a file have no kept sound;/saveleaves the samples out; ten minutes of sound are kept in all (48 kB a second), the oldest replies' dropped first; a reload keeps none.What is sent.
POST /v1/audio/speechwith{"input": TEXT, "response_format": "pcm", "words": 12, "first_words": 5}(the last one with the first text of a voice only;voiceandspeedas chosen) and the page's key: the samples come while they are made. A stop closes the request, which ends the server's synthesis (cancelledinspeak_stats). A text above the server'smax_input(speak_stats, 4096 bytes unless--speak-max) is cut at sentence ends into parts of at most that many bytes, asked for one after the other (a sentence above the limit is cut at a blank, a word above it where it must).How it is played. WebAudio, no
<audio>element (which would ask the route with aGET): each piece read from the response (an odd last byte kept for the next) becomes anAudioBufferof 16-bit samples divided by 32768, at the rate ofX-Sample-Rate, started where the one before ends, or now when the sound already played has run out. TheAudioContextruns at the server's rate (24 kHz), so that no buffer is resampled on its own: 24 kHz with OuteTTS, 32 kHz with Soprano (--speak-model soprano-1.1-80m), whose samples the page plays at that rate, and keeps at that rate for a reply heard again; nothing in the page holds a rate as a constant. The page'sContent-Security-Policyis unchanged:fetch()to the server's own origin and WebAudio are within it.What is read. The reply without its reasoning (
<think>to</think>: a reply that is all reasoning has nothing to read), with "Code omitted." in place of each fenced code block, a link as its text and a bare address as "link", an image as its description, and without the marks of Markdown: emphasis, inline code's quotes, headings'#, list bullets and numbers, quote marks, rules, a table's bars (a row is read as its cells) and HTML tags. A heading, a list item, a table's row and a paragraph end as a sentence. The synthesizer itself reads English letters and numbers and drops the rest; a text with none of them is answered "nothing in it can be read aloud" and nothing is asked of the server.A reply read while it is written. With Read aloud on, the voice does not wait for the reply's end. Each time the reply grows the page works out which of its sentences no later text can change, and gives the new ones to the voice: the sentences of the lines that are whole, and, of the line being written, those ended by
.,!or?and followed by a blank ("3." waits: it may become "3.14"), as long as no mark of Markdown is left in them (an emphasis, a code quote or a link still open, a*, a_or a|whose meaning the rest of the line decides: that line then waits for its end; a_inside a word is none). A code block being written is told once, when its fence opens; reasoning is never read, whether it comes between<think>and</think>in the text or apart from it, asreasoning_contentin the stream's deltas (a harmony model's analysis): the page folds that one above the reply the same way and keeps it out of the message it stores and sends back. A template that opens the reasoning itself sends no<think>, so the page cannot know reasoning from the answer until</think>comes: after/thinking onthe voice waits for one of the two marks (or for the reply's end), and without it the voice starts again from the answer when the closing mark shows that what was being said was reasoning. The voice asks the server for one text at a time, in order: when a request ends, every sentence that waits goes in the next one, as far asmax_inputtakes them, so a voice slower than the writer makes fewer and longer requests, each still sent pass by pass. Every request asks for passes of twelve words; the first one of a reply (and of any text spoken) also asks for its first five words as a pass of their own ("first_words": 5), so that the first sound waits for five words and joins the rest without a seam, the passes being written under one prompt (Speech). A 503 (the server's voice busy with other clients) is asked again two seconds later, six times at most, before it is said; the Speak button of a reply says it at once. When the reply ends its last words follow. Cut at every one of their 1050 positions, 20 replies (emphasis across sentences, links, code blocks open and closed, lists, tables, reasoning, numbers) give sentences that are the first ones of the whole reply's, and never fewer than at the cut before. What it gains is the time the reply takes to be written after its first sentence. A reply of 1500 tokens from SmolLM2-135M (asked for with a system prompt that makes it tell a long story) was first heard 36.7, 37.7, 68.2 and 74.9 s after it was sent when read at its end, which came after 29 to 54 s; read while written, 11.2 to 11.4 s after it was sent in four runs, the reply then taking 78 to 109 s to end, since the voice works on the same cores. For a short reply of a fast model there is nothing to gain: three sentences (53 tokens, written in 0.3 to 1.9 s; 1.6 to 4.0 s beside the voice) were first heard 4.9 to 23.0 s after sending when read at the end and 4.8 to 19.4 s when read while written, over nine runs each on a CPU shared with other work: that spread is the machine's.The delay on a CPU. Nothing is heard before the first pass is made, and a pass is whole sentences of at most 12 words together (
words). The choice, on a first sentence of 11 words: 8 cuts it in two passes (2.3 s then 1.4 s of sound, a silence inside the sentence), 12 speaks it whole (3.1 s), 24 waits for two sentences (4.7 s of sound before anything is heard). On the CPU of the measures below OuteTTS 0.2 makes 0.1 to 0.45 s of sound a second, by how much other work the machine had at the time: three sentences of 8, 7 and 8 words (6.6 s of sound, three passes; the first is 2.4 s) were first heard 5.9 to 11.4 s after they were asked for in the page (5.2 s at the quietest, asked of the route directly; 18 to 27 s at the busiest), one sentence of 8 words after 6.6 to 15.6 s. Then each pass waits for the next, with a silence between them (2 to 5 s at the quietest) wherever the model is slower than the voice: at a sentence's end when the sentence fits a pass. A reply is heard well after it is written: the row says so until the first sound, and Stop is at hand.Refusals. Without
--speak-modelthe Speak buttons and the switch are not shown, the list of commands greys/speakwith "no speech synthesis here" and/speakanswers it, with how to start the server (--speak-model outetts-0.2-500m). A server whose synthesizer already has four texts waiting answers 503: the page notes that the voice is busy, and the reply stays with its button to ask again. A synthesis that fails is noted with the server's reason.A browser that will not play. A browser lets a page play sound once its user has clicked it or pressed a key in it: the page makes or wakes its
AudioContextin the handler of the button, of the switch or of the Enter that sends/speak, so this is met only when speech starts with no such gesture before it. Then the sound is made and scheduled but not played, the row reads "waiting for a click on the page to play", a note says so, and the first click or key in the page plays it from its start.The voice loaded ahead. A server loads its speech model at the first request (above). When Read aloud is switched on, or a conversation started, on a server whose voice is not loaded yet (
loadedfalse inspeak_stats), the page asks at once for one word that it does not play, and drops the request as soon as its first sound comes, or as soon as a real text is to be said: the first sentence of the first reply then finds the model in memory. It counts as one textcancelled(or spoken) inspeak_stats. The Speak button and/speak TEXTask for their own text directly.Limits. The synthesizer speaks English. The kept sound lives as long as the page. With Soprano the chooser holds its one voice (
soprano), and the first sound of two sentences was made 0.2 to 0.5 s after the request on the CPU of the measures: the delays above are OuteTTS's.
A spoken conversation in the dashboard
On a server started with both --transcribe-model and
--speak-model, opened where the browser gives the
microphone (above), the chat has a Talk switch beside
Read aloud. The switch, /talk on or /talk
alone starts a conversation that needs no key and no button:
- The page listens. The microphone is opened once and stays open. A row above the message box shows the state, the microphone's level and two settings. A turn starts when the level (the meter's: the square root of the largest sample of the last 43 ms, from 0 to 1) stays at or above the level setting (0.25) for a quarter of a second, and ends after a pause (1.2 s) under it: the chat window's two lengths (gui.md). The level is not the window's -40 dB: the page reads the largest sample of a block, after the browser's gain control, where a recording's background alone reaches 0.13 to 0.2 on the meter (-35 to -28 dB), so that a turn would never end at -40 dB (0.1 on the meter); 0.25 is -24 dB of peak. Silence starts no turn, and nothing is posted for it.
- The turn is transcribed and sent. Its samples, with
the half second before its start so that its first sound is in them, go
to
POST /v1/audio/transcriptionsas the Record button's do (a 16 kHz WAV file built in the page), and the text is sent as a message without a press. A turn in which no word was heard (no letter and no digit, or a remark between brackets on a noise) sends nothing. - The reply is spoken while it is written (above), whatever the Read aloud switch says.
- When the voice has ended the page listens again.
- The states. The row reads
listening,transcribing,thinking(the reply is asked for, no sound yet) orspeaking. The row of the voice above it keeps its Stop speaking button: it stops the voice, the reply goes on as text, and the page listens again once the reply has ended. - Interrupting. While a reply is written or spoken
the microphone stays open. A voice at or above one and a half times the
level for 250 ms stops the voice at once (its sound cut, its request
closed, which ends the server's synthesis:
cancelledinspeak_stats), stops the reply (the request dropped as/stopdoes: what was written is kept in the conversation), and is the start of the next turn, the half second before it included. - The settings.
pause(0.3 to 5 s) andlevel(0.02 to 0.9) are the two fields of the row, used from the next block of samples on. A speaker who pauses inside sentences wants a longer pause (the test's sentence, an orator's, has two silences of a second in it and is heard whole with 1.5 s). They are kept in the browser, with the box below, the voice and its speed: the page opens again with them./talktakesonoroffonly: the command is the same in every chat, the settings are the page's. - A level that follows the room. With follow the room ticked (it is when the page is first opened), the level in use is the setting or, when it is more, what the room's noise asks for: 1.6 times the noise and 0.05, 0.9 at most. The noise is the quietest level heard in the last three seconds (the quietest block of each second is kept): a voice leaves gaps between its words where the level falls back to the room's, a fan does not. In a quiet room nothing changes (a background of 0.13 on the meter asks for 0.26); a fan at 0.4, above the setting, starts a turn that would never end with a fixed level: followed, the level is 0.69 a second later, the false turn ends after the pause (nothing is sent for it: no word is heard), and a voice above the fan is a turn again. The row says so ("raised to 0.69 by the room's noise"). The level never goes under the setting: a quiet room does not make the page more sensitive to what is left of its own voice. Unticked, the level is the setting alone. It is a rule on levels, checked on levels made for it: a speaker who leaves no gap for three seconds over a loud background raises it too, and is cut at the next pause.
- What is said while a turn is transcribed. The detector goes on while the server writes a turn. A voice that starts again then is the next turn: its samples are kept, the first text is held back until that turn has ended, it is transcribed in its order, and both leave as one message. A noise in which no word is heard adds nothing, but holds the message until its pause has passed.
- Echo, and headphones. The voice played by the
loudspeakers returns into the microphone. The page asks the browser for
its echo cancellation by name:
getUserMedia({audio: {echoCancellation: true, noiseSuppression: true, autoGainControl: true}}). (The Record button asks{audio: true}, which Chrome answers with the three on as well.) When the track's settings say the browser gave none, a warning in the conversation says to wear headphones: without cancellation the voice is heard as the user's, interrupts itself and is sent back as a turn. What is left of the played sound after the canceller is what the higher level and the longer hold of an interruption are for. This was not verified with a loudspeaker and a microphone: the check drives Chrome with a file in place of the microphone, which hears nothing of what the page plays. It proves the constraints asked and the settings Chrome reports (echoCancellation,noiseSuppressionandautoGainControltrue, 48 kHz), and that a voice stops the reply; how much of a loudspeaker's sound a given machine lets through is to be tried on that machine, and headphones make the question moot. - Leaving. The switch,
/talk offor Escape: the microphone is released (its track stopped, the audio context closed: the browser's recording mark goes out), the voice is stopped, a turn half said is dropped. A reply being written goes on, as text./listenand the Record button are refused while a conversation holds the microphone. - What to expect on a CPU. The pause, then Whisper,
then the model's first sentence, then the voice's first pass, which is
the long one: with Whisper base, SmolLM2-135M and OuteTTS 0.2 on the CPU
of the measures the first sound came 6.3 s after the sentence's last
word when the machine was quiet (1.5 s of pause, 0.9 s of transcription
and sending, then 3.8 s for the model's first sentence and the voice's
first pass, of five words), and 20 to 37 s after it when other work
shared the cores and the first pass was of twelve words (14.7 to 29 s of
them after the message was sent). The voice then comes in bursts, a pass
being made slower than it is heard (above). A reply can be interrupted
from its first word, before any sound:
thinkingis interrupted likespeaking. The first turn after the server starts also loads Whisper (a fraction of a second here) unless the server was started with--preload, as the check starts its own; the voice is asked to load when the conversation starts (above). - The voice. The one chosen beside the switches (above): the server's own
(
oklm serve --speak-voice FILEwith a speaker file of OuteTTS, or the model's,en_male_1) unless another of those it offers is picked. Cloning a voice is not possible: making a speaker file from a recording needs WavTokenizer's encoder and a word aligner, which OKLM does not have (speech.md). - Limits. Whisper may write words for a noise loud enough to start a turn ("Thank you."): they are sent. The text of a turn appears when it is sent, not while speaking. The voices speak English: a conversation in another language is transcribed (below) and answered in text that the voice reads badly.
- The language of a conversation. The one chosen in
the language chooser (above). On
autothe first turn in which words are heard is sent without a language, and the language the model found for it is told for every later turn of that conversation: a short turn ("Yes.", "And then?") is not guessed again, where detection on a second of sound is at its weakest. Choosing a language in the chooser during the conversation replaces it;autochosen again lets the next turn be found anew.
The page's parser is a copy in JavaScript of the library's, and the
two are held together: tests/server/dashboard_cases.tsv
lists lines with what they are (a message, a command with its arguments,
or the error), test_dashboard_chat asks the library's
parser for each and checks the page's table of how each command reads
its arguments against the library's, and
tools/check_dashboard.sh asks the page's parser for the
same lines in a headless Chrome or Chromium (driven by Node over the
DevTools protocol, nothing installed), then drives the chat on a server
it starts with SmolLM2-135M on the CPU: every command above, the list of
commands under the keyboard, files chosen, dropped and pasted, the
limits and the refusals, with bge-small-en-v1.5 as the server's
embedding model a document of 3510 tokens kept for retrieval (its row
and its note, the question sent with three passages, the one that
answers among them, in a quarter of the context, the message before sent
again without its passages, a kept document detached before its message
left out of the search, /stats, /reset
dropping what the server kept), and with SmolVLM-256M and its projector
a pasted picture sent and answered, and with Whisper base beside
SmolLM2-135M and whisper.cpp's jfk.wav given to the browser
as its microphone (--use-fake-device-for-media-stream,
--use-file-for-fake-audio-capture) the Record button and
/listen (the sentence in the message box, the WAV file
posted, Enter, Escape, the longest recording,
/listen send), the sample dropped and transcribed from its
row, the refusals (no transcription model, an origin that is not secure,
the microphone refused or missing), with OuteTTS 0.2 beside SmolLM2-135M
the voice (the request sent with its key, the buffers scheduled and
played to their end in a browser with no sound card, what was played
written as a WAV file and transcribed again by Whisper base, a stop
cancelling the server's synthesis, a text in two parts, the Speak
button, /speak in its forms, the switch, a reply of three
sentences asked of the voice while it is written, a busy voice asked
again, the refusal to play before a click, a busy server; the first
words asked as a pass of their own, a reply's kept sound heard again
with the same samples and nothing asked of the server, pause and resume
by the button and by /speak, another speed asked of the
server again, a second speaker file chosen and named in the request, the
choices found again in another tab, and a server started without
--preload whose voice the switch loads ahead) and no track
left live, with the three models together a spoken conversation (the
microphone is a file the script builds from the sample and silence: the
turn ended by the pause and sent, the states in their order, the first
sound scheduled while a long reply is still being written, the page
listening again after the voice, nothing sent for silence, the three
ways out leaving no live track; and, in a second browser, the sentence's
end said again while the first reply is spoken, which stops the voice
and the reply and is the next message; and, on levels and transcriptions
made in the page, the level that follows a fan, two turns said one
during the other's transcription sent as one message, the settings kept
across a reload), and on a server with an adapter (two training steps on
the spot), an index of one file, a directory of two indexes and a fixed
context /model (the adapter chosen, an unknown name told
what the server serves), /models, /rag (on,
off, alone, an index of the directory by its name: the request naming
it, the sources that index's, the server's figures counting its run
apart; a path and an unknown name refused by the page and by the route)
and /context refused, and on a server that asks for a key
the page before and after the key is given; with Whisper base the
language chooser (auto and the model's 99 codes, the language found
shown, a code and the translation route in the request, the choice kept
in another tab, the language of a conversation's first turn kept for its
next ones), and with Soprano 1.1 a voice at 32 kHz (the buffers' rate,
the context's, the seconds played against the server's, what was played
heard again by Whisper base, the kept sound at that rate, the chooser
with its one voice), and a video (CHECKS=video, on the tiny
model the server test writes with the tiny Qwen2-VL projector of the
fixtures, else on SmolVLM-256M: the fixture video dropped, its row with
its name and no picture, the request's video_url part
holding the file's base64 before the text and nothing about the
sampling, one item encoded by the server and the reply written, the part
sent again with the next message and kept by /export json;
a file the browser gives no type taken as a video by its extension;
bytes that are no video refused by the server, its message shown; and,
in the chat group, a video on a server without a projector refused by
the page). It is skipped with a message when no such browser or no Node
22 is installed, and leaves pictures of the page and the first recording
in build/check-dashboard/.
The browsers the page was run in
sh tools/check_dashboard.sh drives Chrome or Chromium
over its DevTools protocol;
OKLM_CHECK_BROWSER=firefox sh tools/check_dashboard.sh
drives Firefox, headless with a profile of its own, over WebDriver BiDi
(tools/check_dashboard_firefox.mjs: Firefox no longer
speaks Chrome's protocol), through the same groups and the same checks,
its pictures in build/check-dashboard-firefox/.
| Chrome 157 | Firefox 153 ESR | |
|---|---|---|
| the chat, the commands and their list, files chosen, dropped and pasted, the kept documents, the adapters and the indexes, the key | run | run: nothing differs in the page |
a video attached (CHECKS=video, added on
2026-11-14) |
run, Chrome 157 | not run |
a full context (CHECKS=full, added on 2026-11-16, the
walk every driver shares: tools/check_dashboard_full.mjs):
a cut reply gone on with by itself, by its Continue button and by
/continue, the pieces the reply written in one go; the
bound; a limit the person gave; the conversation compacted by itself and
with /compact; a reply the context cut where the context is
fixed; the context made larger for a reply it cut; a reasoning reply cut
inside its reasoning and inside its answer. 50 checks: SmolLM2-135M in a
context of 256 tokens with replies of 24, Qwen3-0.6B with replies of
48 |
run | run: nothing differs in the page |
| the file chooser | opened by /attach and by the Attach button |
the same; Firefox tells its driver the first chooser of a tab only, so the second is seen as the page's click on its file input |
Record, /listen, a dropped recording, the language |
the microphone is a sound file
(--use-file-for-fake-audio-capture) |
the same checks with the file played into a stream that takes the
place of getUserMedia's (Firefox has no file microphone):
the page's recording, its resampling from 48 kHz and its WAV file are
Firefox's; the sentence is heard word for word |
| the browser's own capture device | Chrome's fake device plays the file | Firefox's fake device (media.navigator.streams.fake, a
tone) is recorded by the Record button and posted (Whisper hears nothing
in a tone, and the page says so) |
Speak, /speak, Read aloud, pause, the kept sound, the
voices, the speed, Soprano at 32 kHz |
run | run: the same buffers, the same seconds, an
AudioContext at the server's rate |
| Talk: a turn ended by the pause, the reply spoken while written, the page listening again, a voice interrupting | run | run, on the file played as the microphone |
| echo cancellation | Chrome reports it on (echoCancellation true on its fake
device) |
Firefox's fake device reports it off: the page's warning to wear headphones is seen there. A real microphone was not tried in either |
What differs between the two, found by running:
- An
AudioContextneeds a page its user has acted on, in both. In Firefox the activation a driver gives lasts one evaluation, so the check carries it on each one in the microphone's groups; a person who clicked Record or Talk has given it. Nothing was changed in the page for it: it already makes and resumes its contexts in the handlers of its buttons, and tells when a browser will not play. getSettings()of a track names the three voice processings in both; a stream that is no device has none, and the page then does not warn (it warns when the browser saysechoCancellationis false).- A paste made by a script carries no file in Firefox
(its
ClipboardEventignores the data given to it): the check puts the picture in the clipboard and presses Control and V, so the picture sent to SmolVLM there went through a real paste. - No difference was found in the page's own code: the
streamed replies, the samples read from
fetch(), the buffers scheduled, the resampling, drag and drop, the clipboard, the downloads andlocalStoragebehave the same.
WebKit. Safari cannot be run on Linux, but its
engine can:
OKLM_CHECK_BROWSER=/usr/lib/x86_64-linux-gnu/webkitgtk-6.0/MiniBrowser sh tools/check_dashboard.sh
runs the page in WebKitGTK 2.52's MiniBrowser under Xvfb, driven through
its remote inspector (tools/check_dashboard_webkit.mjs; no
WebDriver is installed, so the walk of the group chat is
shorter than Chrome's and Firefox's; the group full is the
same walk, its 50 checks run there with nothing differing in the page,
what /export writes read from the page and not from a
download). Run there: the page and its commands, the list of commands, a
streamed reply, /stats, a document dropped and one pasted,
the Attach button reaching the file input, the language chooser, a text
spoken by Soprano (8 buffers at 32 kHz, their seconds the server's, all
played to their end, the AudioContext at 32 kHz), pause and
resume, a reply's kept sound heard again; no error in the page and
nothing changed in it. Not run there: Record and Talk,
because MiniBrowser asks for the microphone in a dialog of its window
that the inspector cannot answer (the page waits, its button reading
Stop, and is left cleanly: that much is checked). And WebKitGTK is not
Safari: its sound and its capture go through GStreamer, Safari's through
the system's, and Safari's rules for playing sound and for the
microphone on a phone are its own. Safari itself was not
run.
Room in a full context
When the context is what cut a reply, the page makes room before it goes on with it, as the chat window does (gui.md):
- A larger context first, where the server can hold
one: twice the size, the model's trained context at most, asked of the
server's own resize (
POST /api/context) after its dry run for that size ("dry_run": true) said that it fits the memory, that the cache holds a conversation filling it, and that it takes no more than half of the free memory (a cache that large is long to make). The note is the window's:The context was full (256 tokens): it is made larger, 512 tokens, and the reply goes on.A reply cut and gone on with across the resize is the reply written in one go (the server evaluates the conversation again in its new context). - Never on a server started with
--fixed-context(the page readsresizablefromGET /api/contextand asks for nothing), nor when the dry run refuses, when the trained context is reached, or while the server answers other requests (its resize answers 409): the conversation is then compacted, when/autocompactis on, and the note says so, or that nothing makes room. - A message the server refused for its length is
answered the same way when
/autocompactis on (... and the message is answered.); a context that is nearly full, not full, is compacted before the next message as before: the context is made larger only when it is what stopped a reply or a message.
Measured in the page (CHECKS=full, SmolLM2-135M on the
CPU, a context of 256 tokens, replies of 24, temperature 0): a story cut
by the context after its ninth piece had the server's context made 512
by two requests ({"n_ctx":512,"dry_run":true}, then
{"n_ctx":512}), went on to the bound of 12 pieces in a row,
and its text is the reply asked of the server in one go with as many
tokens; on the same server started with --fixed-context no
request for another context is made and the page says that the reply
alone fills the context.
A model that reasons, in the page: Qwen3-0.6B Q8_0
with /thinking on and replies of 48 tokens writes a reply
of 408 tokens in 9 pieces, 6 cut inside its reasoning and 2 inside its
answer; each cut reply goes back as the open last message (its
<think> block open, then closed and followed by the
answer's start), the page shows one reply with its reasoning folded
apart from its answer, and its text is the reply in one go, in Chrome,
Firefox and WebKitGTK alike. It is deterministic at temperature 0 on the
CPU, and is part of CHECKS=full when
qwen3-0.6b is installed beside the chat model.
The figures as JSON
What the tabs read, for any program: the expert map, the model's layout and the profile.
GET /api/experts:
{"moe":true,"n_layer":24,"n_expert":32,"n_expert_used":8,"recording":true,
"last_tokens":1,"answer_tokens":127,"cumulative_tokens":635,
"last":[[1,0,0,0,1,...],...], "answer":[[45,18,33,...],...], "cumulative":[[...],...]}Each matrix has n_layer rows of n_expert
counts: how many times the layer's router chose the expert for a token,
last in the last decode step (over the sequences decoding
in it), answer over the answers being generated (the last
one's once it ended), cumulative over every step recorded
since the server started. A dense layer's row stays 0. Each token
counted adds n_expert_used to its layer's row, so a row
adds up to n_expert_used times its *_tokens;
with one sequence decoding, last has exactly
n_expert_used experts chosen per layer.
- What is counted. The tokens of decode steps: a
prompt's rows are left out; a request verifying drafts
(
--spec-ngram,--draft) counts every row it evaluated, the rejected drafts included. - When. The first request for the map turns the
recording on (from the next step), and it turns off again when nobody
has asked for a minute: the counts (and
cumulative) cover the steps run while it is on. Off, the forward pass only tests a pointer. - Where. In the forward pass, after each MoE layer's
experts are chosen (
model_expert_count(),src/model/expert_map.c, behindoklm_expert_record()): each (token, rank) pair adds one to its sequence's (layer, expert) counter; after the step the scheduler reads and resets each sequence's counters (oklm_seq_expert_counts()). On the CPU the chosen experts are already in host memory, and the cost is a few increments per token and layer: decoding Granite 3.1 1B-A400M (24 layers, 8 of 32 experts) on the CPU with and without recording, alternating every 8 steps, gave medians of 12.93 and 13.00 ms a step, means 13.43 and 13.40 ms, within noise. A layer on a device reads its chosen experts back first, one read per MoE layer and step, but only while recording.
GET /api/model:
The model the server holds, laid out for a program: what
oklm info --json gives for the file, where each layer
actually is, and what the context allocated. It passes the access check
/api/stats passes (a key when the server has keys) and
answers 405 to another method. The model part and the placement do not
change while the server runs; context follows a resize
(POST /api/context).
| Member | Type | What |
|---|---|---|
id |
string | the model's name in the API (--model-id) |
backend |
string | cpu, or the backend of the device that holds the
head |
model |
object | the object of oklm info --json (convert.md) without
metadata: file, shards,
file_size, format, arch,
name, n_params, context_length,
vocab_size, quantization,
hparams, tokenizer, totals,
verdict, verdict_text,
arch_support, notes, layout (the
blocks with their tensors by role, the embeddings, the head); with
?tensors=1 also tensors, every tensor. null
for a model that is not a GGUF file (a Hugging Face checkpoint has no
tensor table to lay out) |
placement |
object | the object oklm plan --json calls
placement (memory-plan.md), for the context
the server runs: n_gpu_layers, n_cpu_moe,
devices (each with its name,
first_layer, layers, head:
several when the layers are spread over devices), layers
(each with index, on: cpu or
device, device: the device's name or null,
bytes, host_bytes, device_bytes,
kv_bytes, and for a layer with routed experts
experts_on and experts_bytes),
embeddings, head, other,
mtp_bytes, host_bytes,
device_bytes, kv_layer_bytes,
kv_host_bytes, kv_device_bytes. null with
model |
context |
object | the context's memory, measured from its allocations
(oklm_context_mem_used()), below |
adapters |
array | each --lora-module adapter: id (1 for the
first), name, scale, file |
context:
| Member | What |
|---|---|
n_ctx |
tokens a sequence |
n_seq, n_batch |
sequences of the cache, tokens a forward pass |
n_gpu_layers, n_cpu_moe |
layers on devices; the first layers whose routed experts stay in host memory |
kv_layer |
one layer's key/value cache |
kv |
host: the cache of the layers on the CPU |
device_kv |
devices: the cache of their layers |
state |
host: a hybrid stack's recurrent state, and the states and journals
it can hold for cut sequences (state_keep.bytes: those
allocated so far) |
activations |
host: the activation buffers |
scratch |
host: the rest of the context's allocations |
expert_cache |
host: the routed-expert cache's slots and pinned experts (0 without
--expert-cache) |
host_total |
kv + state + activations + scratch + expert_cache |
repacked, repacked_from,
repacked_experts |
the CPU matrix product's repacked copies of the weights, the mapped
bytes they stand for, and the part that is the routed experts' copies
(on the weights' side, not in host_total) |
kv_blocks, kv_blocks_free,
kv_block_size |
the paged cache's blocks, those free, tokens a block (0 for a contiguous cache) |
state_keep |
a model with recurrent blocks (qwen35,
qwen35moe): capacity and kept,
the states the context can keep and keeps; bytes, what they
and the journals hold; cuts, the sequences cut and brought
back (a request reusing a cached prefix, a rejected draft);
journal_tokens and forward_tokens, the tokens
those cost through the recurrence alone and through a forward pass (hybrid-layers-design.md).
Zeros for any other model |
In placement, the weights' bytes are the file's:
host_bytes + device_bytes + mtp_bytes is
model.layout.bytes, and each layer's
host_bytes + device_bytes its block's bytes in
model.layout.blocks. The device's activation buffers are
not measured (oklm plan --json projects them as
context.device_buffers).
A mixture-of-experts model on the CPU
(oklm serve -m granite-3.1-1b-a400m-instruct-Q4_K_M.gguf --backend cpu -c 2048 --parallel 4;
the blocks after the first and the layers between the first and the last
cut, lines wrapped):
{
"id":"granite-3.1-1b-a400m-instruct-Q4_K_M",
"backend":"cpu",
"model":{"file":"models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf","shards":1,"file_size":821847360,
"format":{"version":3,"alignment":32,"metadata_entries":46},"arch":"granitemoe",
"name":"Granite 3.1 1b A400M Instruct","n_params":1334628352,"context_length":131072,"vocab_size":49155,
"quantization":"Q4_K_M","hparams":{"block_count":24,"context_length":131072,"embedding_length":1024,
"feed_forward_length":512,"attention.head_count":16,"attention.head_count_kv":8,"rope.freq_base":1500000,
"attention.layer_norm_rms_epsilon":9.99999997e-07,"expert_count":32,"expert_used_count":8,"vocab_size":49155,
"rope.dimension_count":64,"attention.scale":0.015625,"embedding_scale":12,"residual_scale":0.219999999,
"logit_scale":6},"tokenizer":{"kind":"gpt2","pre":"refact","tokens":49155,"chat_template":true},
"totals":{"tensors":242,"params":1334628352,"bytes":820107736,"types":{"F32":73,"Q4_K":144,"Q6_K":25}},
"verdict":"loadable","verdict_text":"loadable","arch_support":"supported","notes":[],"layout":{"n_layer":24,
"n_blocks":24,"tensors":242,"bytes":820107736,"params":1334628352,"embeddings":{"tensors":1,"bytes":41290200,
"params":50334720,"roles":{"embeddings":{"tensors":1,"bytes":41290200,"params":50334720,"types":{"Q6_K":1}}}},
"blocks":[{"index":0,"mtp":false,"tensors":10,"bytes":34680832,"params":53512192,"roles":{"attention":{"tensors":4,
"bytes":1904640,"params":3145728,"types":{"Q4_K":3,"Q6_K":1}},"routed_experts":{"tensors":3,"bytes":32636928,
"params":50331648,"types":{"Q4_K":2,"Q6_K":1}},"routers":{"tensors":1,"bytes":131072,"params":32768,
"types":{"F32":1}},"norms":{"tensors":2,"bytes":8192,"params":2048,"types":{"F32":2}}}}],"head":{"tensors":1,
"bytes":4096,"params":1024,"roles":{"norms":{"tensors":1,"bytes":4096,"params":1024,"types":{"F32":1}}}},
"other":{"tensors":0,"bytes":0,"params":0,"roles":{}}}},
"placement":{"n_gpu_layers":0,"n_cpu_moe":0,"devices":[],"layers":[{"index":0,"on":"cpu","device":null,
"bytes":34680832,"host_bytes":34680832,"device_bytes":0,"kv_bytes":16777216,"experts_on":"cpu",
"experts_bytes":32636928},{"index":23,"on":"cpu","device":null,"bytes":34680832,"host_bytes":34680832,
"device_bytes":0,"kv_bytes":16777216,"experts_on":"cpu","experts_bytes":32636928}],"embeddings":{"on":"cpu",
"device":null,"bytes":41290200},"head":{"on":"cpu","device":null,"bytes":4096},"other":{"on":"cpu","device":null,
"bytes":0},"mtp_bytes":0,"host_bytes":820107736,"device_bytes":0,"kv_layer_bytes":16777216,
"kv_host_bytes":402653184,"kv_device_bytes":0},
"context":{"n_ctx":2048,"n_seq":4,"n_batch":512,"n_gpu_layers":0,"n_cpu_moe":0,"kv_layer":16777216,"kv":402653184,
"state":0,"activations":185741312,"scratch":7532160,"repacked":862617600,"repacked_from":44089344,
"repacked_experts":814743552,"expert_cache":0,"device_kv":0,"host_total":595926656,"kv_blocks":512,
"kv_blocks_free":512,"kv_block_size":16,
"state_keep":{"capacity":0,"kept":0,"bytes":0,"cuts":0,"journal_tokens":0,"forward_tokens":0}},
"adapters":[]
}A dense model answers the same shape, its blocks' roles
attention, dense_ffn and norms,
its layers without experts_on:
{
"id":"SmolLM2-135M-Instruct-Q8_0",
"backend":"cpu",
"model":{"file":"models/smollm2-135m/SmolLM2-135M-Instruct-Q8_0.gguf","shards":1,"file_size":144811360,
"format":{"version":3,"alignment":32,"metadata_entries":37},"arch":"llama","name":"Smollm2 135M 8k Lc100K Mix1 Ep2",
"n_params":134515008,"context_length":8192,"vocab_size":49152,"quantization":"Q8_0","hparams":{"block_count":30,
"context_length":8192,"embedding_length":576,"feed_forward_length":1536,"attention.head_count":9,
"attention.head_count_kv":3,"rope.freq_base":100000,"attention.layer_norm_rms_epsilon":9.99999975e-06,
"vocab_size":49152,"rope.dimension_count":64},"tokenizer":{"kind":"gpt2","pre":"smollm","tokens":49152,
"chat_template":true},"totals":{"tensors":272,"params":134515008,"bytes":143025408,"types":{"F32":61,"Q8_0":211}},
"verdict":"loadable","verdict_text":"loadable","arch_support":"supported","notes":[],"layout":{"n_layer":30,
"n_blocks":30,"tensors":272,"bytes":143025408,"params":134515008,"embeddings":{"tensors":1,"bytes":30081024,
"params":28311552,"roles":{"embeddings":{"tensors":1,"bytes":30081024,"params":28311552,"types":{"Q8_0":1}}}},
"blocks":[{"index":0,"mtp":false,"tensors":9,"bytes":3764736,"params":3540096,"roles":{"attention":{"tensors":4,
"bytes":940032,"params":884736,"types":{"Q8_0":4}},"dense_ffn":{"tensors":3,"bytes":2820096,"params":2654208,
"types":{"Q8_0":3}},"norms":{"tensors":2,"bytes":4608,"params":1152,"types":{"F32":2}}}}],"head":{"tensors":1,
"bytes":2304,"params":576,"roles":{"norms":{"tensors":1,"bytes":2304,"params":576,"types":{"F32":1}}}},
"other":{"tensors":0,"bytes":0,"params":0,"roles":{}}}},
"placement":{"n_gpu_layers":0,"n_cpu_moe":0,"devices":[],"layers":[{"index":0,"on":"cpu","device":null,
"bytes":3764736,"host_bytes":3764736,"device_bytes":0,"kv_bytes":6291456},{"index":29,"on":"cpu","device":null,
"bytes":3764736,"host_bytes":3764736,"device_bytes":0,"kv_bytes":6291456}],"embeddings":{"on":"cpu","device":null,
"bytes":30081024},"head":{"on":"cpu","device":null,"bytes":2304},"other":{"on":"cpu","device":null,"bytes":0},
"mtp_bytes":0,"host_bytes":143025408,"device_bytes":0,"kv_layer_bytes":6291456,"kv_host_bytes":188743680,
"kv_device_bytes":0},
"context":{"n_ctx":2048,"n_seq":4,"n_batch":512,"n_gpu_layers":0,"n_cpu_moe":0,"kv_layer":6291456,"kv":188743680,
"state":0,"activations":117575680,"scratch":3830912,"repacked":142884864,"repacked_from":142884864,
"repacked_experts":0,"expert_cache":0,"device_kv":0,"host_total":310150272,"kv_blocks":512,"kv_blocks_free":512,
"kv_block_size":16,
"state_keep":{"capacity":0,"kept":0,"bytes":0,"cuts":0,"journal_tokens":0,"forward_tokens":0}},
"adapters":[]
}GET /api/profile:
{"requests":[{"id":5,"kind":"generate","finish":"length","wall_ms":3995.4,"queue_ms":0.2,
"prefill_ms":71.0,"decode_ms":3924.1,"tail_ms":0.05,"sample_ms":16.0,"detok_ms":0.3,
"prefill_tokens":3,"cached_tokens":16,"decode_tokens":128,"end_ms":1794074130666}],
"decode_steps":{"total":635,"kept":635,"mean_ms":41.1,"p50_ms":15.4,"p90_ms":72.4,"p99_ms":474.8,
"max_ms":1088.3,"bucket_ms":[0.25,0.5,1,...,2048],"hist":[0,0,0,0,0,0,375,153,...]}}Per request, oldest first, from timestamps the scheduler takes:
| Member | What |
|---|---|
queue_ms |
from its submission until a sequence took it |
prefill_ms |
from then until its prompt was evaluated and its first token sampled (its share of the steps it ran in, other requests' tokens alongside) |
decode_ms |
from then until it ended |
tail_ms |
from then until its last holder let go of it: the answer built, or the stream's last write |
wall_ms |
the sum of those four: its time from submission to its end |
sample_ms, detok_ms |
within prefill and decode: sampling its tokens and checking them against its grammar; turning them into text and looking for stop strings |
prefill_tokens, cached_tokens,
decode_tokens |
prompt rows evaluated (again after a preemption), found cached, tokens generated |
kind, finish |
generate or score (a decision's scoring
request), how it ended |
decode_steps covers the last 1024 steps that decoded
without a prompt piece beside: hist[i] counts those under
bucket_ms[i] (and over the one before), the last bucket
those over 2048 ms. A request cancelled while it waited is not
profiled.
Scheduler
src/server/sched.c.
- The context. One context with
--parallelsequences, each-ctokens long. A scheduler thread runs one forward pass per step,oklm_eval_batch(), over every active sequence. - Each step. Each request generating adds its last
token. Prompts then fill what is left of the batch
(
n_batch, 512 by default), so a long prompt is cut into pieces and the others keep generating meanwhile. The decode rows go in sequence order (the requests are chosen oldest first, then sorted), so the same requests give the same batch from step to step: a backend that records a decode step once (Vulkan) replays it until a request ends or joins (vulkan.md). - Joining. New requests join between steps, each in the free sequence that reuses the most of what it holds: the rows of its prompt's prefix it holds, less the rows it would discard. Only the rest is evaluated: a conversation sent again with one more message costs its new tokens.
- Parking. A short unrelated request between two
turns of a long conversation (an agent's title or classifier aside)
takes an empty or a short sequence, not the one holding the
conversation, whose prefix stays for its next turn. With the paged
cache, a sequence's blocks also stay cached prefixes after it is reused,
until the pool needs them: the free blocks go first, then idle sequences
give theirs back, and the cached blocks are taken least recently
released first, a sequence's last blocks before its first ones, so an
eviction shortens a cached prefix from its end. Measured
(
test_parkingintests/server/test_server.c, a conversation of 897 tokens, a side request of 42, the next turn of 939): with the default pool the next turn finds 896 tokens cached (its full blocks); with the contiguous cache 897 (before the side request stopped taking the conversation's sequence: 1); with a pool of 61 blocks (the next turn's size and one more) and a side request of 270 tokens, which must evict, 688: the side request's blocks came from the conversation's end. - Per request. A request has its own sampler (and
grammar). It ends on EOS or a control token, a stop string, a completed
grammar (JSON, required calls),
max_tokens, or a full sequence. - Same answers.
oklm_eval_batch()gives each sequence what separate evaluations would, so a request gets the same answer alone or batched (greedy, or with the same seed). - Clients going away. A client that goes away (a failed stream write) frees its sequence at the next step. A stop of the server ends every request and refuses new ones.
- Library use. The HTTP layer
(
src/server/server.c) runs on nilorea's HTTP/1.1 server: TLS, deadlines on every phase, keep-alive, 32 connections by default. Streams use its chunked bodies, whose write deadline counts per piece.
The library entry points, for embedding the server (the GUI):
oklm_server_params sp = oklm_server_default_params();
oklm_context_params cp = oklm_context_default_params();
oklm_server* s;
char err[256];
cp.n_ctx = 4096;
if (oklm_server_new(model, &cp, &sp, &s, err, sizeof(err)) == OKLM_OK) {
oklm_server_run(s); /* until oklm_server_stop(s), from any thread or a signal handler */
oklm_server_free(&s);
}Measured
OpenAI's own client. The official Python client (
openai3.42) against Qwen2.5-1.5B Q4_K_M, with an API key:models.list;- a chat answer ("5" to "What is 2 + 3?");
- a stream of 10 pieces with its usage;
- a tool call
get_weather({"city":"Rome"}), then the answer after the tool's result ("The current temperature in Rome is 21 degrees Celsius and the sky is sunny."); - a JSON-schema answer
{"name": "Paris", "country": "France"}; AuthenticationErrorfor a wrong key.
Batching. Eight requests of 128 tokens each, Qwen2.5-0.5B Q4_K_M, four sequences, i7-11700K / gfx1100 (2026-11-01):
Backend One by one Eight at once Speed-up Same answers CPU 62 tokens/s 154 tokens/s 2.5x yes OpenCL 266 tokens/s 581 tokens/s 2.2x yes More requests than 64 at once, Qwen2.5-0.5B Q4_K_M on OpenCL, the short workload of
tools/bench_serve.py, 2 requests a client (2026-11-04):Clients --parallel 64Default (512 sequences) 64 5026 tokens/s, TTFT p50 0.09 s 4979, 0.09 s 128 5178, TTFT 1.59 s 6388, TTFT 0.14 s 256 5203, TTFT 4.57 s 7265, TTFT 0.22 s A long prompt among streams, Qwen3-4B Q4_K_M on OpenCL,
-c 40960, four streams generating up to 700 tokens and after 4 s a prompt of 32,872 tokens (tools/bench_stall.py, 2026-11-04), the streams' gaps between tokens:Before During its prefill Its first token Before these changes median 12.1 ms median 27.6 ms, max 389 225 s Now 12.0 ms median 25.9 ms, p90 31.2, max 32.9 111 s Now, --prefill-share 0.2512.1 ms median 22.3 ms, max 28.9 110 s The prompt is read twice as fast (the attention kernels below) and the one stall at its start is gone (its first pieces were sized from short prompts' costs); the streams still run at about half speed while it is read: a piece's attention over the 30k keys before it costs about as much as their step.
Images, SmolVLM-256M (Q8_0, i7-11700K / gfx1100, 2026-11-03), the newspaper picture of llama.cpp's tests asked about three times, 40 tokens at most; the answers those of
oklm describe, byte for byte:Request OpenCL CPU Rows found cached The picture, cold 4.43 s 4.24 s 0 of 349 The same request again 0.13 s 0.21 s 336 of 349 (the image's 320) A second turn resending the picture 0.11 s 0.18 s 336 of 377 The cold request is mostly the encoder (5 tiles of SigLIP, on the CPU).
A draft model, Qwen3-4B Q4_K_M with Qwen3-0.6B Q8_0 drafting (2026-11-04), one request at a time, the same requests as
--spec-ngram 0(a rewrite of a C function, a story, a sampled list), the answers byte for byte those of plain decoding:Backend Plain With the draft model Trials won CPU 6.3, 9.4, 9.1 tokens/s 7.0, 10.5, 10.3 1 of 3 OpenCL 94.8, 113.8, 112.9 90.8, 109.9, 113.1 0 of 4 On the CPU, where a token's cost follows the weights read, the 0.6B draft pays (11 to 13 %); on the gfx1100 a small model's step costs about half of the 4B's (launches, not bandwidth), so the drafts never win and the trials cost 2 to 4 %. Before the draft's memory was taken out of the cache's default size, the two contexts overcommitted the device and the target ran 30 % slower.
LoRA adapters served beside the weights (
--lora-module, OpenCL, gfx1100, the short workload oftools/bench_serve.pywith--ignore-eos, 128 tokens a request, 2 a client, 2026-11-04), tokens/s;--model a,b,...sends request i to the (i mod n)-th name:Model, adapters (rank 8) Clients The model One adapter Two adapters mixed The model and two adapters mixed SmolLM2-135M F16, q and v 1 376 291 258 333 16 4289 3135 2977 3164 64 8989 6513 6619 6683 Qwen2.5-0.5B Q4_K_M, q, k, v and o 1 298 229 233 265 16 2911 2311 2184 2200 64 5164 4114 4129 4181 Mixing adapters costs nothing more than one: a batch of several runs multiplies each row by its own adapter in one gathered product per weight. Before that, each run of rows had its own products, and four adapters mixed fell to 482 tokens/s at 64 clients (8930 for the model). The cost is the adapters' launches, about two per adapted weight a step: an adapter of all seven projections of SmolLM2 keeps 46 % of the model's throughput (4110 tokens/s at 64 clients). Since the adapters of Q, K and V (and of gate and up) are one product where the device fuses those projections (ADR-0021), the same measure at 64 clients gives, for the same adapter shapes trained again: all seven 4177 to 5223 tokens/s, q and v 6984 to 7449, Qwen2.5-0.5B's q, k, v, o 4093 to 4251; the model alone 9061 and 5147. Batches mixing the model and an adapter vary twofold from one run to the next with how requests fall into batches (all seven: 3011 to 3750 tokens/s, the mean of three runs). A sequence with an adapter gives the merged model's logits within 2e-3 (the merged file rounds W + s B A to F16), a sequence without one the model's.
Sounds, Ultravox 1B (Llama-3.2-1B Q8_0, CPU),
jfk.wavasked about twice: the answer ofoklm describe; 10.8 s cold, 2.1 s again (224 of 228 rows cached, the sound's 188), of which the 40 tokens generated take 2 s: the first token comes in 0.07 s. With the model on a device the encoder's blocks are there too: a new sound and 48 tokens in 0.42 to 0.60 s (sharing the device).Grammar-forced drafts (
--spec-grammar), Qwen2.5-0.5B-Instruct Q4_K_M, CPU, 4 threads, the machine shared with other jobs (load 14 to 21 on 16 cores), prompt lookup on as by default, greedy requests sent six times each (the first prefills, the next five reuse the cached prompt); decode tokens a second from/api/profile, median of the five:request plain forced drafts same answer a required tool call (hermes, 4 arguments), 40 tokens 33.3 t/s 54.4 t/s (1.6x) yes a JSON schema (5 keys), 37 tokens 33.5 t/s 65.2 t/s (1.9x) yes 108 of the 192 forced tokens were kept (the key's closing quote often merges with what follows, and the schema's optional spaces cut the runs). Timed end to end with
curl(a sequence of runs at another moment) the gain disappeared into the machine's load, so these are the scheduler's own decode times.
Tests
/api/commands: test_auth_and_errors (401
without a key, 405 on a POST, the fields, /context
available and /rag, /model not, on a server
with neither an index nor a fixed context), the fixed-context server
(/context not available, with its reason) and the retrieval
server (/rag available) in
tests/server/test_server.c.
/api/activations: test_activations in
tests/server/test_server.c (401 without the key and 404
with oklm serve --observe in its message on a server that
does not record; recording with a lens of three tokens, on a tiny
mixture of experts: the lens of both blocks, the record before the last
read again by serial with its position one before, 404 for
a serial not kept and 400 for one that is not a number;
"serial":0 before a request, then the norms of both blocks,
three experts a block with their weights and the pieces, no row until a
block is asked for, block 1's rows adding up to 1 from the request
after; 400 for a block, a sequence or a list out of range; the record
read some 40 times while a request of 300 tokens decodes, each whole and
its serial never going back; the recording kept by a context made
again).
tests/server/test_server.c builds a tiny random-weight
Llama with its own trained tokenizer and talks to the server through
sockets:
Access. The access rules (loopback, TLS and keys elsewhere, certificate and key together), and keys (missing, wrong, a prefix of a key, the second of two).
Routes. 404, 405 and the invalid requests.
Answers.
- A chat answer whole, then streamed: the pieces add up to it, with
the same finish reason, then
[DONE]and the usage chunk. - The prompt taken from the cache the second time.
- Temperature 0 the same whole, streamed and repeated with the prompt
cached, with and without
--deterministic; with it the cached prefix stops on the--prefill-chunkgrid (test_determinism; the CPU's products do not depend on the piece, so it checks the guarantee and the grid, the GPU's rounding is in the measures above). - A stop string cutting the answer.
- Raw completions, whole and streamed.
- A reply continued (
test_continue_reply): the last message left open bycontinue_final_messageor byprefix, its prompt shorter than the closed turn's; the same request without either word answered as before; the word refused on a last message that is not the model's, on one with tool calls, and on a harmony server; on SmolLM2-135M, when its file is there, two greedy replies cut and continued equal to the replies written in one go, whole and streamed, the cut reply's rows read from the cache.
- A chat answer whole, then streamed: the pieces add up to it, with
the same finish reason, then
Grammars. A JSON schema's enum, and a required tool call, under their grammars, in the Hermes and the GLM-4.5 wire formats. With
--spec-grammara JSON schema (greedy and sampled) and a tool call answer as without, with and without prompt lookup, forced drafts proposed and accepted (spec_forcedin/api/stats).Anthropic Messages API. Keys as
x-api-key(withanthropic-versionandanthropic-betaheaders) or Bearer, a missing or wrong one refused withauthentication_error; a whole message and the same streamed, the events parsed and their order checked, the text the same; the system prompt as a string and as blocks giving the same answer;max_tokensand a stop sequence as stop reasons, whole and streamed; a forcedtool_choicegiving atool_useblock whose input is of the schema, whole and streamed (input_json_delta); the turn after, with the call and itstool_resultin the history; a prefill opening<think>answered with athinkingblock, whole and streamed; the thinking budget (budget_tokensmissing, under 1024 or not belowmax_tokensrefused; at 1024, the reasoning the first 1024 tokens of an uncapped answer's, then atextblock, whole and streamed alike, the closing tokens inoutput_tokens); the token count growing with the system prompt and the tool result, and equal to a message'sinput_tokens + cache_read_input_tokens; errors in Anthropic's shape (nomax_tokens, a bad role, not JSON, an image without--mmproj, an unknown tool named, 405, 404); the model list's Anthropic members. With the projector, animageblock gives the answer the same picture gives as an OpenAIimage_url.Model layout (
test_model_route)./api/modelneeds the key and refuses POST; itsmodelcarries the tiny model's architecture, parameters, vocabulary and block count, a block a layer with itsattention,dense_ffnandnormsroles whose bytes add up to the block's, the parts adding up to the file's tensors; every layer is placed, its bytes on either side adding up to its block's, the totals to the file's; the context's parts add up tohost_totaland its cache to the layers'; after a resize the layout is the same and the context's figures are the new ones;?tensors=1lists every tensor; with two adapters (test_adapters) both are named with their ids.Expert map (
test_experts). A dense model answers"moe":false; a tiny MoE model (the qwen3moe fixture, 6 experts, 3 a token, with the trained tokenizer) after two answers: the last step chose exactly 3 experts per layer, each layer's counts add up to 3 per token counted, an answer counts its generated tokens (less a last one cut by the limit), the cumulative counts add the answers up.Profiles (
test_profile). For whole and streamed answers, the phases add up to the request's time (within 0.01 ms), which the client's own time bounds; the token counts are the usage's; the decode steps' histogram holds every step kept and its percentiles are ordered; a decision's requests are profiled asscore; 405.Dashboard (
test_dashboard_panels). The page carries the expert map, the profiles and their endpoints.The dashboard's chat (
test_dashboard_chat). The page carries the list of commands, the row of attached files, the Attach button with its file input and the routes they call, the Record button with its level and its time, the transcription route, the row and the switch of the voice with the speech route asked forpcm, the Talk switch with the row of its state and its two settings, the microphone asked for with echo cancellation, the turn detector and the sentences said while a reply is written, the voice's name, and no outside resource (no worklet, no recorder of the browser and no<audio>element either); its table of how each command's arguments are read is the library's; the library's parser answers the lines oftests/server/dashboard_cases.tsvas the file says;/talkis the table's row ([on|off], a choice, for the window and the page, needing a recognizer and a voice). The page itself runs intools/check_dashboard.sh(The dashboard's chat).Decisions by scoring (
test_decide). A choice, a yes / no and a score on one state: each question's probabilities sum to 1, its label is the most probable, its confidence andp_yes/expectedfollow from them; the choice's probabilities equal, within 1e-4, those computed directly from the same rendered prompt (each option evaluated token by token on a fresh context, a log-softmax of the test's own), and so dooklm_decide_eval()'s (the chat's path); the options after the first find the shared prefix cached (cached_tokens> 0); the same request by the other route gives the same answers; keys as Bearer orx-api-key, a missing or wrong one 401; 400 in OpenAI's shape for a missing state, no or empty questions, empty or missing criteria, an unknown or missing type, a score without levels or with bad ones, two options of the same tokens, a bad description, a body not an object; 405; a JSON state.tests/model/test_decide.c:/decide's labels parsed and refused, the log-softmax, the probabilities, confidence and expected value of known scores, and the sum against the mean.Stress. 24 clients sending requests of random lengths at random moments (some sharing a prefix, some streamed, some sampled, a quarter hung up on) to a cache of 24 blocks for 8 sequences, with swapping: requests preempt and swap each other, the cache's bookkeeping checked after every step (
OKLM_KV_CHECK=1); at the end every block is free, every swap given back, every request answered or cancelled.Preemption. Four requests on a cache too small for them at once: the answers given alone, by recompute and by swapping (requests brought back from host memory).
Concurrency.
- Six requests at once on four sequences give the answers they gave one by one.
- A full queue gets 503.
- A stop in the middle of a stream ends both streams early, and the server returns.
Draft model. The tiny model drafting for itself: greedy and sampled answers those of plain decoding, one by one and at once, drafts proposed and accepted; a draft of another vocabulary refused; the sequences resolved by default (
oklm_server_parallel()).Images. A tiny random SigLIP projector of the model's width (
tests/mm/tiny_mmproj.h) and PPM pictures as base64: the same picture twice gives the same answer, its rows found cached and its embeddings reused; another picture finds none of them; the contiguous cache gives the same answer; a raw completion with two pictures inmultimodal_datagives the same text twice, cached the second time, and one marker for two pictures is refused; a remote URL, and media without--mmproj, are refused.Encoders on a device (
test_device_encoders, on OpenCL and on Vulkan when the machine has them). The image checks above and the rows that see each other, with the model and the encoder's blocks on the device; then a server whose image encoder, recognizer and synthesizer are on the model's device (vision_deviceand bothbackendmembers of/api/statssay so): a picture's answer is the one computed with the encoder on the CPU, the transcription the CPU's text, the sound the synthesizer's when called directly with that backend; sixteen requests at once (four new pictures, four sounds, four texts to speak, four completions) are each answered as alone. Then a server with a tiny audio encoder (tests/mm/tiny_audio_mmproj.h: a random Whisper encoder and Ultravox projector) on the model's device (audio_device): a sound's answer is the one computed with the encoder on the CPU, and four new sounds and four completions at once are each answered as alone. A backend the machine does not have stops--transcribe-backendand--speak-backendat start. A completion is streamed while six new pictures arrive one after another: all six are answered, and no two events of the stream are more than two seconds apart (a few milliseconds on the fixtures: the bound catches a stream that waits for as long as others keep coming, not a block's time). The device's pass lock itself is tested without a device (test_threads:test_pass_handoff,test_pass_stream,test_pass_devices).device_memoryholds a count for each of the three, and as pending only what the backend's own buffers have not reached of their reserve (the three were loaded at the start); a--kv-memthat leaves a lazily loaded recognizer no room, and a device said to have 1 MiB (OKLM_GPU_MEM_MB=1), are each refused at start by their message.test_small_deviceneeds no device: the tests' counting device (tests/mm/count_backend.h: the CPU's kernels behind a device's interface, a memory of a chosen size and a bound on what it gives), registered as a backend of its own. A recognizer loaded at the start on a device with room for 4 KiB is refused by its message and starts once the room is there; with the model there too and the recognizer left to its first request, the--kv-memthat leaves it no room and a device of 1 MiB are refused by theirs; a server on a device of 64 MiB answers a completion, its cache the blocks that fit less the recognizer's share,device_memoryholding that share as pending, andGET /api/contextand a dry run of the same length plan that cache.Uploads.
/api/ingest: text, JSON, HTML, a PDF and a DOCX sent as the body, each answering the text the library's reader gives for the file; the kind from the content, the name or the declared type; the cuts by characters (at a character) and by tokens (one character more is a token too many); the name without its directories; the refusals and the 413 one byte above--upload-max; the figures./v1/audio/transcriptionson a tiny random Whisper checkpoint the test writes: the body form and the multipart form answering the text ofoklm_whisper_transcribe()called directly,textandverbose_json, four requests at once, every refusal, the model listed by/v1/models, the figures; 404 without the model; and on Whisper base withjfk.wavwhen they are installed, the sentence exactly./v1/audio/speechon a tiny speech model the test writes (a language model whose every word is two audio codes, with the tiny vocoder of the speech tests and a speaker file): a WAV file, raw samples and a streamed WAV file holding the bytes ofoklm_speech_synthesize()called directly, over one pass and several; the headers; the options; the passes of a text under one prompt and under one each, a shorter first pass; a FLAC file, whole and streamed, decoded to the same samples;GET /v1/audio/voices, a voice of a voice directory spoken with, a file that is no speaker file not listed, a voice the model cannot use answered 400; every refusal and the 413 one byte above--speak-max; five requests at once, then twelve long ones (five spoken, the others 503); a client that hangs up counted ascancelledwith the synthesizer free again; a server stopped while it speaks;/v1/models, the figures,listenandspeakamong the features of/api/commands; 404 without the model, 401 without the key; and on OuteTTS with Whisper base when they are installed, a sentence spoken then heard again.
tests/harness/test_procreg.c checks the registry
oklm serve writes itself in: its directory, a command line
quoted and split back into its arguments (quotes, backslashes, an open
quote refused), a command started detached with both outputs in its log,
one started in a directory the caller names (and refused when the
directory is not there), its entry and its log readable by the user
only, the listing and its filter by kind, a log read from its tail and
followed, a process stopped by SIGTERM and one that ignores it killed
after the delay, the current process registered and removed, the entries
of a process that ended and of a pid used again removed, a started
process that registers itself keeping the log it was started with.