OKLMdocumentation v1.0.3
On this page

Serving design

How serving_spec.md maps onto OKLM: the module each phase extends, or where new code goes, the decisions the spec leaves to its S0, and the baseline its exit criteria compare with. What exists, with evidence, is the "Serving" section of gap-analysis.md; the cache layout is adr/0019-paged-kv-cache.md.

Status: proposed (S0, 2026-11-02); reviewed before S1.

Starting point

  • What exists. oklm_eval_batch() runs the new tokens of several sequences in one forward pass, equal to separate evaluations (bit for bit on the CPU). The server's scheduler (src/server/sched.c) is already continuous batching with chunked prefill: each step, decode tokens first, prompt pieces in what is left of n_batch, requests joining between steps, FCFS.
  • What bounds it. The context holds --parallel sequences of -c cells each, reserved and resident from the start. Nothing runs beyond --parallel; beyond --queue requests get 503; beyond 32 connections clients wait at the HTTP layer. A prefix is reused only by the request that lands on the sequence holding it; live requests never share one.
  • What costs time. Attention and the cache writes run once per sequence and layer, each a kernel on OpenCL; sampling runs per request on the host over a full vocabulary row; forward, read-back and sampling never overlap.

Phase by phase

Phase Existing Extended New
S1 Prerequisites, baseline oklm_eval_batch(), oklm_seq_tokens (model.h); check_sequences (test_model.c); oklm_eval() and llama.cpp as references oklm serve --workers N (the HTTP cap, already in oklm_server_params), by default --parallel + --queue; cancellation of non-streamed requests on disconnect (moved from S7); the batch API gains block tables only in S2 tools/bench_serve.py (written in S0), its runs recorded here and in server.md
S2 Paged KV cache k_cache / v_cache per layer, kv_type_ok, rotation, fit_layers (model.c); model_attend oklm_context_params: kv_block_size, kv_blocks (0: the contiguous cache); model_attend takes the paged path; oklm_seq_truncate frees blocks; oklm_seq_copy shares full blocks and copies the tail src/model/kv_pool.c (pool per layer and device, allocator, block tables, slot mapping); OKLM_OP_ATTENTION_PAGED in include/oklm/core/ops.h with a CPU implementation in src/core/ and an OpenCL kernel in kernels.cl; step A gathers into a scratch view and calls oklm_attention
S3 Scheduler sched.c: queue, slots, step, take_queued, finish The step keeps its order (decodes, then prompts); admission by free blocks; prompt chunks oldest request first, each capped (--prefill-chunk, default n_batch, moved from S4) src/server/engine.c replacing the slots: waiting and running lists, decode reservation, preemption by recompute (the newest running request goes back to the head of waiting), FCFS
S4 Chunked prefill fed per slot, the budget after decodes Its fairness rule done in S3; after S5, the 30k-token prompt measured against TPOT Nothing else: S4 is mostly done
S5 Prefix caching assign's per-sequence common prefix; cached_tokens assign becomes a lookup of full blocks Block hashes and the hash -> block map in kv_pool.c (ADR-0019), the LRU of refcount-0 blocks, the copy-on-write rule
S6 Model runner the OpenCL backend's in-order queue, att_part, tiled attention; oklm_sampler One paged attention launch per layer for every sequence; one scatter kernel per layer for the new keys and values; the step's metadata (positions, slot mapping, block tables, logits rows) uploaded as one buffer oklm_sampler_batch (temperature, top-k, top-p, min-p, penalties per row, greedy rows reduced on the device to an argmax); the next step's metadata built while the device runs (decision 1)
S7 Serving layer one scheduler thread, per-request text and condition variable, OpenAI routes, /api/stats Cancellation of non-streamed requests done in S1; /api/stats gains KV usage, prefix hit rate, preemptions, queue time; the dashboard shows them Nothing new: S7 is mostly done
S8 Extensions grammars per request; oklm_speculative (one sequence); quantized caches; layer split over devices Grammars unchanged; quantized and split caches through ADR-0019 Speculative decoding in the engine (draft tokens in private tail blocks, rolled back by truncation); LoRA stays out of scope

Decisions

1. Backends

CPU and OpenCL, nothing else.

  • What replaces CUDA graphs. OpenCL has no portable graph capture; the cost to remove is the count of enqueues and argument sets per step, which today grows with the sequences (three per sequence and layer).
    • The paged attention kernel takes every sequence of the step at once (block tables, lengths, query offsets): one launch per layer, decode and prefill rows alike.
    • The new keys and values go to their slots in one scatter launch per layer.
    • Kernel arguments that do not change between steps are set once; the per-step metadata lives in one device buffer written by one upload.
    • Greedy rows are reduced to an argmax on the device, so only one integer per sequence comes back; sampled rows still come back whole.
    • The CPU needs none of this: its operations are calls, and the worker team (ADR-0016) runs the paged attention over (sequence, head) pairs.
  • The memory budget at start. No forward pass needs profiling: activation buffers are sized from n_batch exactly (model.c:1822).
    • OpenCL: the pool takes what fit_layers leaves after weights, activations and its margin (8 %, at least 256 MiB), measured against CL_DEVICE_GLOBAL_FREE_MEMORY_AMD when the device has cl_amd_device_attribute_query, else against the total (OKLM_GPU_MEM_MB still overrides, as the GPU is often shared).
    • CPU: --kv-mem SIZE; by default the memory today's default server takes for its cache (4 sequences × -c cells: 192 MiB for Qwen2.5-0.5B at -c 4096), not tied to the new --parallel, so an upgrade keeps its footprint and gains concurrency (a short request uses about 4 % of the cells it reserves today).
    • Several devices: one pool per device for its layers, every pool with the same block count (block ids are global); the smallest device sets the count.

2. Determinism

  • CPU. Batched output equals the reference token for token and the logits bit for bit, whatever the batch composition, chunking, preemption or cache hits. This holds today for composition and chunking (check_sequences); the paged attention keeps it by visiting keys in position order with the reduction the contiguous kernel uses, so block boundaries change no sum. Cache hits are exact because a row's keys and values do not depend on the batch it was computed in (CPU), and a quantized row is quantized the same way wherever it is written.
  • OpenCL. Kernels and reduction orders are chosen by batch size (tiled attention from 16 or 32 queries, matrix kernels by width), so a prefix computed in one batch shape and reused in another may differ in the last bits. The rule is M3's against llama.cpp: identical greedy output, except where the reference's top-2 margin is a near tie (0.15 at most); the tests print and allow those positions.

3. Model features with a paged cache

The contiguous cache stays as the reference and as the fallback; a context created with kv_blocks = 0 behaves exactly as today.

Feature With the paged cache Fallback
Sliding windows (Gemma 2, 3) Unchanged: every layer keeps every cell today, the window is applied in attention (ap.window). One block table serves all layers. None needed. Freeing blocks outside the window would need one table per layer group (Gemma 3's 5:1 local layers): later, if memory asks for it.
Latent attention (DeepSeek V2, V3) A key pool only, one row of kv_lora + n_rot per token; the values read the first kv_lora values of the key rows, as now. Step A (gather) until the paged kernel takes the latent layout.
Quantized and rotated caches Rows are per token and quantized along head_dim, so any block size holds whole quantization blocks (the spec's multiple of 32 is not needed). Rotation is applied before the write, unchanged. None needed.
Several OpenCL devices Each device's pool holds its layers' blocks under the same ids; the block tables are uploaded to each device per step. None needed.
Weight streaming (--gpu-stream) Its device mirrors become a mirror pool under the same ids. Contiguous cache in S2 (the two cannot be combined at first; a clear error says so).
Speculative decoding, MTP Draft tokens go to the sequence's private tail blocks; a rejection truncates, freeing blocks past the end. The draft model has its own context and pool. The MTP context (mtp) keeps its contiguous cache. In the server since S8: prompt lookup and a draft model; MTP only in oklm run.
Multimodal prompts Embedding rows enter through oklm_seq_tokens.embd as now, with media their content hashes. A block holding them hashes the media's content hash and the row index, not token ids (ADR-0019). In the server since S8 (--mmproj).
Grammars Per request, on the sampling side: unaffected by blocks or prefix hits. None needed.
GPT-2 Learned positions, plain attention: paged as the Llama family. None needed.

4. Where the code goes

  • Model library (src/model/). kv_pool.c with the pool per layer and device, the allocator, block tables, slot mapping, block hashes, the hash -> block map and the LRU, behind oklm_context_params (kv_block_size, kv_blocks). The map lives next to the allocator, not in the engine as the spec's decision 4 says: reclaiming a cached block must drop its hash in the same operation, and the CLI's own oklm chat gains prefix reuse for free. The engine decides what to look up and when.
  • Public API. oklm_eval_batch() keeps its signature; sequences become cheap (metadata only), so n_seq grows to the batch width the engine wants (default 64) without memory. Added: block-level queries (free blocks, a sequence's blocks), oklm_seq_lookup_prefix() (attach every cached full block of a token prefix, return how many tokens it covers), statistics (used, cached, free blocks, hits).
  • Core. OKLM_OP_ATTENTION_PAGED beside OKLM_OP_ATTENTION: CPU in src/core/ops.c, OpenCL in kernels.cl and backend_cl.c.
  • Server (src/server/). engine.c replaces sched.c's fixed slots with the queues, the step, admission by blocks and preemption. server.c keeps its routes; --parallel becomes the most requests running at once (default 64 on OpenCL, 16 on the CPU, where throughput stops growing at about 16), --queue the most waiting (default 256; 503 only beyond it), --workers the connections (default --parallel
    • --queue, 512 at most, threads with small stacks: a worker holds its connection while the request waits, so fewer workers than that leaves queued clients stuck at connect).
  • Tools. tools/bench_serve.py (standard library only).
  • ADR-0019 records the layout, block size, hash and eviction before S2 starts.

5. Benchmarks

  • Machine. i7-11700K (8 cores, 16 threads, 128 GB), gfx1100 under OpenCL, as every other figure. The GPU is shared with other work; disturbed runs are rerun and marked.
  • Models. Qwen2.5-0.5B, Qwen3-4B and Llama-3.2-1B, Q4_K_M.
  • Workloads, all through tools/bench_serve.py, greedy, streamed:
    • short: distinct questions of about 30 to 50 prompt tokens, 128 output tokens at most, at 1, 4, 16 and 64 clients;
    • agent: a shared system prompt (1100 words, 1319 Qwen tokens) and a distinct question, levels 1, 16 (cold), 16 (warm), 64 on a fresh server; later the real agent's system prompt and tool definitions with a growing conversation (oklm agent --url);
    • long prompt: 30k tokens arriving mid-generation (S4), Qwen3-4B with -c 40960.
  • Reference. llama.cpp's llama-server at the pinned test checkout, same models and workloads; built for the CPU only (the checkout's configuration), so OpenCL figures have no llama.cpp counterpart.
  • Recorded here per phase, and the summary in server.md.

6. Tests

  • tests/server/test_server.c and the model tests keep passing at every phase; every new test runs under ASan and TSan like the rest, and under Wine.
  • tests/model/test_kv_pool.c (new): the allocator invariants (refcounts never negative, no block both free and in a table, the free count back to its start), the hash map and LRU, copy-on-write, and the spec's stress test (random arrivals, lengths and cancellations on a tiny pool forcing constant preemption).
  • check_sequences gains a paged variant: the paged context against the contiguous one, bit for bit on the CPU, within 1e-5 on OpenCL, with block boundaries inside prompts, shared prefixes and truncation.
  • test_server.c gains: preemption (a tiny pool, greedy answers equal to one by one), a prefix hit across live requests (cached_tokens), blocks returned within one step after a cancel (/api/stats), 64 clients with no 503.

Baseline

Measured 2026-11-02 on the i7-11700K / gfx1100 machine, OKLM at b81afc0 built with make -j16 (release), oklm serve -c 4096 (F16 cache, n_batch 512) with --backend cpu or --backend opencl; --parallel 4 --queue 16 (the defaults) or --parallel 16 --queue 64. tools/bench_serve.py --retry, greedy (temperature 0), streamed, one warm-up request, all requests answered. TTFT counts the time in the queue and the waits after 503 answers; TPOT is per request, then the median. Another session's builds and tests ran on the CPU during the runs (load 4 to 8): CPU figures vary by about 15 % between reruns, both runs shown where they differ.

Short prompts

About 48 prompt tokens (Qwen2.5), 26 (Qwen3), 53 (Llama), 128 output tokens at most (64 for Qwen3-4B on the CPU); two requests per client for Qwen2.5-0.5B, one otherwise.

Model, backend --parallel Clients tokens/s TTFT p50 TTFT p90 TPOT p50 503 retries
Qwen2.5-0.5B, CPU 4 1 62.1 0.03 s 0.03 s 16.1 ms 0
4 4 151.3 0.05 s 0.13 s 25.9 ms 0
4 16 149.9 10.1 s 10.6 s 26.0 ms 0
4 64 152.2 (rerun 132.1) 33.7 s 60.7 s 26.0 ms 1030 (1209)
16 1 60.8 0.04 s 0.04 s 16.4 ms 0
16 4 149.6 0.04 s 0.13 s 26.3 ms 0
16 16 376.4 (rerun 328.6) 0.20 s 0.36 s 41.2 ms 0
16 64 387.2 (rerun 345.8) 16.0 s 16.3 s 39.6 ms 0
64 16 377.6 0.18 s 0.53 s 40.8 ms 0
64 64 504.0 8.0 s 9.0 s 60.3 ms 0
Qwen2.5-0.5B, OpenCL 4 1 216.2 0.01 s 0.01 s 4.7 ms 0
4 4 488.4 0.02 s 0.03 s 7.4 ms 0
4 16 570.9 2.7 s 2.7 s 6.9 ms 0
4 64 570.2 7.9 s 18.8 s 6.9 ms 277
16 1 269.3 0.01 s 0.01 s 3.7 ms 0
16 4 560.9 0.01 s 0.03 s 7.0 ms 0
16 16 754.2 0.04 s 0.06 s 21.0 ms 0
16 64 718.6 8.4 s 8.8 s 21.7 ms 0
Qwen3-4B, CPU 4 1 11.0 0.10 s 0.10 s 90.5 ms 0
4 4 30.2 0.55 s 0.56 s 125.9 ms 0
4 16 33.2 8.0 s 23.5 s 115.9 ms 0
16 16 72.8 2.9 s 2.9 s 177.2 ms 0
16 64 84.6 13.2 s 38.2 s 166.4 ms 0
Qwen3-4B, OpenCL 4 1 110.4 0.01 s 0.01 s 9.0 ms 0
4 4 245.6 0.10 s 0.10 s 15.6 ms 0
4 16 242.7 2.1 s 6.4 s 16.4 ms 0
4 64 240.8 14.9 s 29.7 s 16.1 ms 270
16 1 111.2 0.01 s 0.01 s 9.0 ms 0
16 4 238.1 0.10 s 0.10 s 16.1 ms 0
16 16 335.5 0.23 s 0.24 s 46.2 ms 0
16 64 326.6 6.4 s 18.9 s 48.1 ms 0
Llama-3.2-1B, OpenCL 16 1 281.0 0.00 s 0.00 s 3.5 ms 0
16 4 596.5 0.04 s 0.04 s 6.4 ms 0
16 16 784.2 0.10 s 0.14 s 19.7 ms 0
16 64 792.0 2.7 s 7.9 s 19.7 ms 0

What it shows:

  • Throughput stops at --parallel: 4 sequences give the same tokens/s at 4, 16 and 64 clients, the rest waits (TTFT of seconds to a minute).
  • With the default --queue 16, 64 clients get 503 answers by the hundred (and, once, a reset connection on the CPU).
  • --parallel 64 reaches 64 clients only up to the 32 HTTP workers: TTFT p50 of 8 s is the second half waiting for a connection.
  • On OpenCL, TPOT triples from 4 to 16 sequences (7 to 21 ms at 0.5B, 16 to 46 ms at 4B) for 1.3 to 1.4 times the throughput: the per-sequence attention launches (three per sequence and layer) dominate the step.

After S1

Measured 2026-11-03 at bc73f62 plus the S1 changes, Qwen2.5-0.5B Q4_K_M on the CPU, --parallel 64 (queue 16, so 88 workers), the short workload as above (load 6 to 7 from other work):

Clients tokens/s TTFT p50 TTFT p90 TPOT p50 503 retries Baseline (32 workers)
16 385.4 0.17 s 0.50 s 39.9 ms 0 377.6, 0.18 s, 0.53 s, 40.8 ms
64 535.1 0.40 s 1.95 s 113.1 ms 0 504.0, 8.0 s, 9.0 s, 60.3 ms

All 64 clients now run at once: the TTFT median falls from 8 s to 0.4 s and TPOT doubles (64 sequences in a step instead of 32). Two faults only 64 concurrent requests reached were fixed on the way: a burst larger than --queue was refused although every sequence was free (the queue did not count the free sequences), and the forward pass ran out of view headers with that many sequences in one batch (22 of 128 requests failed before).

S2, step A

The paged cache with step A (2026-11-03), Qwen2.5-0.5B Q4_K_M on OpenCL, --parallel 16 --queue 64, the short workload:

Cache Clients tokens/s TTFT p50 TPOT p50
contiguous 1 248.7 0.01 s 4.1 ms
16 726.5 0.04 s 21.8 ms
64 726.9 8.36 s 21.7 ms
paged (--kv-mem 0) 1 155.1 0.01 s 6.9 ms
16 381.2 0.06 s 40.8 ms
64 368.1 16.36 s 43.0 ms

The answers are identical, the memory follows the tokens (16 sequences on the CPU: 1478 MiB against 2052), but gathering costs half the throughput: once sequences interleave in the pool (or one reuses a prefix), every token copies its sequence's rows again in every layer. Step B's kernel reads the blocks in place; until then the server keeps the contiguous cache unless --kv-mem is given.

S2, step B

Attention through the block table (2026-11-03), same workload, run back to back:

Cache Clients tokens/s TTFT p50 TPOT p50
OpenCL contiguous 1 273.2 0.01 s 3.7 ms
16 775.9 0.04 s 20.4 ms
64 782.5 7.87 s 20.3 ms
OpenCL paged 1 274.8 0.01 s 3.6 ms
16 745.9 0.04 s 21.2 ms
64 749.5 8.23 s 21.2 ms
CPU contiguous 16 385.4 0.17 s 40.1 ms
CPU paged 16 367.7 0.16 s 41.6 ms

The paged attention launch costs 15.5 µs against 13.5 (the profiler), the rest of the step is unchanged. A first run lost a layer to the CPU: the default pool filled the device, and the layer fit counts every device layer's cache as one buffer within the largest allocation; the pool is now sized under that limit. With step B the server takes the paged cache by default.

After S3

The engine with its defaults (2026-11-03: oklm serve -c 4096, the paged cache, 64 sequences on OpenCL, 16 on the CPU, queue 256), the short workload. Another user's jobs held about 13 cores during the CPU run: its 16-client figure measured 369 to 380 tokens/s alone.

Backend Clients tokens/s TTFT p50 TTFT p90 TPOT p50 503 Baseline (--parallel 4)
OpenCL 1 270.6 0.01 s 0.01 s 3.7 ms 0 216.2
4 556.0 0.01 s 0.03 s 7.1 ms 0 488.4
16 750.4 0.04 s 0.06 s 21.0 ms 0 570.9, TTFT 2.7 s
64 745.5 0.13 s 0.41 s 83.5 ms 0 570.2, TTFT 7.9 s, 277 retries
CPU 16 235.0 (disturbed) 0.21 s 0.38 s 41.8 ms 0 149.9, TTFT 10.1 s
64 365.8 (disturbed) 16.42 s 17.52 s 40.9 ms 0 152.2, TTFT 33.7 s, 1030 retries

Every client is served at once without a refusal and the time to the first token falls from seconds to a tenth of one, but the throughput at 16 and 64 clients is 1.3 times the baseline on OpenCL (2.4 on the CPU), not the "several times" S3 aims at: on OpenCL each step runs one attention launch per sequence and layer, and TPOT grows with the sequences (84 ms at 64). That is decision 1's S6 work (one paged launch per layer for every sequence).

Shared system prompt (agent)

A 1319-token system prompt (1100 words of fixed text), a distinct question, 64 output tokens at most, one request per client, a fresh server with --parallel 16 --queue 64, levels run in this order.

Model, backend Level tokens/s TTFT p50 TTFT p90 TPOT p50 Cached tokens, mean
Qwen2.5-0.5B, CPU 1 client 28.4 1.13 s 1.13 s 17.9 ms 0
16, cold 42.1 9.3 s 19.3 s 212.1 ms 82
16, warm 257.4 0.08 s 0.08 s 61.8 ms 1318
64 242.7 4.4 s 13.0 s 61.5 ms 1303
Qwen2.5-0.5B, OpenCL 1 client 151.6 0.16 s 0.16 s 4.2 ms 0
16, cold 268.3 1.07 s 2.26 s 38.3 ms 81
16, warm 491.4 0.07 s 0.13 s 31.6 ms 1318
64 526.1 2.0 s 5.9 s 29.8 ms 1303
Qwen3-4B, OpenCL 1 client 34.6 1.16 s 1.16 s 10.9 ms 0
16, cold 47.6 8.5 s 17.0 s 184.9 ms 81
16, warm 219.1 0.08 s 0.08 s 72.8 ms 1318
64 203.4 5.2 s 15.5 s 74.6 ms 1303

What it shows: the 16 cold clients compute the same 1319 tokens 15 more times (each server evaluated about 22,200 prompt tokens and found 105,800 in its caches over 97 requests); once each of the 16 sequences holds the prompt, TTFT drops to 0.08 s. A paged cache with shared prefix blocks makes the cold level cost one prefill instead of sixteen, and needs one copy of the prompt in memory instead of sixteen.

After S5

The same agent workload (2026-11-03), --parallel 16 --queue 64, a fresh server, the prefix cache shared by the requests. The tool's warm-up request now seeds the cache, so the first level starts warm.

Backend Level tokens/s TTFT p50 TTFT p90 TPOT p50 Cached tokens, mean Baseline
OpenCL 1 client 246.4 0.01 s 0.01 s 4.0 ms 1318 151.6, cold
16, cold 635.5 0.08 s 0.08 s 24.3 ms 1297 268.3, TTFT 1.07 s, 81 cached
16, warm 654.9 0.04 s 0.04 s 24.1 ms 1318 491.4
64 641.8 1.64 s 4.84 s 24.3 ms 1303 526.1, TTFT 2.0 s
CPU 1 client 55.8 0.02 s 0.02 s 17.9 ms 1318 28.4, cold
16, cold 256.4 0.47 s 0.47 s 55.9 ms 1297 42.1, TTFT 9.3 s, 82 cached
16, warm 288.4 0.08 s 0.08 s 55.0 ms 1318 257.4
64 265.9 3.99 s 11.90 s 55.4 ms 1303 242.7, TTFT 4.4 s

The cold level now costs one prefill instead of sixteen: 2.4 times the throughput on OpenCL, 6 on the CPU, and the time to the first token falls from 1 and 9 seconds to 0.08 and 0.47. Requests arriving together would each compute the prompt before any of its blocks is full; the queue's head waits instead when a running request is computing blocks it shares (a few steps of wait against a whole prefill).

S4: a long prompt among streams

Qwen3-4B Q4_K_M on OpenCL, -c 40960, the paged cache (2026-11-03): four streams generating 700 tokens, and after 4 s a request of 38,872 prompt tokens. The streams' gaps between tokens:

Prompt pieces Before During its prefill Its answer
fixed, 512 tokens median 17.6 ms median 3496 ms, max 7.6 s 306 s
sized to one decode step (--prefill-share 1, at least 16 tokens) 17.7 ms median 86.7 ms, p90 122, max 402 327 s
the same, at least 4 tokens (the default) 17.6 ms median 40.4 ms, p90 43.5, max 403 318 s

The paged and contiguous caches read such a prompt at the same speed (11,664 tokens in 31 s either way); it is the attention over tens of thousands of keys that costs, about 8 ms a token at 30k. Sizing the pieces keeps the streams moving at about twice their speed instead of stopping for seconds, the prompt still being read at full speed once they end; flat would need that attention much faster (S6).

S6, first part: fewer launches

One write of the new keys and one of the values per layer (set_rows), one attention launch for every sequence's decode token (per-token positions and tables), the step's metadata uploaded once (2026-11-03, Qwen2.5-0.5B on OpenCL, the defaults, the short workload):

Clients tokens/s TTFT p50 TPOT p50 After S3 Baseline (--parallel 4)
1 265.1 0.01 s 3.8 ms 270.6 216.2
4 718.7 0.01 s 5.5 ms 556.0 488.4
16 1078.4 0.05 s 14.6 ms 750.4 570.9
64 1354.3 0.10 s 46.6 ms 745.5 570.2, TTFT 7.9 s

1.9 to 2.4 times the baseline at 16 and 64 clients. A first run held 64 short prompts arriving together behind --prefill-share (TTFT 4.9 s): the hold now applies to long prompts only (more than one piece left), the short ones going whole. The S4 case after it: the streams at 13.2 ms a token, 29.5 during the 38,872-token prefill.

S6, second part: sampling

Timing the scheduler's steps at 64 clients showed 23 ms of forward and 24 ms of sampling a step: each greedy request copied its 151,936 logits and scanned them in a loop the compiler cannot vectorize. A greedy request with no penalty now reads them in place with eight running maxima (the same token, the first of the largest):

Backend Clients tokens/s TTFT p50 TPOT p50 Before Baseline (--parallel 4)
OpenCL 1 310.6 0.01 s 3.2 ms 265.1 216.2
4 989.6 0.02 s 3.9 ms 718.7 488.4
16 2205.6 0.04 s 7.0 ms 1078.4 570.9
64 3340.2 0.08 s 18.5 ms 1354.3 570.2
CPU 16 345.4 0.41 s 44.6 ms 149.9
64 410.9 14.69 s 36.5 ms 152.2

On OpenCL 3.9 times the baseline at 16 clients and 5.9 at 64, the several times S3 asks for; the CPU, bound by its arithmetic at 16 sequences, 2.3 and 2.7 (the machine had other work, load 4 to 5).

S6, third part: the argmax on the device

A greedy request (no penalty, no grammar) marks its item argmax_only: the best token of every such row is computed on the device in one launch and only the ids are read back, not 151,936 floats each (2026-11-03, the same runs):

Clients tokens/s TTFT p50 TPOT p50 Before Baseline (--parallel 4)
1 308.1 0.01 s 3.2 ms 310.6 216.2
4 1069.1 0.01 s 3.6 ms 989.6 488.4
16 2901.0 0.04 s 5.2 ms 2205.6 570.9
64 5125.6 0.08 s 11.9 ms 3340.2 570.2

5.1 times the baseline at 16 clients, 9.0 at 64; the answers are those of the contiguous cache.

S6, last part: candidates on the device, and what is left

A sampler that draws (temperature above 0) needs only its top_k best logits, plus the tokens a penalty may lower: those are found on the device for every row in one launch (an exact radix selection, ties to the lower token as on the CPU) and read back, 2 x k numbers instead of 151,936; the host draws from them the token it would draw from the whole row. Temperature 0.8, the same runs:

Clients tokens/s TPOT p50 Before
1 270.5 3.7 ms 295.6
16 2654.3 5.7 ms 1716.9
64 4917.4 12.4 ms 2250.9

The rest of S6, measured rather than built:

  • Launch overhead: at one client a token takes 2.55 ms of kernels in 3.2 ms (the profiler's kernel times), at four clients 3.4 ms in 3.6 to 3.9: decode steps are dominated by compute, the exit criterion; no graph equivalent.
  • Overlap: at 64 clients the host works 0.22 ms between steps of 11 ms (2 %), the kernels running as they are queued; pipelining steps would win at most that.

S7: the serving layer

Most of it was there (one loop owning the scheduler and the device, per-request channels, the OpenAI routes, oklm agent --url). Added: /api/stats and the dashboard give the KV cache's use, the prefix hit rate, the mean wait for a sequence and the preemptions; with the paged cache a finished or abandoned request gives its blocks back at once, its full blocks staying as cached prefixes, so the use shown is the running requests'. The agent workload after S6 and S7 (OpenCL, --parallel 16 --queue 64): 16 cold clients 1628 tokens/s (636 after S5), 64 clients 1656 (642), a hit rate of 97.6 %.

S8: speculative decoding by prompt lookup

A greedy request proposes up to --spec-ngram (4) tokens that followed its last tokens earlier in its own text, verified in the same step: the device gives the best token after each, the drafts that match are kept with the model's own next token, the cache cut past them. The answers are plain decoding's (2026-11-03, Qwen2.5-0.5B on OpenCL):

No drafts 4 drafts 8 drafts
Rewriting a 3000-character C function, one client 253 tokens/s 569 668
Short workload, 4 clients 1032 1002 964
Short workload, 16 clients 2889 2891 2831
Short workload, 64 clients 5059 5049 5043

Unguarded, drafts cost a third of the throughput at 16 clients (2852 to 1806 tokens/s): they are proposed only while at most four requests decode, and a request keeps proposing while it gains half a token a step. The same rewrite with and without, byte for byte.

S8: images and audio through the server

With --mmproj, a chat message's image_url, video_url and input_audio parts (base64 only) become markers in the rendered conversation, then rows: oklm_mm_build() lays each item out as oklm describe evaluates it (tiles between their separators, a sound's embeddings) and keys each media row by its item's content hash (decoded pixels and slicing side, or samples) and its index. The encoders run in the request's thread as it arrives, one request at a time (on the model's device when it has one, each image a pass the scheduler's steps wait for: server.md); the embeddings of recent items are kept by the same hash, so a picture sent again skips its encoder too. The scheduler cuts prompt pieces at the edges of an item and gives its embeddings in place of the tokens'; the pool hashes and compares the media keys with the tokens (ADR-0019's marker), so a cached prefix holds a picture's blocks and only that picture finds them. SmolVLM-256M on the newspaper picture (2026-11-03), 40 tokens at most:

OpenCL CPU Rows cached
Cold 4.43 s 4.24 s 0 of 349
The same request again 0.13 s 0.21 s 336 of 349
A second turn resending the picture 0.11 s 0.18 s 336 of 377

The answers are oklm describe's; the rows fed in pieces give oklm_mm_eval()'s logits exactly, and a sequence attached to cached media blocks those of one sequence alone, on the CPU and OpenCL. Ultravox on jfk.wav (CPU): 10.8 s cold, the first token 0.07 s when sent again.

S8: a draft model in the server, sampled requests

--draft MODEL gives the server a second context of the draft model, paged, a sequence per slot, sized for five requests drafting at once. A slot's draft sequence is brought to the slot's tokens only when it drafts (its common prefix kept, the rest fed in the first round with the pending token), then a greedy token a round, every drafting request in one batch a round; after verification it keeps what the request kept. Verification is now one loop for every request: row by row, the token the request takes there (the argmax, or its sampler's draw from that row's logits, one draw a token as plain decoding makes) while it is the draft's, so sampled requests draft too, by prompt lookup or by the draft model, with plain decoding's answers.

Two findings. The paged cache's default size took the device's memory less the target's weights: with a draft model beside it the device was overcommitted and the target ran 30 % slower, drafting or not; oklm_context_device_bytes() now takes the draft's share out. And a small model's step on the gfx1100 costs about half of a 4B's (launches, not bandwidth), so drafts lose there; the server measures the tokens a request gets a second with drafts against plain decode steps over 16 steps, and rests the draft model 256 steps when it lost:

Qwen3-4B, a Qwen3-0.6B draft Plain Draft model Trials won
CPU (rewrite, story, sampled) 6.3, 9.4, 9.1 tokens/s 7.0, 10.5, 10.3 1 of 3
OpenCL 94.8, 113.8, 112.9 90.8, 109.9, 113.1 0 of 4

M14's partials

Prompt attention (S4). tests/model/bench_prefill.c times a prompt piece deep in a context (Qwen3-4B Q4_K_M, OpenCL, the paged cache, milliseconds for the piece, 2026-11-04):

Depth Piece Before Head groups Half tiles, registers
1,024 16 32.4 27.9 22.3
1,024 512 536.8 534.1 401.3
16,384 4 85.4 67.8 31.1
16,384 16 215.4 118.9 61.6
16,384 64 416.0 414.5 210.4
16,384 512 3379.8 3356.8 1583.9

Three changes. A tiled block's rows are now the queries of the query heads that read one KV head, so a small piece reads each key once for its heads (before: once per query head, and pieces under 16 tokens took the one-work-group-per-head-and-token kernel). F16 caches get their own kernel whose tiles are halves in local memory (about two thirds of the float kernel's, so more work-groups share a compute unit). And the output accumulators loop to a compile-time bound: indexed by a runtime one they lived in scratch memory, which made the value pass cost more than the scores. Also measured and kept: the key tile written without bank conflicts (2 to 6 %), and the next tile loaded while one is used (no measurable change on its own). Measured and dropped: the query row in registers (2.5 times slower: spilled), the value tile as floats (slower: occupancy), the tiled kernel for single decode tokens (no faster: 4 of 32 rows used). A decode token at 16k still costs 27 ms of attention, once per query head; a decode kernel sharing the KV head's reads is left for later.

Piece sizing (S4). A piece's cost is fitted, by recursive least squares with forgetting over the steps, as a x tokens + b x rows x (position + tokens / 2), rows its tokens and at least 8 (a tiled block), so a piece is sized at its own depth; until fitted, pieces of 16 tokens. The long prompt's one stall (389 ms at its start, its first piece sized from short prompts' costs) is gone; the streams' median gap during it stays about twice their own (tools/bench_stall.py, see server.md): the floor is the attention of the smallest piece over every key before it, about a decode step at 30k keys.

Free memory (S2). The pool is sized within what the device says is free (AMD's query), so other programs and models count; the context's buffers are allocated up front, so the rest is computed, not sampled.

Sequences (S3). By default the batch's rows or the pool's blocks, the fewer; a step uploads only its own sequences' block tables. 256 clients of the short workload: 5203 to 7265 tokens/s, TTFT p50 4.57 to 0.22 s.

Swapping (S3). --swap-mem: a preempted request of 64 positions or more moves its rows to host memory and brings back, running again, what the prefix cache no longer holds; the same answers as by recompute.

Testing strategy. oklm_kv_check() checks the pool's invariants; the server runs it after every step with OKLM_KV_CHECK=1, under a stress test of random arrivals, lengths, shared prefixes and hang-ups on 24 blocks with swapping.

LoRA multiplexing (S8.5). Adapters served beside the weights (ADR-0021): each sequence has one adapter or none, set while it is empty. Every adapted projection adds s B (A x) to its rows, from per-weight stacks of every adapter (slot 0 zeros). One adapter's run of rows multiplies its slices, and a mixed batch multiplies each row by its own slot in one gathered product. The paged cache salts each block's hash with its adapter, and the scheduler reuses rows or waits for a prefix only between requests of the same adapter.

llama.cpp's server

llama-server at the pinned checkout (commit 8642541, CPU build, GGML_NATIVE, 8 threads by default), -np 16 -c 65536 (4096 per slot, no unified cache), the same tool and workloads. A first short-prompt run fell during a build burst on the machine (210 tokens/s at 16 clients) and was rerun:

Workload Level tokens/s TTFT p50 TTFT p90 TPOT p50
Short, Qwen2.5-0.5B 1 68.4 0.03 s 0.03 s 14.5 ms
4 223.0 0.10 s 0.11 s 17.2 ms
16 388.5 0.42 s 0.53 s 37.6 ms
64 398.6 15.8 s 16.3 s 36.6 ms
Agent, Qwen2.5-0.5B 1 25.8 1.44 s 1.44 s 16.5 ms
16, cold 31.2 16.3 s 29.2 s 217.6 ms
16, warm 263.3 0.11 s 0.12 s 59.8 ms
64 213.6 5.5 s 15.3 s 62.2 ms

At this configuration llama.cpp shows the same shape: per-slot prefix reuse, no sharing between live slots, a cold level that recomputes the shared prompt per slot. OKLM's CPU server is close behind at 16 and 64 clients (329 to 376 against 389 tokens/s at 16), further behind at 4 (150 against 223 tokens/s: TPOT 26 against 17 ms), and ahead on the cold agent level (42 against 31 tokens/s).

Memory per request

Today a request holds -c cells of every layer whatever it uses, F16 keys and values, reserved at start:

Model Per token Per request (-c 4096) Used, short (about 176 tokens) Used, agent (about 1384 tokens)
Qwen2.5-0.5B (24 layers, 2 × 64 KV) 12 KiB 48 MiB 2.1 MiB (4 %) 16 MiB (34 %)
Llama-3.2-1B (16 layers, 8 × 64 KV) 32 KiB 128 MiB 5.5 MiB (4 %) 43 MiB (34 %)
Qwen3-4B (36 layers, 8 × 128 KV) 144 KiB 576 MiB 22 MiB (4 %) 195 MiB (34 %)

The reservation is resident: on the CPU the server's RSS is 1.50 GB at --parallel 4, 2.09 GB at 16 and 4.45 GB at 64 (49 MiB a sequence, Qwen2.5-0.5B); the rest is the weights, the activations and the host logits buffer (n_batch × n_vocab floats, 297 MiB for Qwen's vocabulary). On OpenCL the cache is in device memory (192 MiB, 768 MiB; 2.3 GiB and 9 GiB for Qwen3-4B at 4 and 16 sequences) and the host RSS stays near 0.65 GB for Qwen2.5-0.5B and 2.7 GB for Qwen3-4B. With 16-token blocks the same 768 MiB holds 65,536 tokens: about 370 short requests (11 blocks each), or about 800 agent requests sharing the prompt's 82 full blocks with 5 private blocks each, instead of 16 requests of either kind.

Decisions taken

Reviewed and agreed on 2026-11-02, beyond the six above; ADR-0019 is accepted with the amendments they make.

  1. The prefix map in the model library (decision 4), next to the allocator. The engine sees a narrow interface: one call attaching the cached full blocks of a prefix to a sequence and returning how many tokens they cover, and usage counts for /api/stats.
  2. Defaults. --parallel 64 on OpenCL, 16 on the CPU; --queue 256; --workers --parallel + --queue, 512 at most; the CPU pool the memory of today's default cache (4 × -c) unless --kv-mem; OpenCL's pool measured against AMD's free-memory query when present, else the total less the weights and the 8 % margin.
  3. Block size 16, a context parameter with no command-line option until S6 measures 32 against 16 with the tiled kernels; only the OpenCL default changes, and only if 32 is clearly faster.
  4. --gpu-stream and the paged cache exclusive, refused at startup with a clear message; revisited only on demand.
  5. Speculative decoding and multimodal prompts in the server in S8, but designed for from S2: the block hash and the batch input carry the embedding-row marker (media hash and row index) from the start; speculative drafts need only ADR-0019's copy-on-write rule.
  6. Order. S1 (with cancellation of non-streamed requests on disconnect and --workers covering the queue, moved from S7), S2, S3 (with S4's prefill fairness), S5, S4's measurement, S6, S7, S8.