OKLMdocumentation v1.0.3
On this page

Plan: vLLM-Style Serving Mode for a Custom LLM Framework

A phased plan for adding high-throughput, multi-request serving to an existing LLM inference framework. Each phase leaves you with something that works and can be tested.

Status: S0 done (roadmap M14, 2026-11-02): the audit is the "Serving" section of gap-analysis.md, the design note serving-design.md, the cache layout adr/0019-paged-kv-cache.md, accepted with amendments on 2026-11-02 after review (the decisions taken are listed in serving-design.md); S1 is next.


Before starting (OKLM)

What this plan needs settled before its S0 can start, how its names map onto OKLM's, and what already exists. The plan was written for a generic framework on CUDA; OKLM runs on the CPU and on OpenCL devices.

Phases. The plan's phases are numbered S0 to S8 here, as the other specifications number theirs (P, G, C): S0 is the audit, S1 the plan's Phase 0 (prerequisites and baseline), S2 to S8 its Phases 1 to 7.

What OKLM already does. Much of the plan exists in some form; S0 measures each part against the plan rather than assuming:

The plan OKLM today Where
Packed (ragged) forward of several sequences oklm_eval_batch(): the new tokens of up to n_seq sequences in one forward pass, logits equal to separate evaluations (bit for bit on the CPU, in the tests on OpenCL) include/oklm/model/model.h, inference.md
Reference path oklm_eval() on one sequence; llama.cpp built as a test-only oracle (build/llama.cpp) adr/0015-test-references.md
KV cache contiguous: n_ctx cells for each of n_seq sequences, reserved up front; F16, F32 or quantized (-ctk/-ctv: q8_0, q4_0, q5_0, iq4_nl...), optionally rotated src/model/model.c, inference.md
Continuous batching the server's scheduler: one oklm_eval_batch() a step, generating requests add their token, prompts fill what is left of n_batch, new requests join between steps; but at most --parallel requests (fixed sequences) and a --queue src/server/sched.c, server.md
Chunked prefill prompts cut to the batch left after decodes, so others keep generating src/server/sched.c
Prefix caching per sequence: a request takes the free sequence holding the longest prefix of its prompt; oklm_seq_copy() copies a shared prompt. No sharing of blocks between live requests, no cache of freed prefixes src/server/sched.c, model.c
Cancellation a client that goes away frees its sequence at the next step server.md
OpenAI API, streaming, tools, JSON schemas oklm serve (ADR-0018), checked with OpenAI's Python client src/server/server.c
Metrics /api/stats and the dashboard src/server/dashboard.c
Grammars (S8.1) per request, with tool calls and JSON schemas ADR-0009
Speculative decoding (S8.2) a draft model, or a model's own MTP block, per context include/oklm/model/speculative.h
Quantized KV (S8.3) the cache types above
Several devices (S8.4) layers split over OpenCL devices; no tensor split (decided against, see the roadmap's "Closed") opencl.md
Batch speed today 8 requests: 2.5x (CPU) and 2.2x (OpenCL) the throughput of one by one, Qwen2.5-0.5B server.md

So the real work is the paged cache (memory that follows the tokens used, not --parallel times -c), sharing prompt blocks between live requests and keeping freed ones, preemption, an engine no longer bounded by a fixed number of sequences, and the measurements; S0 confirms or corrects this.

Decisions S0 takes (with the design note):

  1. Backends. CPU and OpenCL. "CUDA graphs" (S6) become what lowers launch overhead on OpenCL (fewer enqueues per step, persistent argument buffers, the existing per-layer kernels fused where it pays), and the CPU needs none. Startup profiling (S2) queries the device's free memory on OpenCL and a configured budget on the CPU.
  2. Determinism. On the CPU, batched output equals the reference token for token, whatever the batch composition, chunking, preemption or cache hits (as oklm_eval_batch() already guarantees). On OpenCL some kernels and reduction orders are chosen by batch size, so the rule is the one M3 used against llama.cpp: identical, except where the reference's own top-2 margin is a near tie (0.15 at most), and the tests say which.
  3. Models the paged cache must keep serving, or the fallback for each (the contiguous cache stays as the reference): sliding-window layers (Gemma 2 and 3), latent attention (DeepSeek V2 and V3's compressed cache), quantized and rotated caches (a cache row is one token quantized along head_dim, so a block of any size holds whole quantization blocks), models split over several devices, speculative decoding and MTP (draft tokens written to blocks then rolled back), multimodal prompts (image and audio embeddings in the prefix: a block holding them hashes the media's content hash and the row's index, not token ids nor floats; the hash format carries this from S2, the server takes media in S8), grammars per request.
  4. Where the code goes. The cache allocator and block tables in the model library (src/model/), next to the contiguous cache, behind oklm_context_params, with the prefix hash map and its LRU (evicting a cached block drops its hash in the same operation; the engine only asks for a prefix to be attached and reads usage counts); the engine (queues, step, admission, preemption) in src/server/ replacing sched.c's fixed sequences; the API of oklm_eval_batch() extended rather than replaced. The KV cache layout is a convention: an ADR (ADR-0019) records the block size, the layout, the hash and the eviction policy before S2.
  5. Benchmarks. The i7-11700K / gfx1100 machine of every other figure; Qwen2.5-0.5B and Qwen3-4B (Q4_K_M) and Llama-3.2-1B; the plan's 1, 4, 16 and 64 concurrent requests, plus an agent workload (the same system prompt and tool definitions, a growing conversation per client: what oklm agent --url sends) and a long-prompt one (S4's 30k tokens needs a model with that context: Qwen3-4B, 40k). llama.cpp's server (llama-server, continuous batching and a unified cache) is the throughput reference, built like the other oracles.
  6. Tests. tests/server/test_server.c and the model tests keep passing; S1 adds the benchmark tool (tools/ or oklm bench-serve), the allocator invariants and the stress test of the plan's last section go in a test of their own, run under ASan and TSan like the rest.

S0: Audit of the existing capacities (mandatory)

As each of the other specifications started (see agent-specs.md):

  • Search the code for each part of S1 to S8: the forward pass and oklm_eval_batch(), the KV cache and its types, sched.c's step, joining, chunking and freeing, prefix reuse and oklm_seq_copy(), the stats, cancellation, the speculative and grammar paths, the OpenCL memory queries, Windows.
  • Output: a "Serving" section appended to gap-analysis.md, each part of the plan ✅ present, 🟡 partial, ❌ missing or ⚠️ conflicting with OKLM's conventions, with file and line evidence; and docs/serving-design.md, mapping each phase onto the existing modules and settling the decisions above (ADR-0019 drafted).
  • Measure the baseline the exit criteria compare with: today's throughput, TTFT and TPOT at 1, 4, 16 and 64 clients (64 beyond --parallel, so the queue's effect shows), memory per request.
  • Reviewed before any code. Each later phase searches again first and builds only what is missing or partial, as the other specifications do.

Core Ideas

vLLM-style performance comes from three ideas that build on each other:

  1. Paged KV cache. The cache is split into fixed-size blocks, the way an OS splits memory into pages, so memory is never reserved for the maximum context of every request.
  2. Continuous batching. The batch is rebuilt at every decode step. New requests join and finished ones leave immediately.
  3. Prefix caching. KV blocks for shared prompt prefixes are reused across requests.

Everything else (chunked prefill, CUDA graphs, speculative decoding) is optimization on top of these three.


S1 (Phase 0): Prerequisites and Baseline

Tasks

    • one flat token tensor for the whole batch
    • per-token positions
    • per-sequence seq_start offsets
    • no padding
    • throughput (tokens/s)
    • TTFT (time to first token)
    • TPOT (time per output token)
    • at 1, 4, 16 and 64 concurrent requests

Exit criteria

  • Packed forward gives the same logits as the unbatched forward for a mixed batch.

S2 (Phase 1): Paged KV Cache

Data structures

KVCachePool:
  k_cache[num_layers][num_blocks][block_size][num_kv_heads][head_dim]
  v_cache[...same shape...]

BlockAllocator:
  free_list: stack of block ids
  ref_count[num_blocks]
  allocate() -> block_id
  free(block_id)              # decrement refcount; return to free_list at 0

Sequence:
  token_ids: list
  block_table: list[block_id] # logical block i -> physical block
  num_computed_tokens: int

Tasks

    • Step A (correctness): gather each sequence's blocks into a contiguous buffer and call the existing attention.
    • Step B (speed): write a paged attention kernel that reads K and V directly through the block table.
      • decode path: 1 query token per sequence
      • prefill path: many query tokens with a causal mask

Exit criteria

  • Greedy outputs match the reference exactly.
  • Memory use scales with tokens actually used, not max context.

S3 (Phase 2): Scheduler with Continuous Batching

Queues

  • waiting: requests not started
  • running: requests that hold KV blocks
  • swapped (optional): requests whose KV was moved to CPU

Engine step

budget = max_num_batched_tokens

1. For each running sequence: reserve 1 decode token.
   If it needs a new block and none is free:
     preempt the lowest-priority running sequence
     (free its blocks, move it back to waiting).

2. While budget remains and waiting is non-empty:
     admit a request if its prompt blocks can be allocated.

3. Build packed batch -> model forward -> sample -> append tokens.

4. Finished sequences (EOS, max_tokens, stop string):
     free blocks, emit result.

Tasks

Exit criteria

  • Throughput at 16 or more concurrent requests is several times the baseline.
  • No starvation or deadlock under memory pressure.

S4 (Phase 3): Chunked Prefill

Long prompts otherwise block decode steps for every other request.

Tasks

Exit criteria

  • A 30k-token prompt arriving mid-generation causes no visible stall for other streams (TPOT stays flat).

S5 (Phase 4): Prefix Caching

The biggest win for agent workloads, which resend the same system prompt and tool definitions over and over.

Tasks

Exit criteria

  • The second request sharing a 4k-token system prompt has a TTFT close to zero for that prefix.

S6 (Phase 5): Model Runner Optimizations

Tasks

    • capture graphs for batch sizes 1, 2, 4, 8, …
    • pad up to the nearest captured size and replay

Exit criteria

  • Decode step time at small batch sizes is dominated by compute, not launch overhead.

S7 (Phase 6): Serving Layer

Tasks

Exit criteria

  • Existing agent tooling works against the endpoint without modification.
  • Cancelled requests return their blocks within one engine step.

S8 (Phase 7): Optional Extensions (in order of value)

OKLM has items 1 to 3 for its contiguous cache already (and layers split over devices instead of item 4): with the paged cache they must keep working, which S0 lists and S2 to S5 check.

  1. [ ] Structured output / tool-call grammars: token masking during sampling.
  2. [ ] Speculative decoding: a draft model or n-gram drafting, verified in a single forward pass. Speeds up single-stream latency.
  3. [ ] Quantized weights and KV cache: an FP8 or INT8 KV cache roughly doubles how many tokens fit.
  4. [ ] Tensor parallelism: only if models don't fit on one GPU.
  5. [x] LoRA multiplexing: several adapters served in one batch (ADR-0021, oklm serve --lora-module).

Testing Strategy (applies to every phase)

  • Determinism test: greedy output from the batched engine matches the reference token for token, whatever the batch composition, chunking, preemption or cache hits.
  • Stress test: random arrivals, random lengths, random cancellations, and a tiny KV pool to force constant preemption. Assert no block leaks (the free count returns to its initial value).
  • Allocator invariants:
    • refcounts are never negative
    • no block appears in both the free list and a live block table

Suggested Order and Effort

Phase Outcome Relative effort
S0 Audit of what exists, design note, ADR-0019 drafted, baseline measured Low
S1–S2 (0–1) Paged memory, correctness Medium
S3 (2) Continuous batching without a fixed number of sequences, preemption Medium (OKLM batches already)
S5 (4) Prefix caching across requests (big for agents) Low–Medium
S4 (3) Smooth latency with long prompts Low (OKLM chunks already)
S6 (5) Decode speed Medium–High (kernels)
S7 (6) Usable server Low (OKLM serves already)
S8 (7) Extensions Varies (OKLM has grammars, speculation, quantized caches)

If agents are the main use case, do Phase 4 before Phase 3 (S5 before S4): OKLM's agent (oklm agent --url) resends the same system prompt and tool definitions on every turn. Decided for OKLM: S5 follows S3; S4 is reduced to its exit criterion (the 30k-token prompt measured), its fairness rule moved into S3.