OKLMdocumentation v1.0.3
On this page

Retrieval (RAG)

Retrieval-augmented generation in OKLM: text embeddings, documents cut into chunks and kept in one index file, searched by vectors, by keywords or both, optionally reranked; then answers in run, chat, serve and the agent written from the passages found and citing them, with the model judging what it retrieved. The agent also retrieves its tools. This page is the user guide, then the evaluation and the residual risks, then the reference: how each part works and its own measures. The design and its decisions are in rag-design.md, the index file format in adr/0022-rag-index-format.md.

Status: maintained. Every capability below is implemented, tested in make test and measured (2026-11-08 to 2026-11-09).

Quick start

# an index of the docs, with an embedding model (bge's query instruction recorded in the index)
oklm rag index docs README.md --out docs.gguf -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf \
    --query-prefix "Represent this sentence for searching relevant passages: "
oklm rag search docs.gguf "how is the KV cache quantized" -k 5      # hybrid: vectors and BM25 fused
oklm run -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --rag docs.gguf \
    --template-vars '{"enable_thinking": false}' -p "How do I keep the prefix cache across server restarts?"

The answer cites its sources as [n], and a summary line on stderr says what was retrieved and judged. Run oklm rag index again after the files change: only the changed files are embedded again.

The same from a window: oklm-gui --tool rag builds and updates an index with its progress, shows which sources are stale, searches it, runs the evaluation and starts a chat window on it (gui.md).

Commands

Command What it does Details
oklm embed -m MODEL TEXT... (--file F, --json, --query, --query-prefix, --passage-prefix, --pooling) one vector a text: bert and nomic-bert encoders, Qwen3-Embedding; GGUF or a Hugging Face checkpoint Embeddings
oklm rag index PATH... --out INDEX [-m MODEL] [--progress-json FILE|-] read, chunk, embed and write one GGUF index; run again, it updates by file hash; its progress as JSON lines Chunking and the index, Output for programs
oklm rag index ... --graph --gen-model M also extract each chunk's entities and relations with a chat model Graph retrieval
oklm rag index ... --caption-model VLM also caption the documents' figures with a vision model, each caption a chunk Figures in the index
oklm rag search INDEX QUERY [-k 10] [--mode dense|sparse|hybrid] [--fusion rrf|weighted] [--rerank R] [--rag-graph on] [--json|--json-full] search an index Hybrid search, Output for programs
oklm rag info INDEX [--sources] [--json] the encoder, the chunker, the counts, the graph and figures; with both options each source and whether its file changed since The index file, Output for programs
oklm rag eval DIR --index INDEX [-m MODEL] nDCG@10, MRR@10, recall@k and latency per mode on a BEIR-format set The retrieval evaluation
oklm rerank -m R -q QUERY PASSAGE... [--file F] [--top N] [--json] passages scored against a query by a cross-encoder (or a chat model), best first Reranking

Without -m, oklm rag index uses the hash encoder (a hashed bag of words, no model file): fine for tests and keyword-like lookups, not for meaning. An index remembers its embedding model; search, updates and --rag reload it and refuse another one.

Answers from an index: --rag

oklm run, oklm chat, oklm serve and oklm agent take --rag INDEX and these options (in serve they are the server's defaults, which a request's rag field changes):

Option Default
--rag INDEX off the index; its embedding model is loaded on the CPU to embed the question
--rag-dir DIR none oklm serve only: a directory of indexes a request may name, "rag": {"index": "NAME"}; the settings below go with it as they go with --rag (server.md)
--rag-k N 5 passages retrieved (1 to 50)
--rag-mode dense|sparse|hybrid hybrid (dense without BM25 postings) the search
--rag-crag on|off on corrective retrieval: each passage judged, the question rewritten when none is relevant, "no reliable source" rather than a guess (Corrective retrieval); off puts the top k passages in the prompt as they are
--rag-rewrite on|off on a follow-up rewritten as a standalone query from the conversation before it is searched (chat, serve, the agent's passages per message; Conversational retrieval)
--rag-followup rewrite|join|both both what a follow-up searches: the previous question and the rewrite (both), the rewrite alone, or the previous question and the follow-up as written (join, no generation)
--rag-route on|off off the model decides between no retrieval, one search and several (multi-hop sub-questions; Adaptive retrieval)
--rag-rerank MODEL, --rag-rerank-top N none, 20 the candidates reordered by a reranker before the best k are kept (Reranking the candidates)
--rag-graph on|off off the index's graph walked and fused with the search (an index built with --graph)
--rag-critique on|flag|off off the answer checked against its sources; on holds it and generates it once more when a cited statement is unsupported, flag lets it stream and only flags it (run, chat, serve, and the agent, in process or on --url; Self-critique)
--rag-log FILE none every retrieval appended as one JSON line

What the model is given, and each stage of the pipeline, are in Retrieval in generation.

The server

oklm serve adds three things (server.md):

  • POST /v1/embeddings with --embed-model EMB: OpenAI's request and response (In the server).
  • POST /v1/rerank (and /rerank) with --rerank-model R: Cohere's and Jina's shape (query, documents, top_n, return_documents; Reranking, in the server).
  • With --rag INDEX, a chat completion or an Anthropic message opts in with its rag field: "rag": true takes the server's settings, "rag": {"k": 8, "crag": false, "route": true, "rewrite": false, "followup": "join", "graph": true, "critique": true, "max_tokens": 1024} changes them for the request. The answer carries the run in a rag member (streamed: on the last chunk, or the Anthropic stream's message_delta), with the texts of the sources cited; /api/stats counts the runs (Retrieval in generation, in the server).

The agent

Option Setting Default
--rag-project rag.enabled off the built-in Retrieve tool over an index of the project's documentation the agent builds and keeps up to date itself (Agentic retrieval)
--rag INDEX none Retrieve over a fixed index instead
--rag-memory rag.memory off Retrieve's scope memory / all: the project's earlier sessions and memory files
--rag-inject always|route rag.inject off passages given with each user message; route asks the router first
rag.model bge-small-en-v1.5-q8_0.gguf the embedding model (hash: the hash encoder; the hash encoder too when the file is not found)
rag.queryPrefix none put before each query
rag.include, rag.exclude the documentation globs, none the files indexed
rag.index, rag.maxBytes ~/.local/state/oklm/rag/project-HASH.gguf, 200 MB the index file, the bytes indexed at most
--tool-rag K toolSearch.topK off tool selection by retrieval: every MCP tool deferred, the deferred tools that fit each message given with it (Tool selection by retrieval)
toolSearch.model bge-small-en-v1.5-q8_0.gguf the tool index's embedding model (none: BM25 alone)
toolSearch.minScore 0.62 the cosine a tool needs to be given
toolSearch.mode hybrid the tools' ranking: hybrid, dense or bm25 (Tool selection, measured)

The --rag-* options of the table above apply to the agent's retrieval too; --rag-critique checks its final answers (Self-critique in the agent) and needs a judge: the in-process model (-m) or the served one (--url, On a served model); with --mock there is none, and the agent refuses it. Retrieve only reads: every permission mode runs it as it runs Read, and --disallowedTools Retrieve removes it. With --debug, each index update, retrieval and tool choice is a line of the debug log.

Defaults, and why

Capability Default Reason (measured)
Hybrid search on better than dense or BM25 alone in nDCG@10 on both sets (SciFact 73.04 against 71.27 dense, docs 70.61 against 64.53 BM25)
Weighted fusion, 0.5 / 0.5 on (--fusion rrf the other) better than reciprocal rank fusion on both sets: 73.04 against 71.29 nDCG@10 on SciFact, 70.61 against 69.78 on the docs; other weights lose on one set or the other
Exact dense search, no approximate index always 27 ms a query over a million chunks, under the 100 ms that would call for one
Reranking off the MiniLM cross-encoder lowers SciFact nDCG@10 from 71.64 to 68.95 over the hybrid search, for 0.6 to 1.2 s a query on the GPU; on the docs it lifts recall@1 from 30.2 to 37.4 % (interval +0.5 to +14.0 points; nDCG@10 70.61 to 71.07, within the noise) and answers one more question of 30, while naive retrieval's faithfulness falls from 88.7 % to 81.1 % (interval −19.8 to +2.5)
Corrective retrieval on 92.1 % right against 89.5 % naive (one question; 86.8 % against 84.2 % on 2026-11-10, interval of the difference −5.3 to +10.5 points), 8 of 8 unanswerable questions declined against 7, 22 % fewer prompt tokens, for 0.6 s a question on the GPU (22 s on the CPU alone)
Conversational rewrite, searched with the previous question (--rag-followup both) on recall@5 of a follow-up 83.3 % against 26.9 % searched alone, 71.1 % for the rewrite alone and 82.2 % for the previous question joined with the follow-up, for 0.1 s; it acts only when there are earlier turns
Router (adaptive retrieval) off chose "one search" for 38 of 38 questions: same answers, three scoring requests more; of three other wordings, the one that routes (few-shot examples) splits 10 single questions and answers 32 of 38 against 34; GLM-4.7-Flash splits 46 of 58 requests
Graph retrieval off on the docs' 20 two-section questions recall@5 93.6 % against 89.6 % (recall@10 100 % both), the single-section checks 85 % against 95 %, and the extraction costs 4.3 s a docs chunk with an 8 B model (about 3 hours for these docs); the synthetic set gains far more (28 of 30 pairs in the top 5 against 7), where every second fact is unnamed
Self-critique off faithfulness 87.5 % to 96.1 % as the same model judges it, accuracy unchanged; four to eight scoring requests an answer; a stream is flagged, not answered again, unless the client asks
Figure captions off 18 of 20 chart questions answered against 0 with SmolVLM2-2.2B captions, but it needs a vision model at index time (about 25 s a figure on the CPU)
Retrieval in the agent (rag.enabled) off indexing the project is the user's decision; on the documentation tasks 7 or 8 of 8 passed against 2 of 8
Passages per message (rag.inject) off 8 of 8 with fewer tokens on tasks that all need the docs, but it adds passages to requests that need none
Tool selection by retrieval (--tool-rag) off top-1 61.8 % against 27.0 % for ToolSearch's keywords, but the agent evals (few MCP tools) were unchanged within single-run noise
Tool ranking hybrid (toolSearch.mode) on (dense, bm25 the others) top-1 61.8 % against 57.9 % dense; dense defines more of a request's tools at the 0.62 floor (72.5 % against 68.5 %) only by giving more definitions (2.28 a request against 1.66): at the same number given (dense at 0.64), 67.9 % against 68.5 %

RAG evaluation

Every capability before and after, on 2026-11-09 unless said. The retrieval rows are reproduced by make rag-eval; every other row is a single run of a set made for it, with Qwen3-8B Q4_K_M on a Radeon RX 7900 XT (OpenCL) as the generator and the judge, thinking off, temperature 0, and bge-small-en-v1.5 as the embedding model. Each row links to the section that has the details; Reproducing it gives the commands.

Retrieval (Hybrid retrieval, measured). bge-small-en-v1.5 F32 with its query instruction. SciFact: BEIR's test split, 300 queries over 5,183 abstracts. Docs: 120 questions over OKLM's documentation (1,007 sections) frozen at a pinned revision.

Mode SciFact nDCG@10 MRR@10 R@10 R@100 Docs nDCG@10 MRR@10 R@10 R@100
dense 71.27 68.22 83.6 94.2 60.48 62.79 77.1 99.0
sparse (BM25) 67.41 64.21 79.5 90.5 64.53 68.49 80.7 98.5
hybrid, RRF 71.29 67.85 83.7 97.2 69.78 73.08 85.2 99.6
hybrid, weighted (the default) 73.04 69.51 86.1 97.2 70.61 74.20 86.7 99.4

Dense SciFact equals MTEB's published 71.27 for the model. On a second public set, BEIR NFCorpus (323 medical queries over 3,633 documents, graded relevance), dense gives 34.38 against the model card's 34.31, BM25 32.09, hybrid 36.67 by reciprocal rank and 36.37 weighted (Hybrid retrieval, measured). On 112 questions over OKLM's own sources, the right file ranks in the top 10 for 81 % with bge-small (hybrid, 59.32 nDCG@10) and 85 % with Qwen3-Embedding-0.6B (63.63 dense); the code-aware chunker scores no better than the recursive one (Retrieval on code, measured).

Reranking (Reranking, measured). SciFact, bge-small's hybrid candidates (an index of 5,773 chunks of at most 500 tokens, so its "before" differs from the table above):

Reranker Candidates Queries nDCG@10 before after Added per query (GPU)
ms-marco-MiniLM-L-6-v2 top 100 300 71.64 68.95 1231 ms
ms-marco-MiniLM-L-6-v2 first 50 of the top 100 300 71.64 69.50 618 ms
bge-reranker-base F16 top 50 first 100 78.03 74.73 2612 ms
qwen2.5-0.5b-instruct through decide.c top 50 first 50 84.41 20.44 1039 ms

Generation (Retrieval in generation, measured, Self-critique, measured, Adaptive retrieval, measured). 38 questions: 30 of the docs set, each with the facts a right answer states, and 8 the docs do not answer, right when declined. Faithfulness: cited (sentence, source) pairs the same model judges supported.

Mode Accuracy Answerable Declined Faithfulness Invented citations Prompt tokens / q s / q
no retrieval 15.8 % 4 / 30 2 / 8 - - 31 4.4
naive (top 5 in the prompt) 89.5 % 27 / 30 7 / 8 85.3 % 0 1,100 1.6
corrective (the default) 92.1 % 27 / 30 8 / 8 86.5 % (87.5 %, 89.2 % in two later runs) 0 860 2.2
corrective, routed 92.1 % 27 / 30 8 / 8 88.3 % 0 860
corrective, self-critique 92.1 % 27 / 30 8 / 8 96.1 % 0 2.4

Multi-hop (Graph retrieval, measured). 30 questions needing two articles of a synthetic encyclopedia (100 articles); the graph extracted by Qwen3-8B; retrieval with weighted fusion (the default), answers measured earlier with reciprocal rank:

Graph Both articles in top 5 recall@5 recall@10 Right, naive (RRF) Right, corrective (RRF)
off 7 / 30 61.7 % 78.3 % 8 5
on 28 / 30 96.7 % 100 % 23 23

The 10 single-hop control questions: 10 of 10 right either way. On the documentation (the 20 two-section questions over an 82-section part of the docs set, weighted fusion):

Graph Two-section recall@5 recall@10 nDCG@10 Single-section checks (5) recall@5
off 89.6 % 100 % 88.82 95.0 %
on 93.6 % 100 % 88.49 85.0 %

Figures (Figures in the index, measured). 20 questions answered only by a chart or diagram in five documents (Markdown, PDF, DOCX, HTML, PNG); captions by the vision model named (Q8_0), the recommended one with the current caption prompt:

Index Figure's caption in top 5 Right, naive Right, corrective
text only - 0 / 20 2 / 20 (guesses)
captions, SmolVLM-256M (earlier prompt) 20 / 20 9 / 20 6 / 20
captions, SmolVLM2-2.2B (recommended) 20 / 20 18 / 20 17 / 20

Conversational (Conversational retrieval, measured). 30 follow-ups that do not name their subject, naive retrieval of 5:

Query searched recall@5 hit@5
the follow-up alone 26.9 % 11 / 30
rewritten from the conversation 71.1 % 26 / 30
previous question and follow-up joined 82.2 % 30 / 30
previous question and the rewrite (the default) 83.3 % 30 / 30
the original question (upper bound) 84.4 % 30 / 30

Tool selection (Tool selection, measured, make tool-eval). 105 tools (27 built-in, 12 skills, 4 sub-agent types, 62 MCP tools), 152 labelled requests, bge-small-en-v1.5 Q8_0, tokens of Qwen3-8B's tokenizer:

Ranking top-1 recall@5 MRR
ToolSearch's keyword score 27.0 % 46.6 % 0.376
BM25 53.3 % 70.8 % 0.620
dense 57.9 % 81.0 % 0.692
hybrid (the default) 61.8 % 76.4 % 0.692
Prompt Tokens Request's tools defined from the first turn
every tool offered 8,206 100 %
deferred, named (ToolSearch) 3,978 (+318 a ToolSearch round) 26.6 %
deferred, --tool-rag 5 3,902 68.5 %

Agent evals (make eval, Qwen3-8B Q4_K_M in-process, each run once). Tool selection, the 17 tasks of tests/evals (Tool selection, measured):

Run Passed Turns a task Tokens in all
off 13 of 17 5 526,676
off, again 12 of 17 4 425,197
--tool-rag 5 (floor 0.62) 12 of 17 6 729,730

Retrieval in the agent, the 8 tasks that need the project's docs or an earlier session (Agentic retrieval, measured):

Mode Passed Turns a task Tokens in all Seconds in all
no retrieval 2 of 8 7 503,121 438
passages with each message (--rag-inject always) 8 of 8 3 137,045 151
Retrieve tool (--rag-project --rag-memory) 7 of 8 4 184,240 192
routed passages (--rag-inject route) 8 of 8 2 116,470 146

Latency and token cost per stage. i7-11700K (8 threads) and the RX 7900 XT; each figure from the section linked.

Stage Cost Section
Embedding, bge-small F32 286 texts/s CPU, 731 GPU (24-token lines); SciFact's 290-token abstracts 16/s CPU, 46/s GPU Embeddings, measured
One query: embed and search, SciFact dense 3.1 ms p50, BM25 0.06 ms, hybrid 3.3 ms (GPU) Hybrid retrieval, measured
Dense search alone, a million chunks (384 dims) 27 ms p50 F16, 14.5 ms Q8_0 The index, measured
Index build without embedding, a million chunks 5.9 s (chunking, BM25, write); the embedding model's time dominates The index, measured
Reranking (MiniLM) 618 ms for 50 candidates, 1231 ms for 100 (GPU); 1.9 s, 3.8 s (CPU) Reranking, measured
Generation from 5 passages 1.6 s a question, 1,100 prompt tokens (naive); 20.9 s on the CPU alone Retrieval in generation, measured, On the CPU alone
Judging the passages, rewrites +0.6 to 1.1 s a question; prompt 22 % shorter (860 tokens); +22 s on the CPU alone same
Follow-up rewrite +0.1 s Conversational retrieval, measured
Router three scoring requests a question Adaptive retrieval, measured
Self-critique four to eight scoring requests an answer, a second generation for 11 of 38 Self-critique, measured
Self-critique of a stream 280 ms median after the last token; verdicts equal to the whole answers' on the same text (19 of 19) Self-critique on streams and in the agent, measured
Graph extraction (index time) 1.15 s a short synthetic chunk, 4.3 s a docs chunk (Qwen3-8B, GPU); the walk under a millisecond Graph retrieval, measured
Captions (index time) 9 figures in 250 s (SmolVLM2-2.2B, CPU; SmolVLM-256M 39 s) Figures in the index, measured
The agent's indexes the tool index of 43 to 52 entries 0.22 to 0.25 s, a few ms a query, 136 tokens a message at top 5; a 10-file project about 1 s (CPU) Tool selection, measured

Reproducing it

make rag-eval reproduces the retrieval table: both sets (--sets scifact,nfcorpus,code,docs adds NFCorpus and the code set), the four modes, bge-small-en-v1.5 F32, the outputs in build/rag-eval/ (report.md, one table a set, the rankings in TREC format). The published run embedded on the GPU:

make rag-eval RAG_ARGS="--backend opencl"        # about 3 minutes; the indexes kept for the next run
make rag-eval RAG_ARGS="--reference"             # also the Python metrics over the same rankings, and the embedding references
make tool-eval                                   # the tool selection tables (CPU, about a minute)

The other rows need an 8 B model on the GPU and minutes to hours, so they are commands, not targets. The generation and conversational rows run against a server over the docs set's index that make rag-eval built:

build/release/bin/oklm serve -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --backend opencl --port 7431 \
    --rag build/rag-eval/docs-bge-small-en-v1.5-f32.gguf
python3 tools/rag_gen_eval.py --url http://127.0.0.1:7431 --modes none,naive,crag,route,critique
python3 tools/rag_followup_eval.py --url http://127.0.0.1:7431
python3 tools/rag_stream_critique_eval.py --url http://127.0.0.1:7431     # streamed verdicts against whole ones

Multi-hop and figures (each its own index, then a server over it):

B=build/release/bin/oklm E=models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf
Q="Represent this sentence for searching relevant passages: "
$B rag index tests/rag/multihop_set/corpus --out build/rag-eval/multihop.gguf -m $E --query-prefix "$Q" \
    --graph --gen-model models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --gen-backend opencl
$B serve -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --backend opencl --port 7431 --rag build/rag-eval/multihop.gguf
python3 tools/rag_set_eval.py --set multihop --index build/rag-eval/multihop.gguf --url http://127.0.0.1:7431 --rag '{"k":5,"crag":false}'
python3 tools/rag_set_eval.py --set multihop --index build/rag-eval/multihop.gguf --search-args "--rag-graph on" \
    --url http://127.0.0.1:7431 --rag '{"k":5,"crag":false,"graph":true}'
python3 tools/rag_docs_set.py --subset 20 --out build/rag-eval/docs-mh      # the docs' two-section questions, 82 sections
for g in off on; do $B rag eval build/rag-eval/docs-mh --index build/rag-eval/docs-mh-graph.gguf -m $E --query-prefix "$Q" \
    --backend opencl --graph --gen-model models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --gen-backend opencl \
    --rag-graph $g --mode hybrid --split multihop; done                       # --split lookup: the single-section checks
$B rag index tests/rag/figures_set/corpus --out build/rag-eval/figures.gguf -m $E --query-prefix "$Q" \
    --caption-model models/smolvlm2-2.2b/SmolVLM2-2.2B-Instruct-Q8_0.gguf
$B serve -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --backend opencl --port 7431 --rag build/rag-eval/figures.gguf
python3 tools/rag_set_eval.py --set figures --index build/rag-eval/figures.gguf --url http://127.0.0.1:7431 --rag '{"k":5,"crag":false}'

"crag":true gives the corrective columns; the text-only figure row is the same index built without --caption-model. The router's wordings (the server over any index, or none): python3 tools/rag_route_eval.py --url http://127.0.0.1:7431. The agent evals:

T=$(ls tests/evals | grep -E '^(0[1-9]|1[0-7])-' | paste -sd, -)        # the 17 tasks of the tool selection rows
make eval EVAL_ARGS="--only $T --backend opencl"                         # retrieval off
make eval EVAL_ARGS="--only $T --backend opencl --tool-rag 5"            # tool selection by retrieval
D=$(ls tests/evals | grep -E '^(1[89]|2[0-5])-' | paste -sd, -)          # the 8 tasks that need the docs or a session
make eval EVAL_ARGS="--only $D --backend opencl --rag-project --rag-memory --rag-inject always --disallowedTools Retrieve"
make eval EVAL_ARGS="--only $D --backend opencl --rag-project --rag-memory"   # the Retrieve tool
make eval EVAL_ARGS="--only $D --backend opencl --rag-project --rag-memory --rag-critique on --rag-log $PWD/critique.jsonl"

--rag-inject route --disallowedTools Retrieve gives the routed row, and no --rag-* option the row without retrieval.

Residual risks

  • Cost. An embedding model makes indexing the slow part: bge-small embeds 16 abstracts a second on the CPU, so a large tree takes minutes, and the agent indexes only the documentation by default (rag.include). Graph extraction (4.3 s a docs chunk with an 8 B model; about 3 hours for these docs' 2,461 chunks) and captions are index-time costs paid once per changed file. At answer time, corrective retrieval adds five scoring requests a question, the router three, self-critique four to eight and sometimes a second generation.
  • Latency. Search itself is milliseconds; what a user waits for is the judge (+0.6 s a question with Qwen3-8B on a GPU), a reranker (0.6 to 1.2 s on the GPU, 2 to 4 s on the CPU) and generation. On the CPU alone, with an 8 B generator, every scoring request costs a prompt evaluation: a question takes 20.9 s with naive retrieval and 35.2 s with corrective retrieval, 22 s of it judging, against 1.5 s and 2.3 s on the GPU (On the CPU alone, measured).
  • Index drift. An index is a snapshot. oklm rag index must be run again after files change (it re-embeds only changed files, by content hash, and gives the bytes of a rebuild); --rag INDEX never updates the index it reads. The agent's project and memory indexes are checked before each search (names, sizes, times) and updated by hash, so its own edits are found; a file changed with an unchanged size and time within the clock's resolution would be missed until the next change. Passages are cited by file and lines as they were indexed: a reader should open the file, not trust the line numbers of an old index.
  • Embedding-model identity. Vectors mean something only for the model that made them: the index records the model's identity (a hash of its architecture, sizes, tensor table and a weight sample), its prefixes, pooling and normalization, and refuses a search or update with another one (another quantization of the same model included). The model is found by the path recorded at build time, relative to where the index was built; moving the index elsewhere needs --model.
  • Judge leniency. Faithfulness, the critique and corrective retrieval's verdicts are all the same 8 B model judging its own answers and passages. Hand checks put the faithfulness figure within about ±15 points of a reader (lenient on numbers taken from another source, strict on paraphrase); self-critique's 96.1 % is the figure of the judge it optimizes for.
  • Small sets. Every row is a mean over 38 to 323 questions, and the questions drawn are the main uncertainty: a 95 % interval spans about ±4 nDCG@10 points on the retrieval sets, ±5 on the code set and ±10 to ±13 accuracy points on the 38 answers (Error bars). Of the comparisons the defaults rest on, hybrid against either retriever alone on the docs and NFCorpus, weighted against reciprocal rank on SciFact, Qwen3-Embedding against bge-small on code and reranking's recall@1 on the docs hold; the others (fusions on the docs and NFCorpus, chunkers, reranking's nDCG@10 and answers, corrective against naive accuracy) are inside their intervals. Run to run, the generation rows do not move: four runs of each configuration of 2026-11-10 gave the same verdict on every question and the same faithfulness to the pair (requests one at a time at temperature 0). The agent evals, whose turns and timings vary, passed 13 and 12 of 17 without retrieval with only two failures in common, and the 2026-11-09 generation rows (89.5 % naive, 92.1 % corrective) are two answers above 2026-11-10's (84.2 %, 86.8 %) with weighted fusion the default in between; every new miss had a judged section among its passages.
  • The agent on a server other than OKLM's. Without /v1/decide the judge, router and critic ask chat completions and read the first token's log-probabilities, or only the answer's first word (a verdict of 0 or 1, 0.3 when it names neither); both fallbacks are tested against a fake server, not measured on a real one. A server that answers neither leaves the agent's retrieval naive and refuses --rag-critique (On a served model).
  • Streams. A streamed answer (server, flag) is flagged, not taken back: a client that does not read the verdict shows the unsupported answer.
  • Not measured. Graph retrieval over the whole docs set (measured on an 82-section part of it: the full extraction takes about 3 hours), so its effect on the 120 questions' nDCG@10 is not measured; other generators than Qwen3-8B for the generation rows; the agent's retrieval on a server without /v1/decide; the agent evals repeated.

Embeddings

An embedding model turns a text into one vector; texts of close meaning get vectors of high cosine. OKLM runs three kinds:

Kind general.architecture Models checked Pooling (default)
BERT encoders (BERT, RoBERTa and XLM-R, which llama.cpp's converter writes as bert) bert BAAI/bge-small-en-v1.5 (GGUF F32 and Q8_0, and its checkpoint) CLS
nomic-bert (rope, SwiGLU) nomic-bert nomic-embed-text-v1.5 (GGUF F32 and Q8_0, and its checkpoint) mean
Decoder embedding models qwen3 (any Llama-family architecture) Qwen3-Embedding-0.6B (GGUF F16 and Q8_0) last token

The pooling is the file's (<arch>.pooling_type, or a checkpoint's sentence-transformers 1_Pooling/config.json), else the family's above; --pooling overrides it. Vectors are L2-normalized unless --no-normalize. Texts are tokenized with the model's special tokens: [CLS] ... [SEP] for a WordPiece vocabulary (a GGUF bert vocabulary too, as llama.cpp does), and a decoder's closing token when its tokenizer adds one (Qwen3-Embedding's <|endoftext|>). A text longer than the context (512 tokens for bge, 2048 for nomic, 1024 by default for a decoder, -c to change it) is cut, its closing token kept.

Instruction prefixes are the caller's: bge's model card asks for Represent this sentence for searching relevant passages: before retrieval queries, nomic's for search_query: / search_document: , Qwen3-Embedding's for Instruct: TASK\nQuery: before queries. --query-prefix and --passage-prefix add them (--query marks the texts as queries).

oklm embed -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf "What is the boiling point of water?"
oklm embed -m models/bge-small-en-v1.5 --json --file passages.txt
oklm embed -m models/nomic-embed-text-v1.5/nomic-embed-text-v1.5.Q8_0.gguf \
    --query --query-prefix "search_query: " "how do I quantize a model"

Plain output is one vector a line (space-separated); --json prints OpenAI's embeddings response. A summary goes to stderr: texts, tokens, time, texts per second.

In the library

#include "oklm/rag/embed.h"

oklm_embed_params p = oklm_embed_default_params(); /* the model's pooling, normalized, CPU */
oklm_embedder* e = NULL;
float v[2 * 384];
const char* texts[2] = {"first text", "second text"};
oklm_embedder_new(model, &p, &e);
oklm_embed(e, texts, 2, 0 /* passages */, v);   /* row i for texts[i] */
oklm_embedder_free(&e);

oklm_embed_tokens() takes token ids instead. An embedder owns an embedding context (oklm_context_params.embeddings): no output head and no logits, each text encoded alone from position 0, several texts packed into one forward pass (64 texts or 2048 tokens by default). For an encoder the context holds no KV cache and each text attends over its own tokens in both directions; a decoder runs causally as usual and its final normalized hidden state of the last token is the embedding.

In the server

oklm serve -m MODEL --embed-model EMB answers POST /v1/embeddings with EMB (a second model beside the generative one): OpenAI's request (input a string, strings, token ids or arrays of ids; encoding_format float or base64; dimensions only the model's width) and response with usage.prompt_tokens (server.md).

Differences from llama.cpp

  • GELU. llama.cpp's bert uses the tanh approximation of GELU; the checkpoints are trained with the exact (erf) one, which OKLM uses. OKLM_EMBED_GELU_TANH=1 switches to the tanh form, for the comparison with llama-embedding only.
  • Token types. Every token of a single text is type 0, as llama.cpp; in a pair for a cross-encoder the passage is type 1 (Reranking).
  • OpenCL. The products, norms and activations run on the device; attention in both directions runs on the CPU (the device's attention kernel is causal), which costs a round trip a layer.

Embeddings, measured

On 2026-11-08 (tools/embed_compare.py, 1000 lines of these docs, 24,253 bge tokens; 1 - cosine, worst and mean over the strings):

Model, file Against Worst Mean
bge-small-en-v1.5, F32 GGUF llama-embedding, same file 1.0e-5 9.3e-7
same, OKLM with OKLM_EMBED_GELU_TANH=1 llama-embedding 8.0e-8 3.9e-8
bge-small-en-v1.5, F32 GGUF and checkpoint numpy float64 (tools/embed_ref.py) 8.0e-14 3.9e-14
bge-small-en-v1.5, Q8_0 GGUF llama-embedding 1.9e-4 9.5e-5
nomic-embed-text-v1.5, F32 GGUF llama-embedding 2.5e-7 8.7e-8
nomic-embed-text-v1.5, F32 GGUF and checkpoint numpy float64 4.5e-13 9.7e-14
Qwen3-Embedding-0.6B, F16 GGUF llama-embedding 8.8e-7 2.7e-7

The F32 / F16 files meet the design's proof (1e-4 against llama.cpp, 1e-5 against numpy) with room; bge's remaining 1e-5 against llama.cpp is the GELU (the tanh switch takes it to 8e-8). The Q8_0 file parts by up to 1.9e-4: both engines round the activations to 8 bits before each Q8_0 product, and their float orders differ, so a few roundings flip.

Speed, the same 1000 strings (oklm embed --file, model loading excluded; CPU: 8 threads of an i7-11700K (AVX-512), GPU: a Radeon RX 7900 XT (gfx1100) through OpenCL, attention on the CPU; a shared machine; since 2026-11-14 the encoders' attention runs on the device, see below):

Model CPU texts/s CPU tokens/s GPU texts/s GPU tokens/s
bge-small-en-v1.5 F32 286 6,923 731 17,723
bge-small-en-v1.5 Q8_0 491 11,915 772 18,710
nomic-embed-text-v1.5 F32 60 1,456 479 11,618
Qwen3-Embedding-0.6B F16 19 378 228 4,555

On 2026-11-14 OpenCL's attention kernel took bidirectional attention, which the backend handed back to the CPU until then. oklm embed --file on 953 lines of these docs (21,441 bge tokens), the same build with the attention handed back and on the device, three runs each: bge-small-en-v1.5 F32 750 to 764 texts/s before and 3,240 to 3,260 after, nomic-embed-text-v1.5 F32 503 to 510 and 1,579 to 1,592 (the vectors differ in their last places: test_embed and test_rerank hold the tiny encoders to their references on OpenCL as before).

Since 2026-11-14 a batch of several texts attends in one launch a block on a device (oklm_attention_spans(): each row sees its own text's rows) where it took a launch a text. oklm embed --file on 364 lines of these docs, the whole command with its loading, two runs: bge-small-en-v1.5 F32 0.76 s before and 0.71 s after on OpenCL, 0.25 to 0.36 s and 0.23 s on Vulkan (2.4 to 3.2 s on the CPU); nomic-embed-text-v1.5 F32 1.15 s and 1.08 to 1.10 s on OpenCL, 0.59 s and 0.57 s on Vulkan (12.3 to 12.9 s on the CPU). On OpenCL the attention's kernel time went from 55.8 to 11.0 ms and from 81.2 to 21.7 ms (opencl.md). The vectors are 2.7e-7 to 4e-7 from the CPU's, as before (2e-7 to 4e-7); the CPU's are the same bits.

Retrieval quality, BEIR SciFact test split (make rag-eval, exact cosine search over the 5,183 abstracts, BEIR's nDCG@10):

Model Query instruction nDCG@10 Published (MTEB) MRR@10 Recall@10
bge-small-en-v1.5 F32 Represent this sentence for searching relevant passages: 71.27 71.27 68.22 83.6
bge-small-en-v1.5 F32 none 72.00 68.45 84.5

With the model card's instruction OKLM gives MTEB's published figure exactly (embeddings-benchmark/results, the leaderboard's data: 71.273; the model card's own table: 71.275); the OpenCL run gives the same 71.27. The abstracts are long (about 290 tokens), so the corpus embeds at 16 documents a second on the CPU and 46 on the GPU.

The retrieval evaluation

oklm rag eval DIR --index INDEX [--model MODEL] [options] scores retrieval on a BEIR-format set: DIR/corpus.jsonl (_id, title, text), DIR/queries.jsonl (_id, text) and DIR/qrels/test.tsv (query-id, corpus-id, grade; --split for another file). Each document becomes one source of the index (its title, a blank line, its text), chunked as oklm rag index chunks (--chunker, --chunk-size; --chunk-size 0 keeps each document one chunk) and embedded with the model given (--query-prefix, --passage-prefix, --backend as for oklm rag index; the vectors F32 unless --vec). The index is kept: a second run with the same model and settings embeds nothing. Each judged query then runs through the same search as oklm rag search, for every mode asked (--mode dense,sparse,hybrid, --fusion rrf,weighted: weighted alone by default, --rrf-k, --weights); its chunks are mapped to their documents (a document ranks where its best chunk does) down to --depth documents (100). One row a configuration:

oklm rag eval data/beir/scifact --index build/scifact.gguf \
    -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf --chunk-size 0 \
    --query-prefix "Represent this sentence for searching relevant passages: " --fusion rrf,weighted
Column
nDCG@10 BEIR's (trec_eval's): gain the grade, discount log2(rank + 1), over the ideal ordering of every judged document
MRR@10 1 / the rank of the first relevant document within 10
R@1 ... R@100 relevant documents within the first k, over all the query's relevant documents (judged ones missing from the corpus included)
P@5, P@10 relevant documents within the first k, over k (trec_eval's and BEIR's precision: a ranking shorter than k still divides by k)
p50 / p95 ms one query from its text to its ranking: embedding it, searching, fusing

--run PREFIX writes the rankings in TREC run format (PREFIX.MODE[-FUSION][-rerank].trec). --json prints the table as an array of objects on standard output, one a row (the lines about the indexing stay on standard error; below, the five-document set of the tests):

Field
set, model the set's directory as given; the embedding model's name
mode dense, sparse or hybrid
fusion rrf or weighted for a hybrid row, "" otherwise
rrf_k, w_dense, w_sparse the fusion's settings (used or not)
queries, documents, chunks judged queries run, documents of the corpus, chunks of the index
ndcg@10, mrr@10, recall@1, recall@5, recall@10, recall@20, recall@100, p@5, p@10 the columns above, as percentages (0 to 100)
rerank_top the candidates reranked for each query, 0 without --rerank
ms_p50, ms_p95, ms_mean a query's latency in milliseconds: median, 95th percentile, mean
[
 {"set":"build/test_rag_eval/set","model":"oklm-hash-bow","mode":"dense","fusion":"","rrf_k":60,"w_dense":0.5,"w_sparse":0.5,"queries":3,"documents":5,"chunks":5,"ndcg@10":92.0063,"mrr@10":100.0000,"recall@1":83.3333,"recall@5":83.3333,"recall@10":83.3333,"recall@20":83.3333,"recall@100":83.3333,"p@5":20.0000,"p@10":10.0000,"rerank_top":0,"ms_p50":0.003,"ms_p95":0.195,"ms_mean":0.067},
 {"set":"build/test_rag_eval/set","model":"oklm-hash-bow","mode":"hybrid","fusion":"weighted","rrf_k":60,"w_dense":0.5,"w_sparse":0.5,"queries":3,"documents":5,"chunks":5,"ndcg@10":92.0063,"mrr@10":100.0000,"recall@1":83.3333,"recall@5":83.3333,"recall@10":83.3333,"recall@20":83.3333,"recall@100":83.3333,"p@5":20.0000,"p@10":10.0000,"rerank_top":0,"ms_p50":0.004,"ms_p95":0.007,"ms_mean":0.005}
]

--rerank MODEL [--rerank-top N] reorders each query's first N candidate chunks (50) with a reranker, as oklm rag search --rerank does, before they are mapped to documents (on the device of --backend; the time of a query then includes it; rows marked + rerank N). --graph --gen-model MODEL [--gen-backend B] extracts the graph into the index as oklm rag index does, and --rag-graph on searches with it (Graph retrieval). In the library: include/oklm/rag/eval.h (oklm_rag_eval_set_load(), oklm_rag_eval_index(), oklm_rag_eval_run(), oklm_rag_eval_run_reranked(), and the metric functions oklm_rag_ndcg(), oklm_rag_recall(), oklm_rag_precision(), oklm_rag_rr()). A judged query whose ranking is empty (BM25 finding none of its words) counts, with 0 for every metric.

make rag-eval [RAG_MODEL=...] [RAG_ARGS=...] (tools/rag_eval.sh, outputs in build/rag-eval/, report.md and one table a set) runs it on both sets with bge-small-en-v1.5 by default (bge models get the model card's query instruction):

  • scifact: BEIR SciFact's test split (5,183 abstracts, 300 queries; fetched by sh tools/fetch_datasets.sh --rag into data/, CC BY-NC, local measurement only), each abstract one chunk;
  • nfcorpus (not in the default list, --sets scifact,nfcorpus,docs): BEIR NFCorpus's test split (3,633 medical documents, 323 queries judged 1 or 2; fetched the same way, local measurement only), each document one chunk;
  • code (not in the default list either): the source-code question set, tests/rag/code_set/, 112 questions over OKLM's sources at a pinned revision, each file a document chunked by its path (Retrieval on code, measured);
  • docs: the documentation question set, tests/rag/docs_set/: 120 questions over OKLM's own documentation (70 lookups, 30 how-to, 20 needing two sections), each judged against the sections that answer it (grade 2) or help (grade 1), 276 judgments over 51 files. The corpus is README.md and docs/ as they were at the revision in tests/rag/docs_set/revision, extracted by tools/rag_docs_set.py (git show) and cut at the headings of levels 1 to 3 into 1,007 sections, each a document PATH#ANCHOR; so later edits of the docs do not move the labels. python3 tools/rag_docs_set.py --check verifies that every judged section exists. It also writes qrels/KIND.tsv for each kind of question (--split multihop scores the 20 two-section ones), and --subset N a small set for graph extraction: the two-section questions and five lookups over the sections judged for them and N others.

RAG_ARGS="--sets scifact --backend opencl" picks the sets and passes options on. With --reference it also runs tools/rag_eval.py: SciFact dense retrieval from oklm embed's vectors and a numpy search, scored by its own Python metrics, and those metrics over the C evaluation's rankings (--score-run), which give the same figures to the last digit on both sets; then the reference comparisons of every embedding model present under models/ (tools/embed_compare.py): 1 - cosine against llama-embedding on the same GGUF (llama.cpp built as a test tool, adr/0015-test-references.md) and against the numpy float64 encoder tools/embed_ref.py on the checkpoint, over 1000 lines of OKLM's documentation.

Hybrid retrieval, measured

On 2026-11-09, bge-small-en-v1.5 F32 with its query instruction, F32 vectors, the defaults (50 candidates a retriever, or the depth when larger; weights 0.5 / 0.5; RRF k = 60); the corpus embedded on the GPU (OpenCL), queries timed one at a time on it. make rag-eval RAG_ARGS="--backend opencl" reproduces both tables to the printed digits (the latencies within half a millisecond).

SciFact (300 queries, 5,183 abstracts):

Mode nDCG@10 MRR@10 R@1 R@5 R@10 R@20 R@100 P@5 P@10 p50 / p95 ms
dense 71.27 68.22 57.6 75.9 83.6 87.3 94.2 16.7 9.4 3.1 / 4.3
sparse (BM25) 67.41 64.21 54.1 73.6 79.5 83.9 90.5 15.9 8.7 0.06 / 0.09
hybrid, RRF 71.29 67.85 56.8 77.1 83.7 89.3 97.2 16.9 9.4 3.3 / 4.6
hybrid, weighted 73.04 69.51 58.4 78.7 86.1 91.1 97.2 17.3 9.6 3.2 / 4.4

The documentation set (120 queries, 1,007 sections in 2,655 chunks of up to 256 tokens):

Mode nDCG@10 MRR@10 R@1 R@5 R@10 R@20 R@100 P@5 P@10 p50 / p95 ms
dense 60.48 62.79 27.2 61.3 77.1 86.7 99.0 26.2 17.1 3.9 / 4.3
sparse (BM25) 64.53 68.49 28.3 69.0 80.7 89.4 98.5 29.8 17.7 0.07 / 0.08
hybrid, RRF 69.78 73.08 30.8 70.8 85.2 91.1 99.6 30.2 18.9 4.1 / 4.6
hybrid, weighted 70.61 74.20 30.2 73.5 86.7 92.1 99.4 31.5 19.2 4.1 / 4.8

NFCorpus (--sets nfcorpus, measured 2026-11-10; 323 queries, 3,633 documents; graded 1 or 2, so nDCG@10 is low beside recall for every mode):

Mode nDCG@10 MRR@10 R@1 R@5 R@10 R@20 R@100 P@5 P@10 p50 / p95 ms
dense 34.38 53.02 5.4 12.7 16.2 20.0 31.1 32.8 25.8 2.7 / 2.9
sparse (BM25) 32.09 52.69 5.7 12.3 15.4 18.5 24.5 29.3 22.9 0.01 / 0.05
hybrid, RRF 36.67 57.61 6.9 13.7 17.5 20.7 31.4 33.2 26.7 2.8 / 3.1
hybrid, weighted 36.37 56.96 6.5 13.6 17.0 20.9 31.7 33.8 26.6 2.8 / 3.1

Dense SciFact gives MTEB's published 71.27, as tools/rag_eval.py does, and P@10 9.43 as the model card has it (P@5 16.73 against its 16.8). Dense NFCorpus gives 34.38 against the card's 34.31 (MRR@10 53.02 against 52.68, P@10 25.76 against 25.73): close, not equal, the card's run being on MTEB's copy of the set. BM25 ranks nothing for 18 of NFCorpus's queries (one word, such as "eggnog" or "Fosamax", found in no document); they count with 0. On NFCorpus both fusions beat each retriever alone by 2 to 4.5 nDCG@10, and reciprocal rank is 0.3 ahead of weighted fusion, the only set of the three where it is: within the noise of 323 queries, so weighted fusion stays the default.

Hybrid retrieval is at least as good as each retriever alone on both sets in nDCG@10 and in recall at every cut-off from 5 on (RRF ties dense on SciFact at 10: 83.7 against 83.6; its MRR and recall@1 there are a little under dense's); weighted fusion is the better one on both, and the default since 2026-11-09 (--fusion rrf for reciprocal rank). The fusion settings were left as they were: weights 0.7 / 0.3 or 0.3 / 0.7 lose on one set or the other, and RRF with k = 20 gains 0.7 nDCG@10 on SciFact and 0.2 on the docs, too little on two sets to leave the usual constant. BM25 is strong on the documentation set because its questions use the docs' own names for things (commands, flags, file formats); the hashed bag of words reaches 39.39 nDCG@10 there in dense mode, the model 60.48.

Retrieval on code, measured

The code set (tests/rag/code_set/, --sets code): 112 questions over OKLM's own sources as a developer new to them asks them, each judged against the files whose code answers it (grade 2) or helps (grade 1, a header declaring the API, a caller), 197 judgments. 31 questions name an identifier, a flag, a route or a message (symbol); 60 describe what the code does without any name and without the wording of its comments (behavior); 21 need two files or more (multifile). The corpus is every .c, .h, .cl, .py and .sh file under src/, include/, apps/, tools/, tests/ and fuzz/, and the Makefile, at the revision in tests/rag/code_set/revision (517 files, 10.7 MB; tests/security/corpus/ left out), extracted by tools/rag_code_set.py (--check verifies the set). Each file is one document, chunked by its path as oklm rag index chunks a file. A file ranks where its best chunk does, so the measure is the file a reader opens.

On 2026-11-10, the GPU (OpenCL), 256-token chunks, the defaults otherwise; bge-small-en-v1.5 F32 with its query instruction unless named:

Embedder, chunker Mode Chunks nDCG@10 MRR@10 R@1 R@5 R@10 R@100 P@5 p50 ms
bge-small, auto (code) dense 19,948 55.49 55.94 24.1 62.4 79.8 100.0 21.4 14.6
sparse (BM25) 47.57 55.12 24.0 55.7 69.9 97.8 18.7 0.6
hybrid, RRF 57.68 61.81 26.6 66.1 81.5 99.6 22.3 16.5
hybrid, weighted 59.32 63.40 30.5 67.1 81.4 99.6 22.9 16.5
bge-small, recursive dense 18,733 57.27 57.45 28.0 64.0 80.7 99.6 21.6 13.7
sparse (BM25) 49.62 57.76 27.2 57.2 70.7 96.6 18.9 0.6
hybrid, RRF 59.64 64.79 32.4 65.8 80.7 99.1 21.8 15.6
hybrid, weighted 60.12 65.91 32.4 68.0 80.5 98.7 22.3 15.6
bge-small, fixed dense 19,800 53.59 55.51 24.9 61.0 75.3 98.2 20.7 14.4
hybrid, weighted 58.01 61.25 28.9 67.1 81.1 99.1 22.3 16.5
nomic-embed-text-v1.5 F32, auto dense 19,948 47.46 48.87 20.5 55.8 70.8 97.8 20.2 17.2
hybrid, weighted 53.83 56.22 23.6 60.3 78.8 100.0 20.5 20.1
Qwen3-Embedding-0.6B F16, auto dense 15,933 63.63 64.26 30.3 75.5 83.6 98.2 25.5 17.7
hybrid, RRF 59.83 64.42 29.5 71.7 84.7 98.7 24.3 21.4
hybrid, weighted 62.09 67.06 32.1 73.8 85.3 100.0 25.2 21.0

nDCG@10 by kind of question (tools/rag_eval.py --score-run RUN --data build/rag-eval/code --split KIND):

Configuration symbol (31) behavior (60) multifile (21)
bge-small auto, dense 63.04 52.79 52.04
bge-small auto, BM25 57.44 44.53 41.68
bge-small auto, hybrid weighted 68.96 56.79 52.30
bge-small recursive, hybrid weighted 69.98 59.37 47.71
bge-small fixed, hybrid weighted 65.81 58.26 45.81
Qwen3-Embedding auto, dense 75.73 58.87 59.36
Qwen3-Embedding auto, hybrid weighted 70.86 59.86 55.52

What it shows:

  • The code-aware chunker does not pay on this set. Cut at functions, it gives 59.32 nDCG@10 hybrid against 60.12 for the recursive cutter (paragraphs, then lines) and 58.01 for fixed windows, and 55.49 against 57.27 dense: differences of one or two points over 112 questions, within the noise. It is ahead only on the two-file questions (52.30 against 47.71) and a point or two behind on the others; why was not looked into. auto stays the default: it is never far behind, and its chunks begin and end where a reader expects.
  • Hybrid helps a small encoder and costs a stronger one. With bge-small, weighted fusion adds 3.8 nDCG@10 to dense; BM25 alone is strong on the questions that name things (57.44) and weak on the others (44.53). With Qwen3-Embedding, dense alone is the best nDCG@10 (63.63 against 62.09 weighted), BM25 pulling the named questions down (75.73 to 70.86) while lifting MRR, recall@1 and recall@10 a little.
  • The embedder matters most. Qwen3-Embedding-0.6B is 8 points of nDCG@10 above bge-small dense and 16 above nomic-embed, for 21 % more query time and 1.6 times the indexing time (383 s against 239 s for the 517 files). nomic-embed with its search_query: / search_document: prefixes is the weakest of the three here (dense 47.46; 57.93 on the named questions, level with BM25's 57.44).
  • Recall is high, the first rank is not. The right file is in the first 10 for 80 to 85 % of the questions and in the first 100 for nearly all, but first for only a quarter to a third (R@1 24 to 32 %); the first 5 hold 62 to 76 % of the judged files with bge-small and Qwen3-Embedding.

Reproduce with make rag-eval RAG_ARGS="--sets code --backend opencl" (bge-small, auto, about 5 minutes); the other rows add --chunker recursive|fixed or -m MODEL --query-prefix TEXT to oklm rag eval build/rag-eval/code (Qwen3-Embedding's prefix: Instruct: Given a web search query, retrieve relevant passages that answer the query and a line Query: ).

Tests

tests/rag/test_embed.c in make test, with no download: tiny random bert and nomic-bert encoders (tests/rag/fixtures/, made by tools/gen_embed_fixtures.py as llama.cpp's converter writes them, and as Hugging Face checkpoints) against tools/embed_ref.py within 1e-5 (tokens equal), the tanh switch, batching against one text at a time, normalization and pooling, [CLS] ... [SEP] from a GGUF WordPiece vocabulary, a decoder's last-token embedding against oklm_get_hidden(), the encoder refused outside an embedding context, and OpenCL against the reference within 1e-4. tests/server/test_server.c checks /v1/embeddings (every input form, base64, usage, refusals, 404 without an embedding model). tests/rag/test_eval.c: the metrics against values computed by hand, a small BEIR-format set read and scored (unjudged queries skipped, a judged document missing from the corpus counted, the TREC run written), the tiny bert fixture as an encoder (identity, unit vectors, the checkpoint directory's name), an index built with it and searched with the model the index names, another model, another prefix and the hash encoder refused, oklm rag eval with the model, and the reranked run (no reranker or no candidate giving the plain rankings, each query's first document the one a search reranked by hand puts first with the tiny cross-encoder, --rerank on the command line, a missing reranker refused). tests/rag/test_rag.c: retrieval in generation with scripted stages (the context's entries and numbering, the token budget leaving a passage out and cutting the first, the prompt and its no-source form, the citations, the corrective state machine: relevant answers, nothing relevant rewrites and retries, then the web when allowed, else no source; the thresholds, a failing judge, the reranker hook, the log line, the counters), the retriever of a hash index, and the judge and rewriter on a tiny model (the judge equal to decide.h's scores on a fresh context, its prefix reuse exact). test_rag in tests/server/test_server.c: the rag field off and on, corrective and naive, refused fields, a server without an index, /api/stats and the log. test_rag_http there: the stages on a served model against the tiny model's server (the probe finds /v1/decide; a corrective run judged through it, the rewriter and the condenser through chat completions, the router's three probabilities and the critic's); test_rag_http_fallback: a fake server answering /v1/decide 404, read by its log-probabilities, then without them by its text, the probe done once and no template variables sent. tests/harness/test_cli_rag.c: the agent's --rag-critique refused with --mock and with a --url server that answers nothing; --caption-prompt only with --caption-model. test_captions in tests/rag/test_graph.c: the default prompt reaching the captioner and not recorded, a prompt of its own reaching it, recorded, and re-captioning on a change. test_vlm_devices in tests/rag/test_rag.c: the vision model on a tiny model and projector, its projector found beside it; on OpenCL and on Vulkan its image encoder is on the model's device and the caption is the one the same context gives with the encoder on the CPU.

Chunking and the index

Commands

oklm rag index docs README.md src --out docs.gguf          # build, or update in place
oklm rag index docs --out docs.gguf --chunker fixed --chunk-size 128 --overlap 16
oklm rag search docs.gguf "how is the KV cache quantized" -k 5
oklm rag search docs.gguf "oklm_matmul" --mode sparse --json
oklm rag info docs.gguf --sources
oklm rag index docs --out docs.gguf --progress-json -       # each file's fate and the embedding's advance, as JSON lines
oklm rag info docs.gguf --sources --json                   # every source, and whether its file changed since
oklm rag search docs.gguf "how is the KV cache quantized" --json-full   # the settings, the times, each score

oklm rag index PATH... --out INDEX reads every file given and every file under the directories given (names starting with . are left out, and so are binary files, files over 64 MiB and the index itself), turns each into text with oklm ingest's reader (text, Markdown, code, JSON, HTML, DOCX, PDF), cuts it into chunks, embeds them and writes the index. Run again on the same index, it hashes each file: unchanged files keep their chunks and vectors (copied, not embedded again), changed and new ones are chunked and embedded, files no longer given are dropped. The result is the same file, byte for byte, as a rebuild from scratch. The index is written beside the target and renamed over it, so a search running meanwhile sees the old index or the new one; INDEX.lock keeps two writers apart (a lock left by a process that is gone is taken over).

Option Default
--model MODEL (-m) hash:384 the encoder: an embedding model (a GGUF file or a Hugging Face checkpoint directory, as oklm embed takes it), or hash[:DIM], the hashed bag of words below
--query-prefix TEXT, --passage-prefix TEXT none the model's instructions, recorded in the index (Embeddings)
--backend cpu|opencl|auto cpu where the model runs
--chunker auto|fixed|recursive|code|semantic auto auto: code for source files, recursive for the rest
--chunk-size N 256 target tokens per chunk
--overlap N 32 (fixed), 0 tokens shared by consecutive fixed windows
--min N / --max N 32 / 512 a smaller chunk joins a neighbor; the hard cap (a function, a joined chunk, a semantic group)
--vec f32|f16|q8_0 f16 the vectors' type (Q8_0 needs a dimension multiple of 32)
--no-bm25, --no-stem no keyword postings; no plural stripping
--rebuild ignore the existing index (also one of another encoder)
--threads N all cores worker threads of the products
--progress-json FILE the build's progress written to FILE as JSON lines (Output for programs); -: standard output, the summary then on standard error

Changing the chunker settings or the vector type chunks and embeds everything again (and says so). An index built with another encoder is refused, for an update as for a search: vectors of another model (another quantization of it included), or of the same model with other prefixes, pooling or normalization, mean nothing.

An embedding model is loaded once per command and records its identity in the index: oklm_model_id() (a hash of the architecture, the sizes, the tensor table and a sample of the weights), general.name (a checkpoint's directory name), the architecture, the path given, the dimension, the pooling, the normalization, the prefixes and the context. The chunk sizes are then counted with the model's tokenizer, and a chunk longer than the model's context (512 tokens for bge) is cut when embedded. An existing index supplies what is not given: oklm rag index docs --out docs.gguf updates an index with the model, prefixes and pooling it was built with, and oklm rag search reloads the model the index names (rag.embed.file, as given at build time, so relative to the directory the index was built from) unless --model names another copy of it; the GGUF file and its Hugging Face checkpoint have the same identity.

Output for programs

What a program that drives the commands reads. Every object is UTF-8 JSON; a field named here is always present unless its row says otherwise, and a reader ignores fields it does not know. Sizes are bytes, times of day Unix seconds, hashes and model ids 16 hexadecimal digits in a string (64 bits do not fit a JSON number). Without these options the commands print what they printed before.

oklm rag index ... --progress-json FILE writes one object a line to FILE (truncated first; - is standard output, and the summary a person reads then goes to standard error with the errors), each line flushed as it is written. The stream opens once the options are accepted (a usage error, exit status 2, writes nothing) and always finishes with an end line. In order:

type When Fields
start the encoder is loaded, the build begins index (the path given to --out); sources (the paths given, as given); encoder; chunker; vectors (F32, F16, Q8_0); bm25, rebuild, graph, captions (booleans: the postings kept, --rebuild, --graph, --caption-model)
scan the directories are walked files: the file lines that follow
file one a file, as it is read and chunked (figures captioned then too) path (as the index records it); action; chunks (made, copied or dropped); done and total (this line's number among the file lines, and scan's files)
embed before the first batch (done 0), then at most every quarter second, and after the last done, total (the new chunks; unchanged files' vectors are copied, not counted); per_second (chunks a second since the first line); seconds (since the first line)
graph with --graph, as embed done, total (every chunk of the index: extracted, or kept with its entities), per_second, seconds
end the index is written, or the build failed ok; error (a string, only when ok is false); index; files (added, updated, unchanged, removed, skipped: the file lines counted by action); chunks (added, updated, unchanged, removed: their chunks); sources and chunks_total (what the index now holds); chunks_embedded; rebuilt_all (true when the chunker, the vectors' type or the captioner changed, so every file was chunked again); index_bytes (the index file's size, 0 when there is none); seconds (the whole command); seconds_embed
action
added not in the index before: chunked and embedded
updated in the index with another content (or rebuilt_all): chunked and embedded again
unchanged the same content hash: its chunks and vectors copied
removed in the index before, no longer given or no longer readable: its chunks dropped
skipped given but not a document (unreadable, binary, over 64 MiB) and not in the index before

No embed line is written when nothing is new. With --rebuild the old index is not read: every file is added. When the encoder or a generator cannot be loaded the stream is the end line alone. encoder and chunker are the objects of rag info --json below.

{"type":"start","index":"docs.gguf","sources":["docs","gone.txt"],"encoder":{"name":"oklm-hash-bow","arch":"hash","file":"","model_id":"34242a421855fddc","dimensions":384,"pooling":"mean","normalize":true,"max_tokens":0,"query_prefix":"","passage_prefix":""},"chunker":{"kind":"auto","size":256,"overlap":0,"min":32,"max":512,"breakpoint":95},"vectors":"F16","bm25":true,"rebuild":false,"graph":false,"captions":false}
{"type":"scan","files":5}
{"type":"file","path":"gone.txt","action":"skipped","chunks":0,"done":1,"total":5}
{"type":"file","path":"docs/new.md","action":"added","chunks":1,"done":2,"total":5}
{"type":"file","path":"docs/rag.md","action":"unchanged","chunks":245,"done":3,"total":5}
{"type":"file","path":"docs/roadmap.md","action":"updated","chunks":46,"done":4,"total":5}
{"type":"file","path":"docs/how-it-works.md","action":"removed","chunks":122,"done":5,"total":5}
{"type":"embed","done":0,"total":47,"per_second":0.000,"seconds":0.000}
{"type":"embed","done":47,"total":47,"per_second":52496.486,"seconds":0.001}
{"type":"end","ok":true,"index":"docs.gguf","files":{"added":1,"updated":1,"unchanged":1,"removed":1,"skipped":1},"chunks":{"added":1,"updated":46,"unchanged":245,"removed":122},"sources":3,"chunks_total":292,"chunks_embedded":47,"rebuilt_all":false,"index_bytes":640128,"seconds":0.008,"seconds_embed":0.001}

oklm rag info INDEX --json prints one object:

Field
index the path given
version, minor the format's (rag.version, rag.minor)
file_bytes the index file's size
root the directory the index was built from: relative source paths are under it
created, updated when the index was first built and last written
encoder name (the model's general.name, oklm-hash-bow for the hash encoder); arch (bert, nomic-bert, qwen3..., hash); file (the model file as given at build time, "" for the hash encoder); model_id; dimensions; pooling (mean, cls, last, none); normalize (boolean); max_tokens (the model's context, 0: unbounded); query_prefix, passage_prefix ("": none)
chunker kind (auto, fixed, recursive, code, semantic), size, overlap, min, max (tokens), breakpoint (the semantic chunker's percentile)
counts sources, chunks, figures (chunks that caption a figure), text_bytes, vector_bytes
vectors the vectors' type: F32, F16 or Q8_0
bm25 null without postings (dense search only), else terms, postings, avg_len (terms a chunk), k1, b, analyzer
graph null without a graph, else entities, mentions, relations, model (the extractor's name)
captions null when no captioner was used, else figures, model, prompt ("": the default question)
sources only with --sources: one object a source file, in path order
states only with --sources: unchanged, changed, missing, the sources counted by state
Field of a source
path as recorded: as given to rag index, / separators
kind what it was read as: text, json, html, docx, pdf
bytes, mtime the file's size and modification time when it was indexed
hash the content hash an update compares
chunk0, chunks its first chunk's id and its chunk count (its chunks are consecutive)
state the file now: unchanged (the same content hash), changed (another size or hash: rag index would chunk and embed it again), missing (no regular file there: rag index on its directory would drop it)

The state is found by reading each file (hashed unless its size already differs), which is why it comes with --sources only. A relative path is looked up under root, so the answer does not depend on the directory the command runs in. Files added beside the sources since are not in the index and so not listed: rag index reports them as added.

{"index":"docs.gguf","version":1,"minor":0,"file_bytes":724800,
 "root":"/home/user/project","created":1794517391,"updated":1794517391,
 "encoder":{"name":"bge-small-en-v1.5","arch":"bert","file":"models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf","model_id":"0cddcf8fa7cad783","dimensions":384,"pooling":"cls","normalize":true,"max_tokens":512,"query_prefix":"Represent this sentence for searching relevant passages: ","passage_prefix":""},
 "chunker":{"kind":"auto","size":256,"overlap":0,"min":32,"max":512,"breakpoint":95},
 "counts":{"sources":3,"chunks":343,"figures":0,"text_bytes":216479,"vector_bytes":263424},"vectors":"F16",
 "bm25":{"terms":2765,"postings":23387,"avg_len":108.003,"k1":1.2,"b":0.75,"analyzer":"unicode-lower-s-1"},
 "graph":null,
 "captions":null,
 "sources":[
  {"path":"README.md","kind":"text","bytes":21521,"mtime":1794516719,"hash":"ecb6d2a034981666","chunk0":0,"chunks":34,"state":"unchanged"},
  {"path":"docs/rag.md","kind":"text","bytes":163435,"mtime":1794516719,"hash":"1a9960288bea5dbe","chunk0":34,"chunks":257,"state":"changed"},
  {"path":"docs/roadmap.md","kind":"text","bytes":32124,"mtime":1794516719,"hash":"f94290e491763cd9","chunk0":291,"chunks":52,"state":"missing"}
 ],
 "states":{"unchanged":1,"changed":1,"missing":1}
}

oklm rag search INDEX QUERY --json prints the hits as an array, best first; --json-full prints one object with the settings, the times and the same hits with every score they have:

Field (--json-full)
index, query as given
mode dense, sparse or hybrid: the mode searched (dense when the index has no postings and --mode is not given)
fusion weighted or rrf when lists were fused (hybrid, or a graph walk), else null
k, candidates the hits asked for; the hits taken from each retriever
rrf_k, w_dense, w_sparse the fusion's settings (used or not)
graph, graph_hops whether the graph was walked (--rag-graph on and the index has one), and how far (0 when not)
model the embedding model's name, null in sparse mode
rerank null, or model (the path given) and top (the candidates it scored)
found dense, sparse, graph: the candidates each retriever gave
timings_ms milliseconds, 0 for a stage that did not run: open (the index), encoder (loading the embedding model), embed (the query), dense, sparse, graph, fuse, rerank_load, rerank, total (the command up to its output)
hits the hits, best first
Field of a hit --json --json-full
rank yes yes from 1
chunk yes yes the chunk's id in the index
score yes yes what the hits are ordered by: the reranker's score with --rerank, else the fused score, else the one retriever's
dense, dense_rank when found by the vectors the dot product with the query (the cosine: the vectors are normalized) and the rank among the dense candidates
sparse, sparse_rank when found by the keywords the BM25 score and the rank among its candidates
graph, graph_rank when found by the graph walk the walk's score and rank
fused, fused_rank when fusion is not null the fused score and the rank before any reranking (weighted: each list min-max normalized, so the best of both lists scores 1)
rerank with --rerank the reranker's score (the same as score)
from yes yes the retrievers that found it: dense, sparse, graph, joined by +
source yes yes the source file's path
start, end yes yes the chunk's byte range in the extracted text (end excluded)
line0, line1 yes yes its first and last line, from 1
page, tokens yes the PDF page (0: not recovered) and the chunk's size in tokens
heading yes yes the heading path or the code symbol, "" when none
image a figure's caption only a figure's caption only the figure the chunk describes
text yes yes the chunk's text

A score is absent, not zero, when its retriever did not return the chunk among its candidates. Scores of different kinds do not compare: a cosine, a BM25 score, a fused score and a reranker's logit each rank within their own list.

{"index":"docs.gguf",
 "query":"how is an index updated when a file changes",
 "mode":"hybrid","fusion":"weighted","k":2,"candidates":50,"rrf_k":60,"w_dense":0.5,"w_sparse":0.5,"graph":false,"graph_hops":0,
 "model":"bge-small-en-v1.5","rerank":{"model":"models/ms-marco-MiniLM-L-6-v2","top":10},
 "found":{"dense":50,"sparse":50,"graph":0},
 "timings_ms":{"open":0.116,"encoder":17.491,"embed":16.050,"dense":0.035,"sparse":0.035,"graph":0.000,"fuse":0.012,"rerank_load":24.990,"rerank":450.894,"total":517.455},
 "hits":[
  {"rank":1,"chunk":80,"score":6.34369,"dense":0.796433,"dense_rank":1,"sparse":13.9372,"sparse_rank":1,"fused":1,"fused_rank":1,"rerank":6.34369,"from":"dense+sparse","source":"docs/rag.md","start":26742,"end":27741,"line0":377,"line1":390,"page":0,"tokens":251,"heading":"Retrieval (RAG) > Residual risks","text":"with corrective retrieval, ..."},
  {"rank":2,"chunk":128,"score":3.72353,"dense":0.756906,"dense_rank":3,"sparse":6.48501,"sparse_rank":27,"fused":0.456977,"fused_rank":5,"rerank":3.72353,"from":"dense+sparse","source":"docs/rag.md","start":54942,"end":55834,"line0":867,"line1":881,"page":0,"tokens":232,"heading":"Retrieval (RAG) > Chunking and the index > Commands","text":"`oklm rag index PATH... --out INDEX` reads every file given ..."}
 ]
}

oklm rag eval --json prints its table as an array of rows (The retrieval evaluation). In the library, a build reports through oklm_rag_index_params.progress, oklm_rag_index_source_state() compares a source with its file, and oklm_rag_query_ex() gives each hit's scores and the stages' times.

Chunkers

All work on the extracted text and return byte ranges of it (trimmed of white space), their lines, their size in tokens and a heading. Cuts fall between words; only a run of more than 256 bytes without white space is cut inside, at a UTF-8 boundary. Sizes are counted with the embedding model's tokenizer, word by word, when there is one; the hash encoder has none, and a word then counts one token per run of eight letters or digits and one per punctuation character (close to WordPiece on English).

  • fixed: windows of size tokens; the next window starts overlap tokens before the end of the previous one.
  • recursive: Markdown sections (ATX headings, not inside code fences) are kept apart and give each chunk its heading path (Guide > Install > Windows); a section larger than size is cut at blank lines, then line ends, then sentence ends, then words, and the pieces are joined again up to size. A chunk under min tokens joins the next one (the last one joins the previous one) when the two fit in max.
  • code: C-like sources (C, C++, Java, JavaScript, TypeScript, Go, Rust, C#...) cut at the end of each top-level block or statement, with comments, strings and character literals skipped and extern "C" and namespace blocks looked into; Python cut at each top-level def or class with its decorators and the comment lines just above (triple-quoted strings and open brackets skipped). A function, class or type of up to max tokens is a chunk of its own named after its symbol; a larger one is cut like a recursive section; the declarations between them are joined up to size. Other source files (shell, Ruby, Makefiles...) use the recursive chunker without Markdown headings. The language server's symbols are not used.
  • semantic: the text cut into sentences (within sections and paragraphs), each embedded; a chunk ends where the cosine distance between neighbors exceeds the breakpoint percentile (95) of all neighbor distances, or at max tokens.

PDF pages are not recovered from the extracted text: page is 0.

The index file

One GGUF file (ADR-0022): general.type = rag-index, the encoder's identity (rag.embed.*: model id, name, architecture, dimension, pooling, normalization, prefixes), the chunker (rag.chunk.*), the sources (paths, size, modification time, content hash, kind) and their chunks (text, byte range, lines, tokens, heading, hash), the vectors as one [dim, n] tensor, and the BM25 postings. oklm rag info prints it all; gguf-py and gguf-dump read it.

Dense search maps the file and multiplies the query's vector by the vectors tensor where it lies in the mapping (oklm_matmul(), the CPU's SIMD kernels and worker threads), then takes the top k (oklm_topk()): exact cosine similarity, no approximation, the lower chunk id first on ties. It is checked against a brute-force double loop in make test: the same top 10, scores within 1e-6, for F32 and F16 vectors.

The hash encoder

oklm_rag_encoder_hash(dim): each word of the text (the BM25 analyzer's terms) adds 1 + log(count) to one of dim buckets, chosen and signed by the word's 64-bit hash; the vector is L2-normalized. Deterministic, no model file, fast: it finds the chunks that share words with the query, weighted by how often, and nothing more (no synonyms, no meaning). It is what the tests and the measures below use, and what oklm rag uses when no embedding model is given.

The index, measured

tests/rag/bench_index.c (build/release/bin/tests/rag/bench_index CHUNKS 384 200 0 f16): synthetic documents of 100 paragraphs of 48 words (one chunk each) over a 20,000-word vocabulary drawn with a Zipf-like law, the hash encoder at 384 dimensions, F16 vectors, on an i7-11700K (8 cores, 16 threads); queries of six words, 200 per mode, each timed from the text to the hits (embedding the query included).

Chunks Build of which embedding chunking, BM25, write Open File Dense p50 / p95 Sparse p50 / p95 Hybrid p50 / p95
10,000 0.12 s 0.05 s 0.07 s 0.3 ms 14 MB 0.04 / 0.04 ms 0.05 / 0.09 ms 0.11 / 0.15 ms
100,000 1.11 s 0.46 s 0.60 s 2.3 ms 140 MB 2.40 / 2.58 ms 0.46 / 0.87 ms 3.09 / 3.52 ms
1,000,000 10.9 s 4.6 s 5.9 s 22 ms 1.40 GB 27.2 / 28.4 ms 5.7 / 9.0 ms 33.4 / 37.4 ms
1,000,000, Q8_0 vectors 11.9 s 5.7 s 5.8 s 22 ms 1.04 GB 14.5 / 15.0 ms 5.7 / 8.9 ms 20.5 / 24.1 ms

A dense query reads the whole vector tensor: 768 MB of F16 at a million chunks in 27 ms is 28 GB/s, the memory bandwidth the design counted on (about 30 ms); at ten thousand chunks the 7.7 MB stay in the cache. The build time is the hash encoder's (a real model embeds at milliseconds a chunk: its time dominates) plus chunking, the postings and the write; opening checks every table (offsets, postings, chunk to source).

Exact search stays well under the design's 100 ms threshold for an approximate index at a million chunks, so none is added (decision 6 of rag-design.md).

oklm rag search INDEX QUERY [-k 10] [--mode dense|sparse|hybrid] [--fusion rrf|weighted] [--rrf-k 60] [--weights 0.5,0.5] [--candidates 50] [--rerank MODEL [--rerank-top 50]] [--json]. The default mode is hybrid when the index has postings (the default), dense otherwise; --rerank reorders the candidates (Reranking).

  • sparse (BM25): the postings are built from the chunk texts at index time and stored in the same file: terms sorted by bytes, each with the chunks holding it (ascending) and its frequency there, and every chunk's length in terms. The analyzer (unicode-lower-s-1) takes runs of Unicode letters, digits and marks (_ and everything else separate words, so oklm_matmul is oklm and matmul), lowercases them as Python's str.lower() does, and strips English plurals (the S-stemmer: ponies to pony, dogs to dog; --no-stem keeps words as they are). A chunk scores, over the query's distinct terms it holds, idf × tf × (k1 + 1) / (tf + k1 × (1 − b + b × len / avg_len)), idf = ln(1 + (N − df + 0.5) / (df + 0.5)), k1 = 1.2, b = 0.75; checked in make test against a corpus computed by hand.
  • hybrid: the 50 best of each retriever, fused by their scores min-max normalized per list and weighted 0.5 / 0.5 (--weights DENSE,SPARSE) or, with --fusion rrf, by reciprocal rank (score = Σ 1 / (60 + rank), --rrf-k for another constant). --rag, the server and the agent search with the same defaults. Each hit says which retriever found it (dense, sparse or both). Measured in Hybrid retrieval, measured.

In the library: oklm_rag_query() runs a mode; oklm_rag_index_search() (a vector), oklm_rag_index_bm25() (a text), oklm_rag_fuse_rrf() and oklm_rag_fuse_weighted() are the pieces, for a program that brings its own lists.

Reranking

A first retrieval (dense, BM25 or hybrid) reads the query and each chunk apart and compares two summaries. A reranker reads them together: it scores each (query, passage) pair with one model, so it sees which words of the passage answer which words of the query. It is slower (one forward pass a pair, not a dot product), so it reorders only the first candidates of the retrieval: the 50 best by default.

oklm rerank -m models/ms-marco-MiniLM-L-6-v2 -q "How many people live in Berlin?" \
    "Berlin has a population of 3,520,031 registered inhabitants." \
    "New York City is famous for the Metropolitan Museum of Art."
oklm rerank -m models/ms-marco-MiniLM-L-6-v2 -q "what is BM25" --file passages.txt --top 5 --json
oklm rag search docs.gguf "how is the KV cache quantized" --rerank models/ms-marco-MiniLM-L-6-v2 --rerank-top 50 -k 5
oklm serve -m model.gguf --rerank-model models/ms-marco-MiniLM-L-6-v2      # POST /v1/rerank

oklm rerank prints the passages best first with their score and their relevance (--json: /v1/rerank's response). oklm rag search --rerank MODEL retrieves the --rerank-top best candidates (50) as usual, reads each chunk's text from the index, reranks them and prints the -k first; each hit's score is then the reranker's.

Two kinds of rerankers

The model given decides:

  • A cross-encoder: a text encoder with a classification head, as llama.cpp names its tensors (cls: a dense layer then tanh, BERT's pooler or RoBERTa's classifier.dense; cls.output: the projection to the labels). Default: cross-encoder/ms-marco-MiniLM-L-6-v2 (22 M parameters, BERT, 6 layers, trained on MS MARCO), from its Hugging Face checkpoint or a GGUF. The pair is read as [CLS] query [SEP] passage [SEP], the passage's tokens (and its closing [SEP]) of token type 1 as the model was trained; a model with one token type (RoBERTa, XLM-R: the bge rerankers) reads <s> query </s></s> passage </s>. A pair longer than the context (512 tokens) is cut longest first, as Hugging Face's tokenizers cut it: a short query is kept whole and the passage cut. The score is the head's raw logit (llama.cpp's RANK pooling); several pairs share each forward pass.
  • The generator (any chat model, when no cross-encoder is at hand): each pair becomes a yes / no question answered through decide.c (Decisions by scoring): the state Query: ...\n\nPassage: ..., the question "Is the passage relevant to the query: does it answer the query, or help answer it?", the answers Yes / No scored as the continuation of the prompt (one forward pass over the prompt when each answer is one token, the system turn and the query's prefix kept in the cache from one passage to the next). The score is the log-odds log p(yes) − log p(no). A passage too long for the context (2048 tokens) is cut at its end.

Both scores are logits: the sigmoid turns them into a relevance in (0, 1) (/v1/rerank's relevance_score). They rank passages; they are not calibrated probabilities.

In the library

#include "oklm/rag/rerank.h"

oklm_reranker* r = NULL;
float s[2];
const char* docs[2] = {"first passage", "second passage"};
oklm_reranker_new(model, NULL, &r);              /* a cross-encoder, or a chat model (decide.c) */
oklm_rerank(r, "the query", docs, 2, s);         /* s[i]: a logit, higher is more relevant */
oklm_rag_rerank_hits(r, index, "the query", hits, n);   /* oklm_rag_query()'s hits reordered */
oklm_reranker_free(&r);

Below it, oklm_embed_pairs() encodes pairs with any pooling (RANK for the head's values), oklm_embed_pair_tokens() shows the tokens and the first token of type 1, oklm_model_classify() applies the head to a state; oklm_rerank_order() sorts scores (stable) and oklm_rerank_probability() is the sigmoid.

In the server

oklm serve -m MODEL --rerank-model R [--rerank-model-id NAME] [--rerank-backend cpu|opencl|vulkan|auto] answers POST /v1/rerank (and /rerank) in Cohere's and Jina's shape (server.md): query, documents (strings or {"text": ...}), top_n, return_documents; results best first with index, relevance_score and, when asked, document.

Differences from llama.cpp

  • llama.cpp's converter drops BERT's pooler. For BertForSequenceClassification it writes only the classifier (cls.output) and leaves out bert.pooler.dense, which the classifier reads (logits = classifier(tanh(pooler(h[CLS])))). Its GGUFs of the MiniLM cross-encoders (GGUF-my-repo's among them) therefore score another function: on the Berlin example above 0.08 and −0.22 instead of 8.85 and −11.25, and on SciFact the ranking falls (below). OKLM runs such a file as llama.cpp does, and loads the checkpoint with its pooler as cls; oklm convert writes it so, and llama.cpp then applies it too (its RANK head runs cls with tanh when the file has it).
  • Token types. llama.cpp gives every token type 0; OKLM gives the passage type 1, as the model was trained (the scores of the MiniLM cross-encoder move by up to 10 logits without it). OKLM_EMBED_TYPE0=1 makes every token type 0, for the comparison with llama-embedding only.
  • GELU: the exact one, as for embeddings (OKLM_EMBED_GELU_TANH=1 for the comparison).
  • Long pairs are cut longest first, as the model's tokenizer does; llama-embedding refuses a pair longer than its batch.

Reranking, measured

Scores against the two references (tools/embed_compare.py, 990 pairs: 10 queries of these docs, 99 or 100 passages each; absolute difference of the logits, worst and mean):

File Against Worst Mean
checkpoint, and oklm convert's F32 GGUF numpy float64 (tools/embed_ref.py --pairs) 4.0e-6 4.0e-7
oklm convert's F32 GGUF, OKLM with OKLM_EMBED_GELU_TANH=1 OKLM_EMBED_TYPE0=1 llama-embedding --pooling rank -fa off, same file 1.3e-3 2.3e-4
numpy float64 with ggml's F16 GELU table and type 0 (--gelu-f16 --type0) llama-embedding --pooling rank -fa off, same file 4.0e-4 6.2e-5
GGUF-my-repo's F16 GGUF (no pooler), OKLM with both switches llama-embedding --pooling rank (flash attention on), same file 2.5e-4 5.3e-5

The numpy proof (1e-5) holds with room. Against llama.cpp the logits part by up to 1.3e-3, above the 1e-4 aimed at: ggml's CPU GELU rounds its input to F16 and reads an F16 table (GGML_GELU_FP16), about 1e-3 relative a value, and these logits run from −11 to +2. The numpy reference made to round the same way comes within 4.0e-4 of llama.cpp (mean 6e-5), the rest being F16 roundings that flip on differences upstream; the published F16 file, whose scores are a hundred times smaller (no pooler), parts by 2.5e-4. Without the type-0 switch OKLM and llama.cpp part by up to 10 logits: the token types matter to this model.

Quality and cost on BEIR SciFact's test split (300 queries, 5,183 abstracts; bench_rerank, below): the candidates are the hybrid search of an index of the abstracts embedded with bge-small-en-v1.5 F32 (bge's query instruction, BM25 and dense fused by RRF), the reranker reorders the top 50 or 100, nDCG@10 as BEIR scores it. The added time is the reranker's alone, per query (the pairs read, scored and sorted; CPU: 8 threads of an i7-11700K, GPU: a Radeon RX 7900 XT through OpenCL):

Candidates Reranker Queries nDCG@10 before after Added per query, GPU CPU
hybrid top 100 ms-marco-MiniLM-L-6-v2 (checkpoint) 300 71.64 68.95 1231 ms 3793 ms (first 50 queries)
hybrid top 100, first 50 reranked ms-marco-MiniLM-L-6-v2 300 71.64 69.50 618 ms 1902 ms (first 50 queries)
hybrid top 50 bge-reranker-base (GGUF-my-repo F16) first 100 78.03 74.73 2612 ms
hybrid top 50 qwen2.5-0.5b-instruct Q4_K_M (decide.c) first 50 84.41 20.44 1039 ms

Reranking lowers nDCG@10 on SciFact: the design's proof ("after reranking above the hybrid's") is not met with these rerankers. The MiniLM figure is not a fault of the reranker's code: reranking BM25's top 100 with the same model gives 68.8 in the BEIR paper (Thakur et al., 2021: BM25 + CE), and OKLM's 68.95 over the hybrid's top 100 is that figure; bge-small's own dense retrieval (71.27) is already stronger on these scientific claims than a cross-encoder trained on MS MARCO's web questions. bge-reranker-base (XLM-R, 278 M) loses less (−3.3 on the first 100 queries, MiniLM −5.3 on the same queries). The generator through decide.c with a 0.5 B model is not a usable reranker here (its order is far worse than the hybrid's); it is the fallback for when no cross-encoder is present, and a larger chat model is what it needs: on the Berlin example (with "The capital of Germany is Berlin." as a third, unhelpful passage) qwen2.5-0.5b and Qwen3-0.6B call all three relevant (logits 1.2 to 5.3, though the population line ranks first or second), while Qwen3-4B separates them (38.1 against −27.9 and −29.9) (the yes / no prompt runs about 450 tokens a passage, 1 s for 50 on the GPU with a 0.5 B model). The reranker earns its cost where the first retrieval is weaker than it (BM25 alone, the hash encoder, out-of-domain embeddings), not over a strong in-domain embedding model; --rerank is off by default.

The run: build/release/bin/tests/rag/bench_rerank, built with the benches (make bench); bench_rerank EMBED_MODEL RERANK_MODEL [--backend cpu|opencl] [--queries N] [--top 100] indexes the corpus once (build/rag-eval/scifact-bench.gguf, 5,773 chunks of at most 500 tokens, 116 s on the GPU), then for each test query runs oklm_rag_query() (hybrid, --top candidates per retriever and kept) and oklm_rag_rerank_hits(), and prints nDCG@10 before and after with the reranking time per query. A pair longer than 512 tokens is cut (1 pair in 100 on average).

Tests

tests/rag/test_rerank.c in make test, with no download: a tiny random BertForSequenceClassification (tests/rag/fixtures/tiny-ce, as a checkpoint, as llama.cpp writes it with its pooler, and through oklm convert) against tools/embed_ref.py --pairs within 1e-5 on eight pairs (empty query or passage, a long passage, a long query, both long: the tokens, the first type-1 token and the cuts equal to Hugging Face's tokenizer), pairs packed in one call equal to one at a time, the type-0 switch, an embedding model refused, the order and the sigmoid, search hits reordered, OpenCL within 1e-4; with the public model present, MiniLM on the Berlin example within 1e-4 of the numpy reference. tests/server/test_server.c checks /v1/rerank (both document forms, top_n, return_documents, /rerank, every refusal, 404 without a reranker) and the decide.c fallback: the generator's scores equal to the yes / no question scored directly on a fresh context, the same order, the same through the server.

Retrieval in generation

With --rag INDEX, oklm run, oklm chat, oklm serve (per request) and oklm agent (a tool) answer from the passages of a retrieval index built by oklm rag index: the question is searched, the best passages go into the prompt numbered, with their file, lines and heading, and the model is asked to answer from them only, to cite them as [n] and to say when they do not answer. Corrective retrieval (on by default) has the model judge each passage first, rewrite the question when none is relevant, and, when nothing reliable is found, answer that no reliable source was found rather than cite anything.

oklm rag index docs README.md --out docs.gguf -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf \
    --query-prefix "Represent this sentence for searching relevant passages: "
oklm run -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --backend opencl --rag docs.gguf \
    --template-vars '{"enable_thinking": false}' -n 400 -p "How do I keep the prefix cache across server restarts?"
oklm chat -m MODEL --rag docs.gguf --rag-k 8 --rag-log rag.jsonl
oklm chat -m MODEL --rag docs.gguf --rag-route on    # no retrieval, one or several, as the model decides
oklm serve -m MODEL --rag docs.gguf                  # requests opt in with "rag": true
oklm agent -m MODEL --rag docs.gguf                  # the Retrieve tool over that index
oklm agent -m MODEL --rag-project --rag-memory       # over the project's own docs, kept indexed, and earlier sessions

The options are in Answers from an index.

What the model sees

The question is replaced, for that turn only, by one user message:

Answer the question using only the numbered sources below. After each statement, cite the sources that support it
by their numbers in square brackets, like [1] or [2][3]. Cite only these numbers. If the sources do not contain the
answer, say that the sources do not answer the question instead of guessing.

Sources:

[1] docs/server.md:89-121 (Running it > Prefix cache on disk)
...the passage's text...

[2] README.md:96-112 (Quick start)
...

Question: How do I keep the prefix cache across server restarts?

The passages come in the order of the search (and of the reranker, when one is set), as many as fit a budget of 2048 tokens counted with the generator's tokenizer: one that does not fit is left out, the first one is cut to fit when it alone is too long. When nothing reliable was found the message says so instead (No reliable source was found for this question ... Begin your answer with "No reliable source was found." Do not cite any source and do not invent references), then the question. In oklm chat the conversation keeps the question as typed, so the sources of earlier turns do not pile up in the context. oklm run --rag renders the prompt through the model's chat template (with --system when given) and prints only the answer.

Corrective retrieval

Each passage is judged by the generator itself through a decision by scoring (server.md, decide.h): the state is the question then the passage, the question "Does the passage above contain information that answers the question, fully or in part?" with the options yes and no, each scored in one forward pass from the shared prompt (the question's part is evaluated once per query, then each passage's). The probability of yes is the passage's relevance: relevant at 0.5 or more, irrelevant under 0.2, ambiguous between (the design's defaults, rag-design.md).

  1. The question is searched and its k passages judged. One relevant passage or more: the context is the relevant and ambiguous ones, the irrelevant ones dropped (fallback none).
  2. Otherwise the retrieval is poor: the model rewrites the question as a search query (a short greedy generation, one line: "use the specific names, commands, options and terms the documentation would use"), which is searched and judged in turn (fallback rewrite). A rewrite equal to a query already searched is not searched again.
  3. Still nothing relevant: the web, when allowed, which only the agent allows, and only when its WebSearch tool has a service (webSearch.url) and its permission rules let the call run; its results are judged the same way (fallback web).
  4. Still nothing: the context is empty and the prompt asks for the "no reliable source" answer (fallback no_source). No passage is cited, so no citation can be invented from it.

Every run is one record: the question, its rewrites, each search with its passages (chunk, source, lines, heading, score, the retrievers that found it, the reranker's score, the probability of relevance, the verdict, the number it got in the prompt), the fallback, the sources cited, the context's tokens, the times. --rag-log FILE appends it as a JSON line; oklm run and oklm chat print a summary on stderr:

(retrieval: 5 passages in 1 search, 2 relevant, 1 ambiguous; none; [1] docs/server.md:89-121 [2] docs/server.md:15-88 [3] README.md:96-112)

In the server

oklm serve --rag INDEX [--rag-k N] [--rag-mode M] [--rag-crag on|off] [--rag-route on|off] [--rag-rewrite on|off] [--rag-rerank MODEL [--rag-rerank-top N]] [--rag-log FILE] opens the index once; a chat completion, or an Anthropic message (/v1/messages), opts in with its rag field (default off; server.md): "rag": true takes the server's settings, "rag": {"k": 8, "crag": false, "max_tokens": 1024, "route": true, "rewrite": false} changes them for the request. The judge's questions, the router's choice, the rewrite, a follow-up's condensing and a multi-hop request's sub-questions run through the scheduler as requests of their own (a scoring request per option, the first alone so the others find the shared prefix cached; a short greedy generation). A multi-turn request's last user message is condensed from the messages before it. The answer carries the run in a rag member, with the texts of the sources cited, so a client can check the citations: the whole answer's top level; streamed, the last chunk (the one with the finish_reason) of a chat completion, and the message_delta event of an Anthropic stream. /api/stats counts the runs (rag.requests, judged, relevant, ambiguous, irrelevant, rewrites, no_source, context_chunks, context_tokens, errors, condensed, routed, route_none, route_one, route_multi, hops).

In the agent

oklm agent --rag INDEX gives the agent the built-in tool Retrieve (query, optional k) over that index; --rag-project gives it the project's own documentation instead, indexed and kept up to date by the agent (Agentic retrieval, which also covers the session memory and the passages given with each message). It runs the same pipeline: with an in-process model, the judge and the rewriter use a context of their own on it (4096 tokens, made at the first call, so the agent's cache is left as it is); with a model served elsewhere (--url) they are requests to its server (On a served model). Its result is the numbered sources to cite, or "No reliable source was found in the index for this query ... Do not cite a source for it". When nothing relevant is found and WebSearch has a service, the web is searched through the WebSearch tool, its rules and questions included. Each run goes to --rag-log and to the debug log (retrieval {...}).

On a served model

oklm agent --url URL has no model in process; its retrieval stages and its critic are then requests to the server (include/oklm/rag/rag_http.h, src/rag/rag_http.c). At start, when retrieval is on, the agent finds once how the server answers a closed question, and says so on stderr:

The server Judge, router, critic Told
OKLM's (/v1/decide answers) POST /v1/decide: the same question on the same state as in process (a noul for the judge and the critic, a choice of none / one / several for the router), each option scored by decide.c in the server's scheduler, its prefix cache shared with the agent's own requests "judge, router and critic through the server's /v1/decide"
another, with logprobs a chat completion: the state, the question and "Allowed options:" (decide.h's layout), one token at temperature 0 with top_logprobs 10; each alternative whose first word begins exactly one option adds its probability to it, normalized over the options "the server has no /v1/decide: ... log-probabilities"
another, without them the same request read as text (16 tokens): the option its first word names is certain; none named gives yes 0.3, ambiguous at the default thresholds "... read from the answer's text"
neither answers naive retrieval, --rag-critique refused "no judge, router or rewrite"

The probe is one /v1/decide request (a 404, or an answer without probabilities, moves on), then one chat completion asking for log-probabilities (sent again without them when the server refuses the request with a 400). The rewriter, the condenser and the decomposer are chat completions at temperature 0 with the in-process prompts and token limits (64, 96, 160), their answers cleaned the same way; on an OKLM server they carry chat_template_kwargs {"enable_thinking": false}, which another server is not sent (it may refuse an unknown member). A stage whose request fails is handled as an in-process stage that fails: a rewrite or condensing skipped and noted, the passages of a failed judgement left unjudged.

In the library

include/oklm/rag/rag.h. The pipeline is made of stages, each one C interface (a function and its state) a program can replace:

oklm_rag_store store;                 /* the index, its encoder, its search settings */
oklm_rag_store_open("docs.gguf", NULL, NULL, &store, err, sizeof(err));
oklm_rag_retriever rt = oklm_rag_store_retriever(&store);
oklm_rag_model gen;                   /* the generator as judge and rewriter */
oklm_rag_model_init(&gen, ctx, tmpl);
oklm_rag_evaluator ev = oklm_rag_model_evaluator(&gen);
oklm_rag_rewriter rw = oklm_rag_model_rewriter(&gen);
oklm_rag_stages st = {&rt, NULL /* reranker */, &ev, &rw, NULL /* web */};
oklm_rag_result res;
oklm_rag_run(NULL /* defaults */, &st, question, &res);
/* or a turn of a conversation: a follow-up condensed, the request routed */
oklm_rag_condenser co = oklm_rag_model_condenser(&gen);
oklm_rag_router ro = oklm_rag_model_router(&gen);
oklm_rag_decomposer de = oklm_rag_model_decomposer(&gen);
st.condenser = &co, st.router = &ro, st.decomposer = &de;
oklm_rag_params p = oklm_rag_default_params();
p.route = 1;
oklm_rag_history_text(msgs, n_before, 0, &history);   /* the earlier turns, as text */
oklm_rag_run_turn(&p, &st, history, question, &res);
oklm_rag_prompt(&res, question, &user_message);     /* then generate as usual */
oklm_rag_citations(answer, res.n_context, cited, 16, &n_cited, &n_invalid);
oklm_rag_result_json(&res, "mine", &line);
  • Retriever (oklm_rag_retriever): a query to passages; oklm_rag_store_retriever() searches an index (hybrid by default).
  • Reranker (oklm_rag_reranker, optional): a score per passage; candidates (20) passages are handed to it and the best k kept. oklm_rag_cross_reranker(reranker) makes it of a rerank.h reranker (a cross-encoder, or the generator through decide.c).
  • Evaluator (oklm_rag_evaluator): a probability of relevance per passage; oklm_rag_model_evaluator() is the generator's judgement on a context. oklm_rag_judge_state() and oklm_rag_judge_question() give its question to a program that scores it its own way (the server does, through its scheduler).
  • Rewriter (oklm_rag_rewriter): oklm_rag_model_rewriter(), from oklm_rag_rewrite_prompt() and oklm_rag_rewrite_clean().
  • Web (oklm_rag_web, optional): tried last when web is set.
  • Router (oklm_rag_router, optional): a probability for each route (none, one, several); oklm_rag_model_router() is the generator's decision (Adaptive retrieval).
  • Decomposer (oklm_rag_decomposer, optional): a multi-hop request's sub-questions; oklm_rag_model_decomposer().
  • Condenser (oklm_rag_condenser, optional): a follow-up made standalone from the earlier turns; oklm_rag_model_condenser() (Conversational retrieval).

oklm_rag_params: k (5), candidates (20), max_tokens (2048), corrective (1), relevant (0.5), irrelevant (0.2), rewrites (1, at most 3), web (0), route (0), condense (1), max_hops (3), and a token counter (four bytes a token without one). The same stages on a served model are oklm_rag_http_evaluator(), _rewriter(), _router(), _decomposer(), _condenser() and _critic() of an oklm_rag_http client (oklm_rag_http_new(), then oklm_rag_http_probe(): On a served model). A rewriter or web search that fails is noted in the result's error and skipped; an evaluator that fails leaves the passages unjudged (naive retrieval). oklm_rag_stats_add() sums runs into counters.

Reranking the candidates

--rag-rerank MODEL (on run, chat, serve and the agent) puts a reranker (Reranking) between the search and the judge: the search returns --rag-rerank-top candidates (20), the reranker scores each against the query (one model reading both), and the best k by its score go on to the judge and the prompt, in its order. A cross-encoder (models/ms-marco-MiniLM-L-6-v2) runs on the CPU beside the generator; a chat model given there reranks through decide.c. Each passage's score is in the log line (rerank). It is off by default: on SciFact the MiniLM cross-encoder ranked below the hybrid search it reorders (Reranking, measured). In the library: oklm_rag_cross_reranker(reranker) is the stage.

Reranking in generation, measured

On 2026-11-10, the setup of Retrieval in generation, measured (Qwen3-8B Q4_K_M on the GPU, the docs index with bge-small and weighted fusion, 5 passages), with and without --rag-rerank models/ms-marco-MiniLM-L-6-v2 (the first 20 candidates, the reranker on the CPU beside the generator); both runs the same day:

Retrieval Accuracy Answerable Unanswerable declined Faithfulness s / q
naive 84.2 % 25 / 30 7 / 8 88.7 % (102 / 115) 1.9
naive, reranked 86.8 % 26 / 30 7 / 8 81.1 % (103 / 127) 2.3
corrective 86.8 % 25 / 30 8 / 8 88.0 % (95 / 108) 2.8
corrective, reranked 89.5 % 26 / 30 8 / 8 87.7 % (107 / 122) 3.1

The retrieval behind it (oklm rag eval --rerank, bge-small hybrid with weighted fusion, the reranker on the GPU):

Set Reranked nDCG@10 MRR@10 R@1 R@5 R@10 p50 ms
docs (120) none 70.61 74.20 30.2 73.5 86.7 4.0
first 20 71.07 78.30 37.4 73.8 84.4 138
first 50 71.35 78.15 37.4 74.0 85.5 344
code (112) none 59.32 63.40 30.5 67.1 81.4 16.5
first 20 60.60 65.19 34.0 69.3 82.0 185
first 50 59.40 63.52 33.6 65.2 82.4 437
  • On the docs and the code, unlike SciFact, the cross-encoder puts the right section or file first more often (recall@1 +7 and +3.5 points) and leaves nDCG@10 about level (+0.1 to +1.3): it reorders the top, it finds nothing the search had not found.
  • Answers: one more answerable question right in both modes (26 against 25 of 30), a difference of one question, within the noise. Each configuration's misses had a judged section among their passages (the reranker dropped one of d109's two judged sections, and that answer went wrong); the other misses are the generator's.
  • Faithfulness by the judge falls with naive retrieval (81.1 % against 88.7 %): answers cite more often (127 pairs against 115), and 24 of the pairs are judged unsupported against 13; the interval of the difference, −19.8 to +2.5 points, still holds 0. With corrective retrieval, whose judge drops passages after the reranker, it stays level (87.7 % against 88.0 %).
  • Cost: 0.3 to 0.4 s a question on the CPU for 20 candidates.

Reranking stays off: it changes which passage comes first, not what the model answers, and with naive retrieval it may cost faithfulness. With corrective retrieval it does no harm and is worth trying where the right passage is often retrieved but not first.

On the CPU alone, measured

On 2026-11-10, the pipeline above without a GPU: oklm serve --backend cpu (i7-11700K, 8 cores, the server's default threads, the machine otherwise idle, load under 1), Qwen3-8B Q4_K_M, the same index and settings, 12 questions of the answer set (the first 10 answerable ones and 2 unanswerable), python3 tools/rag_gen_eval.py --no-judge (no faithfulness requests, so only the answers are timed). The same questions on the GPU for comparison:

Retrieval GPU s / q CPU s / q of which retrieval (CPU) of which the answer (CPU) prompt / completion tokens
none 4.9 65.3 0 65.3 31 / 397
naive 1.5 20.9 0.01 s 20.9 1,096 / 78
corrective 2.3 35.2 22.5 s (judging 22.1 s) 12.8 858 / 88
  • The CPU decodes about 6 tokens a second with this model, and reads a prompt at about 135 tokens a second: a question with five passages answered in a hundred tokens takes 20 s, most of it reading the passages.
  • Corrective retrieval's five scoring requests share the question's prefix but each reads a passage: 22 s a question on the CPU, 1.1 s on the GPU. Its shorter prompt saves 8 s of the answer, so it costs 14 s more in all, 68 % on top of naive retrieval (53 % on the GPU). On a CPU, naive retrieval is the practical setting for an 8 B generator; corrective retrieval pays where declining unanswerable questions matters more than the wait.
  • The search itself (embedding the query with bge-small, hybrid search) takes 8 to 9 ms either way.
  • 10 of 12 right on the CPU against 11 on the GPU: one answer differs between the back ends (not looked into).

Error bars

tools/rag_stats.py puts a 95 % interval on a row by the bootstrap over its questions (10,000 resamples, a fixed seed) and compares two rows over the same questions paired, from the TREC runs of oklm rag eval --run or the answers of tools/rag_gen_eval.py; runs gives the spread of repeated runs:

build/venv/bin/python -I tools/rag_stats.py trec --data build/rag-eval/docs RUN [RUN2] [--metric mrr@10] [--split KIND]
build/venv/bin/python -I tools/rag_stats.py answers --mode crag A/answers.jsonl [B/answers.jsonl]
build/venv/bin/python -I tools/rag_stats.py runs --mode crag R1/answers.jsonl R2/answers.jsonl ...

The rows' intervals (2026-11-10):

Row Value 95 % interval
SciFact, hybrid weighted, nDCG@10 73.04 68.81 to 77.13
NFCorpus, hybrid weighted 36.37 32.86 to 39.94
docs, hybrid weighted 70.61 65.96 to 75.09
code, bge-small hybrid weighted 59.32 53.86 to 64.71
code, Qwen3-Embedding dense 63.63 57.95 to 69.27
answers, naive, accuracy 84.2 % 71.1 to 94.7
answers, corrective, accuracy 86.8 % 76.3 to 97.4
answers, naive, faithfulness 88.7 % 81.7 to 96.0
answers, corrective, faithfulness 88.0 % 79.7 to 96.7

The comparisons the documentation draws (nDCG@10 unless named; the share of resamples at or below 0 is a one-sided p-value):

Comparison Difference 95 % interval Resamples ≤ 0 Holds
SciFact, hybrid weighted − dense +1.77 −0.41 to +4.02 0.058 no
SciFact, weighted − RRF +1.76 +0.34 to +3.24 0.007 yes
NFCorpus, hybrid weighted − dense +1.99 +0.80 to +3.29 0.001 yes
NFCorpus, RRF − weighted +0.30 −0.39 to +1.00 0.20 no
docs, hybrid weighted − dense +10.14 +5.74 to +14.50 0.000 yes
docs, hybrid weighted − BM25 +6.08 +2.44 to +9.57 0.001 yes
docs, weighted − RRF +0.84 −1.21 to +2.81 0.21 no
code, recursive − code-aware chunker +0.80 −2.57 to +4.07 0.32 no
code, code-aware − fixed chunker +1.30 −2.00 to +4.54 0.22 no
code, code-aware − recursive, two-file questions +4.60 −3.76 to +13.25 0.15 no
code, bge-small hybrid weighted − dense +3.83 −0.71 to +8.41 0.051 no
code, Qwen3-Embedding − bge-small, dense +8.14 +3.05 to +13.22 0.001 yes
code, Qwen3-Embedding dense − hybrid weighted +1.53 −2.54 to +5.52 0.23 no
docs, reranked (20) − not, nDCG@10 +0.46 −3.96 to +4.96 0.43 no
docs, reranked (20) − not, recall@1 +7.15 +0.49 to +14.03 0.019 yes
code, reranked (20) − not, recall@1 +3.50 −2.60 to +9.82 0.13 no
answers, corrective − naive, accuracy +2.63 −5.26 to +10.53 0.39 no
answers, reranked − not, corrective accuracy +2.63 −7.89 to +15.79 0.41 no
answers, reranked − not, naive faithfulness −7.59 −19.78 to +2.45 0.92 no

What holds: hybrid search over either retriever alone wherever the retrievers differ much (the docs, NFCorpus), weighted fusion over reciprocal rank on SciFact, the larger embedder on code, and reranking putting the right section first on the docs. What does not: the choices between fusions on the docs and NFCorpus, between chunkers, and every difference in answers on 38 questions, where one question is 2.6 points. The defaults stay as they are (none of them is contradicted), but those choices are settled by other reasons than these sets: corrective retrieval by declining unanswerable questions and its shorter prompts, the code-aware chunker by where its chunks begin and end.

Repeated runs: four runs of each generation configuration (naive and corrective, with and without reranking) on 2026-11-10 gave the same verdict on all 38 questions and the same faithfulness to the pair. With one request at a time at temperature 0, oklm serve answers deterministically; the run-to-run spread is nil, and the questions' draw is the whole uncertainty.

Retrieval in generation, measured

On 2026-11-09: Qwen3-8B Q4_K_M on the GPU (OpenCL) behind oklm serve --rag, thinking off, temperature 0, at most 400 tokens an answer; the index of the documentation set's frozen corpus (1,007 sections, bge-small- en-v1.5 F32 with its query instruction, hybrid search by reciprocal rank, its retrieval as in the table above); 5 passages. python3 tools/rag_gen_eval.py asks 38 questions three ways: 30 of the documentation set (every fourth, d001 to d117, each with the facts a right answer states in tests/rag/docs_set/answers.jsonl, regular expressions, all required) and 8 that nothing in the docs answers (an enterprise license's price, a Django plugin, the 2018 World Cup, the capital of Australia...), right when the answer says the sources do not answer or no reliable source was found. Faithfulness: each sentence that cites [n] checked against each source it cites by the same model's yes / no judgement (/v1/systemone, "Does the passage support the statement?"); a citation of a number the prompt did not give counts as unsupported.

Mode Accuracy Answerable Unanswerable (abstained) Faithfulness (cited pairs supported) Invented citations No source Rewrites Prompt tokens / q s / q
no retrieval 15.8 % (6 / 38) 4 / 30 2 / 8 - - - - 31 4.4
naive (top 5 stuffed) 89.5 % (34 / 38) 27 / 30 7 / 8 85.3 % (93 / 109) 0 0 0 1,100 1.6
corrective 92.1 % (35 / 38) 27 / 30 8 / 8 86.5 % (96 / 111) 0 7 6 860 2.2
  • Without retrieval the model does not know OKLM (it takes it for another product and guesses); with it, 27 of 30 answers state every fact. The three misses are the same with and without judging: a resume answered from the checkpoint ADR without --resume, a paged block's size read as the pool's (65,536 tokens), and the index format answered without "GGUF".
  • Corrective retrieval judged 220 passages: 139 relevant, 2 ambiguous, 79 irrelevant (the thresholds are nearly never between: the judge is sure). With the irrelevant ones dropped the prompt was 22 % shorter. On the unanswerable questions 7 of 8 runs found nothing relevant, rewrote the query once (six rewrites searched; the seventh, empty or the question again, was not) and answered "No reliable source was found." Naive retrieval answered 7 of 8 "The sources do not answer the question", but cited a passage for "the capital of Australia is Canberra [1]"; corrective retrieval said no source was found, then Canberra from its own knowledge, marked as such. Some of those answers go on with a general-knowledge guess after the notice (the prompt allows it when said); none cites anything.
  • No answer cited a number outside its sources in either mode.
  • Faithfulness by the judge, checked by hand on 20 (sentence, source) pairs, 10 judged unsupported and 10 supported: two of them are not statements (a "Citations:" line of an answer that listed its sources at the end); of the 18 others the hand verdict agrees with the judge on 11 (7 of 10 judged supported, 4 of 8 judged unsupported). The judge is lenient where a number comes from another source than the one cited (chunk sizes of 128 and 16 credited to the chunkers' description; the pool's 65,536 tokens read as a block's), and strict on paraphrase ("only Ogg Theora is decoded natively" judged unsupported by the passage saying so). The figure is the model's view, within about ±15 points.
  • Cost: judging five passages and the occasional rewrite add 0.6 s a question to the 1.6 s of naive retrieval on this GPU (the judge's prompts share the question's prefix; their passages are the cost).

Adaptive retrieval

Not every request needs the documentation, and some need several parts of it. With --rag-route on (run, chat, serve; "route": true per request; the agent's --rag-inject route) the generator first decides, in one forward pass, how many searches a request needs: a decision by scoring (server.md, decide.h) whose state is Request: and the request, whose question is "A search engine over the project's documentation can be used to answer the request above. How many searches does answering it need?", with three options:

Route Option's description What happens
none it needs no documentation: a greeting, small talk, general knowledge, arithmetic, or work on text it gives nothing is searched; the request goes to the model as it was asked
one one search: it asks about one thing the pipeline of Retrieval in generation
several several searches: it asks two or more separate questions, or combines or compares distinct things multi-hop: the request is split, each part searched, the passages merged

The most probable route is taken. Multi-hop: the generator writes the separate questions the request asks (a short greedy generation, "at most three, each one standalone, one per line"); each is retrieved as one request is (searched, judged, rewritten when nothing relevant is found); the passages of the sub-questions that found something are merged rank by rank (the first of each, then the second...), a chunk found twice kept once, within the token budget, and numbered as one context. A decomposition into fewer than two questions is a single retrieval of the request. A router or decomposer that fails is noted in the run's error and the request is retrieved once.

Every decision is logged: the run's JSON line holds route (none, one, multi), route_p (the three probabilities), hops (the sub-questions) and each round's hop; oklm run and oklm chat print (retrieval: routed one (none 0.03, one 0.85, several 0.12); ...). It is off by default: measured below, it does not answer better than always retrieving.

Adaptive retrieval, measured

On 2026-11-09, as Retrieval in generation, measured (Qwen3-8B Q4_K_M on OpenCL behind oklm serve --rag, the documentation set's index, 5 passages, the same 38 questions), python3 tools/rag_gen_eval.py --modes crag,route, one run:

Mode Accuracy Answerable Unanswerable (abstained) Faithfulness Invented citations Prompt tokens / q
corrective 92.1 % (35 / 38) 27 / 30 8 / 8 89.2 % (99 / 111) 0 860
corrective, routed 92.1 % (35 / 38) 27 / 30 8 / 8 88.3 % (98 / 111) 0 860

The router chose one for all 38, with a probability of 0.99 or more for 36 of them: the eight questions the docs do not answer (the capital of Australia, the 2018 World Cup...) were not routed to none, and the two-part questions (d097 "How do I train a LoRA adapter, and then serve several such adapters...", d109, d113) got at most 0.0035 for several. So the answers are those of corrective retrieval (the faithfulness differs by one judged pair). With this model and question, routing costs three scoring requests a question and changes nothing; it stays off, logged with its probabilities for whoever tries another model or wording.

Other wordings and another model (2026-11-09, python3 tools/rag_route_eval.py, one run). The router's decision asked through /v1/systemone with the same rendering, over 58 requests: the 38 questions (one expected, except several for the 5 of kind multihop and none for the two general-knowledge ones, u04 and u06; the six other unanswerable ones ask about OKLM, so a search that finds nothing is right), the 15 other two-section questions of the docs set (several) and five requests that need no documentation (a greeting, thanks, arithmetic, a translation, a summary of a given sentence; none). The wordings: actions (the options as what to do: answer directly, search once, split and search each part), examples (the question followed by five examples, none from the set) and both.

Model, wording none one several matching none expected (7) one (31) several (20) ms a decision
Qwen3-8B, current 2 56 0 33 2 31 0 138
Qwen3-8B, actions 0 58 0 31 0 31 0 134
Qwen3-8B, examples 2 28 28 40 2 21 17 174
Qwen3-8B, both 3 50 5 39 3 31 5 178
GLM-4.7-Flash Q4_K_M, current 0 12 46 25 0 5 20 201
GLM-4.7-Flash Q4_K_M, examples 9 6 43 33 7 6 20 223
GLM-4.7-Flash Q4_K_M, both 0 11 47 24 0 4 20 242

Only the few-shot examples make Qwen3-8B route at all, and they split ten single questions too (some do ask two things: d001 "which size names ..., and what do the -V and -A suffixes mean"); the greeting and the arithmetic go to none with every wording but actions, the thanks, the translation and the summary never do. GLM-4.7-Flash, the larger model here, calls almost every request several. Answers with the examples wording built in (tools/rag_gen_eval.py --modes crag,route, the same run):

Mode Accuracy Answerable Declined Faithfulness Invented citations Prompt tokens / q s / q
corrective 89.5 % (34 / 38) 26 / 30 8 / 8 90.3 % 0 866 2.7
corrective, routed by examples 84.2 % (32 / 38) 24 / 30 8 / 8 82.8 % 1 1,149 3.2

It routed 14 answerable questions to several searches and none to no search (u04 and u06 among the eight routed to one). The decompositions mostly reword one question three times ("How do I create a retrieval index with fixed-size chunks?", "How do I set overlapping chunks...?", "How do I configure chunk overlap...?"), so the context holds three near-copies of the same sections; two answers that were right became wrong (d073, d105). Corrective retrieval's 89.5 % here against 92.1 % earlier is the run-to-run difference of the server (see the roadmap's determinism item). No wording routes sensibly without losing answers, so the prompt is unchanged and the router stays off.

Conversational retrieval

A follow-up ("and how do I turn it off?") searched as it is finds nothing: its subject is in the turns before it. With --rag-rewrite on (the default) the generator first rewrites it as a standalone query from the conversation: the last user and assistant messages (six at most, each cut at 1200 bytes, the reasoning dropped, 4000 bytes in all) and the follow-up, with the instruction "Rewrite the follow-up message at the end as a standalone search query that can be understood without the conversation: replace pronouns and references ... If it is already standalone, repeat it unchanged", decoded greedily on one line. By default (--rag-followup both) the previous user question and that standalone query, joined by a space, are what is searched (and judged against): the previous question brings back the subject when the rewrite drops it. --rag-followup rewrite searches the rewrite alone, join the previous question and the follow-up as written (no generation; also what both falls back to when the rewrite fails). The prompt keeps the question as the user wrote it, since the conversation is in the context. Where: oklm chat --rag (each turn after the first), a multi-turn request to oklm serve --rag (the messages before the last user message; "rewrite": false per request turns it off, "followup" picks the mode), and the agent's passages per message (--rag-inject). A first message has no history and nothing is rewritten; oklm run has a single message. The run's line holds the query searched in standalone.

Conversational retrieval, measured

tests/rag/docs_set/followups.jsonl: 30 conversations made of two-part questions of the documentation set, the first part asked and answered, the second asked as a follow-up that does not name its subject ("How do I stop contexts from applying them?" after a question about oklm tune's cache), judged by the qrels of the question it came from. python3 tools/rag_followup_eval.py against oklm serve --rag (Qwen3-8B Q4_K_M on OpenCL, the documentation set's index, hybrid search with weighted fusion, naive retrieval of 5 passages so only the query differs), 2026-11-09, one run:

Query searched rag field recall@5 hit@5 s / q
the follow-up alone (no rewrite) "rewrite": false 26.9 % 11 / 30 0.57
the follow-up rewritten from the conversation "followup": "rewrite" 71.1 % 26 / 30 0.68
the previous question and the follow-up joined "followup": "join" 82.2 % 30 / 30 0.49
the previous question and the rewrite (the default) "followup": "both" 83.3 % 30 / 30 0.42
the original two-part question (upper bound) one message 84.4 % 30 / 30 0.55

The rewrite more than doubles the recall of a follow-up, for 0.1 s of generation, but alone it does not reach the plain join of the previous question and the follow-up on this set, where every follow-up is about the question just before it: the 8B model's rewrite sometimes keeps the follow-up's words and drops the subject ("What does the --clip option limit?" without oklm rl; "Who allocates the output tensors of an operation?"). Searching the previous question with the rewrite gives the subject back and keeps what the rewrite resolved: it equals the join on 29 conversations and does better on one (a LoRA follow-up whose rewrite names the option meant: 2 of 3 sections against 1), 1.1 points under the original question. It is the default; join costs no generation, and rewrite searches without the previous turn's words, for a conversation that moved on (which this set does not test). The seconds are noisy (the GPU was shared). An earlier run with reciprocal rank fusion, the default then, gave 32.8 % alone, 70.3 % rewritten (66.9 % with a first instruction; telling it to always name the subject gave 70.3 %), 79.2 % joined and 80.3 % for the original question.

Agentic retrieval

The agent can search its project's documentation the way it reads files: oklm agent --rag-project (or the setting rag.enabled) gives it the built-in tool Retrieve, over an index of the project the agent builds and keeps up to date itself, and lets the model decide when to search, what for, and to search again with other words when the passages do not answer.

oklm agent -m MODEL --rag-project                    # Retrieve over the project's docs
oklm agent -m MODEL --rag-project --rag-memory       # and over earlier sessions
oklm agent -m MODEL --rag-project --rag-inject always   # passages given with each message too

The project's index. Its sources are the project's files the include globs select (rag.include; by default the documentation: **/*.md, *.markdown, *.mdx, *.rst, *.adoc, *.txt), minus the exclude globs (rag.exclude, a directory's name leaves its files out), the files the .gitignore files ignore and those a Read deny rule covers (the walk is Glob's), within rag.maxBytes (200 MB); each recorded under its path relative to the project. It is a RAG index file in the state directory (~/.local/state/oklm/rag/project-HASH.gguf, HASH of the project's path; rag.index names another file), so the project's tree stays as it was. Nothing is read before the first search; before each search the agent lists the files and compares their names, sizes and modification times with the last listing: unchanged, nothing is read; otherwise the index is updated by content hash (a file that did not change keeps its chunks and vectors, a changed or new one is embedded, a removed one dropped), so an edit the agent just made is found by its next search. The embedding model is rag.model (a path, a file name looked for as the models are, hash for the hash encoder; bge-small-en-v1.5-q8_0.gguf by default, on the CPU, the hash encoder when it is not found), with rag.queryPrefix before each query (none by default); the search is hybrid. --rag INDEX gives Retrieve a fixed index instead, searched as it is.

The tool. Retrieve (query, k, and scope with the memory) runs the pipeline of Retrieval in generation on the model's query: judged and rewritten by the generator on a context of its own (an in-process model) or by the server (--url), the web last when WebSearch has a service, the reranker when --rag-rerank is given. Its result is the numbered sources to cite, or that no reliable source was found. It only reads, so the permission modes treat it as Read inside the project: it runs without a question in every mode, plan mode included; a deny rule Retrieve (--disallowedTools Retrieve) removes it from the tools offered. It is a core tool (in the prompt from the first turn) when retrieval is on and absent otherwise; the design planned it deferred, but a small model rarely loads a deferred tool, and the user turned it on to be used (deferredTools can still list it).

The session memory. With --rag-memory (the setting rag.memory) Retrieve takes a scope: project (the default), memory or all. The memory's index holds the project's earlier sessions (each transcript of .oklm/sessions/ as session/ID: its user and assistant messages as text, the agent's reminders left out; the current session left out, its conversation being in the context) and the memory files the agent loaded (AGENTS.md and the others, their imports). It is ~/.local/state/oklm/rag/memory-HASH.gguf, updated the same way before a search of the memory: a session that went on is embedded again, the others kept. A separate tool was the other choice; one tool with a scope keeps the prompt shorter and lets all merge both (the passages of each taken in turn).

Passages with each message. --rag-inject always (rag.inject) runs the pipeline on each user message before the model's first turn, the message condensed from the conversation when it is a follow-up, and gives the passages found as a reminder after the message ("Passages retrieved for this request ... cite them as [n]"); nothing when nothing relevant was found. --rag-inject route asks the router first, so a request that needs no documentation gets nothing, and a multi-hop one the passages of each part. With the memory on, both are searched. This is the fixed retrieval of run and chat brought to the agent; Retrieve stays offered unless denied. Sub-agents use Retrieve but get no passages per message.

The settings are in The agent.

The run options of Retrieval in generation (--rag-k, --rag-mode, --rag-crag, --rag-rewrite, --rag-rerank, --rag-log) apply to the agent's retrieval as well; with --debug each index update and each run is a line of the debug log (retrieval index: 6 files, 6 embedded, 0 kept ..., retrieval {...} with surface agent for a Retrieve call, agent-message for the passages of a message). In the library: include/oklm/harness/rag_index.h (oklm_rag_project_update(), oklm_rag_memory_update(), oklm_rag_transcript_text()), the tool context's retrieve search and oklm_agent_params.rag_context for the passages per message. When Retrieve is offered, the system prompt ends with one line saying what it searches and to use it before guessing (a small model rarely calls a tool it is not told when to use).

Agentic retrieval, measured

Eight agent eval tasks need the project's documentation or an earlier session (tests/evals/18-* to 25-*): a made-up tool, "quarry", its README and nine pages of docs (tests/evals/quarry-docs.sh), each fact a task needs stated once among other sections: a configuration key to set (18), a command to give (19), two facts of one page (20), constants of the code to fix by the documented policy (21), the meaning of an exit status (25), the reason for a limit (24); and two that only an earlier session's transcript answers (22: create the release branch "we agreed on"; 23: a changelog entry "following the convention I gave you"). Each is judged by what it leaves (a file's content, a branch). sh tools/eval_agent.sh MODEL --only 18-...,25-... --backend opencl with Qwen3-8B Q4_K_M on OpenCL, in-process, every mode with the project's index and the memory (--rag-project --rag-memory), each run once, 2026-11-09:

Mode Options Passed Turns a task Tokens in all Seconds in all
no retrieval none 2 of 8 7 503,121 438
(a) fixed retrieval (passages with each message) --rag-inject always --disallowedTools Retrieve 8 of 8 3 137,045 151
(b) agentic Retrieve (Retrieve offered) 7 of 8 4 184,240 192
(c) adaptive routing --rag-inject route --disallowedTools Retrieve 8 of 8 2 116,470 146

Without retrieval the agent guesses (a cache_size key, a branch named release/quarry-1.4) or greps for words the docs do not use; it cannot see .oklm/sessions at all (Glob and Grep skip it), and one task looped for 32 turns. Fixed retrieval puts the right section in front of the model with the request, so most tasks end in two turns. The agentic tool passes as much on the documentation tasks, for a turn more (the call; it called Retrieve in six tasks, and found 19's command with Grep), and searches the memory when the request says "in an earlier session" (22: Retrieve {"query": "quarry 1.4 branch name", "scope": "memory"}); it missed 23, where it edited the changelog without searching ("Fix:" for the convention's "Fixed:"). The routed mode matches the fixed one, with fewer tokens in this run; since the router chooses one for almost every request (Adaptive retrieval, measured), the difference is the noise of single runs. Every request of these tasks needs the docs, so fixed retrieval wins here; on requests that need no documentation it would add passages for nothing, which is what the tool avoids. The index of each task's 10 files was built in about a second on the CPU, before the first search.

On a served model. The same tasks with the agent on --url, Qwen3-8B Q4_K_M served by oklm serve --backend opencl -c 32768, corrective retrieval on (the default), its judge, router, rewriter and condenser the server's (On a served model): sh tools/eval_agent.sh MODEL --only 18-...,25-... --url URL --rag-project --rag-memory, each run once, 2026-11-09, beside the in-process rows above:

Mode Model Passed Turns a task Tokens in all Seconds in all
(b) agentic Retrieve in process 7 of 8 4 184,240 192
(b) agentic Retrieve served (--url) 7 of 8 3 150,567 277
(c) adaptive routing in process 8 of 8 2 116,470 146
(c) adaptive routing served (--url) 8 of 8 2 107,409 195

The pass rates are the in-process ones, and the task missed is the same (23: the changelog edited without a search, "Fix:" for "Fixed:"). The tokens are the agent's turns only (the judge's and the router's scoring requests run in the server, 86 /v1/decide requests over the 16 runs, the probes included); the seconds are longer by the HTTP round trips and by the scoring requests queuing behind the agent's own in one server.

Graph retrieval

Some questions need two facts kept in two places: "in which town was the architect who designed the Varnel Bridge born?" is answered by the bridge's article (designed by Orsolya Kett) and the architect's (born in Lenmark), and the second never names the bridge, so neither the vectors nor the keywords of the question find it. Graph retrieval keeps the entities and relations each chunk states, extracted by a generator when the index is built, and walks them at query time from the question's entities to the chunks they lead to.

oklm rag index docs --out docs.gguf -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf \
    --graph --gen-model models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --gen-backend opencl
oklm rag search docs.gguf "where was the designer of the Varnel Bridge born" --rag-graph on
oklm run -m MODEL --rag docs.gguf --rag-graph on -p "..."
oklm serve -m MODEL --rag docs.gguf               # a request opts in with "rag": {"graph": true}
Option Default
--graph --gen-model MODEL (rag index) off extract each chunk's entities and relations with this chat model and keep them in the index
--gen-backend cpu|opencl|vulkan|auto (rag index) cpu where the generator (and the captioner: its language model and the blocks of its image encoder) runs
--rag-graph on|off (rag search, run, chat, serve, agent) off fuse the graph walk's list with the search; ignored, with a notice, for an index without a graph
--graph-hops 1|2 (rag search) 2 how far the walk goes

Extraction. Each chunk is put in a prompt ("Extract the knowledge graph of the text below ... answer with the JSON object only", reasoning off) and decoded greedily under the grammar of a JSON schema (OKLM_RAG_GRAPH_SCHEMA: entities, a list of names; relations, objects of subject, relation, object), at most 512 tokens. The answer is read leniently (oklm_rag_graph_parse(): a closed reasoning block and a code fence skipped, a relation's ends added to the entities, duplicates dropped, at most 32 of each); an answer cut by the limit or holding no JSON gives an empty extraction, not an error. Entities are linked by their normalized name (lowercased, punctuation runs made one space, a leading "the", "a" or "an" dropped): "The Varnel Bridge" and "Varnel bridge" are one node. Linking by embedding similarity was not added: on the measured set the names are written alike wherever they appear, and the hits' entities (below) already link a question worded otherwise.

The walk. The seeds are the entities whose names appear in the question (every run of up to eight of its words, normalized, looked up) with weight 1, and those of the first stage's three best hits with weight 0.5 divided by the hit's rank. A chunk that mentions a seed is reached in one hop; a chunk that mentions an entity related to a seed (either way) in two, at half the seed's weight. A chunk's score sums, over the entities that reach it, the weight times the entity's rarity, ln(1 + chunks / chunks mentioning it); entities mentioned by more than a tenth of the chunks (at least 20) are not walked through. The ranked list (from graph) is fused with the dense and BM25 lists by the search's own fusion (weighted fusion, the default: weight 0.5; reciprocal rank, the graph list's terms weighted 1).

With corrective retrieval. The judge reads each passage alone against the question, so the second fact of a chain ("Orsolya Kett was born in Lenmark") is judged irrelevant to a question about a bridge. A passage the graph found is therefore kept, as ambiguous, when another passage of the same search is relevant; other irrelevant passages are dropped as before. Without that rule, corrective retrieval threw away what the graph had found: 6 of 30 right with the graph on, against 23 with the rule.

oklm rag info prints the graph (162 entities, 267 mentions, 162 relations, extracted by Qwen3-8B-Q4_K_M.gguf); the file format is ADR-0022's minor 1. An update keeps a kept chunk's entities when the same generator (by file name) built them, and extracts every chunk again with another one.

In the library (include/oklm/rag/graph.h): the extractor is a stage (oklm_rag_extractor, set in oklm_rag_index_params.graph), the generator's is oklm_rag_generator_open() and oklm_rag_generator_extractor(); oklm_rag_graph_search() is the walk, oklm_rag_search_params.graph (with graph_hops, graph_seeds, w_graph) turns it on in oklm_rag_query() and so in every retriever of a store; oklm_rag_graph_find(), oklm_rag_graph_entity() and oklm_rag_graph_relation() read the graph.

Graph retrieval, measured

On 2026-11-09: the multi-hop set of tests/rag/multihop_set/ (tools/rag_multihop_set.py, committed): 100 short articles of an invented encyclopedia, five kinds of chains (a work and its architect's birthplace, a company and its founder's university, an album and its band's city, a ship and its shipyard's port, a novel and its author's instrument), ten chains of each, six asked; 30 questions that need both articles of a chain, and 10 single-hop questions (the first link) as the control. Index: bge-small-en-v1.5 F32 with its query instruction, hybrid search by reciprocal rank (the default then; oklm rag search --fusion rrf); the graph extracted by Qwen3-8B Q4_K_M on OpenCL: 100 chunks in 115 s (1.15 s a chunk), 162 entities, 162 relations. Retrieval by oklm rag search (an article found when one of its chunks is in the first k); answers by Qwen3-8B behind oklm serve --rag (5 passages, temperature 0, thinking off), right when they name the chain's last fact. python3 tools/rag_set_eval.py --set multihop, one run.

Multi-hop (30) both articles in top 5 recall@5 both in top 10 recall@10 right, naive retrieval right, corrective
graph off 8 63.3 % 16 76.7 % 8 5
graph on 23 88.3 % 30 100 % 23 23
  • The graph brings the second article into the top 5 for 23 of 30 questions (8 without) and into the top 10 for all 30; with it the answers follow the retrieval exactly: the 23 questions whose two articles were among the five passages were answered right, in both modes. Without it, corrective retrieval answered 5 (4 of its 8 found pairs: the judge drops the second fact).
  • The 10 single-hop questions: their article first in every configuration (recall@5 100 %), 10 of 10 answered right with and without the graph; the graph's extra candidates did not push it out.
  • The seven multi-hop misses with the graph have the second article at ranks 6 to 10: the question's own words (architect, born, town) rank other articles of the same kind above it in both lists, which the graph's single list does not outweigh in reciprocal rank fusion.
  • Cost: 0.5 s a question to answer either way; the walk itself is sub-millisecond on this index; the extraction is the cost of the build (1.15 s a chunk on this GPU with an 8 B model).
  • The set is synthetic and small, written for this measure: it shows what the walk adds where the facts are split, not how often real documents split them (below, the documentation).

With weighted fusion (the default since; the same index, oklm rag search without --fusion, retrieval only, one run): the graph's list weighs more against the two others than under reciprocal rank.

Multi-hop (30), weighted fusion both articles in top 5 recall@5 both in top 10 recall@10
graph off 7 61.7 % 17 78.3 %
graph on 28 96.7 % 30 100 %

The single-hop controls stay at 10 of 10 in the top 5; five more pairs than under reciprocal rank (23) have both articles in the top 5.

On the documentation (2026-11-09). The docs set's corpus is 1,007 sections, 2,461 chunks; Qwen3-8B extracts a docs chunk in 4.3 s on this GPU (longer chunks, many more names than the synthetic articles), about 3 hours for the whole set, so the graph was built over a part of it: tools/rag_docs_set.py --subset 20, the sections judged for the 20 two-section questions and for five single-section lookups (the checks), and 20 other sections as distractors: 82 sections, 362 chunks, extracted in 1,554 s, 3,123 entities, 4,920 mentions, 2,562 relations (1.2 MB of index). Scored by oklm rag eval --split multihop|lookup with --rag-graph off|on, hybrid search, one run:

Docs part (82 sections) fusion queries nDCG@10 recall@5 recall@10
two-section, graph off weighted 20 88.82 89.6 % 100 %
two-section, graph on weighted 20 88.49 93.6 % 100 %
two-section, graph off RRF 20 86.38 89.4 % 100 %
two-section, graph on RRF 20 86.40 89.8 % 100 %
single-section checks, graph off weighted 5 89.18 95.0 % 100 %
single-section checks, graph on weighted 5 88.77 85.0 % 100 %
  • The documentation's two-section questions name both of their subjects ("train a LoRA adapter, and then serve several such adapters"), so vectors and BM25 already find both sections: recall@10 is 100 % with or without the graph, and the graph adds 4 points of recall@5 at an unchanged nDCG@10. The synthetic set gains because its second fact is never named in the question; real questions here rarely chain that way.
  • On the single-section checks, the graph's extra candidates push a judged section out of the top 5 once (recall@5 95 % to 85 %).
  • With 82 sections the search is easier than over all 1,007 (recall@10 100 % here against 86.7 % for the whole set), so these figures are not comparable with the retrieval table; the comparison is graph off against on over the same part. The 120 questions' nDCG@10 with the graph was not measured (the full extraction).
  • Answer accuracy with the graph was not measured on the docs: only five of the 20 two-section questions have the facts of a right answer, and their sections are found without it.

So --rag-graph stays off: on these documents it is a few points of recall@5 on two-part questions, a loss on single-section ones, and hours of extraction; it pays where facts chain through names a question does not give, as in the synthetic set.

Self-critique

After the answer, the generator checks it: each statement that cites a source is asked against every source it cites ("Does the passage support the statement?"), and the answer against the question ("Does the answer address the question?"), both yes / no decisions by scoring (decide.h, one forward pass each). When a cited statement is not supported (a probability of yes under 0.5, or a citation of a number the prompt did not give), the answer is generated once more with the critique fed back:

Your previous answer was checked against the sources. These statements are not supported by the source they cite:
- "The pool holds 65,536 tokens per block." [3]
Answer the question again. State only what the numbered sources say, cite after each statement a source that says it,
and leave out anything they do not state. If the sources do not answer the question, say so.

The new answer is checked the same way and the one with the larger share of supported (statement, source) pairs is kept (the new one on a tie); when it still has an unsupported statement, it is flagged: in the record, and on stderr in oklm run. Usefulness is scored and recorded, not acted on.

oklm run -m MODEL --rag docs.gguf --rag-critique on -p "How do I keep the prefix cache across restarts?"
oklm serve -m MODEL --rag docs.gguf --rag-critique on   # or per request: "rag": {"critique": true}

on or flag. A retry is possible only while the answer has not been shown. on holds the answer until it is checked: oklm run and oklm chat generate it quietly, retry when a statement is unsupported, and print the one kept. flag lets the answer stream as usual; it is checked once complete and the verdict follows it, with no retry (a streamed answer cannot be taken back, and a second answer printed after it is a second answer, not a correction). Both print a line on stderr after the answer: (critique: 4 of 5 cited pairs supported, useful 0.98; answered again, the new answer kept), or ...; FLAGGED: statements not supported by their sources. The run's line in --rag-log gains a critique member (retried, flagged, chosen, the passes with their pairs, statement, cite, p_supported, p_useful).

oklm chat -m MODEL --rag docs.gguf --rag-critique flag   # replies stream; a verdict line after each retrieved turn
oklm run -m MODEL --rag docs.gguf --rag-critique flag -p "..."

In oklm chat every turn that retrieved is critiqued, except a reply that is a tool call; the critic uses the chat's context, so the next turn evaluates the conversation again instead of reusing the cache.

In the server. The retry of a whole answer is the request's prompt with the critique after its user message, decoded greedily through the scheduler; answers that may be tool calls are not critiqued. A request opts in with "rag": {"critique": true} or {"critique": {"retry": true|false}}; --rag-critique on|flag makes it the default (flag: no retry of whole answers either). The answer's rag member carries the record as critique, a top-level critique member the verdict on the answer kept:

"critique": {"flagged": true, "retried": false, "chosen": 0, "supported": 0.75, "pairs": 4,
  "unsupported": [{"statement": "The pool holds 65,536 tokens per block.", "cite": 3}], "useful": 0.97, "ms": 412.3}

A stream is sent as it is generated; the critique runs on its whole text, and the verdict rides on the last chunk of a chat completion (the one with the finish_reason, beside rag) or on the message_delta of an Anthropic stream. The stream's answer is never retried unless the request asks with "critique": {"retry": true}: then, when a statement is unsupported and the second answer is kept as better, it comes as one more event, a chat.completion.critique_retry object (its choices[0]. message, before data: [DONE]) or a critique_retry event (content blocks, before message_stop), with the verdict on it. A client that does not ask gets the flag only. /api/stats counts critiqued, critique_retries and critique_flagged, streams included.

Self-critique in the agent

oklm agent -m MODEL --rag-project --rag-critique on|flag checks each final answer (a turn without tool calls) that cites [n]: against the passages its message retrieved, every Retrieve call's and those given with the message (--rag-inject). Each retrieval numbers its sources from 1, so a citation [n] is checked against source n of each of them and counts as supported when one of them supports it (the last eight retrievals of the message are kept). The critic is the in-process model on the retrieval's own context (the agent's cache is not touched).

With on, an unsupported statement sends the agent back once: the critique (the unsupported statements, then "Answer the question again. State only what the numbered sources say...") is added as a reminder, as a blocking Stop hook's reason is, and the loop goes on; the model may answer from what it has or call Retrieve again. The second answer is checked too and flagged when still unsupported; it is not sent back again. With flag, the answer is only flagged. The loop was chosen over a quiet regeneration because the answer has already streamed to the transcript, a reminder is how the loop already sends the model back (Stop hooks), and the model keeps its tools for the retry. Each verdict is a notice in the transcript (critique: 1 of 3 cited pairs supported, useful 0.91; answering again with the critique), a critique line of the debug log (--debug) and an agent-critique line of --rag-log (attempt, retry, question, runs, critique). With --url the critic is the served model, through /v1/decide on an OKLM server (On a served model); with --mock there is no model to judge with, and the agent refuses --rag-critique.

Reflection tokens. Self-RAG has the model write [Retrieve], [IsRel], [IsSup] and [IsUse] tokens while it answers; that needs a model trained to write them, and none of the models OKLM runs is. The same judgements are asked after the answer instead, of any chat model.

In the library (include/oklm/rag/critique.h): oklm_rag_critique_run(params, critic, retrieval, question, answer, regenerate, user, &critique) is the state machine; the critic is a stage (oklm_rag_critic: support and useful), the generator's on a context is oklm_rag_model_critic(); the regeneration is the surface's callback (it knows its prompt); oklm_rag_critique_json() is the record.

Self-critique, measured

On 2026-11-09, as Retrieval in generation, measured: Qwen3-8B Q4_K_M on OpenCL behind oklm serve --rag, the documentation set's index (bge-small-en-v1.5, hybrid), 5 passages, corrective retrieval, the same 38 questions (30 answerable, 8 not), temperature 0, thinking off; python3 tools/rag_gen_eval.py --modes crag,critique, one run. Faithfulness as there: each sentence citing [n] judged against each source it cites by the served model ("Does the passage support the statement?").

Mode Accuracy Answerable Unanswerable (abstained) Faithfulness (cited pairs supported) Invented citations s / q
corrective 92.1 % (35 / 38) 27 / 30 8 / 8 87.5 % (98 / 112) 0 2.7
corrective, self-critique 92.1 % (35 / 38) 27 / 30 8 / 8 96.1 % (98 / 102) 0 2.4
  • 38 answers critiqued, 11 answered again (the new answer kept in 9), 4 flagged. The same 98 pairs are supported; the 10 pairs gone are unsupported statements the retry left out or rewrote: "The sources that support this are and." (twice), a command line the sources do not give, a paged block said to hold 65,536 tokens (the pool's size, rewritten "65,536 tokens in 4,096 blocks"), a claim about checked embedding models cited to the wrong source.
  • Accuracy is unchanged: the same 35 answers state every fact, no right answer was lost by the retry.
  • The measure and the critic ask the same question of the same model (the evaluation's judge is the support question by decide.c through /v1/systemone): the critique removes what that judge rejects, so the figure overstates what a reader would see. Read by hand, the statements the retries removed in the examples above were indeed unsupported; on the other side, one flagged answer is flagged for "only Ogg Theora video is decoded natively", a paraphrase of its source that the judge rejects (the strictness on paraphrase noted there). Earlier hand checks put this judge within about ±15 points of a reader.
  • Cost: four to eight more scoring requests an answer, and a second generation for 11 of 38; the time a question was within the noise of one run (2.4 s against 2.7 s).

Self-critique on streams and in the agent, measured

On 2026-11-09, Qwen3-8B Q4_K_M on OpenCL, single runs, thinking off.

Streams. The 38 questions through oklm serve --rag over the docs set's index, corrective retrieval, 5 passages, temperature 0, "critique": {"retry": false}: each asked whole, then streamed (python3 tools/rag_stream_critique_eval.py).

Questions Same answer text Same verdict, same answers Pairs (same answers) Unsupported / flagged (same answers) Added after the last token: median, p90, max Critique time, median
38 19 19 of 19 45 4 / 3 280 ms, 562 ms, 891 ms 261 ms
  • Where the streamed answer is the whole answer's text, its verdict is the same: the same pairs, the same unsupported statements and citations, the same flag. The stream's critique is the whole answer's code path on the same text.
  • 19 of 38 answers differ between the two requests at temperature 0, from the first diverging word on (one example: "On any address other than a loopback address" against "On any other address (non-loopback)"). The second request finds the prompt in the prefix cache and evaluates its last rows in another piece shape, which the OpenCL products round otherwise; the critique is not involved (it runs after the last token). With oklm serve --deterministic 38 of 38 answers and verdicts are the same, the critique's median 816 ms (server.md). Over all 38 the verdicts differ only where the answers do (24 identical verdicts; flagged 9 whole, 11 streamed).
  • The latency a stream gains is the time between its last content chunk and the chunk with the finish reason, which waits for the critique: 280 ms median (one to five support questions and the usefulness question, as scoring requests of the scheduler). The tokens before it are not delayed.

The agent. The 8 retrieval tasks (tests/evals 18 to 25) with the Retrieve tool (--rag-project --rag-memory), without and with --rag-critique on; retries and flags counted from --rag-log's agent-critique lines.

Passed Turns (sum) Prompt tokens Seconds Answers critiqued Retried Flagged
without 6 / 8 28 155,087 164
--rag-critique on 6 / 8 32 181,549 184 1 0 0
  • One final answer of eight cited [n] (3 pairs, all supported, checked in 386 ms); the others act (edit, run, commit) and report without citing, so they are not critiqued. No answer was sent back or flagged: the differences between the rows (18 passed with the option and failed without, 22 the reverse, four more turns, 17 % more prompt tokens) are single-run noise, not the critique, which added no turn here.
  • On these tasks the critique costs nearly nothing and changes nothing; it acts on answers that cite, which in the agent are the question-answering ones.

Figures in the index

Charts and diagrams hold facts the text around them does not repeat. With --caption-model, oklm rag index captions the figures of the documents it indexes with a vision model and indexes each caption as a text chunk of its document, pointing at its figure:

oklm rag index docs report.pdf --out docs.gguf -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf \
    --caption-model models/smolvlm2-2.2b/SmolVLM2-2.2B-Instruct-Q8_0.gguf
oklm rag search docs.gguf "revenue of the North region" --json   # a caption's hit carries "image"
  • Which figures. An image file (.png, .jpg, .gif, .bmp, ...; without a captioner it is skipped as before); the local images a Markdown file shows (an image link: ![ALT] then its path in parentheses) and an HTML file shows (<img src alt>), resolved beside the document (remote and data: ones skipped); the images a DOCX file's relationships name; the image XObjects of a PDF's pages (JPEG through the image decoder, 8-bit gray, RGB, CMYK and ICC-based samples after Flate and PNG predictors). At most 64 a document, smaller than 32 pixels skipped; an image that cannot be read or captioned is counted and skipped.
  • The caption. The vision model is asked "Describe this figure in detail for a search index. Say what kind of figure it is and read out its title and axis labels. Then go through every bar, point, slice or box one by one, and for each give its label and the number written on or next to it." and decodes greedily (at most 256 tokens). The figure and the question are joined as oklm describe joins them (multimodal.md): since 2026-11-14 the question comes right after the figure for SmolVLM and Gemma 3, as their templates write it, where a new line stood between them. The figures of this page were measured with the new line and not again: a caption made since may differ by some words. --caption-prompt TEXT (or @FILE, the file's text) asks something else: the default suits SmolVLM2-2.2B and costs the small models answers (below), so with SmolVLM-256M or 500M the earlier prompt is the one to pass: --caption-prompt "Describe this figure for a search index. Say what kind of figure it is and what it shows, then read out its title, axis labels, legend and every label and number written on it.". Its projector is --caption-mmproj F, else the one mmproj* file beside the model.
  • Which model. Any vision model whose projector OKLM's vision encoder implements (SmolVLM / Idefics3, Gemma 3, Qwen2-VL, LLaVA: see multimodal.md); captions were measured with the SmolVLM models only. SmolVLM2-2.2B-Instruct Q8_0 with its F16 projector (models/smolvlm2-2.2b/, pinned in tests/public_models_large.txt, fetched by sh tools/fetch_models.sh tests/public_models_large.txt) is the recommended one: 18 of 20 chart questions answered against 9 with SmolVLM-256M, for about 25 s a figure on the CPU against 4 (measured). --gen-backend opencl (or vulkan) runs its language model on the GPU, and the blocks of its image encoder with it (multimodal.md): the 9 figures of the set below in 10.1 s on OpenCL and 7.6 s on Vulkan, against 176 s on the CPU (measured). An encoder the device has no room for computes on the CPU, with a notice (the vision encoder runs on the CPU).
  • The chunk. Text Figure: NAME (REF, page N) then the caption, NAME the alternative text or title the document gives the figure (else its file name), heading Figure: NAME, line the line where the document shows it, page its PDF page. The index records the figure (chunk.image: FILE, FILE#page=N&image=NAME, FILE#MEMBER); oklm rag search --json prints it as image, oklm rag info counts the figures and names the captioner. The caption is embedded and in BM25 like any chunk, so it is found by meaning or by its words, and the generator reads it as a source.
  • An update keeps a document's captions with its other chunks (by the document's hash); another captioner, another --caption-prompt (the index records it as rag.caption.prompt, the default as nothing), or no captioner, chunks every source again. An image a Markdown page shows is captioned again only when the page changes.

In the library (include/oklm/rag/caption.h): the captioner is a stage (oklm_rag_captioner, oklm_rag_index_params.caption; its prompt is handed to each caption, OKLM_RAG_CAPTION_PROMPT when NULL), the vision model's is oklm_rag_vlm_open() and oklm_rag_vlm_captioner() (oklm_rag_vlm_encoder_device() puts its image encoder on the model's device or back on the CPU, oklm_rag_vlm_encoder_status() says where it is); oklm_rag_figures_find() lists a document's figures, and oklm_ingest_pdf_images() (ingest.h) a PDF's page images.

Figures in the index, measured

On 2026-11-09: the figure set of tests/rag/figures_set/ (tools/rag_figures_set.py, committed): five documents whose numbers are only in their figures, matplotlib charts with their values printed on them: a Markdown report showing a bar chart and a line chart, a pdflatex PDF with a pie chart, a pandoc DOCX with a horizontal bar chart, an HTML page with a bar chart, and a standalone diagram (PNG); 20 questions, each answered by one figure (a value, the largest or smallest, a step of the diagram). Index: bge-small-en-v1.5 F32, hybrid; the captions by SmolVLM-256M-Instruct Q8_0 on the CPU: 6 figures in 25 s. Answers by Qwen3-8B behind oklm serve --rag (5 passages), right when they state the figure's value. python3 tools/rag_set_eval.py --set figures, one run.

Index figure's caption in top 5 right, naive retrieval right, corrective
text only - (no figure indexed) 0 / 20 2 / 20
captions indexed 20 / 20 9 / 20 6 / 20
  • Retrieval is not what misses: every question found its figure's caption among the five passages. What misses is the caption: the 256 M model reads some charts fully (the battery chart: all four capacities, so 4 of 4 right; the diagram's first step) and others not at all (the pie chart: "The energy mix of Velmor is 38%."; the temperatures: "Average yearly temperature by city is 19.7"; the revenue bars: only the axis range). The two right answers of the text-only index are the model's own guesses ("Embed" after "Chunk", the pipeline's next step in general), not read from anything.
  • Corrective retrieval answers fewer than naive retrieval here: the judge calls a vague caption irrelevant ("the graph shows quarterly revenue by region", nothing about the North's figure), and the answer becomes "no reliable source was found".
  • No joint text-image encoder. The design kept SigLIP's text tower for when captions are not found; here they are always found (20 of 20 in the top 5). A joint encoder retrieves an image by its look, which does not read the numbers a question asks for; what would raise the 9 of 20 is a better reader of charts (a larger vision model at index time, or the image given to a vision-language generator at answer time), not another retriever. So it was not built.

Figures in the index, on a device

With --gen-backend opencl|vulkan the captioner's language model was on the device and its image encoder on the CPU, which was then most of a caption's time. Since 2026-11-14 the encoder's blocks follow the model (oklm_vision_use_backend(), as oklm describe and oklm serve do). The whole oklm rag index of the figure set with SmolVLM2-2.2B-Instruct Q8_0 and the default prompt (9 figures, bge-small-en-v1.5 F32; RX 7900 XT, another job on the CPU, one run each; the captioning, then the command):

--gen-backend encoder on the CPU (before) encoder on the device measured again on 2026-11-15
cpu 176.2 s, 177.1 s not run
opencl 98.3 s, 99.7 s 10.1 s, 11.8 s 9.04 to 9.07 s, 10.7 s
vulkan 95.4 s, 96.8 s 7.6 s, 8.7 s 7.33 to 7.39 s, 8.44 to 8.55 s

The last column is two indexes built from nothing on each backend (load average 1.5 to 4.4), with the whole encoder on the device (its patch embedding and projector too), Vulkan's products at F32's precision and the question written right after the image, as SmolVLM's template has it; 9 figures captioned each time, none skipped. The same command run again onto the index it had just written captioned no figure (0.9 to 1.4 s for the command).

On each backend the index is the one built with the encoder on the CPU: the five hits of the set's 20 questions are the same chunks in the same order, and the captions those hits show (6 of the 9) are the same text. Between backends the captions differ, as they did before (the language model on a device against the CPU: 4 of those 6 captions are the CPU's on either device, and 12 (Vulkan) or 9 (OpenCL) of the 20 lists of hits); the answers through the generator were not measured again, nor the captions' texts and the lists of hits with the day's last builds.

Larger caption models, and the prompt. The same set, index and answering model (Qwen3-8B Q4_K_M on OpenCL behind oklm serve --rag), with SmolVLM-500M-Instruct and SmolVLM2-2.2B-Instruct (Q8_0, F16 projectors, from ggml-org's repositories, pinned in tests/public_models_large.txt) and two caption prompts: the earlier one ("... then read out its title, axis labels, legend and every label and number written on it") and the current default, which asks for the title and axes, then each bar, point, slice or box with its label and number. Index time is the whole oklm rag index on the CPU (9 figures: the Markdown and HTML pages' images are captioned once as the page's figures and once as image files), with other jobs sharing the machine. "Declined" counts corrective answers that begin "No reliable source". One run each.

Captioner Prompt Index time Caption in top 5 Right, naive Right, corrective Declined (corrective)
SmolVLM-256M earlier 25 s (6 figures) 20 / 20 9 / 20 6 / 20 -
SmolVLM-256M current 39 s 20 / 20 7 / 20 6 / 20 6
SmolVLM-500M earlier 49 s 20 / 20 9 / 20 10 / 20 9
SmolVLM-500M current 47 s 20 / 20 4 / 20 6 / 20 11
SmolVLM2-2.2B earlier 189 s 20 / 20 11 / 20 10 / 20 6
SmolVLM2-2.2B current 251 s 20 / 20 18 / 20 17 / 20 2
  • SmolVLM2-2.2B with the current prompt reads every value of the bar, battery, temperature and pie charts and the four steps of the diagram; what it still misses is the line chart's months between January and June ("starts at 120,000 visitors in January and increases steadily to 300,000 in June": March guessed) and the pie's order (all four shares right, but "the smallest segment ... hydro energy at 20%", so the smallest source is answered as hydro instead of gas). With the earlier prompt the same model often gave the title and axes without the values ("line graph" for the line chart, no capacity for the batteries, only the axis range for the temperatures): the prompt is worth 7 answers to it, the model 9 over the 256 M one.
  • The two small models do worse with the current prompt (256M 7 against 9, 500M 4 against 9): asked to go mark by mark, they invent labels or drop the numbers (500M on the pie: four slices named, no percentage). The default was chosen for the recommended captioner; with a small one, --caption-prompt gives it the earlier prompt back.
  • The corrective judge's rejection of vague captions mostly goes away: with complete captions, 2 answers are declined instead of 6 to 11, and corrective retrieval lands one answer under naive retrieval instead of three. The two still declined are the line chart's March (the caption gives no March value) and the pie's smallest share (the caption contradicts its own percentages).
  • Retrieval never missed (20 of 20 in the top 5 for every captioner), so a joint text-image encoder would not have changed these answers.

Tool selection by retrieval

An agent with a hundred tools pays for them in its prompt, or hides them behind ToolSearch, which a small model rarely calls. oklm agent --tool-rag K (or the toolSearch.topK setting; off by default) picks them by retrieval instead.

What is indexed. Every tool the agent may be offered (built-in, the program's, every MCP server's), its model-invocable skills and plugin commands, and its sub-agent types; each entry is one text (the short name, where it comes from, the description cut at 1500 bytes, the parameter names), grouped Type (built-in, MCP, skill, plugin, sub-agent) to Service (the MCP server, the plugin) to Tool, and stored as a source of a RAG index (above) recorded as TYPE/SERVICE/NAME with the hash of its text. The index file is ~/.local/state/oklm/rag/tools-HASH.gguf (HASH of the project's path): a description that did not change keeps its vector across runs, and before each retrieval the agent stages its current set and commits it, which costs a hash when nothing changed and embeds only the changed entries when a server, a skill or a plugin came or went.

The ranking. Hybrid, as oklm rag search does it: the query embedded by toolSearch.model (a path, a file name looked for as the agent looks for models, or none; bge-small-en-v1.5-q8_0.gguf by default, on the CPU beside the generator, 512 tokens) and compared with every entry, BM25 over the same texts, the two lists fused by reciprocal rank (k = 60). toolSearch.mode dense ranks by the cosine alone, bm25 by BM25 alone (the cosine still computed, for the floor below); hybrid is the default. Without the model (none, or not found) the ranking is BM25's alone, whatever the mode. A grouped ranking (services ordered by their best entry, a service's entries together) exists in the library (oklm_tool_index_params.grouped) and is off: it measured below the flat one.

What the model gets. With --tool-rag K every MCP tool is deferred, and the system prompt counts each server's tools (MCP server github: 8 tools deferred ...) instead of naming them. Before each user message's first turn, the message and the previous one (2000 bytes at most) are the query; of the K best entries, the deferred tools whose cosine with the query reaches toolSearch.minScore (0.62) are given to the model: their definitions in a reminder after the message for the in-process model (its prompt, and so its KV cache, unchanged), offered in the tools array for a server (those of the previous message no longer). A request the core tools serve adds nothing. ToolSearch stays, and its word queries are ranked by the same index over the same candidates as before (deferred tools and skills); select:NAME is unchanged. Sub-agents do not retrieve.

Inspecting it. With --debug, each decision is a line of the debug log:

tool index: 52 entries, 52 embedded, 0 kept, 0 dropped (hybrid, 0.25 s)
tool retrieval: query "Use the add tool of the fake MCP server to add 123 and 456, ..." (hybrid); ranked: mcp__fake__add 0.0328 (dense 0.784 #1, bm25 14.34 #1), mcp__fake__big 0.0320 (dense 0.709 #2, bm25 9.12 #3), ...; chosen: mcp__fake__add, mcp__fake__big, mcp__fake__echo, mcp__fake__change, mcp__fake__slow
tool retrieval: query "Rename the function compute_total to sum_prices ..." (hybrid); ranked: ...; chosen: none (below the floor 0.62: EnterWorktree, LSP, ScheduleWakeup, TodoWrite, Grep)

In the library: include/oklm/harness/tool_index.h (oklm_tool_index_stage(), oklm_tool_index_stage_json(), oklm_tool_index_commit(), oklm_tool_index_search()), and oklm_agent_params.tool_index / tool_rag_k; the decisions arrive as OKLM_AGENT_EV_LOG events.

Tool selection, measured

make tool-eval (tests/rag/bench_tool_select.c; TOOL_EVAL_EMBED, TOOL_EVAL_TOKENIZER, TOOL_EVAL_K) builds the index the agent would: its 27 built-in tools, its 12 model-invocable built-in skills, its 4 sub-agent types and the 62 MCP tools of tests/rag/tool_zoo.json (the 13 tools of tools/mcp_zoo.py and nine servers of common kinds: GitHub, calendar, mail, drive, SQL, chat, maps, finance, tasks, translation), 105 entries; then ranks them for the 152 requests of tests/rag/tool_queries.jsonl, each labelled with the tool or tools it needs. bge-small-en-v1.5 Q8_0 on the CPU, tokens counted with Qwen3-8B's tokenizer, 2026-11-09:

Ranking top-1 recall@1 recall@3 recall@5 recall@10 MRR
keyword, the request as query (ToolSearch today) 27.0 % 25.7 % 37.8 % 46.6 % 58.4 % 0.376
keyword, stop words dropped 44.7 % 41.8 % 55.4 % 60.3 % 69.2 % 0.538
BM25 (toolSearch.model none) 53.3 % 49.9 % 62.9 % 70.8 % 77.1 % 0.620
dense 57.9 % 55.2 % 74.8 % 81.0 % 85.9 % 0.692
hybrid (RRF), the default 61.8 % 58.1 % 69.8 % 76.4 % 81.7 % 0.692
hybrid, grouped by service 61.8 % 58.1 % 67.2 % 72.7 % 75.3 % 0.676

The keyword rows score every entry as ToolSearch scores the deferred ones (3 a word in the name, 1 in the text): with the request's own words, articles and verbs match everywhere. Hybrid ranks the right tool first 2.3 times as often; dense alone keeps more of a request's tools in the first 3 to 10 (several MCP tools of one request often share no word with it, which pulls BM25's half of the fusion down); grouping only lowers recall, so it stays off.

What an in-process model is given (tokens; the skills' list, the same in every row, left out):

Prompt system prompt tool definitions per request total request's tools defined without ToolSearch
every tool offered 830 7376 0 8206 100.0 %
deferred, named (ToolSearch today) 1683 2295 0 (+318 a ToolSearch round) 3978 (+318) 26.6 %
deferred, --tool-rag 5 1471 2295 136 3902 68.5 %
deferred, --tool-rag 5, toolSearch.mode dense 1470 2295 177 3942 72.5 %
deferred, --tool-rag 5, toolSearch.mode bm25 1470 2295 105 3870 67.2 %
deferred, --tool-rag 8 1471 2295 170 3936 69.8 %

The last column counts a request's tools that are always there (the core) or given with it: today a deferred tool needs a ToolSearch call the model must think of (a round of 318 tokens when it does), with retrieval 68.5 % of the tools are defined from the first turn, for fewer tokens than naming every deferred tool. The cosine floor (toolSearch.minScore) trades that share against the definitions given for nothing (top 5):

Floor given a request of which not needed request's tools defined
none 3.19 2.62 78.1 %
0.60 2.04 1.53 73.5 %
0.62 1.66 1.20 68.5 %
0.66 0.86 0.47 61.0 %
0.70 0.36 0.12 48.4 %

The ranking (toolSearch.mode, 2026-11-09, make tool-eval: the benchmark runs the index's own search once a mode). The accuracy rows are those of the first table (BM25, dense, hybrid); what each gives at top 5:

Mode floor given a request of which not needed request's tools defined
hybrid 0.62 1.66 1.20 68.5 %
hybrid 0.60 2.04 1.53 73.5 %
dense 0.62 2.28 1.78 72.5 %
dense 0.64 1.67 1.22 67.9 %
bm25 0.62 1.24 0.80 67.2 %

Dense keeps more of a request's tools in its top 5 (81.0 % against 76.4 %), but the floor then lets more definitions through, most of them not needed: at the same number given a request (dense at 0.64, hybrid at 0.62) it defines no more of the request's tools, and hybrid with a floor of 0.60 beats dense at 0.62 on both counts. Hybrid also ranks the right tool first more often (61.8 % against 57.9 %), which ToolSearch's word queries use. So hybrid stays the default and the agent evals were not run again; dense and bm25 are there for a tool set where that balance differs.

The agent evals (make eval, the 17 tasks, Qwen3-8B Q4_K_M, in-process; the tasks have no MCP server but 03-mcp-tool's fake one), each run once, 2026-11-09:

Run passed turns a task tokens in all
off 13 of 17 5 526,676
off, again 12 of 17 4 425,197
--tool-rag 5, final (floor 0.62, tools only) 12 of 17 6 729,730
--tool-rag 5, skills suggested, floor 0.62 12 of 17 6 670,529
--tool-rag 5, no floor, skills suggested 11 of 17 7 777,367

Which tasks fail changes from run to run in both modes (sampling): the two runs without retrieval fail 4 and 5 tasks, only two of them in common. Without a floor every message got two or three deferred tools it did not need (EnterWorktree for a rename, database for a C test) and one run looped for 32 turns; with the floor 11 of the 17 messages get nothing and see the very prompt of a run without retrieval, 03-mcp-tool gets the fake server's tools (passed in every run), and the git tasks get EnterWorktree or LSP. The final configuration passes as many tasks as the second run without it, one fewer than the first: unchanged within the noise of single runs, not shown better on tasks that hardly use deferred tools. Building the index for an agent's 43 to 52 entries takes 0.22 to 0.25 s on the CPU, each message's query a few milliseconds.