Retrieval (RAG)
Retrieval-augmented generation in OKLM: text embeddings, documents
cut into chunks and kept in one index file, searched by vectors, by
keywords or both, optionally reranked; then answers in run,
chat, serve and the agent written from the
passages found and citing them, with the model judging what it
retrieved. The agent also retrieves its tools. This page is the user
guide, then the evaluation and the residual risks, then the reference: how each
part works and its own measures. The design and its decisions are in rag-design.md, the index file format in adr/0022-rag-index-format.md.
Status: maintained. Every capability below is implemented, tested in
make test and measured (2026-11-08 to 2026-11-09).
Quick start
# an index of the docs, with an embedding model (bge's query instruction recorded in the index)
oklm rag index docs README.md --out docs.gguf -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf \
--query-prefix "Represent this sentence for searching relevant passages: "
oklm rag search docs.gguf "how is the KV cache quantized" -k 5 # hybrid: vectors and BM25 fused
oklm run -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --rag docs.gguf \
--template-vars '{"enable_thinking": false}' -p "How do I keep the prefix cache across server restarts?"
The answer cites its sources as [n], and a summary line
on stderr says what was retrieved and judged. Run
oklm rag index again after the files change: only the
changed files are embedded again.
The same from a window: oklm-gui --tool rag builds and
updates an index with its progress, shows which sources are stale,
searches it, runs the evaluation and starts a chat window on it (gui.md).
Commands
| Command | What it does | Details |
|---|---|---|
oklm embed -m MODEL TEXT... (--file F,
--json, --query, --query-prefix,
--passage-prefix, --pooling) |
one vector a text: bert and nomic-bert
encoders, Qwen3-Embedding; GGUF or a Hugging Face checkpoint |
Embeddings |
oklm rag index PATH... --out INDEX [-m MODEL] [--progress-json FILE|-] |
read, chunk, embed and write one GGUF index; run again, it updates by file hash; its progress as JSON lines | Chunking and the index, Output for programs |
oklm rag index ... --graph --gen-model M |
also extract each chunk's entities and relations with a chat model | Graph retrieval |
oklm rag index ... --caption-model VLM |
also caption the documents' figures with a vision model, each caption a chunk | Figures in the index |
oklm rag search INDEX QUERY [-k 10] [--mode dense|sparse|hybrid] [--fusion rrf|weighted] [--rerank R] [--rag-graph on] [--json|--json-full] |
search an index | Hybrid search, Output for programs |
oklm rag info INDEX [--sources] [--json] |
the encoder, the chunker, the counts, the graph and figures; with both options each source and whether its file changed since | The index file, Output for programs |
oklm rag eval DIR --index INDEX [-m MODEL] |
nDCG@10, MRR@10, recall@k and latency per mode on a BEIR-format set | The retrieval evaluation |
oklm rerank -m R -q QUERY PASSAGE... [--file F] [--top N] [--json] |
passages scored against a query by a cross-encoder (or a chat model), best first | Reranking |
Without -m, oklm rag index uses the hash
encoder (a hashed bag of words, no model file): fine for tests and
keyword-like lookups, not for meaning. An index remembers its embedding
model; search, updates and --rag reload it and
refuse another one.
Answers from an index:
--rag
oklm run, oklm chat,
oklm serve and oklm agent take
--rag INDEX and these options (in serve they
are the server's defaults, which a request's rag field
changes):
| Option | Default | |
|---|---|---|
--rag INDEX |
off | the index; its embedding model is loaded on the CPU to embed the question |
--rag-dir DIR |
none | oklm serve only: a directory of indexes a request may
name, "rag": {"index": "NAME"}; the settings below go with
it as they go with --rag (server.md) |
--rag-k N |
5 | passages retrieved (1 to 50) |
--rag-mode dense|sparse|hybrid |
hybrid (dense without BM25 postings) | the search |
--rag-crag on|off |
on | corrective retrieval: each passage judged, the question rewritten
when none is relevant, "no reliable source" rather than a guess (Corrective retrieval); off
puts the top k passages in the prompt as they are |
--rag-rewrite on|off |
on | a follow-up rewritten as a standalone query from the conversation
before it is searched (chat, serve, the
agent's passages per message; Conversational retrieval) |
--rag-followup rewrite|join|both |
both | what a follow-up searches: the previous question and the rewrite
(both), the rewrite alone, or the previous question and the
follow-up as written (join, no generation) |
--rag-route on|off |
off | the model decides between no retrieval, one search and several (multi-hop sub-questions; Adaptive retrieval) |
--rag-rerank MODEL,
--rag-rerank-top N |
none, 20 | the candidates reordered by a reranker before the best
k are kept (Reranking
the candidates) |
--rag-graph on|off |
off | the index's graph walked and fused with the search (an index built
with --graph) |
--rag-critique on|flag|off |
off | the answer checked against its sources; on holds it and
generates it once more when a cited statement is unsupported,
flag lets it stream and only flags it (run,
chat, serve, and the agent, in process or on
--url; Self-critique) |
--rag-log FILE |
none | every retrieval appended as one JSON line |
What the model is given, and each stage of the pipeline, are in Retrieval in generation.
The server
oklm serve adds three things (server.md):
POST /v1/embeddingswith--embed-model EMB: OpenAI's request and response (In the server).POST /v1/rerank(and/rerank) with--rerank-model R: Cohere's and Jina's shape (query,documents,top_n,return_documents; Reranking, in the server).- With
--rag INDEX, a chat completion or an Anthropic message opts in with itsragfield:"rag": truetakes the server's settings,"rag": {"k": 8, "crag": false, "route": true, "rewrite": false, "followup": "join", "graph": true, "critique": true, "max_tokens": 1024}changes them for the request. The answer carries the run in aragmember (streamed: on the last chunk, or the Anthropic stream'smessage_delta), with the texts of the sources cited;/api/statscounts the runs (Retrieval in generation, in the server).
The agent
| Option | Setting | Default | |
|---|---|---|---|
--rag-project |
rag.enabled |
off | the built-in Retrieve tool over an index of the project's documentation the agent builds and keeps up to date itself (Agentic retrieval) |
--rag INDEX |
none | Retrieve over a fixed index instead | |
--rag-memory |
rag.memory |
off | Retrieve's scope memory /
all: the project's earlier sessions and memory files |
--rag-inject always|route |
rag.inject |
off |
passages given with each user message; route asks the
router first |
rag.model |
bge-small-en-v1.5-q8_0.gguf |
the embedding model (hash: the hash encoder; the hash
encoder too when the file is not found) |
|
rag.queryPrefix |
none | put before each query | |
rag.include, rag.exclude |
the documentation globs, none | the files indexed | |
rag.index, rag.maxBytes |
~/.local/state/oklm/rag/project-HASH.gguf, 200 MB |
the index file, the bytes indexed at most | |
--tool-rag K |
toolSearch.topK |
off | tool selection by retrieval: every MCP tool deferred, the deferred tools that fit each message given with it (Tool selection by retrieval) |
toolSearch.model |
bge-small-en-v1.5-q8_0.gguf |
the tool index's embedding model (none: BM25
alone) |
|
toolSearch.minScore |
0.62 | the cosine a tool needs to be given | |
toolSearch.mode |
hybrid |
the tools' ranking: hybrid, dense or
bm25 (Tool selection,
measured) |
The --rag-* options of the table above apply to the
agent's retrieval too; --rag-critique checks its final
answers (Self-critique in the
agent) and needs a judge: the in-process model (-m) or
the served one (--url, On a
served model); with --mock there is none, and the agent
refuses it. Retrieve only reads: every permission mode runs it as it
runs Read, and --disallowedTools Retrieve removes it. With
--debug, each index update, retrieval and tool choice is a
line of the debug log.
Defaults, and why
| Capability | Default | Reason (measured) |
|---|---|---|
| Hybrid search | on | better than dense or BM25 alone in nDCG@10 on both sets (SciFact 73.04 against 71.27 dense, docs 70.61 against 64.53 BM25) |
| Weighted fusion, 0.5 / 0.5 | on (--fusion rrf the other) |
better than reciprocal rank fusion on both sets: 73.04 against 71.29 nDCG@10 on SciFact, 70.61 against 69.78 on the docs; other weights lose on one set or the other |
| Exact dense search, no approximate index | always | 27 ms a query over a million chunks, under the 100 ms that would call for one |
| Reranking | off | the MiniLM cross-encoder lowers SciFact nDCG@10 from 71.64 to 68.95 over the hybrid search, for 0.6 to 1.2 s a query on the GPU; on the docs it lifts recall@1 from 30.2 to 37.4 % (interval +0.5 to +14.0 points; nDCG@10 70.61 to 71.07, within the noise) and answers one more question of 30, while naive retrieval's faithfulness falls from 88.7 % to 81.1 % (interval −19.8 to +2.5) |
| Corrective retrieval | on | 92.1 % right against 89.5 % naive (one question; 86.8 % against 84.2 % on 2026-11-10, interval of the difference −5.3 to +10.5 points), 8 of 8 unanswerable questions declined against 7, 22 % fewer prompt tokens, for 0.6 s a question on the GPU (22 s on the CPU alone) |
Conversational rewrite, searched with the previous question
(--rag-followup both) |
on | recall@5 of a follow-up 83.3 % against 26.9 % searched alone, 71.1 % for the rewrite alone and 82.2 % for the previous question joined with the follow-up, for 0.1 s; it acts only when there are earlier turns |
| Router (adaptive retrieval) | off | chose "one search" for 38 of 38 questions: same answers, three scoring requests more; of three other wordings, the one that routes (few-shot examples) splits 10 single questions and answers 32 of 38 against 34; GLM-4.7-Flash splits 46 of 58 requests |
| Graph retrieval | off | on the docs' 20 two-section questions recall@5 93.6 % against 89.6 % (recall@10 100 % both), the single-section checks 85 % against 95 %, and the extraction costs 4.3 s a docs chunk with an 8 B model (about 3 hours for these docs); the synthetic set gains far more (28 of 30 pairs in the top 5 against 7), where every second fact is unnamed |
| Self-critique | off | faithfulness 87.5 % to 96.1 % as the same model judges it, accuracy unchanged; four to eight scoring requests an answer; a stream is flagged, not answered again, unless the client asks |
| Figure captions | off | 18 of 20 chart questions answered against 0 with SmolVLM2-2.2B captions, but it needs a vision model at index time (about 25 s a figure on the CPU) |
Retrieval in the agent (rag.enabled) |
off | indexing the project is the user's decision; on the documentation tasks 7 or 8 of 8 passed against 2 of 8 |
Passages per message (rag.inject) |
off | 8 of 8 with fewer tokens on tasks that all need the docs, but it adds passages to requests that need none |
Tool selection by retrieval (--tool-rag) |
off | top-1 61.8 % against 27.0 % for ToolSearch's keywords, but the agent evals (few MCP tools) were unchanged within single-run noise |
Tool ranking hybrid (toolSearch.mode) |
on (dense, bm25 the others) |
top-1 61.8 % against 57.9 % dense; dense defines more of a request's tools at the 0.62 floor (72.5 % against 68.5 %) only by giving more definitions (2.28 a request against 1.66): at the same number given (dense at 0.64), 67.9 % against 68.5 % |
RAG evaluation
Every capability before and after, on 2026-11-09 unless said. The
retrieval rows are reproduced by make rag-eval; every other
row is a single run of a set made for it, with Qwen3-8B
Q4_K_M on a Radeon RX 7900 XT (OpenCL) as the generator and the judge,
thinking off, temperature 0, and bge-small-en-v1.5 as the embedding
model. Each row links to the section that has the details; Reproducing it gives the commands.
Retrieval (Hybrid retrieval, measured). bge-small-en-v1.5 F32 with its query instruction. SciFact: BEIR's test split, 300 queries over 5,183 abstracts. Docs: 120 questions over OKLM's documentation (1,007 sections) frozen at a pinned revision.
| Mode | SciFact nDCG@10 | MRR@10 | R@10 | R@100 | Docs nDCG@10 | MRR@10 | R@10 | R@100 |
|---|---|---|---|---|---|---|---|---|
| dense | 71.27 | 68.22 | 83.6 | 94.2 | 60.48 | 62.79 | 77.1 | 99.0 |
| sparse (BM25) | 67.41 | 64.21 | 79.5 | 90.5 | 64.53 | 68.49 | 80.7 | 98.5 |
| hybrid, RRF | 71.29 | 67.85 | 83.7 | 97.2 | 69.78 | 73.08 | 85.2 | 99.6 |
| hybrid, weighted (the default) | 73.04 | 69.51 | 86.1 | 97.2 | 70.61 | 74.20 | 86.7 | 99.4 |
Dense SciFact equals MTEB's published 71.27 for the model. On a second public set, BEIR NFCorpus (323 medical queries over 3,633 documents, graded relevance), dense gives 34.38 against the model card's 34.31, BM25 32.09, hybrid 36.67 by reciprocal rank and 36.37 weighted (Hybrid retrieval, measured). On 112 questions over OKLM's own sources, the right file ranks in the top 10 for 81 % with bge-small (hybrid, 59.32 nDCG@10) and 85 % with Qwen3-Embedding-0.6B (63.63 dense); the code-aware chunker scores no better than the recursive one (Retrieval on code, measured).
Reranking (Reranking, measured). SciFact, bge-small's hybrid candidates (an index of 5,773 chunks of at most 500 tokens, so its "before" differs from the table above):
| Reranker | Candidates | Queries | nDCG@10 before | after | Added per query (GPU) |
|---|---|---|---|---|---|
| ms-marco-MiniLM-L-6-v2 | top 100 | 300 | 71.64 | 68.95 | 1231 ms |
| ms-marco-MiniLM-L-6-v2 | first 50 of the top 100 | 300 | 71.64 | 69.50 | 618 ms |
| bge-reranker-base F16 | top 50 | first 100 | 78.03 | 74.73 | 2612 ms |
| qwen2.5-0.5b-instruct through decide.c | top 50 | first 50 | 84.41 | 20.44 | 1039 ms |
Generation (Retrieval in generation, measured, Self-critique, measured, Adaptive retrieval, measured). 38 questions: 30 of the docs set, each with the facts a right answer states, and 8 the docs do not answer, right when declined. Faithfulness: cited (sentence, source) pairs the same model judges supported.
| Mode | Accuracy | Answerable | Declined | Faithfulness | Invented citations | Prompt tokens / q | s / q |
|---|---|---|---|---|---|---|---|
| no retrieval | 15.8 % | 4 / 30 | 2 / 8 | - | - | 31 | 4.4 |
| naive (top 5 in the prompt) | 89.5 % | 27 / 30 | 7 / 8 | 85.3 % | 0 | 1,100 | 1.6 |
| corrective (the default) | 92.1 % | 27 / 30 | 8 / 8 | 86.5 % (87.5 %, 89.2 % in two later runs) | 0 | 860 | 2.2 |
| corrective, routed | 92.1 % | 27 / 30 | 8 / 8 | 88.3 % | 0 | 860 | |
| corrective, self-critique | 92.1 % | 27 / 30 | 8 / 8 | 96.1 % | 0 | 2.4 |
Multi-hop (Graph retrieval, measured). 30 questions needing two articles of a synthetic encyclopedia (100 articles); the graph extracted by Qwen3-8B; retrieval with weighted fusion (the default), answers measured earlier with reciprocal rank:
| Graph | Both articles in top 5 | recall@5 | recall@10 | Right, naive (RRF) | Right, corrective (RRF) |
|---|---|---|---|---|---|
| off | 7 / 30 | 61.7 % | 78.3 % | 8 | 5 |
| on | 28 / 30 | 96.7 % | 100 % | 23 | 23 |
The 10 single-hop control questions: 10 of 10 right either way. On the documentation (the 20 two-section questions over an 82-section part of the docs set, weighted fusion):
| Graph | Two-section recall@5 | recall@10 | nDCG@10 | Single-section checks (5) recall@5 |
|---|---|---|---|---|
| off | 89.6 % | 100 % | 88.82 | 95.0 % |
| on | 93.6 % | 100 % | 88.49 | 85.0 % |
Figures (Figures in the index, measured). 20 questions answered only by a chart or diagram in five documents (Markdown, PDF, DOCX, HTML, PNG); captions by the vision model named (Q8_0), the recommended one with the current caption prompt:
| Index | Figure's caption in top 5 | Right, naive | Right, corrective |
|---|---|---|---|
| text only | - | 0 / 20 | 2 / 20 (guesses) |
| captions, SmolVLM-256M (earlier prompt) | 20 / 20 | 9 / 20 | 6 / 20 |
| captions, SmolVLM2-2.2B (recommended) | 20 / 20 | 18 / 20 | 17 / 20 |
Conversational (Conversational retrieval, measured). 30 follow-ups that do not name their subject, naive retrieval of 5:
| Query searched | recall@5 | hit@5 |
|---|---|---|
| the follow-up alone | 26.9 % | 11 / 30 |
| rewritten from the conversation | 71.1 % | 26 / 30 |
| previous question and follow-up joined | 82.2 % | 30 / 30 |
| previous question and the rewrite (the default) | 83.3 % | 30 / 30 |
| the original question (upper bound) | 84.4 % | 30 / 30 |
Tool selection (Tool selection, measured,
make tool-eval). 105 tools (27 built-in, 12 skills, 4
sub-agent types, 62 MCP tools), 152 labelled requests, bge-small-en-v1.5
Q8_0, tokens of Qwen3-8B's tokenizer:
| Ranking | top-1 | recall@5 | MRR |
|---|---|---|---|
| ToolSearch's keyword score | 27.0 % | 46.6 % | 0.376 |
| BM25 | 53.3 % | 70.8 % | 0.620 |
| dense | 57.9 % | 81.0 % | 0.692 |
| hybrid (the default) | 61.8 % | 76.4 % | 0.692 |
| Prompt | Tokens | Request's tools defined from the first turn |
|---|---|---|
| every tool offered | 8,206 | 100 % |
| deferred, named (ToolSearch) | 3,978 (+318 a ToolSearch round) | 26.6 % |
deferred, --tool-rag 5 |
3,902 | 68.5 % |
Agent evals (make eval, Qwen3-8B Q4_K_M
in-process, each run once). Tool selection, the 17 tasks of
tests/evals (Tool
selection, measured):
| Run | Passed | Turns a task | Tokens in all |
|---|---|---|---|
| off | 13 of 17 | 5 | 526,676 |
| off, again | 12 of 17 | 4 | 425,197 |
--tool-rag 5 (floor 0.62) |
12 of 17 | 6 | 729,730 |
Retrieval in the agent, the 8 tasks that need the project's docs or an earlier session (Agentic retrieval, measured):
| Mode | Passed | Turns a task | Tokens in all | Seconds in all |
|---|---|---|---|---|
| no retrieval | 2 of 8 | 7 | 503,121 | 438 |
passages with each message (--rag-inject always) |
8 of 8 | 3 | 137,045 | 151 |
Retrieve tool (--rag-project --rag-memory) |
7 of 8 | 4 | 184,240 | 192 |
routed passages (--rag-inject route) |
8 of 8 | 2 | 116,470 | 146 |
Latency and token cost per stage. i7-11700K (8 threads) and the RX 7900 XT; each figure from the section linked.
| Stage | Cost | Section |
|---|---|---|
| Embedding, bge-small F32 | 286 texts/s CPU, 731 GPU (24-token lines); SciFact's 290-token abstracts 16/s CPU, 46/s GPU | Embeddings, measured |
| One query: embed and search, SciFact | dense 3.1 ms p50, BM25 0.06 ms, hybrid 3.3 ms (GPU) | Hybrid retrieval, measured |
| Dense search alone, a million chunks (384 dims) | 27 ms p50 F16, 14.5 ms Q8_0 | The index, measured |
| Index build without embedding, a million chunks | 5.9 s (chunking, BM25, write); the embedding model's time dominates | The index, measured |
| Reranking (MiniLM) | 618 ms for 50 candidates, 1231 ms for 100 (GPU); 1.9 s, 3.8 s (CPU) | Reranking, measured |
| Generation from 5 passages | 1.6 s a question, 1,100 prompt tokens (naive); 20.9 s on the CPU alone | Retrieval in generation, measured, On the CPU alone |
| Judging the passages, rewrites | +0.6 to 1.1 s a question; prompt 22 % shorter (860 tokens); +22 s on the CPU alone | same |
| Follow-up rewrite | +0.1 s | Conversational retrieval, measured |
| Router | three scoring requests a question | Adaptive retrieval, measured |
| Self-critique | four to eight scoring requests an answer, a second generation for 11 of 38 | Self-critique, measured |
| Self-critique of a stream | 280 ms median after the last token; verdicts equal to the whole answers' on the same text (19 of 19) | Self-critique on streams and in the agent, measured |
| Graph extraction (index time) | 1.15 s a short synthetic chunk, 4.3 s a docs chunk (Qwen3-8B, GPU); the walk under a millisecond | Graph retrieval, measured |
| Captions (index time) | 9 figures in 250 s (SmolVLM2-2.2B, CPU; SmolVLM-256M 39 s) | Figures in the index, measured |
| The agent's indexes | the tool index of 43 to 52 entries 0.22 to 0.25 s, a few ms a query, 136 tokens a message at top 5; a 10-file project about 1 s (CPU) | Tool selection, measured |
Reproducing it
make rag-eval reproduces the retrieval table: both sets
(--sets scifact,nfcorpus,code,docs adds NFCorpus and the
code set), the four modes, bge-small-en-v1.5 F32, the outputs in
build/rag-eval/ (report.md, one table a set,
the rankings in TREC format). The published run embedded on the GPU:
make rag-eval RAG_ARGS="--backend opencl" # about 3 minutes; the indexes kept for the next run
make rag-eval RAG_ARGS="--reference" # also the Python metrics over the same rankings, and the embedding references
make tool-eval # the tool selection tables (CPU, about a minute)
The other rows need an 8 B model on the GPU and minutes to hours, so
they are commands, not targets. The generation and conversational rows
run against a server over the docs set's index that
make rag-eval built:
build/release/bin/oklm serve -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --backend opencl --port 7431 \
--rag build/rag-eval/docs-bge-small-en-v1.5-f32.gguf
python3 tools/rag_gen_eval.py --url http://127.0.0.1:7431 --modes none,naive,crag,route,critique
python3 tools/rag_followup_eval.py --url http://127.0.0.1:7431
python3 tools/rag_stream_critique_eval.py --url http://127.0.0.1:7431 # streamed verdicts against whole ones
Multi-hop and figures (each its own index, then a server over it):
B=build/release/bin/oklm E=models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf
Q="Represent this sentence for searching relevant passages: "
$B rag index tests/rag/multihop_set/corpus --out build/rag-eval/multihop.gguf -m $E --query-prefix "$Q" \
--graph --gen-model models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --gen-backend opencl
$B serve -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --backend opencl --port 7431 --rag build/rag-eval/multihop.gguf
python3 tools/rag_set_eval.py --set multihop --index build/rag-eval/multihop.gguf --url http://127.0.0.1:7431 --rag '{"k":5,"crag":false}'
python3 tools/rag_set_eval.py --set multihop --index build/rag-eval/multihop.gguf --search-args "--rag-graph on" \
--url http://127.0.0.1:7431 --rag '{"k":5,"crag":false,"graph":true}'
python3 tools/rag_docs_set.py --subset 20 --out build/rag-eval/docs-mh # the docs' two-section questions, 82 sections
for g in off on; do $B rag eval build/rag-eval/docs-mh --index build/rag-eval/docs-mh-graph.gguf -m $E --query-prefix "$Q" \
--backend opencl --graph --gen-model models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --gen-backend opencl \
--rag-graph $g --mode hybrid --split multihop; done # --split lookup: the single-section checks
$B rag index tests/rag/figures_set/corpus --out build/rag-eval/figures.gguf -m $E --query-prefix "$Q" \
--caption-model models/smolvlm2-2.2b/SmolVLM2-2.2B-Instruct-Q8_0.gguf
$B serve -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --backend opencl --port 7431 --rag build/rag-eval/figures.gguf
python3 tools/rag_set_eval.py --set figures --index build/rag-eval/figures.gguf --url http://127.0.0.1:7431 --rag '{"k":5,"crag":false}'
"crag":true gives the corrective columns; the text-only
figure row is the same index built without --caption-model.
The router's wordings (the server over any index, or none):
python3 tools/rag_route_eval.py --url http://127.0.0.1:7431.
The agent evals:
T=$(ls tests/evals | grep -E '^(0[1-9]|1[0-7])-' | paste -sd, -) # the 17 tasks of the tool selection rows
make eval EVAL_ARGS="--only $T --backend opencl" # retrieval off
make eval EVAL_ARGS="--only $T --backend opencl --tool-rag 5" # tool selection by retrieval
D=$(ls tests/evals | grep -E '^(1[89]|2[0-5])-' | paste -sd, -) # the 8 tasks that need the docs or a session
make eval EVAL_ARGS="--only $D --backend opencl --rag-project --rag-memory --rag-inject always --disallowedTools Retrieve"
make eval EVAL_ARGS="--only $D --backend opencl --rag-project --rag-memory" # the Retrieve tool
make eval EVAL_ARGS="--only $D --backend opencl --rag-project --rag-memory --rag-critique on --rag-log $PWD/critique.jsonl"
--rag-inject route --disallowedTools Retrieve gives the
routed row, and no --rag-* option the row without
retrieval.
Residual risks
- Cost. An embedding model makes indexing the slow
part: bge-small embeds 16 abstracts a second on the CPU, so a large tree
takes minutes, and the agent indexes only the documentation by default
(
rag.include). Graph extraction (4.3 s a docs chunk with an 8 B model; about 3 hours for these docs' 2,461 chunks) and captions are index-time costs paid once per changed file. At answer time, corrective retrieval adds five scoring requests a question, the router three, self-critique four to eight and sometimes a second generation. - Latency. Search itself is milliseconds; what a user waits for is the judge (+0.6 s a question with Qwen3-8B on a GPU), a reranker (0.6 to 1.2 s on the GPU, 2 to 4 s on the CPU) and generation. On the CPU alone, with an 8 B generator, every scoring request costs a prompt evaluation: a question takes 20.9 s with naive retrieval and 35.2 s with corrective retrieval, 22 s of it judging, against 1.5 s and 2.3 s on the GPU (On the CPU alone, measured).
- Index drift. An index is a snapshot.
oklm rag indexmust be run again after files change (it re-embeds only changed files, by content hash, and gives the bytes of a rebuild);--rag INDEXnever updates the index it reads. The agent's project and memory indexes are checked before each search (names, sizes, times) and updated by hash, so its own edits are found; a file changed with an unchanged size and time within the clock's resolution would be missed until the next change. Passages are cited by file and lines as they were indexed: a reader should open the file, not trust the line numbers of an old index. - Embedding-model identity. Vectors mean something
only for the model that made them: the index records the model's
identity (a hash of its architecture, sizes, tensor table and a weight
sample), its prefixes, pooling and normalization, and refuses a search
or update with another one (another quantization of the same model
included). The model is found by the path recorded at build time,
relative to where the index was built; moving the index elsewhere needs
--model. - Judge leniency. Faithfulness, the critique and corrective retrieval's verdicts are all the same 8 B model judging its own answers and passages. Hand checks put the faithfulness figure within about ±15 points of a reader (lenient on numbers taken from another source, strict on paraphrase); self-critique's 96.1 % is the figure of the judge it optimizes for.
- Small sets. Every row is a mean over 38 to 323 questions, and the questions drawn are the main uncertainty: a 95 % interval spans about ±4 nDCG@10 points on the retrieval sets, ±5 on the code set and ±10 to ±13 accuracy points on the 38 answers (Error bars). Of the comparisons the defaults rest on, hybrid against either retriever alone on the docs and NFCorpus, weighted against reciprocal rank on SciFact, Qwen3-Embedding against bge-small on code and reranking's recall@1 on the docs hold; the others (fusions on the docs and NFCorpus, chunkers, reranking's nDCG@10 and answers, corrective against naive accuracy) are inside their intervals. Run to run, the generation rows do not move: four runs of each configuration of 2026-11-10 gave the same verdict on every question and the same faithfulness to the pair (requests one at a time at temperature 0). The agent evals, whose turns and timings vary, passed 13 and 12 of 17 without retrieval with only two failures in common, and the 2026-11-09 generation rows (89.5 % naive, 92.1 % corrective) are two answers above 2026-11-10's (84.2 %, 86.8 %) with weighted fusion the default in between; every new miss had a judged section among its passages.
- The agent on a server other than OKLM's. Without
/v1/decidethe judge, router and critic ask chat completions and read the first token's log-probabilities, or only the answer's first word (a verdict of 0 or 1, 0.3 when it names neither); both fallbacks are tested against a fake server, not measured on a real one. A server that answers neither leaves the agent's retrieval naive and refuses--rag-critique(On a served model). - Streams. A streamed answer (server,
flag) is flagged, not taken back: a client that does not read the verdict shows the unsupported answer. - Not measured. Graph retrieval over the whole docs
set (measured on an 82-section part of it: the full extraction takes
about 3 hours), so its effect on the 120 questions' nDCG@10 is not
measured; other generators than Qwen3-8B for the generation rows; the
agent's retrieval on a server without
/v1/decide; the agent evals repeated.
Embeddings
An embedding model turns a text into one vector; texts of close meaning get vectors of high cosine. OKLM runs three kinds:
| Kind | general.architecture |
Models checked | Pooling (default) |
|---|---|---|---|
BERT encoders (BERT, RoBERTa and XLM-R, which llama.cpp's converter
writes as bert) |
bert |
BAAI/bge-small-en-v1.5 (GGUF F32 and Q8_0, and its checkpoint) | CLS |
| nomic-bert (rope, SwiGLU) | nomic-bert |
nomic-embed-text-v1.5 (GGUF F32 and Q8_0, and its checkpoint) | mean |
| Decoder embedding models | qwen3 (any Llama-family architecture) |
Qwen3-Embedding-0.6B (GGUF F16 and Q8_0) | last token |
The pooling is the file's (<arch>.pooling_type, or
a checkpoint's sentence-transformers
1_Pooling/config.json), else the family's above;
--pooling overrides it. Vectors are L2-normalized unless
--no-normalize. Texts are tokenized with the model's
special tokens: [CLS] ... [SEP] for a WordPiece vocabulary
(a GGUF bert vocabulary too, as llama.cpp does), and a
decoder's closing token when its tokenizer adds one (Qwen3-Embedding's
<|endoftext|>). A text longer than the context (512
tokens for bge, 2048 for nomic, 1024 by default for a decoder,
-c to change it) is cut, its closing token kept.
Instruction prefixes are the caller's: bge's model card asks for
Represent this sentence for searching relevant passages:
before retrieval queries, nomic's for search_query: /
search_document: , Qwen3-Embedding's for
Instruct: TASK\nQuery: before queries.
--query-prefix and --passage-prefix add them
(--query marks the texts as queries).
oklm embed -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf "What is the boiling point of water?"
oklm embed -m models/bge-small-en-v1.5 --json --file passages.txt
oklm embed -m models/nomic-embed-text-v1.5/nomic-embed-text-v1.5.Q8_0.gguf \
--query --query-prefix "search_query: " "how do I quantize a model"
Plain output is one vector a line (space-separated);
--json prints OpenAI's embeddings response. A summary goes
to stderr: texts, tokens, time, texts per second.
In the library
#include "oklm/rag/embed.h"
oklm_embed_params p = oklm_embed_default_params(); /* the model's pooling, normalized, CPU */
oklm_embedder* e = NULL;
float v[2 * 384];
const char* texts[2] = {"first text", "second text"};
oklm_embedder_new(model, &p, &e);
oklm_embed(e, texts, 2, 0 /* passages */, v); /* row i for texts[i] */
oklm_embedder_free(&e);oklm_embed_tokens() takes token ids instead. An embedder
owns an embedding context (oklm_context_params.embeddings):
no output head and no logits, each text encoded alone from position 0,
several texts packed into one forward pass (64 texts or 2048 tokens by
default). For an encoder the context holds no KV cache and each text
attends over its own tokens in both directions; a decoder runs causally
as usual and its final normalized hidden state of the last token is the
embedding.
In the server
oklm serve -m MODEL --embed-model EMB answers
POST /v1/embeddings with EMB (a second model beside the
generative one): OpenAI's request (input a string, strings,
token ids or arrays of ids; encoding_format float or
base64; dimensions only the model's width) and response
with usage.prompt_tokens (server.md).
Differences from llama.cpp
- GELU. llama.cpp's
bertuses the tanh approximation of GELU; the checkpoints are trained with the exact (erf) one, which OKLM uses.OKLM_EMBED_GELU_TANH=1switches to the tanh form, for the comparison withllama-embeddingonly. - Token types. Every token of a single text is type 0, as llama.cpp; in a pair for a cross-encoder the passage is type 1 (Reranking).
- OpenCL. The products, norms and activations run on the device; attention in both directions runs on the CPU (the device's attention kernel is causal), which costs a round trip a layer.
Embeddings, measured
On 2026-11-08 (tools/embed_compare.py, 1000 lines of
these docs, 24,253 bge tokens; 1 - cosine, worst and mean over the
strings):
| Model, file | Against | Worst | Mean |
|---|---|---|---|
| bge-small-en-v1.5, F32 GGUF | llama-embedding, same file |
1.0e-5 | 9.3e-7 |
same, OKLM with OKLM_EMBED_GELU_TANH=1 |
llama-embedding |
8.0e-8 | 3.9e-8 |
| bge-small-en-v1.5, F32 GGUF and checkpoint | numpy float64 (tools/embed_ref.py) |
8.0e-14 | 3.9e-14 |
| bge-small-en-v1.5, Q8_0 GGUF | llama-embedding |
1.9e-4 | 9.5e-5 |
| nomic-embed-text-v1.5, F32 GGUF | llama-embedding |
2.5e-7 | 8.7e-8 |
| nomic-embed-text-v1.5, F32 GGUF and checkpoint | numpy float64 | 4.5e-13 | 9.7e-14 |
| Qwen3-Embedding-0.6B, F16 GGUF | llama-embedding |
8.8e-7 | 2.7e-7 |
The F32 / F16 files meet the design's proof (1e-4 against llama.cpp, 1e-5 against numpy) with room; bge's remaining 1e-5 against llama.cpp is the GELU (the tanh switch takes it to 8e-8). The Q8_0 file parts by up to 1.9e-4: both engines round the activations to 8 bits before each Q8_0 product, and their float orders differ, so a few roundings flip.
Speed, the same 1000 strings (oklm embed --file, model
loading excluded; CPU: 8 threads of an i7-11700K (AVX-512), GPU: a
Radeon RX 7900 XT (gfx1100) through OpenCL, attention on the CPU; a
shared machine; since 2026-11-14 the encoders' attention runs on the
device, see below):
| Model | CPU texts/s | CPU tokens/s | GPU texts/s | GPU tokens/s |
|---|---|---|---|---|
| bge-small-en-v1.5 F32 | 286 | 6,923 | 731 | 17,723 |
| bge-small-en-v1.5 Q8_0 | 491 | 11,915 | 772 | 18,710 |
| nomic-embed-text-v1.5 F32 | 60 | 1,456 | 479 | 11,618 |
| Qwen3-Embedding-0.6B F16 | 19 | 378 | 228 | 4,555 |
On 2026-11-14 OpenCL's attention kernel took bidirectional attention,
which the backend handed back to the CPU until then.
oklm embed --file on 953 lines of these docs (21,441 bge
tokens), the same build with the attention handed back and on the
device, three runs each: bge-small-en-v1.5 F32 750 to 764 texts/s before
and 3,240 to 3,260 after, nomic-embed-text-v1.5 F32 503 to 510 and 1,579
to 1,592 (the vectors differ in their last places:
test_embed and test_rerank hold the tiny
encoders to their references on OpenCL as before).
Since 2026-11-14 a batch of several texts attends in one launch a
block on a device (oklm_attention_spans(): each row sees
its own text's rows) where it took a launch a text.
oklm embed --file on 364 lines of these docs, the whole
command with its loading, two runs: bge-small-en-v1.5 F32 0.76 s before
and 0.71 s after on OpenCL, 0.25 to 0.36 s and 0.23 s on Vulkan (2.4 to
3.2 s on the CPU); nomic-embed-text-v1.5 F32 1.15 s and 1.08 to 1.10 s
on OpenCL, 0.59 s and 0.57 s on Vulkan (12.3 to 12.9 s on the CPU). On
OpenCL the attention's kernel time went from 55.8 to 11.0 ms and from
81.2 to 21.7 ms (opencl.md). The
vectors are 2.7e-7 to 4e-7 from the CPU's, as before (2e-7 to 4e-7); the
CPU's are the same bits.
Retrieval quality, BEIR SciFact test split
(make rag-eval, exact cosine search over the 5,183
abstracts, BEIR's nDCG@10):
| Model | Query instruction | nDCG@10 | Published (MTEB) | MRR@10 | Recall@10 |
|---|---|---|---|---|---|
| bge-small-en-v1.5 F32 | Represent this sentence for searching relevant passages: |
71.27 | 71.27 | 68.22 | 83.6 |
| bge-small-en-v1.5 F32 | none | 72.00 | 68.45 | 84.5 |
With the model card's instruction OKLM gives MTEB's published figure exactly (embeddings-benchmark/results, the leaderboard's data: 71.273; the model card's own table: 71.275); the OpenCL run gives the same 71.27. The abstracts are long (about 290 tokens), so the corpus embeds at 16 documents a second on the CPU and 46 on the GPU.
The retrieval evaluation
oklm rag eval DIR --index INDEX [--model MODEL] [options]
scores retrieval on a BEIR-format set: DIR/corpus.jsonl
(_id, title, text),
DIR/queries.jsonl (_id, text) and
DIR/qrels/test.tsv (query-id,
corpus-id, grade; --split for another file).
Each document becomes one source of the index (its title, a blank line,
its text), chunked as oklm rag index chunks
(--chunker, --chunk-size;
--chunk-size 0 keeps each document one chunk) and embedded
with the model given (--query-prefix,
--passage-prefix, --backend as for
oklm rag index; the vectors F32 unless --vec).
The index is kept: a second run with the same model and settings embeds
nothing. Each judged query then runs through the same search as
oklm rag search, for every mode asked
(--mode dense,sparse,hybrid,
--fusion rrf,weighted: weighted alone by default,
--rrf-k, --weights); its chunks are mapped to
their documents (a document ranks where its best chunk does) down to
--depth documents (100). One row a configuration:
oklm rag eval data/beir/scifact --index build/scifact.gguf \
-m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf --chunk-size 0 \
--query-prefix "Represent this sentence for searching relevant passages: " --fusion rrf,weighted
| Column | |
|---|---|
| nDCG@10 | BEIR's (trec_eval's): gain the grade, discount log2(rank + 1), over the ideal ordering of every judged document |
| MRR@10 | 1 / the rank of the first relevant document within 10 |
| R@1 ... R@100 | relevant documents within the first k, over all the query's relevant documents (judged ones missing from the corpus included) |
| P@5, P@10 | relevant documents within the first k, over k (trec_eval's and BEIR's precision: a ranking shorter than k still divides by k) |
| p50 / p95 ms | one query from its text to its ranking: embedding it, searching, fusing |
--run PREFIX writes the rankings in TREC run format
(PREFIX.MODE[-FUSION][-rerank].trec). --json
prints the table as an array of objects on standard output, one a row
(the lines about the indexing stay on standard error; below, the
five-document set of the tests):
| Field | |
|---|---|
set, model |
the set's directory as given; the embedding model's name |
mode |
dense, sparse or hybrid |
fusion |
rrf or weighted for a hybrid row,
"" otherwise |
rrf_k, w_dense, w_sparse |
the fusion's settings (used or not) |
queries, documents,
chunks |
judged queries run, documents of the corpus, chunks of the index |
ndcg@10, mrr@10, recall@1,
recall@5, recall@10, recall@20,
recall@100, p@5, p@10 |
the columns above, as percentages (0 to 100) |
rerank_top |
the candidates reranked for each query, 0 without
--rerank |
ms_p50, ms_p95, ms_mean |
a query's latency in milliseconds: median, 95th percentile, mean |
[
{"set":"build/test_rag_eval/set","model":"oklm-hash-bow","mode":"dense","fusion":"","rrf_k":60,"w_dense":0.5,"w_sparse":0.5,"queries":3,"documents":5,"chunks":5,"ndcg@10":92.0063,"mrr@10":100.0000,"recall@1":83.3333,"recall@5":83.3333,"recall@10":83.3333,"recall@20":83.3333,"recall@100":83.3333,"p@5":20.0000,"p@10":10.0000,"rerank_top":0,"ms_p50":0.003,"ms_p95":0.195,"ms_mean":0.067},
{"set":"build/test_rag_eval/set","model":"oklm-hash-bow","mode":"hybrid","fusion":"weighted","rrf_k":60,"w_dense":0.5,"w_sparse":0.5,"queries":3,"documents":5,"chunks":5,"ndcg@10":92.0063,"mrr@10":100.0000,"recall@1":83.3333,"recall@5":83.3333,"recall@10":83.3333,"recall@20":83.3333,"recall@100":83.3333,"p@5":20.0000,"p@10":10.0000,"rerank_top":0,"ms_p50":0.004,"ms_p95":0.007,"ms_mean":0.005}
]
--rerank MODEL [--rerank-top N] reorders each query's
first N candidate chunks (50) with a reranker, as
oklm rag search --rerank does, before they are mapped to
documents (on the device of --backend; the time of a query
then includes it; rows marked + rerank N).
--graph --gen-model MODEL [--gen-backend B] extracts the
graph into the index as oklm rag index does, and
--rag-graph on searches with it (Graph retrieval). In the library:
include/oklm/rag/eval.h
(oklm_rag_eval_set_load(),
oklm_rag_eval_index(), oklm_rag_eval_run(),
oklm_rag_eval_run_reranked(), and the metric functions
oklm_rag_ndcg(), oklm_rag_recall(),
oklm_rag_precision(), oklm_rag_rr()). A judged
query whose ranking is empty (BM25 finding none of its words) counts,
with 0 for every metric.
make rag-eval [RAG_MODEL=...] [RAG_ARGS=...]
(tools/rag_eval.sh, outputs in
build/rag-eval/, report.md and one table a
set) runs it on both sets with bge-small-en-v1.5 by default (bge models
get the model card's query instruction):
- scifact: BEIR SciFact's test split (5,183
abstracts, 300 queries; fetched by
sh tools/fetch_datasets.sh --ragintodata/, CC BY-NC, local measurement only), each abstract one chunk; - nfcorpus (not in the default list,
--sets scifact,nfcorpus,docs): BEIR NFCorpus's test split (3,633 medical documents, 323 queries judged 1 or 2; fetched the same way, local measurement only), each document one chunk; - code (not in the default list either): the
source-code question set,
tests/rag/code_set/, 112 questions over OKLM's sources at a pinned revision, each file a document chunked by its path (Retrieval on code, measured); - docs: the documentation question set,
tests/rag/docs_set/: 120 questions over OKLM's own documentation (70 lookups, 30 how-to, 20 needing two sections), each judged against the sections that answer it (grade 2) or help (grade 1), 276 judgments over 51 files. The corpus is README.md and docs/ as they were at the revision intests/rag/docs_set/revision, extracted bytools/rag_docs_set.py(git show) and cut at the headings of levels 1 to 3 into 1,007 sections, each a documentPATH#ANCHOR; so later edits of the docs do not move the labels.python3 tools/rag_docs_set.py --checkverifies that every judged section exists. It also writesqrels/KIND.tsvfor each kind of question (--split multihopscores the 20 two-section ones), and--subset Na small set for graph extraction: the two-section questions and five lookups over the sections judged for them and N others.
RAG_ARGS="--sets scifact --backend opencl" picks the
sets and passes options on. With --reference it also runs
tools/rag_eval.py: SciFact dense retrieval from
oklm embed's vectors and a numpy search, scored by its own
Python metrics, and those metrics over the C evaluation's rankings
(--score-run), which give the same figures to the last
digit on both sets; then the reference comparisons of every embedding
model present under models/
(tools/embed_compare.py): 1 - cosine against
llama-embedding on the same GGUF (llama.cpp built as a test
tool, adr/0015-test-references.md)
and against the numpy float64 encoder tools/embed_ref.py on
the checkpoint, over 1000 lines of OKLM's documentation.
Hybrid retrieval, measured
On 2026-11-09, bge-small-en-v1.5 F32 with its query instruction, F32
vectors, the defaults (50 candidates a retriever, or the depth when
larger; weights 0.5 / 0.5; RRF k = 60); the corpus embedded on the GPU
(OpenCL), queries timed one at a time on it.
make rag-eval RAG_ARGS="--backend opencl" reproduces both
tables to the printed digits (the latencies within half a
millisecond).
SciFact (300 queries, 5,183 abstracts):
| Mode | nDCG@10 | MRR@10 | R@1 | R@5 | R@10 | R@20 | R@100 | P@5 | P@10 | p50 / p95 ms |
|---|---|---|---|---|---|---|---|---|---|---|
| dense | 71.27 | 68.22 | 57.6 | 75.9 | 83.6 | 87.3 | 94.2 | 16.7 | 9.4 | 3.1 / 4.3 |
| sparse (BM25) | 67.41 | 64.21 | 54.1 | 73.6 | 79.5 | 83.9 | 90.5 | 15.9 | 8.7 | 0.06 / 0.09 |
| hybrid, RRF | 71.29 | 67.85 | 56.8 | 77.1 | 83.7 | 89.3 | 97.2 | 16.9 | 9.4 | 3.3 / 4.6 |
| hybrid, weighted | 73.04 | 69.51 | 58.4 | 78.7 | 86.1 | 91.1 | 97.2 | 17.3 | 9.6 | 3.2 / 4.4 |
The documentation set (120 queries, 1,007 sections in 2,655 chunks of up to 256 tokens):
| Mode | nDCG@10 | MRR@10 | R@1 | R@5 | R@10 | R@20 | R@100 | P@5 | P@10 | p50 / p95 ms |
|---|---|---|---|---|---|---|---|---|---|---|
| dense | 60.48 | 62.79 | 27.2 | 61.3 | 77.1 | 86.7 | 99.0 | 26.2 | 17.1 | 3.9 / 4.3 |
| sparse (BM25) | 64.53 | 68.49 | 28.3 | 69.0 | 80.7 | 89.4 | 98.5 | 29.8 | 17.7 | 0.07 / 0.08 |
| hybrid, RRF | 69.78 | 73.08 | 30.8 | 70.8 | 85.2 | 91.1 | 99.6 | 30.2 | 18.9 | 4.1 / 4.6 |
| hybrid, weighted | 70.61 | 74.20 | 30.2 | 73.5 | 86.7 | 92.1 | 99.4 | 31.5 | 19.2 | 4.1 / 4.8 |
NFCorpus (--sets nfcorpus, measured 2026-11-10; 323
queries, 3,633 documents; graded 1 or 2, so nDCG@10 is low beside recall
for every mode):
| Mode | nDCG@10 | MRR@10 | R@1 | R@5 | R@10 | R@20 | R@100 | P@5 | P@10 | p50 / p95 ms |
|---|---|---|---|---|---|---|---|---|---|---|
| dense | 34.38 | 53.02 | 5.4 | 12.7 | 16.2 | 20.0 | 31.1 | 32.8 | 25.8 | 2.7 / 2.9 |
| sparse (BM25) | 32.09 | 52.69 | 5.7 | 12.3 | 15.4 | 18.5 | 24.5 | 29.3 | 22.9 | 0.01 / 0.05 |
| hybrid, RRF | 36.67 | 57.61 | 6.9 | 13.7 | 17.5 | 20.7 | 31.4 | 33.2 | 26.7 | 2.8 / 3.1 |
| hybrid, weighted | 36.37 | 56.96 | 6.5 | 13.6 | 17.0 | 20.9 | 31.7 | 33.8 | 26.6 | 2.8 / 3.1 |
Dense SciFact gives MTEB's published 71.27, as
tools/rag_eval.py does, and P@10 9.43 as the model card has
it (P@5 16.73 against its 16.8). Dense NFCorpus gives 34.38 against the
card's 34.31 (MRR@10 53.02 against 52.68, P@10 25.76 against 25.73):
close, not equal, the card's run being on MTEB's copy of the set. BM25
ranks nothing for 18 of NFCorpus's queries (one word, such as "eggnog"
or "Fosamax", found in no document); they count with 0. On NFCorpus both
fusions beat each retriever alone by 2 to 4.5 nDCG@10, and reciprocal
rank is 0.3 ahead of weighted fusion, the only set of the three where it
is: within the noise of 323 queries, so weighted fusion stays the
default.
Hybrid retrieval is at least as good as each retriever alone on both
sets in nDCG@10 and in recall at every cut-off from 5 on (RRF ties dense
on SciFact at 10: 83.7 against 83.6; its MRR and recall@1 there are a
little under dense's); weighted fusion is the better one on both, and
the default since 2026-11-09 (--fusion rrf for reciprocal
rank). The fusion settings were left as they were: weights 0.7 / 0.3 or
0.3 / 0.7 lose on one set or the other, and RRF with k = 20 gains 0.7
nDCG@10 on SciFact and 0.2 on the docs, too little on two sets to leave
the usual constant. BM25 is strong on the documentation set because its
questions use the docs' own names for things (commands, flags, file
formats); the hashed bag of words reaches 39.39 nDCG@10 there in dense
mode, the model 60.48.
Retrieval on code, measured
The code set (tests/rag/code_set/,
--sets code): 112 questions over OKLM's own sources as a
developer new to them asks them, each judged against the files whose
code answers it (grade 2) or helps (grade 1, a header declaring the API,
a caller), 197 judgments. 31 questions name an identifier, a flag, a
route or a message (symbol); 60 describe what the code does
without any name and without the wording of its comments
(behavior); 21 need two files or more
(multifile). The corpus is every .c,
.h, .cl, .py and .sh
file under src/, include/, apps/,
tools/, tests/ and fuzz/, and the
Makefile, at the revision in tests/rag/code_set/revision
(517 files, 10.7 MB; tests/security/corpus/ left out),
extracted by tools/rag_code_set.py (--check
verifies the set). Each file is one document, chunked by its path as
oklm rag index chunks a file. A file ranks where its best
chunk does, so the measure is the file a reader opens.
On 2026-11-10, the GPU (OpenCL), 256-token chunks, the defaults otherwise; bge-small-en-v1.5 F32 with its query instruction unless named:
| Embedder, chunker | Mode | Chunks | nDCG@10 | MRR@10 | R@1 | R@5 | R@10 | R@100 | P@5 | p50 ms |
|---|---|---|---|---|---|---|---|---|---|---|
bge-small, auto (code) |
dense | 19,948 | 55.49 | 55.94 | 24.1 | 62.4 | 79.8 | 100.0 | 21.4 | 14.6 |
| sparse (BM25) | 47.57 | 55.12 | 24.0 | 55.7 | 69.9 | 97.8 | 18.7 | 0.6 | ||
| hybrid, RRF | 57.68 | 61.81 | 26.6 | 66.1 | 81.5 | 99.6 | 22.3 | 16.5 | ||
| hybrid, weighted | 59.32 | 63.40 | 30.5 | 67.1 | 81.4 | 99.6 | 22.9 | 16.5 | ||
bge-small, recursive |
dense | 18,733 | 57.27 | 57.45 | 28.0 | 64.0 | 80.7 | 99.6 | 21.6 | 13.7 |
| sparse (BM25) | 49.62 | 57.76 | 27.2 | 57.2 | 70.7 | 96.6 | 18.9 | 0.6 | ||
| hybrid, RRF | 59.64 | 64.79 | 32.4 | 65.8 | 80.7 | 99.1 | 21.8 | 15.6 | ||
| hybrid, weighted | 60.12 | 65.91 | 32.4 | 68.0 | 80.5 | 98.7 | 22.3 | 15.6 | ||
bge-small, fixed |
dense | 19,800 | 53.59 | 55.51 | 24.9 | 61.0 | 75.3 | 98.2 | 20.7 | 14.4 |
| hybrid, weighted | 58.01 | 61.25 | 28.9 | 67.1 | 81.1 | 99.1 | 22.3 | 16.5 | ||
nomic-embed-text-v1.5 F32, auto |
dense | 19,948 | 47.46 | 48.87 | 20.5 | 55.8 | 70.8 | 97.8 | 20.2 | 17.2 |
| hybrid, weighted | 53.83 | 56.22 | 23.6 | 60.3 | 78.8 | 100.0 | 20.5 | 20.1 | ||
Qwen3-Embedding-0.6B F16, auto |
dense | 15,933 | 63.63 | 64.26 | 30.3 | 75.5 | 83.6 | 98.2 | 25.5 | 17.7 |
| hybrid, RRF | 59.83 | 64.42 | 29.5 | 71.7 | 84.7 | 98.7 | 24.3 | 21.4 | ||
| hybrid, weighted | 62.09 | 67.06 | 32.1 | 73.8 | 85.3 | 100.0 | 25.2 | 21.0 |
nDCG@10 by kind of question
(tools/rag_eval.py --score-run RUN --data build/rag-eval/code --split KIND):
| Configuration | symbol (31) | behavior (60) | multifile (21) |
|---|---|---|---|
bge-small auto, dense |
63.04 | 52.79 | 52.04 |
bge-small auto, BM25 |
57.44 | 44.53 | 41.68 |
bge-small auto, hybrid weighted |
68.96 | 56.79 | 52.30 |
bge-small recursive, hybrid weighted |
69.98 | 59.37 | 47.71 |
bge-small fixed, hybrid weighted |
65.81 | 58.26 | 45.81 |
Qwen3-Embedding auto, dense |
75.73 | 58.87 | 59.36 |
Qwen3-Embedding auto, hybrid weighted |
70.86 | 59.86 | 55.52 |
What it shows:
- The code-aware chunker does not pay on this set.
Cut at functions, it gives 59.32 nDCG@10 hybrid against 60.12 for the
recursive cutter (paragraphs, then lines) and 58.01 for fixed windows,
and 55.49 against 57.27 dense: differences of one or two points over 112
questions, within the noise. It is ahead only on the two-file questions
(52.30 against 47.71) and a point or two behind on the others; why was
not looked into.
autostays the default: it is never far behind, and its chunks begin and end where a reader expects. - Hybrid helps a small encoder and costs a stronger one. With bge-small, weighted fusion adds 3.8 nDCG@10 to dense; BM25 alone is strong on the questions that name things (57.44) and weak on the others (44.53). With Qwen3-Embedding, dense alone is the best nDCG@10 (63.63 against 62.09 weighted), BM25 pulling the named questions down (75.73 to 70.86) while lifting MRR, recall@1 and recall@10 a little.
- The embedder matters most. Qwen3-Embedding-0.6B is
8 points of nDCG@10 above bge-small dense and 16 above nomic-embed, for
21 % more query time and 1.6 times the indexing time (383 s against 239
s for the 517 files). nomic-embed with its
search_query:/search_document:prefixes is the weakest of the three here (dense 47.46; 57.93 on the named questions, level with BM25's 57.44). - Recall is high, the first rank is not. The right file is in the first 10 for 80 to 85 % of the questions and in the first 100 for nearly all, but first for only a quarter to a third (R@1 24 to 32 %); the first 5 hold 62 to 76 % of the judged files with bge-small and Qwen3-Embedding.
Reproduce with
make rag-eval RAG_ARGS="--sets code --backend opencl"
(bge-small, auto, about 5 minutes); the other rows add
--chunker recursive|fixed or
-m MODEL --query-prefix TEXT to
oklm rag eval build/rag-eval/code (Qwen3-Embedding's
prefix:
Instruct: Given a web search query, retrieve relevant passages that answer the query
and a line Query: ).
Tests
tests/rag/test_embed.c in make test, with
no download: tiny random bert and nomic-bert
encoders (tests/rag/fixtures/, made by
tools/gen_embed_fixtures.py as llama.cpp's converter writes
them, and as Hugging Face checkpoints) against
tools/embed_ref.py within 1e-5 (tokens equal), the tanh
switch, batching against one text at a time, normalization and pooling,
[CLS] ... [SEP] from a GGUF WordPiece vocabulary, a
decoder's last-token embedding against oklm_get_hidden(),
the encoder refused outside an embedding context, and OpenCL against the
reference within 1e-4. tests/server/test_server.c checks
/v1/embeddings (every input form, base64, usage, refusals,
404 without an embedding model). tests/rag/test_eval.c: the
metrics against values computed by hand, a small BEIR-format set read
and scored (unjudged queries skipped, a judged document missing from the
corpus counted, the TREC run written), the tiny bert
fixture as an encoder (identity, unit vectors, the checkpoint
directory's name), an index built with it and searched with the model
the index names, another model, another prefix and the hash encoder
refused, oklm rag eval with the model, and the reranked run
(no reranker or no candidate giving the plain rankings, each query's
first document the one a search reranked by hand puts first with the
tiny cross-encoder, --rerank on the command line, a missing
reranker refused). tests/rag/test_rag.c: retrieval in
generation with scripted stages (the context's entries and numbering,
the token budget leaving a passage out and cutting the first, the prompt
and its no-source form, the citations, the corrective state machine:
relevant answers, nothing relevant rewrites and retries, then the web
when allowed, else no source; the thresholds, a failing judge, the
reranker hook, the log line, the counters), the retriever of a hash
index, and the judge and rewriter on a tiny model (the judge equal to
decide.h's scores on a fresh context, its prefix reuse exact).
test_rag in tests/server/test_server.c: the
rag field off and on, corrective and naive, refused fields,
a server without an index, /api/stats and the log.
test_rag_http there: the stages on a served model against
the tiny model's server (the probe finds /v1/decide; a
corrective run judged through it, the rewriter and the condenser through
chat completions, the router's three probabilities and the critic's);
test_rag_http_fallback: a fake server answering
/v1/decide 404, read by its log-probabilities, then without
them by its text, the probe done once and no template variables sent.
tests/harness/test_cli_rag.c: the agent's
--rag-critique refused with --mock and with a
--url server that answers nothing;
--caption-prompt only with --caption-model.
test_captions in tests/rag/test_graph.c: the
default prompt reaching the captioner and not recorded, a prompt of its
own reaching it, recorded, and re-captioning on a change.
test_vlm_devices in tests/rag/test_rag.c: the
vision model on a tiny model and projector, its projector found beside
it; on OpenCL and on Vulkan its image encoder is on the model's device
and the caption is the one the same context gives with the encoder on
the CPU.
Chunking and the index
Commands
oklm rag index docs README.md src --out docs.gguf # build, or update in place
oklm rag index docs --out docs.gguf --chunker fixed --chunk-size 128 --overlap 16
oklm rag search docs.gguf "how is the KV cache quantized" -k 5
oklm rag search docs.gguf "oklm_matmul" --mode sparse --json
oklm rag info docs.gguf --sources
oklm rag index docs --out docs.gguf --progress-json - # each file's fate and the embedding's advance, as JSON lines
oklm rag info docs.gguf --sources --json # every source, and whether its file changed since
oklm rag search docs.gguf "how is the KV cache quantized" --json-full # the settings, the times, each score
oklm rag index PATH... --out INDEX reads every file
given and every file under the directories given (names starting with
. are left out, and so are binary files, files over 64 MiB
and the index itself), turns each into text with
oklm ingest's reader (text, Markdown, code, JSON, HTML,
DOCX, PDF), cuts it into chunks, embeds them and writes the index. Run
again on the same index, it hashes each file: unchanged files keep their
chunks and vectors (copied, not embedded again), changed and new ones
are chunked and embedded, files no longer given are dropped. The result
is the same file, byte for byte, as a rebuild from scratch. The index is
written beside the target and renamed over it, so a search running
meanwhile sees the old index or the new one; INDEX.lock
keeps two writers apart (a lock left by a process that is gone is taken
over).
| Option | Default | |
|---|---|---|
--model MODEL (-m) |
hash:384 |
the encoder: an embedding model (a GGUF file or a Hugging Face
checkpoint directory, as oklm embed takes it), or
hash[:DIM], the hashed bag of words below |
--query-prefix TEXT,
--passage-prefix TEXT |
none | the model's instructions, recorded in the index (Embeddings) |
--backend cpu|opencl|auto |
cpu |
where the model runs |
--chunker auto|fixed|recursive|code|semantic |
auto |
auto: code for source files, recursive for the
rest |
--chunk-size N |
256 | target tokens per chunk |
--overlap N |
32 (fixed), 0 | tokens shared by consecutive fixed windows |
--min N / --max N |
32 / 512 | a smaller chunk joins a neighbor; the hard cap (a function, a joined chunk, a semantic group) |
--vec f32|f16|q8_0 |
f16 |
the vectors' type (Q8_0 needs a dimension multiple of 32) |
--no-bm25, --no-stem |
no keyword postings; no plural stripping | |
--rebuild |
ignore the existing index (also one of another encoder) | |
--threads N |
all cores | worker threads of the products |
--progress-json FILE |
the build's progress written to FILE as JSON lines (Output for programs); -:
standard output, the summary then on standard error |
Changing the chunker settings or the vector type chunks and embeds everything again (and says so). An index built with another encoder is refused, for an update as for a search: vectors of another model (another quantization of it included), or of the same model with other prefixes, pooling or normalization, mean nothing.
An embedding model is loaded once per command and records its
identity in the index: oklm_model_id() (a hash of the
architecture, the sizes, the tensor table and a sample of the weights),
general.name (a checkpoint's directory name), the
architecture, the path given, the dimension, the pooling, the
normalization, the prefixes and the context. The chunk sizes are then
counted with the model's tokenizer, and a chunk longer than the model's
context (512 tokens for bge) is cut when embedded. An existing index
supplies what is not given:
oklm rag index docs --out docs.gguf updates an index with
the model, prefixes and pooling it was built with, and
oklm rag search reloads the model the index names
(rag.embed.file, as given at build time, so relative to the
directory the index was built from) unless --model names
another copy of it; the GGUF file and its Hugging Face checkpoint have
the same identity.
Output for programs
What a program that drives the commands reads. Every object is UTF-8 JSON; a field named here is always present unless its row says otherwise, and a reader ignores fields it does not know. Sizes are bytes, times of day Unix seconds, hashes and model ids 16 hexadecimal digits in a string (64 bits do not fit a JSON number). Without these options the commands print what they printed before.
oklm rag index ... --progress-json FILE
writes one object a line to FILE (truncated first; - is
standard output, and the summary a person reads then goes to standard
error with the errors), each line flushed as it is written. The stream
opens once the options are accepted (a usage error, exit status 2,
writes nothing) and always finishes with an end line. In
order:
type |
When | Fields |
|---|---|---|
start |
the encoder is loaded, the build begins | index (the path given to --out);
sources (the paths given, as given); encoder;
chunker; vectors (F32,
F16, Q8_0); bm25,
rebuild, graph, captions
(booleans: the postings kept, --rebuild,
--graph, --caption-model) |
scan |
the directories are walked | files: the file lines that follow |
file |
one a file, as it is read and chunked (figures captioned then too) | path (as the index records it); action;
chunks (made, copied or dropped); done and
total (this line's number among the file
lines, and scan's files) |
embed |
before the first batch (done 0), then at most every
quarter second, and after the last |
done, total (the new chunks; unchanged
files' vectors are copied, not counted); per_second (chunks
a second since the first line); seconds (since the first
line) |
graph |
with --graph, as embed |
done, total (every chunk of the index:
extracted, or kept with its entities), per_second,
seconds |
end |
the index is written, or the build failed | ok; error (a string, only when
ok is false); index; files
(added, updated, unchanged,
removed, skipped: the file lines
counted by action); chunks (added,
updated, unchanged, removed:
their chunks); sources and chunks_total (what
the index now holds); chunks_embedded;
rebuilt_all (true when the chunker, the vectors' type or
the captioner changed, so every file was chunked again);
index_bytes (the index file's size, 0 when there is none);
seconds (the whole command);
seconds_embed |
action |
|
|---|---|
added |
not in the index before: chunked and embedded |
updated |
in the index with another content (or rebuilt_all):
chunked and embedded again |
unchanged |
the same content hash: its chunks and vectors copied |
removed |
in the index before, no longer given or no longer readable: its chunks dropped |
skipped |
given but not a document (unreadable, binary, over 64 MiB) and not in the index before |
No embed line is written when nothing is new. With
--rebuild the old index is not read: every file is
added. When the encoder or a generator cannot be loaded the
stream is the end line alone. encoder and
chunker are the objects of rag info --json
below.
{"type":"start","index":"docs.gguf","sources":["docs","gone.txt"],"encoder":{"name":"oklm-hash-bow","arch":"hash","file":"","model_id":"34242a421855fddc","dimensions":384,"pooling":"mean","normalize":true,"max_tokens":0,"query_prefix":"","passage_prefix":""},"chunker":{"kind":"auto","size":256,"overlap":0,"min":32,"max":512,"breakpoint":95},"vectors":"F16","bm25":true,"rebuild":false,"graph":false,"captions":false}
{"type":"scan","files":5}
{"type":"file","path":"gone.txt","action":"skipped","chunks":0,"done":1,"total":5}
{"type":"file","path":"docs/new.md","action":"added","chunks":1,"done":2,"total":5}
{"type":"file","path":"docs/rag.md","action":"unchanged","chunks":245,"done":3,"total":5}
{"type":"file","path":"docs/roadmap.md","action":"updated","chunks":46,"done":4,"total":5}
{"type":"file","path":"docs/how-it-works.md","action":"removed","chunks":122,"done":5,"total":5}
{"type":"embed","done":0,"total":47,"per_second":0.000,"seconds":0.000}
{"type":"embed","done":47,"total":47,"per_second":52496.486,"seconds":0.001}
{"type":"end","ok":true,"index":"docs.gguf","files":{"added":1,"updated":1,"unchanged":1,"removed":1,"skipped":1},"chunks":{"added":1,"updated":46,"unchanged":245,"removed":122},"sources":3,"chunks_total":292,"chunks_embedded":47,"rebuilt_all":false,"index_bytes":640128,"seconds":0.008,"seconds_embed":0.001}
oklm rag info INDEX --json prints one
object:
| Field | |
|---|---|
index |
the path given |
version, minor |
the format's (rag.version, rag.minor) |
file_bytes |
the index file's size |
root |
the directory the index was built from: relative source paths are under it |
created, updated |
when the index was first built and last written |
encoder |
name (the model's general.name,
oklm-hash-bow for the hash encoder); arch
(bert, nomic-bert, qwen3...,
hash); file (the model file as given at build
time, "" for the hash encoder); model_id;
dimensions; pooling (mean,
cls, last, none);
normalize (boolean); max_tokens (the model's
context, 0: unbounded); query_prefix,
passage_prefix ("": none) |
chunker |
kind (auto, fixed,
recursive, code, semantic),
size, overlap, min,
max (tokens), breakpoint (the semantic
chunker's percentile) |
counts |
sources, chunks, figures
(chunks that caption a figure), text_bytes,
vector_bytes |
vectors |
the vectors' type: F32, F16 or
Q8_0 |
bm25 |
null without postings (dense search only), else
terms, postings, avg_len (terms a
chunk), k1, b, analyzer |
graph |
null without a graph, else entities,
mentions, relations, model (the
extractor's name) |
captions |
null when no captioner was used, else
figures, model, prompt
("": the default question) |
sources |
only with --sources: one object a source file, in path
order |
states |
only with --sources: unchanged,
changed, missing, the sources counted by
state |
| Field of a source | |
|---|---|
path |
as recorded: as given to rag index, /
separators |
kind |
what it was read as: text, json,
html, docx, pdf |
bytes, mtime |
the file's size and modification time when it was indexed |
hash |
the content hash an update compares |
chunk0, chunks |
its first chunk's id and its chunk count (its chunks are consecutive) |
state |
the file now: unchanged (the same content hash),
changed (another size or hash: rag index would
chunk and embed it again), missing (no regular file there:
rag index on its directory would drop it) |
The state is found by reading each file (hashed unless its size
already differs), which is why it comes with --sources
only. A relative path is looked up under root, so the
answer does not depend on the directory the command runs in. Files added
beside the sources since are not in the index and so not listed:
rag index reports them as added.
{"index":"docs.gguf","version":1,"minor":0,"file_bytes":724800,
"root":"/home/user/project","created":1794517391,"updated":1794517391,
"encoder":{"name":"bge-small-en-v1.5","arch":"bert","file":"models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf","model_id":"0cddcf8fa7cad783","dimensions":384,"pooling":"cls","normalize":true,"max_tokens":512,"query_prefix":"Represent this sentence for searching relevant passages: ","passage_prefix":""},
"chunker":{"kind":"auto","size":256,"overlap":0,"min":32,"max":512,"breakpoint":95},
"counts":{"sources":3,"chunks":343,"figures":0,"text_bytes":216479,"vector_bytes":263424},"vectors":"F16",
"bm25":{"terms":2765,"postings":23387,"avg_len":108.003,"k1":1.2,"b":0.75,"analyzer":"unicode-lower-s-1"},
"graph":null,
"captions":null,
"sources":[
{"path":"README.md","kind":"text","bytes":21521,"mtime":1794516719,"hash":"ecb6d2a034981666","chunk0":0,"chunks":34,"state":"unchanged"},
{"path":"docs/rag.md","kind":"text","bytes":163435,"mtime":1794516719,"hash":"1a9960288bea5dbe","chunk0":34,"chunks":257,"state":"changed"},
{"path":"docs/roadmap.md","kind":"text","bytes":32124,"mtime":1794516719,"hash":"f94290e491763cd9","chunk0":291,"chunks":52,"state":"missing"}
],
"states":{"unchanged":1,"changed":1,"missing":1}
}
oklm rag search INDEX QUERY --json
prints the hits as an array, best first;
--json-full prints one object with the
settings, the times and the same hits with every score they have:
Field (--json-full) |
|
|---|---|
index, query |
as given |
mode |
dense, sparse or hybrid: the
mode searched (dense when the index has no postings and
--mode is not given) |
fusion |
weighted or rrf when lists were fused
(hybrid, or a graph walk), else null |
k, candidates |
the hits asked for; the hits taken from each retriever |
rrf_k, w_dense, w_sparse |
the fusion's settings (used or not) |
graph, graph_hops |
whether the graph was walked (--rag-graph on and the
index has one), and how far (0 when not) |
model |
the embedding model's name, null in sparse mode |
rerank |
null, or model (the path given) and
top (the candidates it scored) |
found |
dense, sparse, graph: the
candidates each retriever gave |
timings_ms |
milliseconds, 0 for a stage that did not run: open (the
index), encoder (loading the embedding model),
embed (the query), dense, sparse,
graph, fuse, rerank_load,
rerank, total (the command up to its
output) |
hits |
the hits, best first |
| Field of a hit | --json |
--json-full |
|
|---|---|---|---|
rank |
yes | yes | from 1 |
chunk |
yes | yes | the chunk's id in the index |
score |
yes | yes | what the hits are ordered by: the reranker's score with
--rerank, else the fused score, else the one
retriever's |
dense, dense_rank |
when found by the vectors | the dot product with the query (the cosine: the vectors are normalized) and the rank among the dense candidates | |
sparse, sparse_rank |
when found by the keywords | the BM25 score and the rank among its candidates | |
graph, graph_rank |
when found by the graph walk | the walk's score and rank | |
fused, fused_rank |
when fusion is not null |
the fused score and the rank before any reranking (weighted: each list min-max normalized, so the best of both lists scores 1) | |
rerank |
with --rerank |
the reranker's score (the same as score) |
|
from |
yes | yes | the retrievers that found it: dense,
sparse, graph, joined by + |
source |
yes | yes | the source file's path |
start, end |
yes | yes | the chunk's byte range in the extracted text (end excluded) |
line0, line1 |
yes | yes | its first and last line, from 1 |
page, tokens |
yes | the PDF page (0: not recovered) and the chunk's size in tokens | |
heading |
yes | yes | the heading path or the code symbol, "" when none |
image |
a figure's caption only | a figure's caption only | the figure the chunk describes |
text |
yes | yes | the chunk's text |
A score is absent, not zero, when its retriever did not return the chunk among its candidates. Scores of different kinds do not compare: a cosine, a BM25 score, a fused score and a reranker's logit each rank within their own list.
{"index":"docs.gguf",
"query":"how is an index updated when a file changes",
"mode":"hybrid","fusion":"weighted","k":2,"candidates":50,"rrf_k":60,"w_dense":0.5,"w_sparse":0.5,"graph":false,"graph_hops":0,
"model":"bge-small-en-v1.5","rerank":{"model":"models/ms-marco-MiniLM-L-6-v2","top":10},
"found":{"dense":50,"sparse":50,"graph":0},
"timings_ms":{"open":0.116,"encoder":17.491,"embed":16.050,"dense":0.035,"sparse":0.035,"graph":0.000,"fuse":0.012,"rerank_load":24.990,"rerank":450.894,"total":517.455},
"hits":[
{"rank":1,"chunk":80,"score":6.34369,"dense":0.796433,"dense_rank":1,"sparse":13.9372,"sparse_rank":1,"fused":1,"fused_rank":1,"rerank":6.34369,"from":"dense+sparse","source":"docs/rag.md","start":26742,"end":27741,"line0":377,"line1":390,"page":0,"tokens":251,"heading":"Retrieval (RAG) > Residual risks","text":"with corrective retrieval, ..."},
{"rank":2,"chunk":128,"score":3.72353,"dense":0.756906,"dense_rank":3,"sparse":6.48501,"sparse_rank":27,"fused":0.456977,"fused_rank":5,"rerank":3.72353,"from":"dense+sparse","source":"docs/rag.md","start":54942,"end":55834,"line0":867,"line1":881,"page":0,"tokens":232,"heading":"Retrieval (RAG) > Chunking and the index > Commands","text":"`oklm rag index PATH... --out INDEX` reads every file given ..."}
]
}
oklm rag eval --json prints its table as an array of
rows (The retrieval evaluation).
In the library, a build reports through
oklm_rag_index_params.progress,
oklm_rag_index_source_state() compares a source with its
file, and oklm_rag_query_ex() gives each hit's scores and
the stages' times.
Chunkers
All work on the extracted text and return byte ranges of it (trimmed of white space), their lines, their size in tokens and a heading. Cuts fall between words; only a run of more than 256 bytes without white space is cut inside, at a UTF-8 boundary. Sizes are counted with the embedding model's tokenizer, word by word, when there is one; the hash encoder has none, and a word then counts one token per run of eight letters or digits and one per punctuation character (close to WordPiece on English).
- fixed: windows of
sizetokens; the next window startsoverlaptokens before the end of the previous one. - recursive: Markdown sections (ATX headings, not
inside code fences) are kept apart and give each chunk its heading path
(
Guide > Install > Windows); a section larger thansizeis cut at blank lines, then line ends, then sentence ends, then words, and the pieces are joined again up tosize. A chunk undermintokens joins the next one (the last one joins the previous one) when the two fit inmax. - code: C-like sources (C, C++, Java, JavaScript,
TypeScript, Go, Rust, C#...) cut at the end of each top-level block or
statement, with comments, strings and character literals skipped and
extern "C"andnamespaceblocks looked into; Python cut at each top-leveldeforclasswith its decorators and the comment lines just above (triple-quoted strings and open brackets skipped). A function, class or type of up tomaxtokens is a chunk of its own named after its symbol; a larger one is cut like a recursive section; the declarations between them are joined up tosize. Other source files (shell, Ruby, Makefiles...) use the recursive chunker without Markdown headings. The language server's symbols are not used. - semantic: the text cut into sentences (within
sections and paragraphs), each embedded; a chunk ends where the cosine
distance between neighbors exceeds the
breakpointpercentile (95) of all neighbor distances, or atmaxtokens.
PDF pages are not recovered from the extracted text:
page is 0.
The index file
One GGUF file (ADR-0022):
general.type = rag-index, the encoder's identity
(rag.embed.*: model id, name, architecture, dimension,
pooling, normalization, prefixes), the chunker
(rag.chunk.*), the sources (paths, size, modification time,
content hash, kind) and their chunks (text, byte range, lines, tokens,
heading, hash), the vectors as one [dim, n] tensor, and the
BM25 postings. oklm rag info prints it all; gguf-py and
gguf-dump read it.
Search
Dense search maps the file and multiplies the query's vector by the
vectors tensor where it lies in the mapping
(oklm_matmul(), the CPU's SIMD kernels and worker threads),
then takes the top k (oklm_topk()): exact cosine
similarity, no approximation, the lower chunk id first on ties. It is
checked against a brute-force double loop in make test: the
same top 10, scores within 1e-6, for F32 and F16 vectors.
The hash encoder
oklm_rag_encoder_hash(dim): each word of the text (the
BM25 analyzer's terms) adds 1 + log(count) to one of dim
buckets, chosen and signed by the word's 64-bit hash; the vector is
L2-normalized. Deterministic, no model file, fast: it finds the chunks
that share words with the query, weighted by how often, and nothing more
(no synonyms, no meaning). It is what the tests and the measures below
use, and what oklm rag uses when no embedding model is
given.
The index, measured
tests/rag/bench_index.c
(build/release/bin/tests/rag/bench_index CHUNKS 384 200 0 f16):
synthetic documents of 100 paragraphs of 48 words (one chunk each) over
a 20,000-word vocabulary drawn with a Zipf-like law, the hash encoder at
384 dimensions, F16 vectors, on an i7-11700K (8 cores, 16 threads);
queries of six words, 200 per mode, each timed from the text to the hits
(embedding the query included).
| Chunks | Build | of which embedding | chunking, BM25, write | Open | File | Dense p50 / p95 | Sparse p50 / p95 | Hybrid p50 / p95 |
|---|---|---|---|---|---|---|---|---|
| 10,000 | 0.12 s | 0.05 s | 0.07 s | 0.3 ms | 14 MB | 0.04 / 0.04 ms | 0.05 / 0.09 ms | 0.11 / 0.15 ms |
| 100,000 | 1.11 s | 0.46 s | 0.60 s | 2.3 ms | 140 MB | 2.40 / 2.58 ms | 0.46 / 0.87 ms | 3.09 / 3.52 ms |
| 1,000,000 | 10.9 s | 4.6 s | 5.9 s | 22 ms | 1.40 GB | 27.2 / 28.4 ms | 5.7 / 9.0 ms | 33.4 / 37.4 ms |
| 1,000,000, Q8_0 vectors | 11.9 s | 5.7 s | 5.8 s | 22 ms | 1.04 GB | 14.5 / 15.0 ms | 5.7 / 8.9 ms | 20.5 / 24.1 ms |
A dense query reads the whole vector tensor: 768 MB of F16 at a million chunks in 27 ms is 28 GB/s, the memory bandwidth the design counted on (about 30 ms); at ten thousand chunks the 7.7 MB stay in the cache. The build time is the hash encoder's (a real model embeds at milliseconds a chunk: its time dominates) plus chunking, the postings and the write; opening checks every table (offsets, postings, chunk to source).
Exact search stays well under the design's 100 ms threshold for an approximate index at a million chunks, so none is added (decision 6 of rag-design.md).
Hybrid search
oklm rag search INDEX QUERY [-k 10] [--mode dense|sparse|hybrid] [--fusion rrf|weighted] [--rrf-k 60] [--weights 0.5,0.5] [--candidates 50] [--rerank MODEL [--rerank-top 50]] [--json].
The default mode is hybrid when the index has postings (the default),
dense otherwise; --rerank reorders the candidates (Reranking).
- sparse (BM25): the postings are built from the
chunk texts at index time and stored in the same file: terms sorted by
bytes, each with the chunks holding it (ascending) and its frequency
there, and every chunk's length in terms. The analyzer
(
unicode-lower-s-1) takes runs of Unicode letters, digits and marks (_and everything else separate words, sooklm_matmulisoklmandmatmul), lowercases them as Python'sstr.lower()does, and strips English plurals (the S-stemmer:poniestopony,dogstodog;--no-stemkeeps words as they are). A chunk scores, over the query's distinct terms it holds, idf × tf × (k1 + 1) / (tf + k1 × (1 − b + b × len / avg_len)), idf = ln(1 + (N − df + 0.5) / (df + 0.5)), k1 = 1.2, b = 0.75; checked inmake testagainst a corpus computed by hand. - hybrid: the 50 best of each retriever, fused by
their scores min-max normalized per list and weighted 0.5 / 0.5
(
--weights DENSE,SPARSE) or, with--fusion rrf, by reciprocal rank (score = Σ 1 / (60 + rank),--rrf-kfor another constant).--rag, the server and the agent search with the same defaults. Each hit says which retriever found it (dense,sparseor both). Measured in Hybrid retrieval, measured.
In the library: oklm_rag_query() runs a mode;
oklm_rag_index_search() (a vector),
oklm_rag_index_bm25() (a text),
oklm_rag_fuse_rrf() and
oklm_rag_fuse_weighted() are the pieces, for a program that
brings its own lists.
Reranking
A first retrieval (dense, BM25 or hybrid) reads the query and each chunk apart and compares two summaries. A reranker reads them together: it scores each (query, passage) pair with one model, so it sees which words of the passage answer which words of the query. It is slower (one forward pass a pair, not a dot product), so it reorders only the first candidates of the retrieval: the 50 best by default.
oklm rerank -m models/ms-marco-MiniLM-L-6-v2 -q "How many people live in Berlin?" \
"Berlin has a population of 3,520,031 registered inhabitants." \
"New York City is famous for the Metropolitan Museum of Art."
oklm rerank -m models/ms-marco-MiniLM-L-6-v2 -q "what is BM25" --file passages.txt --top 5 --json
oklm rag search docs.gguf "how is the KV cache quantized" --rerank models/ms-marco-MiniLM-L-6-v2 --rerank-top 50 -k 5
oklm serve -m model.gguf --rerank-model models/ms-marco-MiniLM-L-6-v2 # POST /v1/rerank
oklm rerank prints the passages best first with their
score and their relevance (--json:
/v1/rerank's response).
oklm rag search --rerank MODEL retrieves the
--rerank-top best candidates (50) as usual, reads each
chunk's text from the index, reranks them and prints the -k
first; each hit's score is then the reranker's.
Two kinds of rerankers
The model given decides:
- A cross-encoder: a text encoder with a
classification head, as llama.cpp names its tensors (
cls: a dense layer then tanh, BERT's pooler or RoBERTa'sclassifier.dense;cls.output: the projection to the labels). Default: cross-encoder/ms-marco-MiniLM-L-6-v2 (22 M parameters, BERT, 6 layers, trained on MS MARCO), from its Hugging Face checkpoint or a GGUF. The pair is read as[CLS] query [SEP] passage [SEP], the passage's tokens (and its closing[SEP]) of token type 1 as the model was trained; a model with one token type (RoBERTa, XLM-R: the bge rerankers) reads<s> query </s></s> passage </s>. A pair longer than the context (512 tokens) is cut longest first, as Hugging Face's tokenizers cut it: a short query is kept whole and the passage cut. The score is the head's raw logit (llama.cpp's RANK pooling); several pairs share each forward pass. - The generator (any chat model, when no
cross-encoder is at hand): each pair becomes a yes / no question
answered through decide.c (Decisions by scoring):
the state
Query: ...\n\nPassage: ..., the question "Is the passage relevant to the query: does it answer the query, or help answer it?", the answersYes/Noscored as the continuation of the prompt (one forward pass over the prompt when each answer is one token, the system turn and the query's prefix kept in the cache from one passage to the next). The score is the log-odds log p(yes) − log p(no). A passage too long for the context (2048 tokens) is cut at its end.
Both scores are logits: the sigmoid turns them into a relevance in
(0, 1) (/v1/rerank's relevance_score). They
rank passages; they are not calibrated probabilities.
In the library
#include "oklm/rag/rerank.h"
oklm_reranker* r = NULL;
float s[2];
const char* docs[2] = {"first passage", "second passage"};
oklm_reranker_new(model, NULL, &r); /* a cross-encoder, or a chat model (decide.c) */
oklm_rerank(r, "the query", docs, 2, s); /* s[i]: a logit, higher is more relevant */
oklm_rag_rerank_hits(r, index, "the query", hits, n); /* oklm_rag_query()'s hits reordered */
oklm_reranker_free(&r);Below it, oklm_embed_pairs() encodes pairs with any
pooling (RANK for the head's values),
oklm_embed_pair_tokens() shows the tokens and the first
token of type 1, oklm_model_classify() applies the head to
a state; oklm_rerank_order() sorts scores (stable) and
oklm_rerank_probability() is the sigmoid.
In the server
oklm serve -m MODEL --rerank-model R [--rerank-model-id NAME] [--rerank-backend cpu|opencl|vulkan|auto]
answers POST /v1/rerank (and /rerank) in
Cohere's and Jina's shape (server.md): query,
documents (strings or {"text": ...}),
top_n, return_documents; results
best first with index, relevance_score and,
when asked, document.
Differences from llama.cpp
- llama.cpp's converter drops BERT's pooler. For
BertForSequenceClassificationit writes only the classifier (cls.output) and leaves outbert.pooler.dense, which the classifier reads (logits = classifier(tanh(pooler(h[CLS])))). Its GGUFs of the MiniLM cross-encoders (GGUF-my-repo's among them) therefore score another function: on the Berlin example above 0.08 and −0.22 instead of 8.85 and −11.25, and on SciFact the ranking falls (below). OKLM runs such a file as llama.cpp does, and loads the checkpoint with its pooler ascls;oklm convertwrites it so, and llama.cpp then applies it too (its RANK head runsclswith tanh when the file has it). - Token types. llama.cpp gives every token type 0;
OKLM gives the passage type 1, as the model was trained (the scores of
the MiniLM cross-encoder move by up to 10 logits without it).
OKLM_EMBED_TYPE0=1makes every token type 0, for the comparison withllama-embeddingonly. - GELU: the exact one, as for embeddings
(
OKLM_EMBED_GELU_TANH=1for the comparison). - Long pairs are cut longest first, as the model's
tokenizer does;
llama-embeddingrefuses a pair longer than its batch.
Reranking, measured
Scores against the two references
(tools/embed_compare.py, 990 pairs: 10 queries of these
docs, 99 or 100 passages each; absolute difference of the logits, worst
and mean):
| File | Against | Worst | Mean |
|---|---|---|---|
checkpoint, and oklm convert's F32 GGUF |
numpy float64 (tools/embed_ref.py --pairs) |
4.0e-6 | 4.0e-7 |
oklm convert's F32 GGUF, OKLM with
OKLM_EMBED_GELU_TANH=1 OKLM_EMBED_TYPE0=1 |
llama-embedding --pooling rank -fa off, same file |
1.3e-3 | 2.3e-4 |
numpy float64 with ggml's F16 GELU table and type 0
(--gelu-f16 --type0) |
llama-embedding --pooling rank -fa off, same file |
4.0e-4 | 6.2e-5 |
| GGUF-my-repo's F16 GGUF (no pooler), OKLM with both switches | llama-embedding --pooling rank (flash attention on),
same file |
2.5e-4 | 5.3e-5 |
The numpy proof (1e-5) holds with room. Against llama.cpp the logits
part by up to 1.3e-3, above the 1e-4 aimed at: ggml's CPU GELU rounds
its input to F16 and reads an F16 table (GGML_GELU_FP16),
about 1e-3 relative a value, and these logits run from −11 to +2. The
numpy reference made to round the same way comes within 4.0e-4 of
llama.cpp (mean 6e-5), the rest being F16 roundings that flip on
differences upstream; the published F16 file, whose scores are a hundred
times smaller (no pooler), parts by 2.5e-4. Without the type-0 switch
OKLM and llama.cpp part by up to 10 logits: the token types matter to
this model.
Quality and cost on BEIR SciFact's test split (300 queries, 5,183
abstracts; bench_rerank, below): the candidates are the
hybrid search of an index of the abstracts embedded with
bge-small-en-v1.5 F32 (bge's query instruction, BM25 and dense fused by
RRF), the reranker reorders the top 50 or 100, nDCG@10 as BEIR scores
it. The added time is the reranker's alone, per query (the pairs read,
scored and sorted; CPU: 8 threads of an i7-11700K, GPU: a Radeon RX 7900
XT through OpenCL):
| Candidates | Reranker | Queries | nDCG@10 before | after | Added per query, GPU | CPU |
|---|---|---|---|---|---|---|
| hybrid top 100 | ms-marco-MiniLM-L-6-v2 (checkpoint) | 300 | 71.64 | 68.95 | 1231 ms | 3793 ms (first 50 queries) |
| hybrid top 100, first 50 reranked | ms-marco-MiniLM-L-6-v2 | 300 | 71.64 | 69.50 | 618 ms | 1902 ms (first 50 queries) |
| hybrid top 50 | bge-reranker-base (GGUF-my-repo F16) | first 100 | 78.03 | 74.73 | 2612 ms | |
| hybrid top 50 | qwen2.5-0.5b-instruct Q4_K_M (decide.c) | first 50 | 84.41 | 20.44 | 1039 ms |
Reranking lowers nDCG@10 on SciFact: the design's
proof ("after reranking above the hybrid's") is not met with these
rerankers. The MiniLM figure is not a fault of the reranker's code:
reranking BM25's top 100 with the same model gives 68.8 in the BEIR
paper (Thakur et al., 2021: BM25 + CE), and OKLM's 68.95 over the
hybrid's top 100 is that figure; bge-small's own dense retrieval (71.27)
is already stronger on these scientific claims than a cross-encoder
trained on MS MARCO's web questions. bge-reranker-base (XLM-R, 278 M)
loses less (−3.3 on the first 100 queries, MiniLM −5.3 on the same
queries). The generator through decide.c with a 0.5 B model is not a
usable reranker here (its order is far worse than the hybrid's); it is
the fallback for when no cross-encoder is present, and a larger chat
model is what it needs: on the Berlin example (with "The capital of
Germany is Berlin." as a third, unhelpful passage) qwen2.5-0.5b and
Qwen3-0.6B call all three relevant (logits 1.2 to 5.3, though the
population line ranks first or second), while Qwen3-4B separates them
(38.1 against −27.9 and −29.9) (the yes / no prompt runs about 450
tokens a passage, 1 s for 50 on the GPU with a 0.5 B model). The
reranker earns its cost where the first retrieval is weaker than it
(BM25 alone, the hash encoder, out-of-domain embeddings), not over a
strong in-domain embedding model; --rerank is off by
default.
The run: build/release/bin/tests/rag/bench_rerank, built
with the benches (make bench);
bench_rerank EMBED_MODEL RERANK_MODEL [--backend cpu|opencl] [--queries N] [--top 100]
indexes the corpus once (build/rag-eval/scifact-bench.gguf,
5,773 chunks of at most 500 tokens, 116 s on the GPU), then for each
test query runs oklm_rag_query() (hybrid,
--top candidates per retriever and kept) and
oklm_rag_rerank_hits(), and prints nDCG@10 before and after
with the reranking time per query. A pair longer than 512 tokens is cut
(1 pair in 100 on average).
Tests
tests/rag/test_rerank.c in make test, with
no download: a tiny random BertForSequenceClassification
(tests/rag/fixtures/tiny-ce, as a checkpoint, as llama.cpp
writes it with its pooler, and through oklm convert)
against tools/embed_ref.py --pairs within 1e-5 on eight
pairs (empty query or passage, a long passage, a long query, both long:
the tokens, the first type-1 token and the cuts equal to Hugging Face's
tokenizer), pairs packed in one call equal to one at a time, the type-0
switch, an embedding model refused, the order and the sigmoid, search
hits reordered, OpenCL within 1e-4; with the public model present,
MiniLM on the Berlin example within 1e-4 of the numpy reference.
tests/server/test_server.c checks /v1/rerank
(both document forms, top_n, return_documents,
/rerank, every refusal, 404 without a reranker) and the
decide.c fallback: the generator's scores equal to the yes / no question
scored directly on a fresh context, the same order, the same through the
server.
Retrieval in generation
With --rag INDEX, oklm run,
oklm chat, oklm serve (per request) and
oklm agent (a tool) answer from the passages of a retrieval
index built by oklm rag index: the question is searched,
the best passages go into the prompt numbered, with their file, lines
and heading, and the model is asked to answer from them only, to cite
them as [n] and to say when they do not answer. Corrective
retrieval (on by default) has the model judge each passage first,
rewrite the question when none is relevant, and, when nothing reliable
is found, answer that no reliable source was found rather than cite
anything.
oklm rag index docs README.md --out docs.gguf -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf \
--query-prefix "Represent this sentence for searching relevant passages: "
oklm run -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --backend opencl --rag docs.gguf \
--template-vars '{"enable_thinking": false}' -n 400 -p "How do I keep the prefix cache across server restarts?"
oklm chat -m MODEL --rag docs.gguf --rag-k 8 --rag-log rag.jsonl
oklm chat -m MODEL --rag docs.gguf --rag-route on # no retrieval, one or several, as the model decides
oklm serve -m MODEL --rag docs.gguf # requests opt in with "rag": true
oklm agent -m MODEL --rag docs.gguf # the Retrieve tool over that index
oklm agent -m MODEL --rag-project --rag-memory # over the project's own docs, kept indexed, and earlier sessions
The options are in Answers from an index.
What the model sees
The question is replaced, for that turn only, by one user message:
Answer the question using only the numbered sources below. After each statement, cite the sources that support it
by their numbers in square brackets, like [1] or [2][3]. Cite only these numbers. If the sources do not contain the
answer, say that the sources do not answer the question instead of guessing.
Sources:
[1] docs/server.md:89-121 (Running it > Prefix cache on disk)
...the passage's text...
[2] README.md:96-112 (Quick start)
...
Question: How do I keep the prefix cache across server restarts?
The passages come in the order of the search (and of the reranker,
when one is set), as many as fit a budget of 2048 tokens counted with
the generator's tokenizer: one that does not fit is left out, the first
one is cut to fit when it alone is too long. When nothing reliable was
found the message says so instead
(No reliable source was found for this question ... Begin your answer with "No reliable source was found." Do not cite any source and do not invent references),
then the question. In oklm chat the conversation keeps the
question as typed, so the sources of earlier turns do not pile up in the
context. oklm run --rag renders the prompt through the
model's chat template (with --system when given) and prints
only the answer.
Corrective retrieval
Each passage is judged by the generator itself through a decision by scoring (server.md, decide.h): the state is the question then the passage, the question "Does the passage above contain information that answers the question, fully or in part?" with the options yes and no, each scored in one forward pass from the shared prompt (the question's part is evaluated once per query, then each passage's). The probability of yes is the passage's relevance: relevant at 0.5 or more, irrelevant under 0.2, ambiguous between (the design's defaults, rag-design.md).
- The question is searched and its
kpassages judged. One relevant passage or more: the context is the relevant and ambiguous ones, the irrelevant ones dropped (fallbacknone). - Otherwise the retrieval is poor: the model rewrites the question as
a search query (a short greedy generation, one line: "use the specific
names, commands, options and terms the documentation would use"), which
is searched and judged in turn (fallback
rewrite). A rewrite equal to a query already searched is not searched again. - Still nothing relevant: the web, when allowed, which only the agent
allows, and only when its WebSearch tool has a service
(
webSearch.url) and its permission rules let the call run; its results are judged the same way (fallbackweb). - Still nothing: the context is empty and the prompt asks for the "no
reliable source" answer (fallback
no_source). No passage is cited, so no citation can be invented from it.
Every run is one record: the question, its rewrites, each search with
its passages (chunk, source, lines, heading, score, the retrievers that
found it, the reranker's score, the probability of relevance, the
verdict, the number it got in the prompt), the fallback, the sources
cited, the context's tokens, the times. --rag-log FILE
appends it as a JSON line; oklm run and
oklm chat print a summary on stderr:
(retrieval: 5 passages in 1 search, 2 relevant, 1 ambiguous; none; [1] docs/server.md:89-121 [2] docs/server.md:15-88 [3] README.md:96-112)
In the server
oklm serve --rag INDEX [--rag-k N] [--rag-mode M] [--rag-crag on|off] [--rag-route on|off] [--rag-rewrite on|off] [--rag-rerank MODEL [--rag-rerank-top N]] [--rag-log FILE]
opens the index once; a chat completion, or an Anthropic message
(/v1/messages), opts in with its rag field
(default off; server.md):
"rag": true takes the server's settings,
"rag": {"k": 8, "crag": false, "max_tokens": 1024, "route": true, "rewrite": false}
changes them for the request. The judge's questions, the router's
choice, the rewrite, a follow-up's condensing and a multi-hop request's
sub-questions run through the scheduler as requests of their own (a
scoring request per option, the first alone so the others find the
shared prefix cached; a short greedy generation). A multi-turn request's
last user message is condensed from the messages before it. The answer
carries the run in a rag member, with the texts of the
sources cited, so a client can check the citations: the whole answer's
top level; streamed, the last chunk (the one with the
finish_reason) of a chat completion, and the
message_delta event of an Anthropic stream.
/api/stats counts the runs (rag.requests,
judged, relevant, ambiguous,
irrelevant, rewrites, no_source,
context_chunks, context_tokens,
errors, condensed, routed,
route_none, route_one,
route_multi, hops).
In the agent
oklm agent --rag INDEX gives the agent the built-in tool
Retrieve (query, optional k) over
that index; --rag-project gives it the project's own
documentation instead, indexed and kept up to date by the agent (Agentic retrieval, which also covers the
session memory and the passages given with each message). It runs the
same pipeline: with an in-process model, the judge and the rewriter use
a context of their own on it (4096 tokens, made at the first call, so
the agent's cache is left as it is); with a model served elsewhere
(--url) they are requests to its server (On a served model). Its result is the
numbered sources to cite, or "No reliable source was found in the index
for this query ... Do not cite a source for it". When nothing relevant
is found and WebSearch has a service, the web is searched through the
WebSearch tool, its rules and questions included. Each run goes to
--rag-log and to the debug log
(retrieval {...}).
On a served model
oklm agent --url URL has no model in process; its
retrieval stages and its critic are then requests to the server
(include/oklm/rag/rag_http.h,
src/rag/rag_http.c). At start, when retrieval is on, the
agent finds once how the server answers a closed question, and says so
on stderr:
| The server | Judge, router, critic | Told |
|---|---|---|
OKLM's (/v1/decide answers) |
POST /v1/decide: the same question on the same state as
in process (a noul for the judge and the critic, a
choice of none / one /
several for the router), each option scored by decide.c in
the server's scheduler, its prefix cache shared with the agent's own
requests |
"judge, router and critic through the server's /v1/decide" |
another, with logprobs |
a chat completion: the state, the question and "Allowed options:"
(decide.h's layout), one token at temperature 0 with
top_logprobs 10; each alternative whose first word begins
exactly one option adds its probability to it, normalized over the
options |
"the server has no /v1/decide: ... log-probabilities" |
| another, without them | the same request read as text (16 tokens): the option its first word names is certain; none named gives yes 0.3, ambiguous at the default thresholds | "... read from the answer's text" |
| neither answers | naive retrieval, --rag-critique refused |
"no judge, router or rewrite" |
The probe is one /v1/decide request (a 404, or an answer
without probabilities, moves on), then one chat completion asking for
log-probabilities (sent again without them when the server refuses the
request with a 400). The rewriter, the condenser and the decomposer are
chat completions at temperature 0 with the in-process prompts and token
limits (64, 96, 160), their answers cleaned the same way; on an OKLM
server they carry chat_template_kwargs
{"enable_thinking": false}, which another server is not
sent (it may refuse an unknown member). A stage whose request fails is
handled as an in-process stage that fails: a rewrite or condensing
skipped and noted, the passages of a failed judgement left unjudged.
In the library
include/oklm/rag/rag.h. The pipeline is made of stages,
each one C interface (a function and its state) a program can
replace:
oklm_rag_store store; /* the index, its encoder, its search settings */
oklm_rag_store_open("docs.gguf", NULL, NULL, &store, err, sizeof(err));
oklm_rag_retriever rt = oklm_rag_store_retriever(&store);
oklm_rag_model gen; /* the generator as judge and rewriter */
oklm_rag_model_init(&gen, ctx, tmpl);
oklm_rag_evaluator ev = oklm_rag_model_evaluator(&gen);
oklm_rag_rewriter rw = oklm_rag_model_rewriter(&gen);
oklm_rag_stages st = {&rt, NULL /* reranker */, &ev, &rw, NULL /* web */};
oklm_rag_result res;
oklm_rag_run(NULL /* defaults */, &st, question, &res);
/* or a turn of a conversation: a follow-up condensed, the request routed */
oklm_rag_condenser co = oklm_rag_model_condenser(&gen);
oklm_rag_router ro = oklm_rag_model_router(&gen);
oklm_rag_decomposer de = oklm_rag_model_decomposer(&gen);
st.condenser = &co, st.router = &ro, st.decomposer = &de;
oklm_rag_params p = oklm_rag_default_params();
p.route = 1;
oklm_rag_history_text(msgs, n_before, 0, &history); /* the earlier turns, as text */
oklm_rag_run_turn(&p, &st, history, question, &res);
oklm_rag_prompt(&res, question, &user_message); /* then generate as usual */
oklm_rag_citations(answer, res.n_context, cited, 16, &n_cited, &n_invalid);
oklm_rag_result_json(&res, "mine", &line);- Retriever (
oklm_rag_retriever): a query to passages;oklm_rag_store_retriever()searches an index (hybrid by default). - Reranker (
oklm_rag_reranker, optional): a score per passage;candidates(20) passages are handed to it and the bestkkept.oklm_rag_cross_reranker(reranker)makes it of a rerank.h reranker (a cross-encoder, or the generator through decide.c). - Evaluator (
oklm_rag_evaluator): a probability of relevance per passage;oklm_rag_model_evaluator()is the generator's judgement on a context.oklm_rag_judge_state()andoklm_rag_judge_question()give its question to a program that scores it its own way (the server does, through its scheduler). - Rewriter (
oklm_rag_rewriter):oklm_rag_model_rewriter(), fromoklm_rag_rewrite_prompt()andoklm_rag_rewrite_clean(). - Web (
oklm_rag_web, optional): tried last whenwebis set. - Router (
oklm_rag_router, optional): a probability for each route (none, one, several);oklm_rag_model_router()is the generator's decision (Adaptive retrieval). - Decomposer (
oklm_rag_decomposer, optional): a multi-hop request's sub-questions;oklm_rag_model_decomposer(). - Condenser (
oklm_rag_condenser, optional): a follow-up made standalone from the earlier turns;oklm_rag_model_condenser()(Conversational retrieval).
oklm_rag_params: k (5),
candidates (20), max_tokens (2048),
corrective (1), relevant (0.5),
irrelevant (0.2), rewrites (1, at most 3),
web (0), route (0), condense (1),
max_hops (3), and a token counter (four bytes a token
without one). The same stages on a served model are
oklm_rag_http_evaluator(), _rewriter(),
_router(), _decomposer(),
_condenser() and _critic() of an
oklm_rag_http client (oklm_rag_http_new(),
then oklm_rag_http_probe(): On
a served model). A rewriter or web search that fails is noted in the
result's error and skipped; an evaluator that fails leaves
the passages unjudged (naive retrieval).
oklm_rag_stats_add() sums runs into counters.
Reranking the candidates
--rag-rerank MODEL (on run,
chat, serve and the agent) puts a reranker (Reranking) between the search and the judge: the
search returns --rag-rerank-top candidates (20), the
reranker scores each against the query (one model reading both), and the
best k by its score go on to the judge and the prompt, in
its order. A cross-encoder (models/ms-marco-MiniLM-L-6-v2)
runs on the CPU beside the generator; a chat model given there reranks
through decide.c. Each passage's score is in the log line
(rerank). It is off by default: on SciFact the MiniLM
cross-encoder ranked below the hybrid search it reorders (Reranking, measured). In the library:
oklm_rag_cross_reranker(reranker) is the stage.
Reranking in generation, measured
On 2026-11-10, the setup of Retrieval in generation,
measured (Qwen3-8B Q4_K_M on the GPU, the docs index with bge-small
and weighted fusion, 5 passages), with and without
--rag-rerank models/ms-marco-MiniLM-L-6-v2 (the first 20
candidates, the reranker on the CPU beside the generator); both runs the
same day:
| Retrieval | Accuracy | Answerable | Unanswerable declined | Faithfulness | s / q |
|---|---|---|---|---|---|
| naive | 84.2 % | 25 / 30 | 7 / 8 | 88.7 % (102 / 115) | 1.9 |
| naive, reranked | 86.8 % | 26 / 30 | 7 / 8 | 81.1 % (103 / 127) | 2.3 |
| corrective | 86.8 % | 25 / 30 | 8 / 8 | 88.0 % (95 / 108) | 2.8 |
| corrective, reranked | 89.5 % | 26 / 30 | 8 / 8 | 87.7 % (107 / 122) | 3.1 |
The retrieval behind it (oklm rag eval --rerank,
bge-small hybrid with weighted fusion, the reranker on the GPU):
| Set | Reranked | nDCG@10 | MRR@10 | R@1 | R@5 | R@10 | p50 ms |
|---|---|---|---|---|---|---|---|
| docs (120) | none | 70.61 | 74.20 | 30.2 | 73.5 | 86.7 | 4.0 |
| first 20 | 71.07 | 78.30 | 37.4 | 73.8 | 84.4 | 138 | |
| first 50 | 71.35 | 78.15 | 37.4 | 74.0 | 85.5 | 344 | |
| code (112) | none | 59.32 | 63.40 | 30.5 | 67.1 | 81.4 | 16.5 |
| first 20 | 60.60 | 65.19 | 34.0 | 69.3 | 82.0 | 185 | |
| first 50 | 59.40 | 63.52 | 33.6 | 65.2 | 82.4 | 437 |
- On the docs and the code, unlike SciFact, the cross-encoder puts the right section or file first more often (recall@1 +7 and +3.5 points) and leaves nDCG@10 about level (+0.1 to +1.3): it reorders the top, it finds nothing the search had not found.
- Answers: one more answerable question right in both modes (26 against 25 of 30), a difference of one question, within the noise. Each configuration's misses had a judged section among their passages (the reranker dropped one of d109's two judged sections, and that answer went wrong); the other misses are the generator's.
- Faithfulness by the judge falls with naive retrieval (81.1 % against 88.7 %): answers cite more often (127 pairs against 115), and 24 of the pairs are judged unsupported against 13; the interval of the difference, −19.8 to +2.5 points, still holds 0. With corrective retrieval, whose judge drops passages after the reranker, it stays level (87.7 % against 88.0 %).
- Cost: 0.3 to 0.4 s a question on the CPU for 20 candidates.
Reranking stays off: it changes which passage comes first, not what the model answers, and with naive retrieval it may cost faithfulness. With corrective retrieval it does no harm and is worth trying where the right passage is often retrieved but not first.
On the CPU alone, measured
On 2026-11-10, the pipeline above without a GPU:
oklm serve --backend cpu (i7-11700K, 8 cores, the server's
default threads, the machine otherwise idle, load under 1), Qwen3-8B
Q4_K_M, the same index and settings, 12 questions of the answer set (the
first 10 answerable ones and 2 unanswerable),
python3 tools/rag_gen_eval.py --no-judge (no faithfulness
requests, so only the answers are timed). The same questions on the GPU
for comparison:
| Retrieval | GPU s / q | CPU s / q | of which retrieval (CPU) | of which the answer (CPU) | prompt / completion tokens |
|---|---|---|---|---|---|
| none | 4.9 | 65.3 | 0 | 65.3 | 31 / 397 |
| naive | 1.5 | 20.9 | 0.01 s | 20.9 | 1,096 / 78 |
| corrective | 2.3 | 35.2 | 22.5 s (judging 22.1 s) | 12.8 | 858 / 88 |
- The CPU decodes about 6 tokens a second with this model, and reads a prompt at about 135 tokens a second: a question with five passages answered in a hundred tokens takes 20 s, most of it reading the passages.
- Corrective retrieval's five scoring requests share the question's prefix but each reads a passage: 22 s a question on the CPU, 1.1 s on the GPU. Its shorter prompt saves 8 s of the answer, so it costs 14 s more in all, 68 % on top of naive retrieval (53 % on the GPU). On a CPU, naive retrieval is the practical setting for an 8 B generator; corrective retrieval pays where declining unanswerable questions matters more than the wait.
- The search itself (embedding the query with bge-small, hybrid search) takes 8 to 9 ms either way.
- 10 of 12 right on the CPU against 11 on the GPU: one answer differs between the back ends (not looked into).
Error bars
tools/rag_stats.py puts a 95 % interval on a row by the
bootstrap over its questions (10,000 resamples, a fixed seed) and
compares two rows over the same questions paired, from the TREC runs of
oklm rag eval --run or the answers of
tools/rag_gen_eval.py; runs gives the spread
of repeated runs:
build/venv/bin/python -I tools/rag_stats.py trec --data build/rag-eval/docs RUN [RUN2] [--metric mrr@10] [--split KIND]
build/venv/bin/python -I tools/rag_stats.py answers --mode crag A/answers.jsonl [B/answers.jsonl]
build/venv/bin/python -I tools/rag_stats.py runs --mode crag R1/answers.jsonl R2/answers.jsonl ...
The rows' intervals (2026-11-10):
| Row | Value | 95 % interval |
|---|---|---|
| SciFact, hybrid weighted, nDCG@10 | 73.04 | 68.81 to 77.13 |
| NFCorpus, hybrid weighted | 36.37 | 32.86 to 39.94 |
| docs, hybrid weighted | 70.61 | 65.96 to 75.09 |
| code, bge-small hybrid weighted | 59.32 | 53.86 to 64.71 |
| code, Qwen3-Embedding dense | 63.63 | 57.95 to 69.27 |
| answers, naive, accuracy | 84.2 % | 71.1 to 94.7 |
| answers, corrective, accuracy | 86.8 % | 76.3 to 97.4 |
| answers, naive, faithfulness | 88.7 % | 81.7 to 96.0 |
| answers, corrective, faithfulness | 88.0 % | 79.7 to 96.7 |
The comparisons the documentation draws (nDCG@10 unless named; the share of resamples at or below 0 is a one-sided p-value):
| Comparison | Difference | 95 % interval | Resamples ≤ 0 | Holds |
|---|---|---|---|---|
| SciFact, hybrid weighted − dense | +1.77 | −0.41 to +4.02 | 0.058 | no |
| SciFact, weighted − RRF | +1.76 | +0.34 to +3.24 | 0.007 | yes |
| NFCorpus, hybrid weighted − dense | +1.99 | +0.80 to +3.29 | 0.001 | yes |
| NFCorpus, RRF − weighted | +0.30 | −0.39 to +1.00 | 0.20 | no |
| docs, hybrid weighted − dense | +10.14 | +5.74 to +14.50 | 0.000 | yes |
| docs, hybrid weighted − BM25 | +6.08 | +2.44 to +9.57 | 0.001 | yes |
| docs, weighted − RRF | +0.84 | −1.21 to +2.81 | 0.21 | no |
| code, recursive − code-aware chunker | +0.80 | −2.57 to +4.07 | 0.32 | no |
| code, code-aware − fixed chunker | +1.30 | −2.00 to +4.54 | 0.22 | no |
| code, code-aware − recursive, two-file questions | +4.60 | −3.76 to +13.25 | 0.15 | no |
| code, bge-small hybrid weighted − dense | +3.83 | −0.71 to +8.41 | 0.051 | no |
| code, Qwen3-Embedding − bge-small, dense | +8.14 | +3.05 to +13.22 | 0.001 | yes |
| code, Qwen3-Embedding dense − hybrid weighted | +1.53 | −2.54 to +5.52 | 0.23 | no |
| docs, reranked (20) − not, nDCG@10 | +0.46 | −3.96 to +4.96 | 0.43 | no |
| docs, reranked (20) − not, recall@1 | +7.15 | +0.49 to +14.03 | 0.019 | yes |
| code, reranked (20) − not, recall@1 | +3.50 | −2.60 to +9.82 | 0.13 | no |
| answers, corrective − naive, accuracy | +2.63 | −5.26 to +10.53 | 0.39 | no |
| answers, reranked − not, corrective accuracy | +2.63 | −7.89 to +15.79 | 0.41 | no |
| answers, reranked − not, naive faithfulness | −7.59 | −19.78 to +2.45 | 0.92 | no |
What holds: hybrid search over either retriever alone wherever the retrievers differ much (the docs, NFCorpus), weighted fusion over reciprocal rank on SciFact, the larger embedder on code, and reranking putting the right section first on the docs. What does not: the choices between fusions on the docs and NFCorpus, between chunkers, and every difference in answers on 38 questions, where one question is 2.6 points. The defaults stay as they are (none of them is contradicted), but those choices are settled by other reasons than these sets: corrective retrieval by declining unanswerable questions and its shorter prompts, the code-aware chunker by where its chunks begin and end.
Repeated runs: four runs of each generation configuration (naive and
corrective, with and without reranking) on 2026-11-10 gave the same
verdict on all 38 questions and the same faithfulness to the pair. With
one request at a time at temperature 0, oklm serve answers
deterministically; the run-to-run spread is nil, and the questions' draw
is the whole uncertainty.
Retrieval in generation, measured
On 2026-11-09: Qwen3-8B Q4_K_M on the GPU (OpenCL) behind
oklm serve --rag, thinking off, temperature 0, at most 400
tokens an answer; the index of the documentation set's frozen corpus
(1,007 sections, bge-small- en-v1.5 F32 with its query instruction,
hybrid search by reciprocal rank, its retrieval as in the table above); 5 passages.
python3 tools/rag_gen_eval.py asks 38 questions three ways:
30 of the documentation set (every fourth, d001 to d117, each with the
facts a right answer states in
tests/rag/docs_set/answers.jsonl, regular expressions, all
required) and 8 that nothing in the docs answers (an enterprise
license's price, a Django plugin, the 2018 World Cup, the capital of
Australia...), right when the answer says the sources do not answer or
no reliable source was found. Faithfulness: each sentence that cites
[n] checked against each source it cites by the same
model's yes / no judgement (/v1/systemone, "Does the
passage support the statement?"); a citation of a number the prompt did
not give counts as unsupported.
| Mode | Accuracy | Answerable | Unanswerable (abstained) | Faithfulness (cited pairs supported) | Invented citations | No source | Rewrites | Prompt tokens / q | s / q |
|---|---|---|---|---|---|---|---|---|---|
| no retrieval | 15.8 % (6 / 38) | 4 / 30 | 2 / 8 | - | - | - | - | 31 | 4.4 |
| naive (top 5 stuffed) | 89.5 % (34 / 38) | 27 / 30 | 7 / 8 | 85.3 % (93 / 109) | 0 | 0 | 0 | 1,100 | 1.6 |
| corrective | 92.1 % (35 / 38) | 27 / 30 | 8 / 8 | 86.5 % (96 / 111) | 0 | 7 | 6 | 860 | 2.2 |
- Without retrieval the model does not know OKLM (it takes it for
another product and guesses); with it, 27 of 30 answers state every
fact. The three misses are the same with and without judging: a resume
answered from the checkpoint ADR without
--resume, a paged block's size read as the pool's (65,536 tokens), and the index format answered without "GGUF". - Corrective retrieval judged 220 passages: 139 relevant, 2 ambiguous, 79 irrelevant (the thresholds are nearly never between: the judge is sure). With the irrelevant ones dropped the prompt was 22 % shorter. On the unanswerable questions 7 of 8 runs found nothing relevant, rewrote the query once (six rewrites searched; the seventh, empty or the question again, was not) and answered "No reliable source was found." Naive retrieval answered 7 of 8 "The sources do not answer the question", but cited a passage for "the capital of Australia is Canberra [1]"; corrective retrieval said no source was found, then Canberra from its own knowledge, marked as such. Some of those answers go on with a general-knowledge guess after the notice (the prompt allows it when said); none cites anything.
- No answer cited a number outside its sources in either mode.
- Faithfulness by the judge, checked by hand on 20 (sentence, source) pairs, 10 judged unsupported and 10 supported: two of them are not statements (a "Citations:" line of an answer that listed its sources at the end); of the 18 others the hand verdict agrees with the judge on 11 (7 of 10 judged supported, 4 of 8 judged unsupported). The judge is lenient where a number comes from another source than the one cited (chunk sizes of 128 and 16 credited to the chunkers' description; the pool's 65,536 tokens read as a block's), and strict on paraphrase ("only Ogg Theora is decoded natively" judged unsupported by the passage saying so). The figure is the model's view, within about ±15 points.
- Cost: judging five passages and the occasional rewrite add 0.6 s a question to the 1.6 s of naive retrieval on this GPU (the judge's prompts share the question's prefix; their passages are the cost).
Adaptive retrieval
Not every request needs the documentation, and some need several
parts of it. With --rag-route on (run,
chat, serve; "route": true per
request; the agent's --rag-inject route) the generator
first decides, in one forward pass, how many searches a request needs: a
decision by scoring (server.md, decide.h) whose
state is Request: and the request, whose question is "A
search engine over the project's documentation can be used to answer the
request above. How many searches does answering it need?", with three
options:
| Route | Option's description | What happens |
|---|---|---|
| none | it needs no documentation: a greeting, small talk, general knowledge, arithmetic, or work on text it gives | nothing is searched; the request goes to the model as it was asked |
| one | one search: it asks about one thing | the pipeline of Retrieval in generation |
| several | several searches: it asks two or more separate questions, or combines or compares distinct things | multi-hop: the request is split, each part searched, the passages merged |
The most probable route is taken. Multi-hop: the
generator writes the separate questions the request asks (a short greedy
generation, "at most three, each one standalone, one per line"); each is
retrieved as one request is (searched, judged, rewritten when nothing
relevant is found); the passages of the sub-questions that found
something are merged rank by rank (the first of each, then the
second...), a chunk found twice kept once, within the token budget, and
numbered as one context. A decomposition into fewer than two questions
is a single retrieval of the request. A router or decomposer that fails
is noted in the run's error and the request is retrieved
once.
Every decision is logged: the run's JSON line holds
route (none, one,
multi), route_p (the three probabilities),
hops (the sub-questions) and each round's hop;
oklm run and oklm chat print
(retrieval: routed one (none 0.03, one 0.85, several 0.12); ...).
It is off by default: measured below, it does not answer better than
always retrieving.
Adaptive retrieval, measured
On 2026-11-09, as Retrieval in generation,
measured (Qwen3-8B Q4_K_M on OpenCL behind
oklm serve --rag, the documentation set's index, 5
passages, the same 38 questions),
python3 tools/rag_gen_eval.py --modes crag,route, one
run:
| Mode | Accuracy | Answerable | Unanswerable (abstained) | Faithfulness | Invented citations | Prompt tokens / q |
|---|---|---|---|---|---|---|
| corrective | 92.1 % (35 / 38) | 27 / 30 | 8 / 8 | 89.2 % (99 / 111) | 0 | 860 |
| corrective, routed | 92.1 % (35 / 38) | 27 / 30 | 8 / 8 | 88.3 % (98 / 111) | 0 | 860 |
The router chose one for all 38, with a probability of
0.99 or more for 36 of them: the eight questions the docs do not answer
(the capital of Australia, the 2018 World Cup...) were not routed to
none, and the two-part questions (d097 "How do I train a
LoRA adapter, and then serve several such adapters...", d109, d113) got
at most 0.0035 for several. So the answers are those of
corrective retrieval (the faithfulness differs by one judged pair). With
this model and question, routing costs three scoring requests a question
and changes nothing; it stays off, logged with its probabilities for
whoever tries another model or wording.
Other wordings and another model (2026-11-09,
python3 tools/rag_route_eval.py, one run). The router's
decision asked through /v1/systemone with the same
rendering, over 58 requests: the 38 questions (one
expected, except several for the 5 of kind multihop and
none for the two general-knowledge ones, u04 and u06; the
six other unanswerable ones ask about OKLM, so a search that finds
nothing is right), the 15 other two-section questions of the docs set
(several) and five requests that need no documentation (a
greeting, thanks, arithmetic, a translation, a summary of a given
sentence; none). The wordings: actions (the
options as what to do: answer directly, search once, split and search
each part), examples (the question followed by five
examples, none from the set) and both.
| Model, wording | none | one | several | matching | none expected (7) | one (31) | several (20) | ms a decision |
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B, current | 2 | 56 | 0 | 33 | 2 | 31 | 0 | 138 |
| Qwen3-8B, actions | 0 | 58 | 0 | 31 | 0 | 31 | 0 | 134 |
| Qwen3-8B, examples | 2 | 28 | 28 | 40 | 2 | 21 | 17 | 174 |
| Qwen3-8B, both | 3 | 50 | 5 | 39 | 3 | 31 | 5 | 178 |
| GLM-4.7-Flash Q4_K_M, current | 0 | 12 | 46 | 25 | 0 | 5 | 20 | 201 |
| GLM-4.7-Flash Q4_K_M, examples | 9 | 6 | 43 | 33 | 7 | 6 | 20 | 223 |
| GLM-4.7-Flash Q4_K_M, both | 0 | 11 | 47 | 24 | 0 | 4 | 20 | 242 |
Only the few-shot examples make Qwen3-8B route at all, and they split
ten single questions too (some do ask two things: d001 "which size names
..., and what do the -V and -A suffixes mean"); the greeting and the
arithmetic go to none with every wording but
actions, the thanks, the translation and the summary never
do. GLM-4.7-Flash, the larger model here, calls almost every request
several. Answers with the examples wording
built in (tools/rag_gen_eval.py --modes crag,route, the
same run):
| Mode | Accuracy | Answerable | Declined | Faithfulness | Invented citations | Prompt tokens / q | s / q |
|---|---|---|---|---|---|---|---|
| corrective | 89.5 % (34 / 38) | 26 / 30 | 8 / 8 | 90.3 % | 0 | 866 | 2.7 |
corrective, routed by examples |
84.2 % (32 / 38) | 24 / 30 | 8 / 8 | 82.8 % | 1 | 1,149 | 3.2 |
It routed 14 answerable questions to several searches and none to no search (u04 and u06 among the eight routed to one). The decompositions mostly reword one question three times ("How do I create a retrieval index with fixed-size chunks?", "How do I set overlapping chunks...?", "How do I configure chunk overlap...?"), so the context holds three near-copies of the same sections; two answers that were right became wrong (d073, d105). Corrective retrieval's 89.5 % here against 92.1 % earlier is the run-to-run difference of the server (see the roadmap's determinism item). No wording routes sensibly without losing answers, so the prompt is unchanged and the router stays off.
Conversational retrieval
A follow-up ("and how do I turn it off?") searched as it is finds
nothing: its subject is in the turns before it. With
--rag-rewrite on (the default) the generator first rewrites
it as a standalone query from the conversation: the last user and
assistant messages (six at most, each cut at 1200 bytes, the reasoning
dropped, 4000 bytes in all) and the follow-up, with the instruction
"Rewrite the follow-up message at the end as a standalone search query
that can be understood without the conversation: replace pronouns and
references ... If it is already standalone, repeat it unchanged",
decoded greedily on one line. By default
(--rag-followup both) the previous user question and that
standalone query, joined by a space, are what is searched (and judged
against): the previous question brings back the subject when the rewrite
drops it. --rag-followup rewrite searches the rewrite
alone, join the previous question and the follow-up as
written (no generation; also what both falls back to when
the rewrite fails). The prompt keeps the question as the user wrote it,
since the conversation is in the context. Where:
oklm chat --rag (each turn after the first), a multi-turn
request to oklm serve --rag (the messages before the last
user message; "rewrite": false per request turns it off,
"followup" picks the mode), and the agent's passages per
message (--rag-inject). A first message has no history and
nothing is rewritten; oklm run has a single message. The
run's line holds the query searched in standalone.
Conversational retrieval, measured
tests/rag/docs_set/followups.jsonl: 30 conversations
made of two-part questions of the documentation set, the first part
asked and answered, the second asked as a follow-up that does not name
its subject ("How do I stop contexts from applying them?" after a
question about oklm tune's cache), judged by the qrels of
the question it came from.
python3 tools/rag_followup_eval.py against
oklm serve --rag (Qwen3-8B Q4_K_M on OpenCL, the
documentation set's index, hybrid search with weighted fusion, naive
retrieval of 5 passages so only the query differs), 2026-11-09, one
run:
| Query searched | rag field |
recall@5 | hit@5 | s / q |
|---|---|---|---|---|
| the follow-up alone (no rewrite) | "rewrite": false |
26.9 % | 11 / 30 | 0.57 |
| the follow-up rewritten from the conversation | "followup": "rewrite" |
71.1 % | 26 / 30 | 0.68 |
| the previous question and the follow-up joined | "followup": "join" |
82.2 % | 30 / 30 | 0.49 |
| the previous question and the rewrite (the default) | "followup": "both" |
83.3 % | 30 / 30 | 0.42 |
| the original two-part question (upper bound) | one message | 84.4 % | 30 / 30 | 0.55 |
The rewrite more than doubles the recall of a follow-up, for 0.1 s of
generation, but alone it does not reach the plain join of the previous
question and the follow-up on this set, where every follow-up is about
the question just before it: the 8B model's rewrite sometimes keeps the
follow-up's words and drops the subject ("What does the --clip option
limit?" without oklm rl; "Who allocates the output tensors
of an operation?"). Searching the previous question with the rewrite
gives the subject back and keeps what the rewrite resolved: it equals
the join on 29 conversations and does better on one (a LoRA follow-up
whose rewrite names the option meant: 2 of 3 sections against 1), 1.1
points under the original question. It is the default; join
costs no generation, and rewrite searches without the
previous turn's words, for a conversation that moved on (which this set
does not test). The seconds are noisy (the GPU was shared). An earlier
run with reciprocal rank fusion, the default then, gave 32.8 % alone,
70.3 % rewritten (66.9 % with a first instruction; telling it to always
name the subject gave 70.3 %), 79.2 % joined and 80.3 % for the original
question.
Agentic retrieval
The agent can search its project's documentation the way it reads
files: oklm agent --rag-project (or the setting
rag.enabled) gives it the built-in tool
Retrieve, over an index of the project the agent builds
and keeps up to date itself, and lets the model decide when to search,
what for, and to search again with other words when the passages do not
answer.
oklm agent -m MODEL --rag-project # Retrieve over the project's docs
oklm agent -m MODEL --rag-project --rag-memory # and over earlier sessions
oklm agent -m MODEL --rag-project --rag-inject always # passages given with each message too
The project's index. Its sources are the project's
files the include globs select (rag.include; by default the
documentation: **/*.md, *.markdown,
*.mdx, *.rst, *.adoc,
*.txt), minus the exclude globs (rag.exclude,
a directory's name leaves its files out), the files the
.gitignore files ignore and those a Read deny rule covers
(the walk is Glob's), within rag.maxBytes (200 MB); each
recorded under its path relative to the project. It is a RAG index file
in the state directory
(~/.local/state/oklm/rag/project-HASH.gguf, HASH of the
project's path; rag.index names another file), so the
project's tree stays as it was. Nothing is read before the first search;
before each search the agent lists the files and compares their names,
sizes and modification times with the last listing: unchanged, nothing
is read; otherwise the index is updated by content hash (a file that did
not change keeps its chunks and vectors, a changed or new one is
embedded, a removed one dropped), so an edit the agent just made is
found by its next search. The embedding model is rag.model
(a path, a file name looked for as the models are, hash for
the hash encoder; bge-small-en-v1.5-q8_0.gguf by default,
on the CPU, the hash encoder when it is not found), with
rag.queryPrefix before each query (none by default); the
search is hybrid. --rag INDEX gives Retrieve a fixed index
instead, searched as it is.
The tool. Retrieve (query,
k, and scope with the memory) runs the
pipeline of Retrieval in
generation on the model's query: judged and rewritten by the
generator on a context of its own (an in-process model) or by the server
(--url), the web last when WebSearch has a service, the
reranker when --rag-rerank is given. Its result is the
numbered sources to cite, or that no reliable source was found. It only
reads, so the permission modes treat it as Read inside the project: it
runs without a question in every mode, plan mode included; a deny rule
Retrieve (--disallowedTools Retrieve) removes
it from the tools offered. It is a core tool (in the prompt from the
first turn) when retrieval is on and absent otherwise; the design
planned it deferred, but a small model rarely loads a deferred tool, and
the user turned it on to be used (deferredTools can still
list it).
The session memory. With --rag-memory
(the setting rag.memory) Retrieve takes a
scope: project (the default),
memory or all. The memory's index holds the
project's earlier sessions (each transcript of
.oklm/sessions/ as session/ID: its user and
assistant messages as text, the agent's reminders left out; the current
session left out, its conversation being in the context) and the memory
files the agent loaded (AGENTS.md and the others, their
imports). It is ~/.local/state/oklm/rag/memory-HASH.gguf,
updated the same way before a search of the memory: a session that went
on is embedded again, the others kept. A separate tool was the other
choice; one tool with a scope keeps the prompt shorter and lets
all merge both (the passages of each taken in turn).
Passages with each message.
--rag-inject always (rag.inject) runs the
pipeline on each user message before the model's first turn, the message
condensed from the conversation when it is a follow-up, and gives the
passages found as a reminder after the message ("Passages retrieved for
this request ... cite them as [n]"); nothing when nothing relevant was
found. --rag-inject route asks the router first, so a
request that needs no documentation gets nothing, and a multi-hop one
the passages of each part. With the memory on, both are searched. This
is the fixed retrieval of run and chat brought
to the agent; Retrieve stays offered unless denied. Sub-agents use
Retrieve but get no passages per message.
The settings are in The agent.
The run options of Retrieval in
generation (--rag-k, --rag-mode,
--rag-crag, --rag-rewrite,
--rag-rerank, --rag-log) apply to the agent's
retrieval as well; with --debug each index update and each
run is a line of the debug log
(retrieval index: 6 files, 6 embedded, 0 kept ...,
retrieval {...} with surface
agent for a Retrieve call, agent-message for
the passages of a message). In the library:
include/oklm/harness/rag_index.h
(oklm_rag_project_update(),
oklm_rag_memory_update(),
oklm_rag_transcript_text()), the tool context's
retrieve search and
oklm_agent_params.rag_context for the passages per message.
When Retrieve is offered, the system prompt ends with one line saying
what it searches and to use it before guessing (a small model rarely
calls a tool it is not told when to use).
Agentic retrieval, measured
Eight agent eval tasks need the project's documentation or an earlier
session (tests/evals/18-* to 25-*): a made-up
tool, "quarry", its README and nine pages of docs
(tests/evals/quarry-docs.sh), each fact a task needs stated
once among other sections: a configuration key to set (18), a command to
give (19), two facts of one page (20), constants of the code to fix by
the documented policy (21), the meaning of an exit status (25), the
reason for a limit (24); and two that only an earlier session's
transcript answers (22: create the release branch "we agreed on"; 23: a
changelog entry "following the convention I gave you"). Each is judged
by what it leaves (a file's content, a branch).
sh tools/eval_agent.sh MODEL --only 18-...,25-... --backend opencl
with Qwen3-8B Q4_K_M on OpenCL, in-process, every mode with the
project's index and the memory
(--rag-project --rag-memory), each run once,
2026-11-09:
| Mode | Options | Passed | Turns a task | Tokens in all | Seconds in all |
|---|---|---|---|---|---|
| no retrieval | none | 2 of 8 | 7 | 503,121 | 438 |
| (a) fixed retrieval (passages with each message) | --rag-inject always --disallowedTools Retrieve |
8 of 8 | 3 | 137,045 | 151 |
| (b) agentic Retrieve | (Retrieve offered) | 7 of 8 | 4 | 184,240 | 192 |
| (c) adaptive routing | --rag-inject route --disallowedTools Retrieve |
8 of 8 | 2 | 116,470 | 146 |
Without retrieval the agent guesses (a cache_size key, a
branch named release/quarry-1.4) or greps for words the
docs do not use; it cannot see .oklm/sessions at all (Glob
and Grep skip it), and one task looped for 32 turns. Fixed retrieval
puts the right section in front of the model with the request, so most
tasks end in two turns. The agentic tool passes as much on the
documentation tasks, for a turn more (the call; it called Retrieve in
six tasks, and found 19's command with Grep), and searches the memory
when the request says "in an earlier session" (22:
Retrieve {"query": "quarry 1.4 branch name", "scope": "memory"});
it missed 23, where it edited the changelog without searching ("Fix:"
for the convention's "Fixed:"). The routed mode matches the fixed one,
with fewer tokens in this run; since the router chooses one
for almost every request (Adaptive retrieval, measured),
the difference is the noise of single runs. Every request of these tasks
needs the docs, so fixed retrieval wins here; on requests that need no
documentation it would add passages for nothing, which is what the tool
avoids. The index of each task's 10 files was built in about a second on
the CPU, before the first search.
On a served model. The same tasks with the agent on
--url, Qwen3-8B Q4_K_M served by
oklm serve --backend opencl -c 32768, corrective retrieval
on (the default), its judge, router, rewriter and condenser the server's
(On a served model):
sh tools/eval_agent.sh MODEL --only 18-...,25-... --url URL --rag-project --rag-memory,
each run once, 2026-11-09, beside the in-process rows above:
| Mode | Model | Passed | Turns a task | Tokens in all | Seconds in all |
|---|---|---|---|---|---|
| (b) agentic Retrieve | in process | 7 of 8 | 4 | 184,240 | 192 |
| (b) agentic Retrieve | served (--url) |
7 of 8 | 3 | 150,567 | 277 |
| (c) adaptive routing | in process | 8 of 8 | 2 | 116,470 | 146 |
| (c) adaptive routing | served (--url) |
8 of 8 | 2 | 107,409 | 195 |
The pass rates are the in-process ones, and the task missed is the
same (23: the changelog edited without a search, "Fix:" for "Fixed:").
The tokens are the agent's turns only (the judge's and the router's
scoring requests run in the server, 86 /v1/decide requests
over the 16 runs, the probes included); the seconds are longer by the
HTTP round trips and by the scoring requests queuing behind the agent's
own in one server.
Graph retrieval
Some questions need two facts kept in two places: "in which town was the architect who designed the Varnel Bridge born?" is answered by the bridge's article (designed by Orsolya Kett) and the architect's (born in Lenmark), and the second never names the bridge, so neither the vectors nor the keywords of the question find it. Graph retrieval keeps the entities and relations each chunk states, extracted by a generator when the index is built, and walks them at query time from the question's entities to the chunks they lead to.
oklm rag index docs --out docs.gguf -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf \
--graph --gen-model models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --gen-backend opencl
oklm rag search docs.gguf "where was the designer of the Varnel Bridge born" --rag-graph on
oklm run -m MODEL --rag docs.gguf --rag-graph on -p "..."
oklm serve -m MODEL --rag docs.gguf # a request opts in with "rag": {"graph": true}
| Option | Default | |
|---|---|---|
--graph --gen-model MODEL (rag index) |
off | extract each chunk's entities and relations with this chat model and keep them in the index |
--gen-backend cpu|opencl|vulkan|auto
(rag index) |
cpu | where the generator (and the captioner: its language model and the blocks of its image encoder) runs |
--rag-graph on|off (rag search,
run, chat, serve,
agent) |
off | fuse the graph walk's list with the search; ignored, with a notice, for an index without a graph |
--graph-hops 1|2 (rag search) |
2 | how far the walk goes |
Extraction. Each chunk is put in a prompt ("Extract
the knowledge graph of the text below ... answer with the JSON object
only", reasoning off) and decoded greedily under the grammar of a JSON
schema (OKLM_RAG_GRAPH_SCHEMA: entities, a
list of names; relations, objects of subject,
relation, object), at most 512 tokens. The
answer is read leniently (oklm_rag_graph_parse(): a closed
reasoning block and a code fence skipped, a relation's ends added to the
entities, duplicates dropped, at most 32 of each); an answer cut by the
limit or holding no JSON gives an empty extraction, not an error.
Entities are linked by their normalized name
(lowercased, punctuation runs made one space, a leading "the", "a" or
"an" dropped): "The Varnel Bridge" and "Varnel bridge" are one node.
Linking by embedding similarity was not added: on the measured set the
names are written alike wherever they appear, and the hits' entities
(below) already link a question worded otherwise.
The walk. The seeds are the entities whose names
appear in the question (every run of up to eight of its words,
normalized, looked up) with weight 1, and those of the first stage's
three best hits with weight 0.5 divided by the hit's rank. A chunk that
mentions a seed is reached in one hop; a chunk that mentions an entity
related to a seed (either way) in two, at half the seed's weight. A
chunk's score sums, over the entities that reach it, the weight times
the entity's rarity, ln(1 + chunks / chunks mentioning it); entities
mentioned by more than a tenth of the chunks (at least 20) are not
walked through. The ranked list (from graph)
is fused with the dense and BM25 lists by the search's own fusion
(weighted fusion, the default: weight 0.5; reciprocal rank, the graph
list's terms weighted 1).
With corrective retrieval. The judge reads each passage alone against the question, so the second fact of a chain ("Orsolya Kett was born in Lenmark") is judged irrelevant to a question about a bridge. A passage the graph found is therefore kept, as ambiguous, when another passage of the same search is relevant; other irrelevant passages are dropped as before. Without that rule, corrective retrieval threw away what the graph had found: 6 of 30 right with the graph on, against 23 with the rule.
oklm rag info prints the graph
(162 entities, 267 mentions, 162 relations, extracted by Qwen3-8B-Q4_K_M.gguf);
the file format is ADR-0022's
minor 1. An update keeps a kept chunk's entities when the same
generator (by file name) built them, and extracts every chunk again with
another one.
In the library
(include/oklm/rag/graph.h): the extractor is a stage
(oklm_rag_extractor, set in
oklm_rag_index_params.graph), the generator's is
oklm_rag_generator_open() and
oklm_rag_generator_extractor();
oklm_rag_graph_search() is the walk,
oklm_rag_search_params.graph (with graph_hops,
graph_seeds, w_graph) turns it on in
oklm_rag_query() and so in every retriever of a store;
oklm_rag_graph_find(), oklm_rag_graph_entity()
and oklm_rag_graph_relation() read the graph.
Graph retrieval, measured
On 2026-11-09: the multi-hop set of
tests/rag/multihop_set/
(tools/rag_multihop_set.py, committed): 100 short articles
of an invented encyclopedia, five kinds of chains (a work and its
architect's birthplace, a company and its founder's university, an album
and its band's city, a ship and its shipyard's port, a novel and its
author's instrument), ten chains of each, six asked; 30 questions that
need both articles of a chain, and 10 single-hop questions (the first
link) as the control. Index: bge-small-en-v1.5 F32 with its query
instruction, hybrid search by reciprocal rank (the default then;
oklm rag search --fusion rrf); the graph extracted by
Qwen3-8B Q4_K_M on OpenCL: 100 chunks in 115 s (1.15 s a chunk), 162
entities, 162 relations. Retrieval by oklm rag search (an
article found when one of its chunks is in the first k); answers by
Qwen3-8B behind oklm serve --rag (5 passages, temperature
0, thinking off), right when they name the chain's last fact.
python3 tools/rag_set_eval.py --set multihop, one run.
| Multi-hop (30) | both articles in top 5 | recall@5 | both in top 10 | recall@10 | right, naive retrieval | right, corrective |
|---|---|---|---|---|---|---|
| graph off | 8 | 63.3 % | 16 | 76.7 % | 8 | 5 |
| graph on | 23 | 88.3 % | 30 | 100 % | 23 | 23 |
- The graph brings the second article into the top 5 for 23 of 30 questions (8 without) and into the top 10 for all 30; with it the answers follow the retrieval exactly: the 23 questions whose two articles were among the five passages were answered right, in both modes. Without it, corrective retrieval answered 5 (4 of its 8 found pairs: the judge drops the second fact).
- The 10 single-hop questions: their article first in every configuration (recall@5 100 %), 10 of 10 answered right with and without the graph; the graph's extra candidates did not push it out.
- The seven multi-hop misses with the graph have the second article at
ranks 6 to 10: the question's own words (
architect,born,town) rank other articles of the same kind above it in both lists, which the graph's single list does not outweigh in reciprocal rank fusion. - Cost: 0.5 s a question to answer either way; the walk itself is sub-millisecond on this index; the extraction is the cost of the build (1.15 s a chunk on this GPU with an 8 B model).
- The set is synthetic and small, written for this measure: it shows what the walk adds where the facts are split, not how often real documents split them (below, the documentation).
With weighted fusion (the default since; the same
index, oklm rag search without --fusion,
retrieval only, one run): the graph's list weighs more against the two
others than under reciprocal rank.
| Multi-hop (30), weighted fusion | both articles in top 5 | recall@5 | both in top 10 | recall@10 |
|---|---|---|---|---|
| graph off | 7 | 61.7 % | 17 | 78.3 % |
| graph on | 28 | 96.7 % | 30 | 100 % |
The single-hop controls stay at 10 of 10 in the top 5; five more pairs than under reciprocal rank (23) have both articles in the top 5.
On the documentation (2026-11-09). The docs set's
corpus is 1,007 sections, 2,461 chunks; Qwen3-8B extracts a docs chunk
in 4.3 s on this GPU (longer chunks, many more names than the synthetic
articles), about 3 hours for the whole set, so the graph was built over
a part of it: tools/rag_docs_set.py --subset 20, the
sections judged for the 20 two-section questions and for five
single-section lookups (the checks), and 20 other sections as
distractors: 82 sections, 362 chunks, extracted in 1,554 s, 3,123
entities, 4,920 mentions, 2,562 relations (1.2 MB of index). Scored by
oklm rag eval --split multihop|lookup with
--rag-graph off|on, hybrid search, one run:
| Docs part (82 sections) | fusion | queries | nDCG@10 | recall@5 | recall@10 |
|---|---|---|---|---|---|
| two-section, graph off | weighted | 20 | 88.82 | 89.6 % | 100 % |
| two-section, graph on | weighted | 20 | 88.49 | 93.6 % | 100 % |
| two-section, graph off | RRF | 20 | 86.38 | 89.4 % | 100 % |
| two-section, graph on | RRF | 20 | 86.40 | 89.8 % | 100 % |
| single-section checks, graph off | weighted | 5 | 89.18 | 95.0 % | 100 % |
| single-section checks, graph on | weighted | 5 | 88.77 | 85.0 % | 100 % |
- The documentation's two-section questions name both of their subjects ("train a LoRA adapter, and then serve several such adapters"), so vectors and BM25 already find both sections: recall@10 is 100 % with or without the graph, and the graph adds 4 points of recall@5 at an unchanged nDCG@10. The synthetic set gains because its second fact is never named in the question; real questions here rarely chain that way.
- On the single-section checks, the graph's extra candidates push a judged section out of the top 5 once (recall@5 95 % to 85 %).
- With 82 sections the search is easier than over all 1,007 (recall@10 100 % here against 86.7 % for the whole set), so these figures are not comparable with the retrieval table; the comparison is graph off against on over the same part. The 120 questions' nDCG@10 with the graph was not measured (the full extraction).
- Answer accuracy with the graph was not measured on the docs: only five of the 20 two-section questions have the facts of a right answer, and their sections are found without it.
So --rag-graph stays off: on these documents it is a few
points of recall@5 on two-part questions, a loss on single-section ones,
and hours of extraction; it pays where facts chain through names a
question does not give, as in the synthetic set.
Self-critique
After the answer, the generator checks it: each statement that cites a source is asked against every source it cites ("Does the passage support the statement?"), and the answer against the question ("Does the answer address the question?"), both yes / no decisions by scoring (decide.h, one forward pass each). When a cited statement is not supported (a probability of yes under 0.5, or a citation of a number the prompt did not give), the answer is generated once more with the critique fed back:
Your previous answer was checked against the sources. These statements are not supported by the source they cite:
- "The pool holds 65,536 tokens per block." [3]
Answer the question again. State only what the numbered sources say, cite after each statement a source that says it,
and leave out anything they do not state. If the sources do not answer the question, say so.
The new answer is checked the same way and the one with the larger
share of supported (statement, source) pairs is kept (the new one on a
tie); when it still has an unsupported statement, it is
flagged: in the record, and on stderr in
oklm run. Usefulness is scored and recorded, not acted
on.
oklm run -m MODEL --rag docs.gguf --rag-critique on -p "How do I keep the prefix cache across restarts?"
oklm serve -m MODEL --rag docs.gguf --rag-critique on # or per request: "rag": {"critique": true}
on or flag. A retry is
possible only while the answer has not been shown. on holds
the answer until it is checked: oklm run and
oklm chat generate it quietly, retry when a statement is
unsupported, and print the one kept. flag lets the answer
stream as usual; it is checked once complete and the verdict follows it,
with no retry (a streamed answer cannot be taken back, and a second
answer printed after it is a second answer, not a correction). Both
print a line on stderr after the answer:
(critique: 4 of 5 cited pairs supported, useful 0.98; answered again, the new answer kept),
or ...; FLAGGED: statements not supported by their sources.
The run's line in --rag-log gains a critique
member (retried, flagged, chosen,
the passes with their pairs, statement, cite,
p_supported, p_useful).
oklm chat -m MODEL --rag docs.gguf --rag-critique flag # replies stream; a verdict line after each retrieved turn
oklm run -m MODEL --rag docs.gguf --rag-critique flag -p "..."
In oklm chat every turn that retrieved is critiqued,
except a reply that is a tool call; the critic uses the chat's context,
so the next turn evaluates the conversation again instead of reusing the
cache.
In the server. The retry of a whole answer is the
request's prompt with the critique after its user message, decoded
greedily through the scheduler; answers that may be tool calls are not
critiqued. A request opts in with "rag": {"critique": true}
or {"critique": {"retry": true|false}};
--rag-critique on|flag makes it the default
(flag: no retry of whole answers either). The answer's
rag member carries the record as critique, a
top-level critique member the verdict on the answer
kept:
"critique": {"flagged": true, "retried": false, "chosen": 0, "supported": 0.75, "pairs": 4,
"unsupported": [{"statement": "The pool holds 65,536 tokens per block.", "cite": 3}], "useful": 0.97, "ms": 412.3}A stream is sent as it is generated; the critique
runs on its whole text, and the verdict rides on the last chunk of a
chat completion (the one with the finish_reason, beside
rag) or on the message_delta of an Anthropic
stream. The stream's answer is never retried unless the request asks
with "critique": {"retry": true}: then, when a statement is
unsupported and the second answer is kept as better, it comes as one
more event, a chat.completion.critique_retry object (its
choices[0]. message, before data: [DONE]) or a
critique_retry event (content blocks, before
message_stop), with the verdict on it. A client that does
not ask gets the flag only. /api/stats counts
critiqued, critique_retries and
critique_flagged, streams included.
Self-critique in the agent
oklm agent -m MODEL --rag-project --rag-critique on|flag
checks each final answer (a turn without tool calls) that cites
[n]: against the passages its message retrieved, every
Retrieve call's and those given with the message
(--rag-inject). Each retrieval numbers its sources from 1,
so a citation [n] is checked against source n
of each of them and counts as supported when one of them supports it
(the last eight retrievals of the message are kept). The critic is the
in-process model on the retrieval's own context (the agent's cache is
not touched).
With on, an unsupported statement sends the agent back
once: the critique (the unsupported statements, then "Answer the
question again. State only what the numbered sources say...") is added
as a reminder, as a blocking Stop hook's reason is, and the loop goes
on; the model may answer from what it has or call Retrieve again. The
second answer is checked too and flagged when still unsupported; it is
not sent back again. With flag, the answer is only flagged.
The loop was chosen over a quiet regeneration because the answer has
already streamed to the transcript, a reminder is how the loop already
sends the model back (Stop hooks), and the model keeps its tools for the
retry. Each verdict is a notice in the transcript
(critique: 1 of 3 cited pairs supported, useful 0.91; answering again with the critique),
a critique line of the debug log (--debug) and
an agent-critique line of --rag-log
(attempt, retry, question,
runs, critique). With --url the
critic is the served model, through /v1/decide on an OKLM
server (On a served model); with
--mock there is no model to judge with, and the agent
refuses --rag-critique.
Reflection tokens. Self-RAG has the model write
[Retrieve], [IsRel], [IsSup] and
[IsUse] tokens while it answers; that needs a model trained
to write them, and none of the models OKLM runs is. The same judgements
are asked after the answer instead, of any chat model.
In the library
(include/oklm/rag/critique.h):
oklm_rag_critique_run(params, critic, retrieval, question, answer, regenerate, user, &critique)
is the state machine; the critic is a stage
(oklm_rag_critic: support and
useful), the generator's on a context is
oklm_rag_model_critic(); the regeneration is the surface's
callback (it knows its prompt); oklm_rag_critique_json() is
the record.
Self-critique, measured
On 2026-11-09, as Retrieval in generation,
measured: Qwen3-8B Q4_K_M on OpenCL behind
oklm serve --rag, the documentation set's index
(bge-small-en-v1.5, hybrid), 5 passages, corrective retrieval, the same
38 questions (30 answerable, 8 not), temperature 0, thinking off;
python3 tools/rag_gen_eval.py --modes crag,critique, one
run. Faithfulness as there: each sentence citing [n] judged
against each source it cites by the served model ("Does the passage
support the statement?").
| Mode | Accuracy | Answerable | Unanswerable (abstained) | Faithfulness (cited pairs supported) | Invented citations | s / q |
|---|---|---|---|---|---|---|
| corrective | 92.1 % (35 / 38) | 27 / 30 | 8 / 8 | 87.5 % (98 / 112) | 0 | 2.7 |
| corrective, self-critique | 92.1 % (35 / 38) | 27 / 30 | 8 / 8 | 96.1 % (98 / 102) | 0 | 2.4 |
- 38 answers critiqued, 11 answered again (the new answer kept in 9), 4 flagged. The same 98 pairs are supported; the 10 pairs gone are unsupported statements the retry left out or rewrote: "The sources that support this are and." (twice), a command line the sources do not give, a paged block said to hold 65,536 tokens (the pool's size, rewritten "65,536 tokens in 4,096 blocks"), a claim about checked embedding models cited to the wrong source.
- Accuracy is unchanged: the same 35 answers state every fact, no right answer was lost by the retry.
- The measure and the critic ask the same question of the same
model (the evaluation's judge is the support question by
decide.c through
/v1/systemone): the critique removes what that judge rejects, so the figure overstates what a reader would see. Read by hand, the statements the retries removed in the examples above were indeed unsupported; on the other side, one flagged answer is flagged for "only Ogg Theora video is decoded natively", a paraphrase of its source that the judge rejects (the strictness on paraphrase noted there). Earlier hand checks put this judge within about ±15 points of a reader. - Cost: four to eight more scoring requests an answer, and a second generation for 11 of 38; the time a question was within the noise of one run (2.4 s against 2.7 s).
Self-critique on streams and in the agent, measured
On 2026-11-09, Qwen3-8B Q4_K_M on OpenCL, single runs, thinking off.
Streams. The 38 questions through
oklm serve --rag over the docs set's index, corrective
retrieval, 5 passages, temperature 0,
"critique": {"retry": false}: each asked whole, then
streamed (python3 tools/rag_stream_critique_eval.py).
| Questions | Same answer text | Same verdict, same answers | Pairs (same answers) | Unsupported / flagged (same answers) | Added after the last token: median, p90, max | Critique time, median |
|---|---|---|---|---|---|---|
| 38 | 19 | 19 of 19 | 45 | 4 / 3 | 280 ms, 562 ms, 891 ms | 261 ms |
- Where the streamed answer is the whole answer's text, its verdict is the same: the same pairs, the same unsupported statements and citations, the same flag. The stream's critique is the whole answer's code path on the same text.
- 19 of 38 answers differ between the two requests at temperature 0,
from the first diverging word on (one example: "On any address other
than a loopback address" against "On any other address (non-loopback)").
The second request finds the prompt in the prefix cache and evaluates
its last rows in another piece shape, which the OpenCL products round
otherwise; the critique is not involved (it runs after the last token).
With
oklm serve --deterministic38 of 38 answers and verdicts are the same, the critique's median 816 ms (server.md). Over all 38 the verdicts differ only where the answers do (24 identical verdicts; flagged 9 whole, 11 streamed). - The latency a stream gains is the time between its last content chunk and the chunk with the finish reason, which waits for the critique: 280 ms median (one to five support questions and the usefulness question, as scoring requests of the scheduler). The tokens before it are not delayed.
The agent. The 8 retrieval tasks
(tests/evals 18 to 25) with the Retrieve tool
(--rag-project --rag-memory), without and with
--rag-critique on; retries and flags counted from
--rag-log's agent-critique lines.
| Passed | Turns (sum) | Prompt tokens | Seconds | Answers critiqued | Retried | Flagged | |
|---|---|---|---|---|---|---|---|
| without | 6 / 8 | 28 | 155,087 | 164 | |||
--rag-critique on |
6 / 8 | 32 | 181,549 | 184 | 1 | 0 | 0 |
- One final answer of eight cited
[n](3 pairs, all supported, checked in 386 ms); the others act (edit, run, commit) and report without citing, so they are not critiqued. No answer was sent back or flagged: the differences between the rows (18 passed with the option and failed without, 22 the reverse, four more turns, 17 % more prompt tokens) are single-run noise, not the critique, which added no turn here. - On these tasks the critique costs nearly nothing and changes nothing; it acts on answers that cite, which in the agent are the question-answering ones.
Figures in the index
Charts and diagrams hold facts the text around them does not repeat.
With --caption-model, oklm rag index captions
the figures of the documents it indexes with a vision model and indexes
each caption as a text chunk of its document, pointing at its
figure:
oklm rag index docs report.pdf --out docs.gguf -m models/bge-small-en-v1.5/bge-small-en-v1.5-f32.gguf \
--caption-model models/smolvlm2-2.2b/SmolVLM2-2.2B-Instruct-Q8_0.gguf
oklm rag search docs.gguf "revenue of the North region" --json # a caption's hit carries "image"
- Which figures. An image file (
.png,.jpg,.gif,.bmp, ...; without a captioner it is skipped as before); the local images a Markdown file shows (an image link:![ALT]then its path in parentheses) and an HTML file shows (<img src alt>), resolved beside the document (remote anddata:ones skipped); the images a DOCX file's relationships name; the image XObjects of a PDF's pages (JPEG through the image decoder, 8-bit gray, RGB, CMYK and ICC-based samples after Flate and PNG predictors). At most 64 a document, smaller than 32 pixels skipped; an image that cannot be read or captioned is counted and skipped. - The caption. The vision model is asked "Describe
this figure in detail for a search index. Say what kind of figure it is
and read out its title and axis labels. Then go through every bar,
point, slice or box one by one, and for each give its label and the
number written on or next to it." and decodes greedily (at most 256
tokens). The figure and the question are joined as
oklm describejoins them (multimodal.md): since 2026-11-14 the question comes right after the figure for SmolVLM and Gemma 3, as their templates write it, where a new line stood between them. The figures of this page were measured with the new line and not again: a caption made since may differ by some words.--caption-prompt TEXT(or@FILE, the file's text) asks something else: the default suits SmolVLM2-2.2B and costs the small models answers (below), so with SmolVLM-256M or 500M the earlier prompt is the one to pass:--caption-prompt "Describe this figure for a search index. Say what kind of figure it is and what it shows, then read out its title, axis labels, legend and every label and number written on it.". Its projector is--caption-mmproj F, else the onemmproj*file beside the model. - Which model. Any vision model whose projector
OKLM's vision encoder implements (SmolVLM / Idefics3, Gemma 3, Qwen2-VL,
LLaVA: see multimodal.md); captions
were measured with the SmolVLM models only. SmolVLM2-2.2B-Instruct Q8_0
with its F16 projector (
models/smolvlm2-2.2b/, pinned intests/public_models_large.txt, fetched bysh tools/fetch_models.sh tests/public_models_large.txt) is the recommended one: 18 of 20 chart questions answered against 9 with SmolVLM-256M, for about 25 s a figure on the CPU against 4 (measured).--gen-backend opencl(orvulkan) runs its language model on the GPU, and the blocks of its image encoder with it (multimodal.md): the 9 figures of the set below in 10.1 s on OpenCL and 7.6 s on Vulkan, against 176 s on the CPU (measured). An encoder the device has no room for computes on the CPU, with a notice (the vision encoder runs on the CPU). - The chunk. Text
Figure: NAME (REF, page N)then the caption, NAME the alternative text or title the document gives the figure (else its file name), headingFigure: NAME, line the line where the document shows it, page its PDF page. The index records the figure (chunk.image:FILE,FILE#page=N&image=NAME,FILE#MEMBER);oklm rag search --jsonprints it asimage,oklm rag infocounts the figures and names the captioner. The caption is embedded and in BM25 like any chunk, so it is found by meaning or by its words, and the generator reads it as a source. - An update keeps a document's captions with its other chunks (by the
document's hash); another captioner, another
--caption-prompt(the index records it asrag.caption.prompt, the default as nothing), or no captioner, chunks every source again. An image a Markdown page shows is captioned again only when the page changes.
In the library
(include/oklm/rag/caption.h): the captioner is a stage
(oklm_rag_captioner,
oklm_rag_index_params.caption; its prompt is
handed to each caption, OKLM_RAG_CAPTION_PROMPT when NULL),
the vision model's is oklm_rag_vlm_open() and
oklm_rag_vlm_captioner()
(oklm_rag_vlm_encoder_device() puts its image encoder on
the model's device or back on the CPU,
oklm_rag_vlm_encoder_status() says where it is);
oklm_rag_figures_find() lists a document's figures, and
oklm_ingest_pdf_images() (ingest.h) a PDF's page
images.
Figures in the index, measured
On 2026-11-09: the figure set of tests/rag/figures_set/
(tools/rag_figures_set.py, committed): five documents whose
numbers are only in their figures, matplotlib charts with their values
printed on them: a Markdown report showing a bar chart and a line chart,
a pdflatex PDF with a pie chart, a pandoc DOCX with a horizontal bar
chart, an HTML page with a bar chart, and a standalone diagram (PNG); 20
questions, each answered by one figure (a value, the largest or
smallest, a step of the diagram). Index: bge-small-en-v1.5 F32, hybrid;
the captions by SmolVLM-256M-Instruct Q8_0 on the CPU: 6 figures in 25
s. Answers by Qwen3-8B behind oklm serve --rag (5
passages), right when they state the figure's value.
python3 tools/rag_set_eval.py --set figures, one run.
| Index | figure's caption in top 5 | right, naive retrieval | right, corrective |
|---|---|---|---|
| text only | - (no figure indexed) | 0 / 20 | 2 / 20 |
| captions indexed | 20 / 20 | 9 / 20 | 6 / 20 |
- Retrieval is not what misses: every question found its figure's caption among the five passages. What misses is the caption: the 256 M model reads some charts fully (the battery chart: all four capacities, so 4 of 4 right; the diagram's first step) and others not at all (the pie chart: "The energy mix of Velmor is 38%."; the temperatures: "Average yearly temperature by city is 19.7"; the revenue bars: only the axis range). The two right answers of the text-only index are the model's own guesses ("Embed" after "Chunk", the pipeline's next step in general), not read from anything.
- Corrective retrieval answers fewer than naive retrieval here: the judge calls a vague caption irrelevant ("the graph shows quarterly revenue by region", nothing about the North's figure), and the answer becomes "no reliable source was found".
- No joint text-image encoder. The design kept SigLIP's text tower for when captions are not found; here they are always found (20 of 20 in the top 5). A joint encoder retrieves an image by its look, which does not read the numbers a question asks for; what would raise the 9 of 20 is a better reader of charts (a larger vision model at index time, or the image given to a vision-language generator at answer time), not another retriever. So it was not built.
Figures in the index, on a device
With --gen-backend opencl|vulkan the captioner's
language model was on the device and its image encoder on the CPU, which
was then most of a caption's time. Since 2026-11-14 the encoder's blocks
follow the model (oklm_vision_use_backend(), as
oklm describe and oklm serve do). The whole
oklm rag index of the figure set with
SmolVLM2-2.2B-Instruct Q8_0 and the default prompt (9 figures,
bge-small-en-v1.5 F32; RX 7900 XT, another job on the CPU, one run each;
the captioning, then the command):
--gen-backend |
encoder on the CPU (before) | encoder on the device | measured again on 2026-11-15 |
|---|---|---|---|
cpu |
176.2 s, 177.1 s | not run | |
opencl |
98.3 s, 99.7 s | 10.1 s, 11.8 s | 9.04 to 9.07 s, 10.7 s |
vulkan |
95.4 s, 96.8 s | 7.6 s, 8.7 s | 7.33 to 7.39 s, 8.44 to 8.55 s |
The last column is two indexes built from nothing on each backend (load average 1.5 to 4.4), with the whole encoder on the device (its patch embedding and projector too), Vulkan's products at F32's precision and the question written right after the image, as SmolVLM's template has it; 9 figures captioned each time, none skipped. The same command run again onto the index it had just written captioned no figure (0.9 to 1.4 s for the command).
On each backend the index is the one built with the encoder on the CPU: the five hits of the set's 20 questions are the same chunks in the same order, and the captions those hits show (6 of the 9) are the same text. Between backends the captions differ, as they did before (the language model on a device against the CPU: 4 of those 6 captions are the CPU's on either device, and 12 (Vulkan) or 9 (OpenCL) of the 20 lists of hits); the answers through the generator were not measured again, nor the captions' texts and the lists of hits with the day's last builds.
Larger caption models, and the prompt. The same set,
index and answering model (Qwen3-8B Q4_K_M on OpenCL behind
oklm serve --rag), with SmolVLM-500M-Instruct and
SmolVLM2-2.2B-Instruct (Q8_0, F16 projectors, from ggml-org's
repositories, pinned in tests/public_models_large.txt) and
two caption prompts: the earlier one ("... then read out its title, axis
labels, legend and every label and number written on it") and the
current default, which asks for the title and axes, then each bar,
point, slice or box with its label and number. Index time is the whole
oklm rag index on the CPU (9 figures: the Markdown and HTML
pages' images are captioned once as the page's figures and once as image
files), with other jobs sharing the machine. "Declined" counts
corrective answers that begin "No reliable source". One run each.
| Captioner | Prompt | Index time | Caption in top 5 | Right, naive | Right, corrective | Declined (corrective) |
|---|---|---|---|---|---|---|
| SmolVLM-256M | earlier | 25 s (6 figures) | 20 / 20 | 9 / 20 | 6 / 20 | - |
| SmolVLM-256M | current | 39 s | 20 / 20 | 7 / 20 | 6 / 20 | 6 |
| SmolVLM-500M | earlier | 49 s | 20 / 20 | 9 / 20 | 10 / 20 | 9 |
| SmolVLM-500M | current | 47 s | 20 / 20 | 4 / 20 | 6 / 20 | 11 |
| SmolVLM2-2.2B | earlier | 189 s | 20 / 20 | 11 / 20 | 10 / 20 | 6 |
| SmolVLM2-2.2B | current | 251 s | 20 / 20 | 18 / 20 | 17 / 20 | 2 |
- SmolVLM2-2.2B with the current prompt reads every value of the bar, battery, temperature and pie charts and the four steps of the diagram; what it still misses is the line chart's months between January and June ("starts at 120,000 visitors in January and increases steadily to 300,000 in June": March guessed) and the pie's order (all four shares right, but "the smallest segment ... hydro energy at 20%", so the smallest source is answered as hydro instead of gas). With the earlier prompt the same model often gave the title and axes without the values ("line graph" for the line chart, no capacity for the batteries, only the axis range for the temperatures): the prompt is worth 7 answers to it, the model 9 over the 256 M one.
- The two small models do worse with the current prompt (256M 7
against 9, 500M 4 against 9): asked to go mark by mark, they invent
labels or drop the numbers (500M on the pie: four slices named, no
percentage). The default was chosen for the recommended captioner; with
a small one,
--caption-promptgives it the earlier prompt back. - The corrective judge's rejection of vague captions mostly goes away: with complete captions, 2 answers are declined instead of 6 to 11, and corrective retrieval lands one answer under naive retrieval instead of three. The two still declined are the line chart's March (the caption gives no March value) and the pie's smallest share (the caption contradicts its own percentages).
- Retrieval never missed (20 of 20 in the top 5 for every captioner), so a joint text-image encoder would not have changed these answers.
Tool selection by retrieval
An agent with a hundred tools pays for them in its prompt, or hides
them behind ToolSearch, which a small model rarely calls.
oklm agent --tool-rag K (or the
toolSearch.topK setting; off by default) picks them by
retrieval instead.
What is indexed. Every tool the agent may be offered
(built-in, the program's, every MCP server's), its model-invocable
skills and plugin commands, and its sub-agent types; each entry is one
text (the short name, where it comes from, the description cut at 1500
bytes, the parameter names), grouped Type (built-in, MCP, skill, plugin,
sub-agent) to Service (the MCP server, the plugin) to Tool, and stored
as a source of a RAG index (above) recorded as
TYPE/SERVICE/NAME with the hash of its text. The index file
is ~/.local/state/oklm/rag/tools-HASH.gguf (HASH of the
project's path): a description that did not change keeps its vector
across runs, and before each retrieval the agent stages its current set
and commits it, which costs a hash when nothing changed and embeds only
the changed entries when a server, a skill or a plugin came or went.
The ranking. Hybrid, as oklm rag search
does it: the query embedded by toolSearch.model (a path, a
file name looked for as the agent looks for models, or
none; bge-small-en-v1.5-q8_0.gguf by default,
on the CPU beside the generator, 512 tokens) and compared with every
entry, BM25 over the same texts, the two lists fused by reciprocal rank
(k = 60). toolSearch.mode dense ranks by the
cosine alone, bm25 by BM25 alone (the cosine still
computed, for the floor below); hybrid is the default.
Without the model (none, or not found) the ranking is
BM25's alone, whatever the mode. A grouped ranking (services ordered by
their best entry, a service's entries together) exists in the library
(oklm_tool_index_params.grouped) and is off: it measured
below the flat one.
What the model gets. With --tool-rag K
every MCP tool is deferred, and the system prompt counts each server's
tools (MCP server github: 8 tools deferred ...) instead of
naming them. Before each user message's first turn, the message and the
previous one (2000 bytes at most) are the query; of the K best entries,
the deferred tools whose cosine with the query reaches
toolSearch.minScore (0.62) are given to the model: their
definitions in a reminder after the message for the in-process model
(its prompt, and so its KV cache, unchanged), offered in the tools array
for a server (those of the previous message no longer). A request the
core tools serve adds nothing. ToolSearch stays, and its word queries
are ranked by the same index over the same candidates as before
(deferred tools and skills); select:NAME is unchanged.
Sub-agents do not retrieve.
Inspecting it. With --debug, each
decision is a line of the debug log:
tool index: 52 entries, 52 embedded, 0 kept, 0 dropped (hybrid, 0.25 s)
tool retrieval: query "Use the add tool of the fake MCP server to add 123 and 456, ..." (hybrid); ranked: mcp__fake__add 0.0328 (dense 0.784 #1, bm25 14.34 #1), mcp__fake__big 0.0320 (dense 0.709 #2, bm25 9.12 #3), ...; chosen: mcp__fake__add, mcp__fake__big, mcp__fake__echo, mcp__fake__change, mcp__fake__slow
tool retrieval: query "Rename the function compute_total to sum_prices ..." (hybrid); ranked: ...; chosen: none (below the floor 0.62: EnterWorktree, LSP, ScheduleWakeup, TodoWrite, Grep)
In the library: include/oklm/harness/tool_index.h
(oklm_tool_index_stage(),
oklm_tool_index_stage_json(),
oklm_tool_index_commit(),
oklm_tool_index_search()), and
oklm_agent_params.tool_index / tool_rag_k; the
decisions arrive as OKLM_AGENT_EV_LOG events.
Tool selection, measured
make tool-eval
(tests/rag/bench_tool_select.c;
TOOL_EVAL_EMBED, TOOL_EVAL_TOKENIZER,
TOOL_EVAL_K) builds the index the agent would: its 27
built-in tools, its 12 model-invocable built-in skills, its 4 sub-agent
types and the 62 MCP tools of tests/rag/tool_zoo.json (the
13 tools of tools/mcp_zoo.py and nine servers of common
kinds: GitHub, calendar, mail, drive, SQL, chat, maps, finance, tasks,
translation), 105 entries; then ranks them for the 152 requests of
tests/rag/tool_queries.jsonl, each labelled with the tool
or tools it needs. bge-small-en-v1.5 Q8_0 on the CPU, tokens counted
with Qwen3-8B's tokenizer, 2026-11-09:
| Ranking | top-1 | recall@1 | recall@3 | recall@5 | recall@10 | MRR |
|---|---|---|---|---|---|---|
| keyword, the request as query (ToolSearch today) | 27.0 % | 25.7 % | 37.8 % | 46.6 % | 58.4 % | 0.376 |
| keyword, stop words dropped | 44.7 % | 41.8 % | 55.4 % | 60.3 % | 69.2 % | 0.538 |
BM25 (toolSearch.model none) |
53.3 % | 49.9 % | 62.9 % | 70.8 % | 77.1 % | 0.620 |
| dense | 57.9 % | 55.2 % | 74.8 % | 81.0 % | 85.9 % | 0.692 |
| hybrid (RRF), the default | 61.8 % | 58.1 % | 69.8 % | 76.4 % | 81.7 % | 0.692 |
| hybrid, grouped by service | 61.8 % | 58.1 % | 67.2 % | 72.7 % | 75.3 % | 0.676 |
The keyword rows score every entry as ToolSearch scores the deferred ones (3 a word in the name, 1 in the text): with the request's own words, articles and verbs match everywhere. Hybrid ranks the right tool first 2.3 times as often; dense alone keeps more of a request's tools in the first 3 to 10 (several MCP tools of one request often share no word with it, which pulls BM25's half of the fusion down); grouping only lowers recall, so it stays off.
What an in-process model is given (tokens; the skills' list, the same in every row, left out):
| Prompt | system prompt | tool definitions | per request | total | request's tools defined without ToolSearch |
|---|---|---|---|---|---|
| every tool offered | 830 | 7376 | 0 | 8206 | 100.0 % |
| deferred, named (ToolSearch today) | 1683 | 2295 | 0 (+318 a ToolSearch round) | 3978 (+318) | 26.6 % |
deferred, --tool-rag 5 |
1471 | 2295 | 136 | 3902 | 68.5 % |
deferred, --tool-rag 5,
toolSearch.mode dense |
1470 | 2295 | 177 | 3942 | 72.5 % |
deferred, --tool-rag 5,
toolSearch.mode bm25 |
1470 | 2295 | 105 | 3870 | 67.2 % |
deferred, --tool-rag 8 |
1471 | 2295 | 170 | 3936 | 69.8 % |
The last column counts a request's tools that are always there (the
core) or given with it: today a deferred tool needs a ToolSearch call
the model must think of (a round of 318 tokens when it does), with
retrieval 68.5 % of the tools are defined from the first turn, for fewer
tokens than naming every deferred tool. The cosine floor
(toolSearch.minScore) trades that share against the
definitions given for nothing (top 5):
| Floor | given a request | of which not needed | request's tools defined |
|---|---|---|---|
| none | 3.19 | 2.62 | 78.1 % |
| 0.60 | 2.04 | 1.53 | 73.5 % |
| 0.62 | 1.66 | 1.20 | 68.5 % |
| 0.66 | 0.86 | 0.47 | 61.0 % |
| 0.70 | 0.36 | 0.12 | 48.4 % |
The ranking (toolSearch.mode,
2026-11-09, make tool-eval: the benchmark runs the index's
own search once a mode). The accuracy rows are those of the first table
(BM25, dense, hybrid); what each gives at top 5:
| Mode | floor | given a request | of which not needed | request's tools defined |
|---|---|---|---|---|
| hybrid | 0.62 | 1.66 | 1.20 | 68.5 % |
| hybrid | 0.60 | 2.04 | 1.53 | 73.5 % |
| dense | 0.62 | 2.28 | 1.78 | 72.5 % |
| dense | 0.64 | 1.67 | 1.22 | 67.9 % |
| bm25 | 0.62 | 1.24 | 0.80 | 67.2 % |
Dense keeps more of a request's tools in its top 5 (81.0 % against
76.4 %), but the floor then lets more definitions through, most of them
not needed: at the same number given a request (dense at 0.64, hybrid at
0.62) it defines no more of the request's tools, and hybrid with a floor
of 0.60 beats dense at 0.62 on both counts. Hybrid also ranks the right
tool first more often (61.8 % against 57.9 %), which ToolSearch's word
queries use. So hybrid stays the default and the agent evals were not
run again; dense and bm25 are there for a tool
set where that balance differs.
The agent evals (make eval, the 17
tasks, Qwen3-8B Q4_K_M, in-process; the tasks have no MCP server but
03-mcp-tool's fake one), each run once, 2026-11-09:
| Run | passed | turns a task | tokens in all |
|---|---|---|---|
| off | 13 of 17 | 5 | 526,676 |
| off, again | 12 of 17 | 4 | 425,197 |
--tool-rag 5, final (floor 0.62, tools only) |
12 of 17 | 6 | 729,730 |
--tool-rag 5, skills suggested, floor 0.62 |
12 of 17 | 6 | 670,529 |
--tool-rag 5, no floor, skills suggested |
11 of 17 | 7 | 777,367 |
Which tasks fail changes from run to run in both modes (sampling):
the two runs without retrieval fail 4 and 5 tasks, only two of them in
common. Without a floor every message got two or three deferred tools it
did not need (EnterWorktree for a rename, database for a C
test) and one run looped for 32 turns; with the floor 11 of the 17
messages get nothing and see the very prompt of a run without retrieval,
03-mcp-tool gets the fake server's tools (passed in every
run), and the git tasks get EnterWorktree or LSP. The final
configuration passes as many tasks as the second run without it, one
fewer than the first: unchanged within the noise of single runs, not
shown better on tasks that hardly use deferred tools. Building the index
for an agent's 43 to 52 entries takes 0.22 to 0.25 s on the CPU, each
message's query a few milliseconds.