RAG Spec: Retrieval-Augmented Generation for OKLM and its Agent
Status: done (2026-11-08 to 2026-11-09, roadmap M17). K0: the audit
in gap-analysis.md, the design in rag-design.md, the index format in adr/0022-rag-index-format.md;
K1 to K8 implemented, each with its proof in rag.md (what was built differently is in
rag-design.md's "As built" and "Implemented" sections); K9: the
evaluation summary, the reproduction commands and the residual risks in
rag.md, make rag-eval
reproducing the retrieval table to the printed digits.
Competitive retrieval-augmented generation (RAG) for OKLM:
embeddings, a local index, hybrid search, reranking, retrieval wired
into oklm run, oklm chat,
oklm serve and oklm agent, then the agentic
and agent-centric variants (tool selection by retrieval, adaptive and
corrective retrieval, conversational memory), and last graph,
self-critique and multimodal retrieval. This page is OKLM's plan for it:
the steps, what exists to build on, the decisions, and each step's
proof.
Before starting (OKLM)
Steps. The request's workflow (audit, reuse, gap, implement, verify) and its priorities (P0, P1, P2) are numbered K0 to K9 here, as the other specifications number theirs (S, P, G, C, R, T): K0 is the audit and the design, K1 to K6 are P0 (the baseline), K7 is P1, K8 is P2, and K9 closes with the evaluation and the docs. Each step leaves something that works, is tested and is measured.
Constraints that come from OKLM, not from the request.
- C17 and nothing outside the project for anything that touches a
model or its data (architecture.md): the
request's dependency inventory (LangChain, LlamaIndex, Haystack, FAISS,
Qdrant, pgvector, Elastic, Neo4j, Milvus, Chroma) has no place here. The
encoder, the index, the keyword search, the fusion, the reranker and the
graph are OKLM's own code. Python is allowed only as a test reference
(
build/venv, adr/0015-test-references.md), with llama.cpp'sllama-embeddingas the other oracle. - Linux and Windows (the MinGW build); every feature behind a flag or a setting, off by default, no change to what an existing command does without it.
- Every step updates its docs in the same commit: the reference page, the README quick start for a new command or flag, and how-it-works.md.
- Measured before it is kept: each step adds its metric to the RAG evaluation (K1 creates it) and reports it before and after.
What OKLM already has. The seed of K0's audit, checked against the code on 2026-11-08; K0 verified it with file and line evidence (gap-analysis.md) and corrected the rows marked ⚠️:
| Capability | OKLM today | Where | Verdict |
|---|---|---|---|
| Embedding model forward pass | no text encoder architecture (BERT, nomic-bert, XLM-R), no pooling, no embedding output. ⚠️ The encoder ops exist (LayerNorm with bias, exact GELU, rope, SwiGLU, bidirectional attention), driven by the SigLIP and Whisper encoders; on OpenCL bidirectional attention runs on the CPU | src/model/model.c, src/core/ops.c, src/mm/vision.c | NEW (K1) on REUSED ops |
| Embedding tokenizer | WordPiece exact for BERT / BGE vocabularies, Unigram for XLM-R
(bge-m3); BPE and SentencePiece for decoder-based embedding models. ⚠️
From a GGUF bert vocabulary, [SEP] is not
added (llama.cpp always adds it); models/bge-small-en holds
the v1.5 tokenizer and a v1 config.json, no weights |
src/tok/, tokenizer.md | REUSE, EXTEND (K1: SEP) |
| Decoder-based embeddings (Qwen3-Embedding) | the forward pass exists (Qwen3). ⚠️ The final normalized hidden
state of every token is kept on request
(oklm_context_params.hidden,
oklm_get_hidden()); taking the last row, normalisation and
skipping the output head do not exist |
src/model/arch_llama.c, model.c | EXTEND (K1) |
/v1/embeddings, /v1/rerank |
none; oklm serve speaks OpenAI and Anthropic chat and
completions |
src/server/ | EXTEND (K1, K4) |
| Document loading | oklm ingest / --doc: text, JSON, HTML,
DOCX, PDF (pages) to text, the whole document put in the prompt |
src/mm/ingest.c, multimodal.md | EXTEND (K2: chunks) |
| Chunker | none | — | NEW (K2) |
| Vector store | none | — | NEW (K2) |
| Keyword search | the agent's Grep (regular expressions, unranked) | src/harness/tools.c | NEW (K3: BM25) |
| Reranker | none; scoring by the model exists: System One decisions (questions
answered by levels from one forward pass,
/v1/systemone) |
src/model/decide.c, server.md | NEW cross-encoder (K4); REUSE decide.c as the fallback and as K5's relevance judge |
| Tool selection | deferred tools and ToolSearch (keyword match over tool names and
descriptions); skills with a prompt budget; MCP servers; plugins. ⚠️
ToolSearch is in agent.c (run_tool_search), its prompt
section in prompt.c, not in tools.c |
src/harness/agent.c, prompt.c, skills.c, mcp.c, plugins.c | EXTEND (K6) |
| Agent loop, context injection | the tool loop, reminders, prompt templates, compaction | src/harness/agent.c, prompt.c, compact.c | REUSE |
| Memory | memory files (scopes, imports), session transcripts | src/harness/memory.c, session.c | EXTEND (K7) |
| Web fallback | WebSearch through SearXNG, WebFetch | src/harness/tools.c | REUSE (K5) |
| Multimodal | SigLIP / idefics3 image embeddings fed to the language model (not a joint text-image space); Whisper; video frames | src/mm/vision.c, audio.c | PARTIAL (K8) |
| Graph | none | — | NEW (K8) |
| Caching | prefix cache (also on disk, --prefix-dir), KV files,
the media embedding cache |
src/server/, src/model/kv_file.c | REUSE |
| Observability | debug log, OpenTelemetry export, /api/stats, the
dashboard |
src/harness/telemetry.c, src/server/dashboard.c | EXTEND (retrieval decisions logged) |
| Evaluation | agent evals (make eval, 17 tasks in tests/evals, pass
rate, turns, tokens, time); oklm ppl. ⚠️
sentence-transformers needs PyTorch, which adr/0015-test-references.md
rules out: the second embedding reference is a numpy float64
encoder |
tests/evals, agent-loop.md | EXTEND (K1: the RAG evaluation) |
Decisions to take in K0 (an ADR each where marked),
taken in rag-design.md:
bge-small-en-v1.5 by default, the MiniLM cross-encoder as the default
reranker (the bge rerankers checked), one GGUF file per index,
.oklm/rag/ for the agent (never indexed without asking),
the defaults below, exact search up to a million chunks:
- The default embedding model, and the ones checked: proposed
bge-small-en-v1.5 (English, 33 M, the tokenizer already in
models/) as the default and the test model, bge-m3 or multilingual-e5 for other languages, Qwen3-Embedding-0.6B as the decoder-based one; the reranker bge-reranker-base or bge-reranker-v2-m3. - The index file format (ADR): one file per index, a header with the embedding model's identity (refused on mismatch), chunks with their source, offsets and hash, vectors (F32, F16 or Q8_0), the BM25 inverted lists, later the graph; written to a temporary file and renamed (the platform layer), updated incrementally by file hash.
- Where indexes live: per project (
.oklm/rag/) for the agent, any path foroklm rag; what the agent indexes on its own and what it asks. - Defaults: chunk size and overlap per chunker, top-k, RRF's constant (60), the relevance thresholds of K5.
- Search: exact (brute force, SIMD, threads) first; an approximate index (HNSW or IVF) only if K2's measure shows exact search too slow at the sizes targeted (proposed: exact up to a million chunks).
Steps
K0. Audit and design
- The "RAG" section of gap-analysis.md: every capability of the request's taxonomy (below) with ✅ / 🟡 / ❌ and file:line evidence, from the table above.
- The capability matrix: DONE / PARTIAL / MISSING with the change needed.
docs/rag-design.md: each step mapped to the modules it extends or the new files it adds, the stage contracts (retriever, reranker, evaluator, generator as pluggable components with one C interface each), the API (include/oklm/rag/), the decisions above taken.- Proof: the three documents reviewed; nothing is coded before.
K1. Embeddings (P0, the foundation)
- Encoder architectures in GGUF and from Hugging Face checkpoints:
bert(BGE, e5, MiniLM),nomic-bert,xlm-roberta(bge-m3); pooling (CLS, mean, last token), normalisation; decoder-based embedding models (Qwen3-Embedding) through the existing forward pass with last-token pooling. CPU and OpenCL. oklm_embed()in the library,oklm embed -m MODEL TEXT|--file F,/v1/embeddingsinoklm serve(OpenAI's request and response, batches).- The RAG evaluation starts here (
make rag-eval,build/rag-eval/): a fixed question set over OKLM's own documentation (a corpus that exists, answers checkable), and a small public retrieval set (a BEIR subset such as SciFact or NFCorpus), with recall@k, MRR, nDCG@10, latency and tokens. - Proof: embeddings within 1e-4 cosine of llama.cpp's
llama-embeddingand 1e-5 of a numpy float64 encoder (tools/embed_ref.py, in place of sentence-transformers, which needs PyTorch) on 1000 strings per model; the public set's nDCG@10 equal to the published figure for the model within noise.
K2. Chunking and the index (P0)
- Chunkers: fixed (tokens with overlap), recursive (headings,
paragraphs, sentences), semantic (split where consecutive embeddings
part), code-aware (functions and classes: by braces and indentation, or
the language server's symbols when one runs,
src/harness/lsp.c), on top ofingest. - The index (decision 2):
oklm rag index PATH... --model EMB --out INDEX(incremental: unchanged files kept by hash),oklm rag search INDEX QUERY [-k N],oklm rag info INDEX; exact cosine top-k on the CPU's SIMD kernels and threads. - Proof: search equal to a brute-force reference (the same top-k, scores within 1e-6); an incremental update equal to a rebuild; index and search times measured at 10 k, 100 k and 1 M chunks.
K3. Hybrid retrieval (P0)
- BM25 over the same chunks (inverted lists in the index, the
tokenizer reused, stemming optional), fusion by reciprocal rank (RRF) or
weights,
--mode dense|sparse|hybrid. - Proof: on the evaluation, hybrid recall@10 at least that of dense and of sparse alone, on both sets.
K4. Reranking (P0)
- A cross-encoder reranker (the K1 encoder with its classification
head),
oklm rag search --rerank MODEL,/v1/rerank(Cohere / Jina shaped); the fallback without a reranker model: the generator scores relevance throughdecide.c. - Proof: nDCG@10 after reranking above the hybrid's, and the latency added per query.
K5. Retrieval in generation, and corrective RAG (P0)
oklm run/oklm chat --rag INDEXandoklm serve --rag INDEX(per request opt-in): the top chunks put into the prompt with their sources, the answer citing them.- Corrective RAG: each retrieved chunk judged relevant / ambiguous / irrelevant (decide.c, one forward pass); on a poor retrieval the query rewritten and retried, then WebSearch when allowed, else the answer says that no reliable source was found (graceful degradation, never an invented citation).
- Every retrieval decision logged: the query (and its rewrites), the
chunks with their scores, the verdicts, the fallback taken
(
--rag-log, the agent's debug log,/api/stats). - Proof: answer accuracy and faithfulness (cited chunks support the answer, judged by decide.c and checked by hand on a sample) against no retrieval and against naive retrieval, on the documentation set.
K6. Tool selection by retrieval (P0)
- Hierarchical tool retrieval: the descriptions of built-in tools, MCP tools, skills, sub-agent types and plugin commands embedded once (and again when they change), grouped Type (built-in, MCP, skill, plugin) to Service (an MCP server, a plugin) to Tool; ToolSearch extended to semantic search, and the top-k tools put in the prompt in place of all deferred tools.
- Proof: tool-selection accuracy and prompt tokens on an evaluation with 100+ tools (the MCP tool zoo of mcp-training-design.md plus the built-ins and skills), against the current ToolSearch; the agent evals' pass rate unchanged or better.
K7. Agentic, adaptive and conversational retrieval (P1)
- Agentic: a
Retrievetool over the project's index (built and kept up to date by the agent per decision 3); the model decides when and what to retrieve, and iterates. - Adaptive: a router (decide.c) sending a request to no retrieval, one retrieval or several (multi-hop) by its complexity.
- Conversational: the query rewritten from the conversation before retrieval; session memory retrieval over transcripts and memory files.
- Proof: new agent eval tasks that need the project's documentation or an earlier session; pass rate, turns and tokens against K5's fixed retrieval.
K8. Graph, self-critique and multimodal retrieval (P2)
- Graph RAG: entities and relations extracted by the generator into the index, multi-hop traversal at query time, combined with the chunks.
- Self-RAG: the generator's self-critique of its evidence and answer (support and usefulness scored through decide.c; reflection tokens only with a model trained for them), a retry when unsupported.
- Multimodal: images and charts in documents captioned by the vision model (SmolVLM) and indexed as text first; a joint text-image encoder (the SigLIP text tower) after, if the evaluation shows the captions miss.
- Proof: a multi-hop question set (graph), faithfulness with and without self-critique, and a set of questions answered from figures (multimodal).
K9. Evaluation report and documentation
- The RAG evaluation in full: retrieval precision / recall@k, MRR,
nDCG@10, faithfulness, latency, token cost and tool-selection accuracy,
every capability before and after, in
docs/rag.md's measured section. docs/rag.md(the user guide:oklm embed,oklm rag,--rag, the server endpoints, the agent's tools and settings), the how-it-works.md section, the README quick start, and the residual risks (cost, latency, index drift when files change).- Proof:
make rag-evalreproduces the published table.
Implementation rules
From the request, as they apply to OKLM:
- One step at a time, in order; re-check the audit at the start of each.
- Reuse the existing abstractions; an interface only where a stage needs one. Retriever, reranker, evaluator and generator stay composable, each behind one C interface.
- Every capability ships with its flag or setting (off by default, or
a safe default), its tests in
make test, its entry in the RAG evaluation, and its docs. - Retrieval decisions are logged and visible in the agent's terminal.
- Backward compatible; anything replaced is deprecated with a notice, never silently.
- Verification per step: the full test suite (pass / fail delta), the evaluation before and after, a short "what changed and why", and the residual risk.
Deliverables
| Request deliverable | Where in OKLM |
|---|---|
| 1. Audit report | K0: gap-analysis.md "RAG" |
| 2. Capability matrix | K0: gap-analysis.md "RAG", rag-design.md |
| 3. Gap plan (P0 / P1 / P2) | this page's steps; roadmap M17 |
| 4. Implementation | K1 to K8, one commit series each |
| 5. Verification report | each step's proof; K9's table in rag.md |
| 6. Next-steps backlog | the roadmap's "Next" |
The capability taxonomy
The competitive RAG taxonomy, with the step that covers each:
| Group | Capability | Step |
|---|---|---|
| Foundational | Vector RAG (dense retrieval, top-k, cosine) | K1, K2 |
| Hybrid RAG (dense + BM25, RRF or weighted fusion) | K3 | |
| Graph RAG (entities, relations, multi-hop) | K8 | |
| Agentic / adaptive | Agentic RAG (the model decides when and what to retrieve) | K7 |
| Adaptive RAG (routing by query complexity) | K7 | |
| Self-RAG (self-critique of evidence and answer) | K8 | |
| Corrective RAG (relevance evaluator, fallback) | K5 | |
| Agent-centric | Hierarchical tool RAG (MCP tools and skills, Type to Service to Tool) | K6 |
| Multimodal RAG | K8 | |
| Conversational / memory RAG | K7 | |
| Cross-cutting | Chunking (fixed, semantic, recursive, code-aware) | K2 |
| Reranking | K4 | |
| Query rewriting and expansion | K5, K7 | |
| Evaluation harness | K1, every step, K9 | |
| Caching, observability, cost and latency budgets | K5, K9 | |
| Graceful degradation | K5 |