OKLMdocumentation v1.0.3
On this page

RAG Spec: Retrieval-Augmented Generation for OKLM and its Agent

Status: done (2026-11-08 to 2026-11-09, roadmap M17). K0: the audit in gap-analysis.md, the design in rag-design.md, the index format in adr/0022-rag-index-format.md; K1 to K8 implemented, each with its proof in rag.md (what was built differently is in rag-design.md's "As built" and "Implemented" sections); K9: the evaluation summary, the reproduction commands and the residual risks in rag.md, make rag-eval reproducing the retrieval table to the printed digits.

Competitive retrieval-augmented generation (RAG) for OKLM: embeddings, a local index, hybrid search, reranking, retrieval wired into oklm run, oklm chat, oklm serve and oklm agent, then the agentic and agent-centric variants (tool selection by retrieval, adaptive and corrective retrieval, conversational memory), and last graph, self-critique and multimodal retrieval. This page is OKLM's plan for it: the steps, what exists to build on, the decisions, and each step's proof.


Before starting (OKLM)

Steps. The request's workflow (audit, reuse, gap, implement, verify) and its priorities (P0, P1, P2) are numbered K0 to K9 here, as the other specifications number theirs (S, P, G, C, R, T): K0 is the audit and the design, K1 to K6 are P0 (the baseline), K7 is P1, K8 is P2, and K9 closes with the evaluation and the docs. Each step leaves something that works, is tested and is measured.

Constraints that come from OKLM, not from the request.

  • C17 and nothing outside the project for anything that touches a model or its data (architecture.md): the request's dependency inventory (LangChain, LlamaIndex, Haystack, FAISS, Qdrant, pgvector, Elastic, Neo4j, Milvus, Chroma) has no place here. The encoder, the index, the keyword search, the fusion, the reranker and the graph are OKLM's own code. Python is allowed only as a test reference (build/venv, adr/0015-test-references.md), with llama.cpp's llama-embedding as the other oracle.
  • Linux and Windows (the MinGW build); every feature behind a flag or a setting, off by default, no change to what an existing command does without it.
  • Every step updates its docs in the same commit: the reference page, the README quick start for a new command or flag, and how-it-works.md.
  • Measured before it is kept: each step adds its metric to the RAG evaluation (K1 creates it) and reports it before and after.

What OKLM already has. The seed of K0's audit, checked against the code on 2026-11-08; K0 verified it with file and line evidence (gap-analysis.md) and corrected the rows marked ⚠️:

Capability OKLM today Where Verdict
Embedding model forward pass no text encoder architecture (BERT, nomic-bert, XLM-R), no pooling, no embedding output. ⚠️ The encoder ops exist (LayerNorm with bias, exact GELU, rope, SwiGLU, bidirectional attention), driven by the SigLIP and Whisper encoders; on OpenCL bidirectional attention runs on the CPU src/model/model.c, src/core/ops.c, src/mm/vision.c NEW (K1) on REUSED ops
Embedding tokenizer WordPiece exact for BERT / BGE vocabularies, Unigram for XLM-R (bge-m3); BPE and SentencePiece for decoder-based embedding models. ⚠️ From a GGUF bert vocabulary, [SEP] is not added (llama.cpp always adds it); models/bge-small-en holds the v1.5 tokenizer and a v1 config.json, no weights src/tok/, tokenizer.md REUSE, EXTEND (K1: SEP)
Decoder-based embeddings (Qwen3-Embedding) the forward pass exists (Qwen3). ⚠️ The final normalized hidden state of every token is kept on request (oklm_context_params.hidden, oklm_get_hidden()); taking the last row, normalisation and skipping the output head do not exist src/model/arch_llama.c, model.c EXTEND (K1)
/v1/embeddings, /v1/rerank none; oklm serve speaks OpenAI and Anthropic chat and completions src/server/ EXTEND (K1, K4)
Document loading oklm ingest / --doc: text, JSON, HTML, DOCX, PDF (pages) to text, the whole document put in the prompt src/mm/ingest.c, multimodal.md EXTEND (K2: chunks)
Chunker none — NEW (K2)
Vector store none — NEW (K2)
Keyword search the agent's Grep (regular expressions, unranked) src/harness/tools.c NEW (K3: BM25)
Reranker none; scoring by the model exists: System One decisions (questions answered by levels from one forward pass, /v1/systemone) src/model/decide.c, server.md NEW cross-encoder (K4); REUSE decide.c as the fallback and as K5's relevance judge
Tool selection deferred tools and ToolSearch (keyword match over tool names and descriptions); skills with a prompt budget; MCP servers; plugins. ⚠️ ToolSearch is in agent.c (run_tool_search), its prompt section in prompt.c, not in tools.c src/harness/agent.c, prompt.c, skills.c, mcp.c, plugins.c EXTEND (K6)
Agent loop, context injection the tool loop, reminders, prompt templates, compaction src/harness/agent.c, prompt.c, compact.c REUSE
Memory memory files (scopes, imports), session transcripts src/harness/memory.c, session.c EXTEND (K7)
Web fallback WebSearch through SearXNG, WebFetch src/harness/tools.c REUSE (K5)
Multimodal SigLIP / idefics3 image embeddings fed to the language model (not a joint text-image space); Whisper; video frames src/mm/vision.c, audio.c PARTIAL (K8)
Graph none — NEW (K8)
Caching prefix cache (also on disk, --prefix-dir), KV files, the media embedding cache src/server/, src/model/kv_file.c REUSE
Observability debug log, OpenTelemetry export, /api/stats, the dashboard src/harness/telemetry.c, src/server/dashboard.c EXTEND (retrieval decisions logged)
Evaluation agent evals (make eval, 17 tasks in tests/evals, pass rate, turns, tokens, time); oklm ppl. ⚠️ sentence-transformers needs PyTorch, which adr/0015-test-references.md rules out: the second embedding reference is a numpy float64 encoder tests/evals, agent-loop.md EXTEND (K1: the RAG evaluation)

Decisions to take in K0 (an ADR each where marked), taken in rag-design.md: bge-small-en-v1.5 by default, the MiniLM cross-encoder as the default reranker (the bge rerankers checked), one GGUF file per index, .oklm/rag/ for the agent (never indexed without asking), the defaults below, exact search up to a million chunks:

  1. The default embedding model, and the ones checked: proposed bge-small-en-v1.5 (English, 33 M, the tokenizer already in models/) as the default and the test model, bge-m3 or multilingual-e5 for other languages, Qwen3-Embedding-0.6B as the decoder-based one; the reranker bge-reranker-base or bge-reranker-v2-m3.
  2. The index file format (ADR): one file per index, a header with the embedding model's identity (refused on mismatch), chunks with their source, offsets and hash, vectors (F32, F16 or Q8_0), the BM25 inverted lists, later the graph; written to a temporary file and renamed (the platform layer), updated incrementally by file hash.
  3. Where indexes live: per project (.oklm/rag/) for the agent, any path for oklm rag; what the agent indexes on its own and what it asks.
  4. Defaults: chunk size and overlap per chunker, top-k, RRF's constant (60), the relevance thresholds of K5.
  5. Search: exact (brute force, SIMD, threads) first; an approximate index (HNSW or IVF) only if K2's measure shows exact search too slow at the sizes targeted (proposed: exact up to a million chunks).

Steps

K0. Audit and design

  • The "RAG" section of gap-analysis.md: every capability of the request's taxonomy (below) with ✅ / 🟡 / ❌ and file:line evidence, from the table above.
  • The capability matrix: DONE / PARTIAL / MISSING with the change needed.
  • docs/rag-design.md: each step mapped to the modules it extends or the new files it adds, the stage contracts (retriever, reranker, evaluator, generator as pluggable components with one C interface each), the API (include/oklm/rag/), the decisions above taken.
  • Proof: the three documents reviewed; nothing is coded before.

K1. Embeddings (P0, the foundation)

  • Encoder architectures in GGUF and from Hugging Face checkpoints: bert (BGE, e5, MiniLM), nomic-bert, xlm-roberta (bge-m3); pooling (CLS, mean, last token), normalisation; decoder-based embedding models (Qwen3-Embedding) through the existing forward pass with last-token pooling. CPU and OpenCL.
  • oklm_embed() in the library, oklm embed -m MODEL TEXT|--file F, /v1/embeddings in oklm serve (OpenAI's request and response, batches).
  • The RAG evaluation starts here (make rag-eval, build/rag-eval/): a fixed question set over OKLM's own documentation (a corpus that exists, answers checkable), and a small public retrieval set (a BEIR subset such as SciFact or NFCorpus), with recall@k, MRR, nDCG@10, latency and tokens.
  • Proof: embeddings within 1e-4 cosine of llama.cpp's llama-embedding and 1e-5 of a numpy float64 encoder (tools/embed_ref.py, in place of sentence-transformers, which needs PyTorch) on 1000 strings per model; the public set's nDCG@10 equal to the published figure for the model within noise.

K2. Chunking and the index (P0)

  • Chunkers: fixed (tokens with overlap), recursive (headings, paragraphs, sentences), semantic (split where consecutive embeddings part), code-aware (functions and classes: by braces and indentation, or the language server's symbols when one runs, src/harness/lsp.c), on top of ingest.
  • The index (decision 2): oklm rag index PATH... --model EMB --out INDEX (incremental: unchanged files kept by hash), oklm rag search INDEX QUERY [-k N], oklm rag info INDEX; exact cosine top-k on the CPU's SIMD kernels and threads.
  • Proof: search equal to a brute-force reference (the same top-k, scores within 1e-6); an incremental update equal to a rebuild; index and search times measured at 10 k, 100 k and 1 M chunks.

K3. Hybrid retrieval (P0)

  • BM25 over the same chunks (inverted lists in the index, the tokenizer reused, stemming optional), fusion by reciprocal rank (RRF) or weights, --mode dense|sparse|hybrid.
  • Proof: on the evaluation, hybrid recall@10 at least that of dense and of sparse alone, on both sets.

K4. Reranking (P0)

  • A cross-encoder reranker (the K1 encoder with its classification head), oklm rag search --rerank MODEL, /v1/rerank (Cohere / Jina shaped); the fallback without a reranker model: the generator scores relevance through decide.c.
  • Proof: nDCG@10 after reranking above the hybrid's, and the latency added per query.

K5. Retrieval in generation, and corrective RAG (P0)

  • oklm run / oklm chat --rag INDEX and oklm serve --rag INDEX (per request opt-in): the top chunks put into the prompt with their sources, the answer citing them.
  • Corrective RAG: each retrieved chunk judged relevant / ambiguous / irrelevant (decide.c, one forward pass); on a poor retrieval the query rewritten and retried, then WebSearch when allowed, else the answer says that no reliable source was found (graceful degradation, never an invented citation).
  • Every retrieval decision logged: the query (and its rewrites), the chunks with their scores, the verdicts, the fallback taken (--rag-log, the agent's debug log, /api/stats).
  • Proof: answer accuracy and faithfulness (cited chunks support the answer, judged by decide.c and checked by hand on a sample) against no retrieval and against naive retrieval, on the documentation set.

K6. Tool selection by retrieval (P0)

  • Hierarchical tool retrieval: the descriptions of built-in tools, MCP tools, skills, sub-agent types and plugin commands embedded once (and again when they change), grouped Type (built-in, MCP, skill, plugin) to Service (an MCP server, a plugin) to Tool; ToolSearch extended to semantic search, and the top-k tools put in the prompt in place of all deferred tools.
  • Proof: tool-selection accuracy and prompt tokens on an evaluation with 100+ tools (the MCP tool zoo of mcp-training-design.md plus the built-ins and skills), against the current ToolSearch; the agent evals' pass rate unchanged or better.

K7. Agentic, adaptive and conversational retrieval (P1)

  • Agentic: a Retrieve tool over the project's index (built and kept up to date by the agent per decision 3); the model decides when and what to retrieve, and iterates.
  • Adaptive: a router (decide.c) sending a request to no retrieval, one retrieval or several (multi-hop) by its complexity.
  • Conversational: the query rewritten from the conversation before retrieval; session memory retrieval over transcripts and memory files.
  • Proof: new agent eval tasks that need the project's documentation or an earlier session; pass rate, turns and tokens against K5's fixed retrieval.

K8. Graph, self-critique and multimodal retrieval (P2)

  • Graph RAG: entities and relations extracted by the generator into the index, multi-hop traversal at query time, combined with the chunks.
  • Self-RAG: the generator's self-critique of its evidence and answer (support and usefulness scored through decide.c; reflection tokens only with a model trained for them), a retry when unsupported.
  • Multimodal: images and charts in documents captioned by the vision model (SmolVLM) and indexed as text first; a joint text-image encoder (the SigLIP text tower) after, if the evaluation shows the captions miss.
  • Proof: a multi-hop question set (graph), faithfulness with and without self-critique, and a set of questions answered from figures (multimodal).

K9. Evaluation report and documentation

  • The RAG evaluation in full: retrieval precision / recall@k, MRR, nDCG@10, faithfulness, latency, token cost and tool-selection accuracy, every capability before and after, in docs/rag.md's measured section.
  • docs/rag.md (the user guide: oklm embed, oklm rag, --rag, the server endpoints, the agent's tools and settings), the how-it-works.md section, the README quick start, and the residual risks (cost, latency, index drift when files change).
  • Proof: make rag-eval reproduces the published table.

Implementation rules

From the request, as they apply to OKLM:

  • One step at a time, in order; re-check the audit at the start of each.
  • Reuse the existing abstractions; an interface only where a stage needs one. Retriever, reranker, evaluator and generator stay composable, each behind one C interface.
  • Every capability ships with its flag or setting (off by default, or a safe default), its tests in make test, its entry in the RAG evaluation, and its docs.
  • Retrieval decisions are logged and visible in the agent's terminal.
  • Backward compatible; anything replaced is deprecated with a notice, never silently.
  • Verification per step: the full test suite (pass / fail delta), the evaluation before and after, a short "what changed and why", and the residual risk.

Deliverables

Request deliverable Where in OKLM
1. Audit report K0: gap-analysis.md "RAG"
2. Capability matrix K0: gap-analysis.md "RAG", rag-design.md
3. Gap plan (P0 / P1 / P2) this page's steps; roadmap M17
4. Implementation K1 to K8, one commit series each
5. Verification report each step's proof; K9's table in rag.md
6. Next-steps backlog the roadmap's "Next"

The capability taxonomy

The competitive RAG taxonomy, with the step that covers each:

Group Capability Step
Foundational Vector RAG (dense retrieval, top-k, cosine) K1, K2
Hybrid RAG (dense + BM25, RRF or weighted fusion) K3
Graph RAG (entities, relations, multi-hop) K8
Agentic / adaptive Agentic RAG (the model decides when and what to retrieve) K7
Adaptive RAG (routing by query complexity) K7
Self-RAG (self-critique of evidence and answer) K8
Corrective RAG (relevance evaluator, fallback) K5
Agent-centric Hierarchical tool RAG (MCP tools and skills, Type to Service to Tool) K6
Multimodal RAG K8
Conversational / memory RAG K7
Cross-cutting Chunking (fixed, semantic, recursive, code-aware) K2
Reranking K4
Query rewriting and expansion K5, K7
Evaluation harness K1, every step, K9
Caching, observability, cost and latency budgets K5, K9
Graceful degradation K5