OKLMdocumentation v1.0.3
On this page

RAG design

How rag_spec.md maps onto OKLM: the modules each step extends, the files it adds, the stage contracts, the public API, the command line and server surface, the decisions the spec leaves to its K0, and how each step is tested and measured. What exists, with evidence, is the "RAG" section of gap-analysis.md; the index file is adr/0022-rag-index-format.md.

Status: implemented (2026-11-08 to 2026-11-09): K0 to K8 built and K9's evaluation published (rag.md); "As built in K1" and the "Implemented" sections say what differs from the plan below.

Starting point

  • What exists. The ops an encoder needs (LayerNorm with bias, exact GELU, rope, SwiGLU, bidirectional attention), already driven by the SigLIP and Whisper encoders in src/mm/; the final hidden state of a decoder kept on request (oklm_context_params.hidden); WordPiece and Unigram tokenizers checked against llama.cpp; oklm_matmul() of F32 rows against F16 or Q8_0 matrices and oklm_topk(), on the SIMD kernels and the worker team; the GGUF writer (temporary file, then replace) and memory-mapped reads; oklm ingest; System One scoring (decide.c); ToolSearch over deferred tools and skills; memory files, transcripts, the debug log, the OTLP export and /api/stats.
  • What is missing. The text encoder, pooling, chunkers, the index, BM25 and fusion, the cross-encoder, the pipeline that puts it all into run, chat, serve and agent, and a way to measure retrieval.
  • What bounds it. C17, nothing outside the project on the model or data path; Linux and the MinGW build; every feature off by default.

Layout

include/oklm/rag/
  embed.h       oklm_embedder, oklm_embed(), oklm_rerank(), pooling
  chunk.h       chunkers
  index.h       the index (ADR-0022), exact and BM25 search
  retrieve.h    stage contracts (retriever, reranker, evaluator,
                generator) and the pipeline
  eval.h        retrieval metrics
src/model/arch_bert.c        bert / nomic-bert forward pass (XLM-R is bert)
src/io/names_bert.c          Hugging Face names to GGUF names
src/rag/embed.c              batching, pooling, normalization, pairs
src/rag/chunk.c              fixed, recursive, semantic
src/rag/chunk_code.c         code-aware (braces, indentation, LSP symbols)
src/rag/index.c              build, update, open, exact search
src/rag/bm25.c               analyzer, postings, BM25
src/rag/fuse.c               RRF, weighted
src/rag/pipeline.c           stages composed, corrective loop, trace
src/rag/graph.c              K8: entities, relations, traversal
src/rag/eval.c               recall@k, MRR, nDCG, faithfulness counts
src/server/embed.c           /v1/embeddings, /v1/rerank
src/harness/tool_index.c     K6: embedded tool descriptions
tests/rag/                   test_embed.c, test_chunk.c, test_index.c,
                             test_bm25.c, test_pipeline.c, test_eval.c,
                             docs_questions.jsonl, tool_queries.jsonl
tools/embed_ref.py           numpy float64 encoder (test reference)
tools/rag_eval.sh            make rag-eval

Architectures stay in src/model/ as a family of their own: model.c picks the family by the GGUF general.architecture (model.c:1166-1182), bind_* binds the tensors and model_forward_* runs a range of layers on the context's backends (model.c:2528). An encoder is a third family, ARCH_BERT, so it gets the device copies, the CPU and OpenCL backends, quantized weights and the memory planner for free; src/rag/ only drives it. A oklm_context for an encoder is created with a new embeddings flag: no KV cache (n_ctx is the longest input, keys and values of the current batch only), bidirectional attention per sequence, no output head.

Step by step

Step Existing (reused) Extended New Blast radius
K1 Embeddings ops.h encoder ops; hidden and oklm_get_hidden(); WordPiece and Unigram; oklm_model_id(); the server's router and queue model.c (family ARCH_BERT, embeddings context flag), model_internal.h (token_types, tok_norm, cls, cls_out; per layer attn_out_norm, out_norm), hf_config.c (BertModel, XLMRobertaModel, NomicBertModel rows; positions sliced by pad + 1 for XLM-R), convert.c, tok_gguf.c (SEP for WordPiece), server.c (route), apps/oklm.c (embed), Makefile, backend_cl.c (bidirectional attention, or the CPU fallback kept) arch_bert.c, names_bert.c, embed.c, eval.c, server/embed.c, include/oklm/rag/embed.h and eval.h, tools/embed_ref.py, tools/rag_eval.sh, tests/rag/test_embed.c, test_eval.c, docs_questions.jsonl; a tiny random BERT fixture from tools/gen_model_fixtures.py New public headers only; oklm_context_params gains embeddings (0 keeps today's behavior); a GGUF WordPiece vocabulary now ends with SEP (llama.cpp's behavior; no generative model uses WordPiece). README: oklm embed.
K2 Chunking, index ingest (all kinds, PDF pages); GGUF writer and reader; oklm_matmul, oklm_topk; lsp.c symbols ingest.h (nothing: offsets are computed on the extracted text) chunk.c, chunk_code.c, index.c, include/oklm/rag/chunk.h and index.h, cmd_rag (index, search, info, eval), tests/rag/test_chunk.c, test_index.c, tests/rag/bench_index.c New command and files; nothing existing changes. README: oklm rag.
K3 Hybrid the index; unicode.c word boundaries index.c (postings section), oklm rag search --mode bm25.c, fuse.c, tests/rag/test_bm25.c Index version 1 already reserves the BM25 tensors (ADR-0022): no format change.
K4 Reranking arch_bert.c; decide.c arch_bert.c (cls, cls.output head, RANK pooling), embed.c (pairs), server/embed.c (/v1/rerank), oklm rag search --rerank the decide.c fallback reranker in pipeline.c; tests in test_embed.c, test_pipeline.c Additive.
K5 Generation, corrective attach_doc (gen_opts); agent_push_meta; decide.c; WebSearch; telemetry apps/oklm.c (--rag, --rag-k, --rag-tokens, --rag-log on run, chat, serve), server.c (rag field per request), sched.c nothing, telemetry.c (retrieval events), server stats (rag_* counters), dashboard pipeline.c, include/oklm/rag/retrieve.h, tests/rag/test_pipeline.c Off unless --rag; a server started without --rag refuses the field (400).
K6 Tool selection agent.c run_tool_search, is_deferred, put_deferred; skills, MCP, plugins, agent definitions agent.c (semantic rank, top-k loaded), prompt.c (the deferred section names groups only when toolSearch.mode is semantic), settings.c (toolSearch.*) src/harness/tool_index.c, tests/harness/test_tool_index.c, tests/rag/tool_queries.jsonl Agent prompt bytes unchanged in the default keyword mode.
K7 Agentic, adaptive, conversational tools.c table; session.c, memory.c; compact.c's summary call tools.c (Retrieve), agent.c (rewrite before retrieval), settings.c (rag.*), apps/oklm.c (/rag) router and rewrite in pipeline.c; .oklm/rag/ management; new eval tasks under tests/evals/ A new deferred tool, off unless rag.enabled.
K8 Graph, self-critique, multimodal decide.c; grammars (oklm_json_schema_to_grammar); vision.c and SmolVLM index.c (graph tensors, a minor version), pipeline.c graph.c; captioning in the index builder Additive; index minor version 2.
K9 Report, docs everything above README, how-it-works.md docs/rag.md Docs.

The encoder

OKLM reads the GGUF files llama.cpp writes, so the same file runs in both and llama.cpp is the oracle. What llama.cpp does (pinned tree in build/llama.cpp; conversion/bert.py, src/models/bert.cpp, src/llama-graph.cpp, src/llama-vocab.cpp):

  • Architectures. BertModel (also BertForSequenceClassification), RobertaModel and XLMRobertaModel (bge-m3, bge-reranker-v2-m3) are all written as bert; NomicBertModel as nomic-bert; JinaBertModel as jina-bert-v2 (ALiBi: not in K1). XLM-R's position table is sliced by pad_token_id + 1 at conversion, so positions start at 0 at run time; its tokenizer is t5 (Unigram) with <s> … </s>.
  • Tensors. token_embd, token_types, position_embd (bert only), token_embd_norm (weight and bias); per block attn_qkv or attn_q / attn_k / attn_v (with biases), attn_output, attn_output_norm, ffn_up, ffn_gate (nomic-bert), ffn_down, layer_output_norm; heads cls, cls.output (with biases).
  • Graph. Post-norm: x = LN(tok + type[0] + pos), then per block h = LN(x + attn(x)), x = LN(h + ffn(h)); bert's FFN is down(gelu(up h)), nomic-bert's down(silu(gate h) * up h) without biases, with NEOX rope (base 10000) on Q and K. Attention scale 1/sqrt(head_dim), every key visible within the sequence.
  • Metadata. <arch>.pooling_type (NONE 0, MEAN 1, CLS 2, LAST 3, RANK 4; absent means NONE), <arch>.attention.causal (written false), <arch>.attention.layer_norm_epsilon, tokenizer.ggml.token_type_count, tokenizer.ggml.seperator_token_id (spelled so), <arch>.classifier.output_labels. The converter takes the pooling from sentence-transformers' modules.json and 1_Pooling/config.json, drops the HF pooler.dense, and never writes RANK for a reranker (llama.cpp is told --pooling rank).
  • Differences OKLM keeps on purpose. llama.cpp's bert uses the tanh GELU, Hugging Face the exact one: OKLM follows the checkpoint (hidden_act: gelu is exact) and the tolerance against llama.cpp absorbs it (a test switch to tanh proves the rest is identical). llama.cpp gives every token type 0; OKLM gives the passage type 1 in a pair when the model has two types (the MiniLM cross-encoder was trained so), with the same switch for the oracle comparison.

Decoder embeddings (Qwen3-Embedding): plain qwen3 with pooling LAST; the tokenizer appends <|endoftext|> when its add_eos says so; queries carry the model's instruction prefix. OKLM runs it through ARCH_LLAMA with hidden on and the output head skipped.

Public API

/* include/oklm/rag/embed.h */
typedef enum oklm_pooling {
    OKLM_POOLING_DEFAULT = -1, /* the model's (GGUF key, sentence-transformers config, family default) */
    OKLM_POOLING_NONE = 0,     /* one vector per token */
    OKLM_POOLING_MEAN = 1,
    OKLM_POOLING_CLS = 2,      /* the first token */
    OKLM_POOLING_LAST = 3,     /* the last token (decoder models) */
    OKLM_POOLING_RANK = 4      /* the classification head: one score per pair */
} oklm_pooling;

typedef struct oklm_embed_params {
    oklm_pooling pooling;       /* OKLM_POOLING_DEFAULT */
    int normalize;              /* 1 (default): L2; 0: raw */
    int32_t n_batch;            /* tokens per forward pass (default 2048) */
    int32_t n_ctx;              /* longest input; 0: the model's training context */
    const char* query_prefix;   /* NULL: the model's (e.g. bge's instruction) */
    const char* passage_prefix; /* NULL: the model's (e.g. e5's "passage: ") */
    const oklm_backend* backend;/* NULL: the CPU */
} oklm_embed_params;

oklm_embed_params oklm_embed_default_params(void);
int oklm_embedder_new(const oklm_model* m, const oklm_embed_params* p, oklm_embedder** out);
void oklm_embedder_free(oklm_embedder** e);
int32_t oklm_embedder_dim(const oklm_embedder* e);
oklm_pooling oklm_embedder_pooling(const oklm_embedder* e);
/* n texts to n rows of dim floats; is_query selects the prefix */
int oklm_embed(oklm_embedder* e, const char* const* texts, int32_t n, int is_query, float* out);
int oklm_embed_tokens(oklm_embedder* e, const int32_t* const* tokens, const int32_t* n_tokens, int32_t n, float* out);
/* a cross-encoder (RANK): one raw score per passage */
int oklm_rerank(oklm_embedder* e, const char* query, const char* const* passages, int32_t n, float* scores);
/* include/oklm/rag/chunk.h */
typedef enum oklm_chunker { OKLM_CHUNK_FIXED, OKLM_CHUNK_RECURSIVE, OKLM_CHUNK_SEMANTIC, OKLM_CHUNK_CODE } oklm_chunker;
typedef struct oklm_chunk_params {
    oklm_chunker kind;      /* RECURSIVE for prose, CODE for source files (by extension) */
    int32_t size;           /* target tokens (256) */
    int32_t overlap;        /* tokens shared with the previous chunk (FIXED: 32) */
    int32_t max;            /* hard cap (the encoder's context, 512 for bge) */
    int32_t min;            /* merged with a neighbor below (32) */
    float breakpoint;       /* SEMANTIC: the percentile of neighbor distances that splits (95) */
} oklm_chunk_params;
typedef struct oklm_chunk_span {
    int64_t start, end;     /* bytes in the extracted text */
    int32_t line0, line1;   /* lines, 1-based */
    int32_t page;           /* PDF page, or 0 */
    int32_t n_tokens;
    char heading[256];      /* the Markdown heading path or the symbol name */
} oklm_chunk_span;
int oklm_chunk_text(const oklm_chunk_params* p, const oklm_tokenizer* tok, oklm_embedder* sem,
                    const char* path, const char* text, size_t len, oklm_chunk_span** out, int32_t* n);
/* include/oklm/rag/index.h */
typedef struct oklm_rag_index_params {
    oklm_dtype vec_type;     /* OKLM_DTYPE_F16 (default), OKLM_DTYPE_F32, OKLM_DTYPE_Q8_0 */
    oklm_chunk_params chunk; /* recorded in the file */
    int bm25;                /* 1 (default from K3): keep the postings */
} oklm_rag_index_params;
typedef struct oklm_rag_hit { int64_t chunk; float score; int32_t rank; uint32_t from; /* retriever bit */ } oklm_rag_hit;
typedef struct oklm_rag_chunk {
    const char* source;      /* path as given, relative to the index's root */
    oklm_chunk_span span;
    const char* text;        /* the chunk's text (mapped, not owned) */
    uint64_t hash;
} oklm_rag_chunk;

int oklm_rag_index_open(const char* path, oklm_rag_index** out);            /* mapped, read-only */
int oklm_rag_index_create(const char* path, const oklm_embedder* e, const oklm_rag_index_params* p, oklm_rag_index** out);
int oklm_rag_index_update(oklm_rag_index* x, oklm_embedder* e, const char* const* paths, int32_t n, oklm_rag_update_stats* st);
int oklm_rag_index_commit(oklm_rag_index* x);                               /* temporary file, then replace */
int oklm_rag_index_check_model(const oklm_rag_index* x, const oklm_embedder* e); /* OKLM_ERR_STATE on another model */
int oklm_rag_index_search(const oklm_rag_index* x, const float* query, int32_t k, oklm_rag_hit* out, int32_t* n);
int oklm_rag_index_bm25(const oklm_rag_index* x, const char* query, int32_t k, oklm_rag_hit* out, int32_t* n);
int oklm_rag_index_chunk(const oklm_rag_index* x, int64_t id, oklm_rag_chunk* out);
int oklm_rag_index_info(const oklm_rag_index* x, oklm_rag_index_info* out);
void oklm_rag_index_free(oklm_rag_index** x);

Stage contracts

Four stages, each one C interface (a table of functions and its state), composed by the pipeline; any of them can be replaced by the program (a custom retriever over another store, a remote reranker) the way the agent takes custom tools.

/* include/oklm/rag/retrieve.h */
typedef struct oklm_rag_query {
    const char* text;        /* the query (after rewriting) */
    const float* vec;        /* its embedding, or NULL: the retriever computes it */
    int32_t k;               /* hits wanted */
    const char* filter;      /* glob over sources, or NULL */
} oklm_rag_query;

typedef struct oklm_rag_retriever {
    const char* name;        /* "dense", "bm25", "hybrid", "graph" */
    void* self;
    int (*search)(void* self, const oklm_rag_query* q, oklm_rag_hit* out, int32_t* n);
    void (*free)(void* self);
} oklm_rag_retriever;

typedef struct oklm_rag_reranker {
    const char* name;        /* "cross-encoder", "decide" */
    void* self;
    /* scores[i] for passages[i]; higher is better, comparable within one call only */
    int (*score)(void* self, const char* query, const char* const* passages, int32_t n, float* scores);
    void (*free)(void* self);
} oklm_rag_reranker;

typedef enum oklm_rag_verdict { OKLM_RAG_RELEVANT = 0, OKLM_RAG_AMBIGUOUS = 1, OKLM_RAG_IRRELEVANT = 2 } oklm_rag_verdict;

typedef struct oklm_rag_evaluator {
    const char* name;        /* "decide" */
    void* self;
    /* one verdict and its probability of relevance per passage */
    int (*judge)(void* self, const char* query, const char* const* passages, int32_t n, oklm_rag_verdict* v, float* p_relevant);
    void (*free)(void* self);
} oklm_rag_evaluator;

typedef struct oklm_rag_generator {
    const char* name;        /* "context" (in process), "http" */
    void* self;
    /* a better query from the query and the conversation (NULL: none) */
    int (*rewrite)(void* self, const char* query, const char* history, char** out);
    /* the answer from the query and the numbered passages, streamed */
    int (*answer)(void* self, const char* query, const oklm_rag_chunk* chunks, int32_t n, oklm_rag_text_fn on_text, void* user);
    void (*free)(void* self);
} oklm_rag_generator;

typedef struct oklm_rag_pipeline_params {
    int32_t k_retrieve;      /* candidates per retriever (50) */
    int32_t k_rerank;        /* candidates reranked (50) */
    int32_t k_context;       /* chunks given to the generator (5) */
    int32_t max_tokens;      /* their token budget (2048) */
    int32_t rrf_k;           /* 60 */
    int corrective;          /* evaluate, rewrite, fall back */
    int web;                 /* WebSearch allowed as the last fallback */
    oklm_rag_trace_fn on_trace; void* user; /* every decision, for logs and events */
} oklm_rag_pipeline_params;

int oklm_rag_pipeline_new(const oklm_rag_pipeline_params* p, oklm_rag_retriever* const* r, int32_t n_r,
                          oklm_rag_reranker* rr, oklm_rag_evaluator* ev, oklm_rag_generator* gen, oklm_rag_pipeline** out);
/* retrieval only (no generator call): the chunks a prompt would get, with their verdicts */
int oklm_rag_pipeline_retrieve(oklm_rag_pipeline* pl, const char* query, const char* history, oklm_rag_result* out);
int oklm_rag_pipeline_answer(oklm_rag_pipeline* pl, const char* query, const char* history, oklm_rag_text_fn on_text, void* user, oklm_rag_result* out);
void oklm_rag_pipeline_free(oklm_rag_pipeline** pl);

The trace (oklm_rag_trace: the stage, the query and its rewrites, each hit with its scores per retriever and after fusion and reranking, each verdict and probability, the fallback taken, times per stage) is the one record behind --rag-log, the agent's debug log and events, the OTLP spans and the server's counters.

Command line and server

  • oklm embed -m MODEL [--pooling cls|mean|last|none] [--no-normalize] [--query] [--json] TEXT... | --file F (one text a line), printing one vector a line or OpenAI's JSON.
  • oklm rag index PATH... -m EMB --out INDEX [--chunker fixed|recursive|semantic|code] [--chunk-size N] [--overlap N] [--vec f32|f16|q8_0] [--exclude GLOB]: incremental when INDEX exists (chunks of unchanged files kept by hash, removed files dropped).
  • oklm rag search INDEX QUERY [-k N] [--mode dense|sparse|hybrid] [--rerank MODEL] [--json] (the embedding model named in the index, found beside it or given with -m).
  • oklm rag info INDEX: model identity, chunker, counts, sizes, sources.
  • oklm rag eval INDEX --questions F [--qrels F] [--json]: the metrics.
  • oklm run|chat ... --rag INDEX [--rag-k N] [--rag-tokens N] [--rag-mode M] [--rag-rerank MODEL] [--rag-corrective] [--rag-log F].
  • oklm serve ... --embed-model EMB [--rerank-model R] [--rag INDEX]: POST /v1/embeddings (OpenAI: input a string or an array, model, encoding_format float or base64, dimensions refused unless equal; usage.prompt_tokens), POST /v1/rerank (Cohere and Jina shape: query, documents, top_n, return_documents; results with index and relevance_score), and on chat and completions a rag object ({"k":5}) when started with --rag. Embedding and rerank requests run on their own context beside the scheduler, one at a time, batched within a request, behind the same queue and its 503.
  • Agent (K7): settings rag.enabled, rag.index (default .oklm/rag/project.gguf), rag.model, rag.include, rag.exclude, rag.autoIndex (false); /rag index, /rag status; the Retrieve tool, deferred.

Tool selection by retrieval (K6)

Today (agent.c, prompt.c): the tools in deferred_tools (by default ProcessList, PowerShell, LSP, ScheduleWakeup, Service, NotebookEdit, WebSearch, AskUserQuestion, EnterWorktree, ExitWorktree; the deferredTools setting replaces it; mcp__* is added when the MCP servers' schemas exceed mcp.deferAboveTokens, a quarter of the context window by default) stay out of the offered array but in all_tools_json, which the tool-call grammar accepts (agent_tools, put_tool). The prompt names them, built-ins in one line and each MCP server's tools in a line of their own (put_deferred). Skills past the 2000-token list budget and the built-in integration skills (searchable) are counted, not listed (oklm_prompt_skills_listed). ToolSearch is offered when anything is deferred or unlisted; select:A,B loads exact names, otherwise the query's words are scored 3 in a name and 1 in a description over the deferred tools and the model-invocable skills, best first, at most max_results (5, cap 32); a loaded tool is noted in loaded_tools and, for an HTTP backend, offered from the next turn (agent_retool); the in-process model's prompt stays the same bytes.

K6 keeps that contract and changes the ranking and the grouping:

  • src/harness/tool_index.c embeds each tool's name, description and parameter names, each skill's name and description, each sub-agent type's description and each plugin command's, once per agent start and again when the set changes (MCP list_changed, skills reload, plugins); vectors kept in memory and cached in .oklm/rag/tools.gguf keyed by the text's hash, so a restart embeds only what changed. The embedder is toolSearch.model (default the RAG default model).
  • Groups: Type (built-in, MCP, skill, sub-agent, plugin) to Service (the MCP server, the plugin) to Tool, from the name prefixes and sources already known (mcp__SERVER__TOOL, the plugin's name).
  • toolSearch.mode: keyword (default, today's behavior, bytes unchanged), semantic (the query embedded; services ranked by their best tool, then tools within the top services; the keyword score added as a tie-break), hybrid (RRF of both). select: is unchanged.
  • In semantic mode with toolSearch.prefetch on, before each user turn the top toolSearch.topK (8) tools for the user's message are loaded for an HTTP backend (as ToolSearch would), and the deferred section of the prompt lists the groups and their sizes instead of every name, which is where the tokens are saved with a hundred MCP tools. For the in-process model the prompt is unchanged (its prefix cache), and the top tools come as a reminder.
  • Measured on tests/rag/tool_queries.jsonl (rag-design.md, "Evaluation"), with the agent evals as the guard.

Decisions taken

  1. Default embedding model: BAAI/bge-small-en-v1.5 (BERT, 33 M parameters, 384 dimensions, 512 tokens, CLS pooling, MIT). Small enough for make test's public part and the CPU, a WordPiece tokenizer OKLM already matches, the most common default of local RAG stacks. Checked in K1 as well: bge-m3 (XLM-R, 568 M, 1024 dimensions, 8192 tokens, CLS, multilingual; dense output only, its sparse and multi-vector heads ignored) for other languages, multilingual-e5-small (XLM-R, 118 M, mean pooling, query: / passage: prefixes) for the mean-pooling path, nomic-embed-text-v1.5 (nomic-bert, 137 M, mean, rope and SwiGLU, task prefixes) for the second architecture, Qwen3-Embedding-0.6B (decoder, last token, 1024 dimensions, instruction prefix on queries) as the decoder-based one.
  2. Reranker: cross-encoder/ms-marco-MiniLM-L-6-v2 by default (BERT with a one-label head, 22 M, Apache-2.0): the same code path as the default encoder, fast enough to rerank 50 candidates on the CPU. bge-reranker-v2-m3 (XLM-R, 568 M, multilingual, Apache-2.0) is the quality option, and bge-reranker-base the checked middle one; both exercise the XLM-R path with cls and cls.output. Scores are raw logits (as llama.cpp), sigmoid only in /v1/rerank's relevance_score.
  3. Pooling from the file: <arch>.pooling_type, else sentence-transformers' 1_Pooling/config.json, else the family's default (CLS for bert, mean for nomic-bert, last for decoders); --pooling overrides. L2 normalization on by default (llama-embedding's default too), recorded in the index.
  4. Where indexes live. oklm rag takes any path. The agent's index is .oklm/rag/project.gguf in the project (git-ignored like the rest of .oklm/); the agent never indexes on its own unless rag.autoIndex is set: /rag index asks first, shows the file count and size, follows .gitignore, rag.include / rag.exclude and the read permission rules (a file the agent may not Read is not indexed), and caps the size (rag.maxBytes, 200 MB of text). Once built, the agent refreshes changed files by hash before a Retrieve.
  5. Defaults. Chunks: recursive for prose (Markdown headings, then paragraphs, then sentences) with a 256-token target, 32 minimum, the encoder's context as maximum, no overlap; fixed 256 with 32 overlap; semantic breaks at the 95th percentile of neighbor distances; code by functions and classes up to 512 tokens, split recursively above. Search: 50 candidates per retriever, RRF k = 60 (Cormack et al., robust without tuning), weighted fusion 0.5 / 0.5 of min-max normalized scores when asked, 50 reranked, 5 chunks into the prompt within 2048 tokens; BM25 k1 = 1.2, b = 0.75. Corrective thresholds (K5): relevant when System One's probability of "relevant" is at least 0.5, irrelevant under 0.2, ambiguous between; a retrieval is poor when no chunk is relevant, which rewrites once, then falls back. These are starting points tuned on the documentation set's dev split in K5, not on its test split.
  6. Exact search first. One query is a product of a [dim] row with the [dim, n] vector tensor and a top-k: at a million F16 chunks of 384 values, 768 MB read at the 25 to 35 GB/s the CPU's products reach, about 30 ms; Q8_0 halves it. An approximate index (HNSW first: no training step, incremental) is added only if K2's measure shows the 95th percentile query above 100 ms at the sizes targeted, or the vectors not fitting in memory; the format reserves it (ADR-0022). Expected: exact up to a million chunks.
  7. Index format: one GGUF file (ADR-0022).

Models to download (K1, nothing in K0)

Listed in tests/public_models.txt (or public_models_large.txt for the large ones) with their sha256, fetched by tools/fetch_models.sh:

Model Hugging Face Files
bge-small-en-v1.5 BAAI/bge-small-en-v1.5 model.safetensors, config.json, modules.json, 1_Pooling/config.json (the tokenizer is there already); the GGUF from CompendiumLabs/bge-small-en-v1.5-gguf (F32 and Q8_0)
ms-marco-MiniLM-L-6-v2 cross-encoder/ms-marco-MiniLM-L-6-v2 safetensors, config, tokenizer
bge-reranker-v2-m3 BAAI/bge-reranker-v2-m3; GGUF gpustack/bge-reranker-v2-m3-GGUF large list
bge-m3 BAAI/bge-m3; GGUF gpustack/bge-m3-GGUF large list
multilingual-e5-small intfloat/multilingual-e5-small safetensors, config, tokenizer, pooling config
nomic-embed-text-v1.5 nomic-ai/nomic-embed-text-v1.5; GGUF nomic-ai/nomic-embed-text-v1.5-GGUF
Qwen3-Embedding-0.6B Qwen/Qwen3-Embedding-0.6B; GGUF Qwen/Qwen3-Embedding-0.6B-GGUF (Q8_0, F16)

The third-party GGUF repositories are checked at download (that they come from llama.cpp's converter, from the same revision of the checkpoint); where none fits, the safetensors are loaded directly by OKLM and the GGUF made by oklm convert, compared tensor by tensor with what llama.cpp's converter would write (tools/hf_to_gguf_check.py).

Test references

ADR-0015 rules out PyTorch, and sentence-transformers needs it (build/venv has numpy, gguf-py, safetensors, tokenizers and jinja2, no torch). So:

  • llama.cpp's llama-embedding: tools/gen_llama_ref.sh builds the llama-embedding target beside llama (today only llama, gen_llama_ref.sh:16) and records, per model, the vectors of the reference strings (--embd-output-format json, --embd-normalize -1 for raw values, --pooling), and rerank scores (--pooling rank, query<TAB>passage lines). Proof: cosine at least 1 - 1e-4 on the same GGUF.
  • tools/embed_ref.py: a numpy float64 encoder written from the model definitions (as tools/infer_ref.py is for decoders), reading safetensors with the safetensors package's numpy loader and tokenizing with tokenizers: the exact GELU and the token types, which llama.cpp does not reproduce. Proof: 1e-5 cosine against OKLM on F32 weights.
  • Published figures: the MTEB leaderboard's per-task nDCG@10 of the model on the BEIR subset (made with sentence-transformers), within noise (± 0.5 points), recorded in K1 from the leaderboard, not from memory.
  • Committed fixtures: a tiny random bert and nomic-bert GGUF and their numpy outputs (tools/gen_model_fixtures.py), so make test checks the forward pass with no download.

The 1000 strings per model are the documentation set's chunks and questions (English) and, for the multilingual models, the same through a fixed list of 200 sentences in eight languages committed in tests/rag/strings.txt.

Evaluation

make rag-eval (tools/rag_eval.sh, outputs in build/rag-eval/: report.md, report.json, one row per configuration), the RAG counterpart of make eval:

make rag-eval RAG_MODEL=models/bge-small-en-v1.5/... \
    RAG_ARGS="--sets docs,scifact --mode dense,sparse,hybrid --rerank models/ms-marco-MiniLM-L-6-v2"
  • The documentation set. tests/rag/docs_questions.jsonl, committed: 120 questions over OKLM's own docs and README, frozen at a pinned revision (the corpus is extracted from that revision into build/rag-eval/corpus/, so later doc changes do not move the labels; the revision is bumped with the set). Kinds: 60 lookups (one section answers), 25 how-to (a command and its flags), 20 multi-hop (two sections), 15 unanswerable (nothing in the docs answers; the right answer says so). Each question has relevant: the sections that answer it as file#anchor with a grade (2 primary, 1 supporting), and answer: the facts an answer must contain (strings or regular expressions) and forbidden ones. A chunk counts as relevant when at least half of it lies in a labelled section, so the labels do not depend on the chunker. Labels are written by hand from the sections; oklm rag eval --check verifies that every anchor exists in the frozen corpus and every answer fact appears in its sections. 20 questions are the dev split (thresholds, prompts), 100 the test split (reported).
  • The public set: BEIR SciFact, test split: 5,183 abstracts, 300 queries with graded relevance judgments (CC BY-NC 2.0 in BEIR's table; used locally, never committed or redistributed). Fetched by tools/fetch_datasets.sh --rag (sha256 pinned) into data/beir/scifact/ (data/ is git-ignored, .gitignore:3). NFCorpus (3,633 documents, 323 queries) is the optional second set.
  • Tool selection set (K6). tests/rag/tool_queries.jsonl: 200 requests over 100+ tools (the MCP tool zoo of mcp-training-design.md, the built-ins, the skills and the sub-agent types), each labelled with the tool or tools it needs; accuracy@1 and @5 and prompt tokens.
  • Metrics. Retrieval: recall@1, 5, 10, 20, MRR@10, nDCG@10 (graded), computed by oklm rag eval in C (src/rag/eval.c, unit-tested against hand-computed cases). Generation (K5 on): answer accuracy (every fact present, no forbidden one), abstention accuracy on the unanswerable questions, citation faithfulness (each cited chunk judged supporting by System One, and 20 answers checked by hand per report), tokens put in the prompt. Cost: latency per stage (embed, search, fuse, rerank, judge, generate) at p50 and p95, index build time and size.
  • Before and after. Each step adds its rows and keeps the earlier ones (dense only, then hybrid, then reranked, then corrective), so the report shows what each capability adds; K9 copies the final table into docs/rag.md.

As built, the sets and targets differ from this plan: the docs set has 120 questions with no unanswerable split (the 8 unanswerable questions are in tests/rag/docs_set/answers.jsonl), the tool set 152 requests, make rag-eval scores retrieval only and the generation, graph, figure and agent rows are separate commands (rag.md, Reproducing it).

Tests per step

Step Unit (make test) Integration Evaluation metric
K1 tiny bert / nomic-bert fixtures against numpy (1e-5); pooling and normalization; SEP from a GGUF WordPiece; XLM-R positions; Qwen3 last-token rows equal to oklm_get_hidden()'s; CPU against OpenCL (1e-4) public models (when present): llama-embedding 1e-4 cosine, embed_ref.py 1e-5; /v1/embeddings through the OpenAI Python client (test_server) SciFact nDCG@10 equal to the published figure; docs set recall@10 and MRR (dense)
K2 chunk boundaries on fixtures (headings, code, overlap, sizes, offsets and lines round-trip); index write, open, check-model refusal; search equal to a brute-force double loop (same top-k, 1e-6) incremental update equal to a rebuild; oklm rag end to end on docs/; a truncated or foreign file refused index and search times at 10 k, 100 k and 1 M chunks (synthetic vectors at 1 M)
K3 BM25 against a hand-computed corpus; RRF and weights on fixed lists hybrid search end to end hybrid recall@10 ≥ dense and ≥ sparse, both sets
K4 pair encoding (both tokenizers, token types); cls head on a fixture llama-embedding --pooling rank scores (type-0 switch); /v1/rerank nDCG@10 after reranking above hybrid; latency per query
K5 pipeline with mock stages: budget, citations, verdict thresholds, rewrite then fallback order, trace contents oklm run --rag with a small model; the server's rag field; WebSearch fallback against a stub SearXNG answer accuracy, faithfulness, abstention against no retrieval and naive retrieval
K6 tool index rebuild on change, grouping, ranking with a stub embedder; keyword mode bytes unchanged (prompt snapshot test) ToolSearch semantic over the MCP zoo with a real embedder tool accuracy@1/@5, prompt tokens; agent evals' pass rate unchanged or better
K7 Retrieve tool arguments, permission rules applied to indexing, router decisions with a stub new agent eval tasks (docs, earlier session) pass rate, turns, tokens against K5
K8 graph extraction and traversal on fixtures; self-critique loop with stubs captioned figures indexed multi-hop set, faithfulness with and without self-critique, figure questions
K9 docs lint make rag-eval reproduces the table the full table

Implemented: K2 and K3

What K2 and K3 built (user guide: rag.md) and where it differs from the plan above.

Headers. include/oklm/rag/encoder.h (the encoder interface and the hash encoder), chunk.h (chunkers), index.h (the index, dense and BM25 search, fusion, oklm_rag_query()), cli.h (oklm rag, run in process by the tests). Sources in src/rag/: encoder.c, chunk.c, chunk_code.c, index.c, bm25.c, fuse.c, cli.c, with rag_internal.h. Tests: tests/rag/test_chunk.c, test_index.c, test_bm25.c; the measure tests/rag/bench_index.c.

The encoder interface, and where K1 plugs in. K2 came before K1, so the chunkers and the index take an oklm_rag_encoder (encoder.h) instead of the planned oklm_embedder: a function embed(self, texts, n, is_query, out) filling n rows of dim floats, and the identity ADR-0022 records (model_id, name, arch, file, dim, pooling, normalize, query_prefix, passage_prefix, max_tokens) plus the model's tokenizer for chunk sizes. Two encoders: the hash encoder (oklm_rag_encoder_hash(), deterministic, what the tests and the measures use), and an embedding model. The hook for K1 is one function, open_encoder() in src/rag/cli.c, which today refuses anything but hash[:DIM] ("embedding models are not available in this build (they need oklm_embed)"). With K1 it loads the model (oklm_model_load()), creates the embedder (oklm_embedder_new()) and fills the struct: embed a three-line wrapper calling oklm_embed(self, texts, n, is_query, out), self the embedder, dim = oklm_embedder_dim(), pooling = oklm_embedder_pooling(), model_id = oklm_model_id(), name and arch from general.name and general.architecture, file the path, the prefixes and normalization from the embed parameters, max_tokens the context, tok the model's tokenizer; the same in cmd_search when the index names a model (rag.embed.file, or --model). Nothing else changes: the index refuses vectors of another identity by itself (oklm_rag_index_check_encoder()).

Differences from the plan.

  • API: one call builds or updates and commits (oklm_rag_index_build() for paths, oklm_rag_index_build_sources() for documents in memory with a loader for changed ones), instead of create / update / commit; the description type is oklm_rag_info. The output depends only on the sources, the encoder, the settings and now: an update writes the bytes of a rebuild (tested).
  • auto chunker added (code for source files by extension, recursive otherwise), the default, recorded as auto in rag.chunk.kind.
  • Chunk sizes are counted word by word (with the tokenizer when there is one, else an approximation), so cuts never fall inside a word.
  • Code chunks by braces and indentation only; the language server's symbols are left for later. PDF pages are not recovered (page 0).
  • Semantic groups are capped at max, not at size.
  • The BM25 analyzer splits at _ and strips plurals by default (unicode-lower-s-1; unicode-lower-1 without); a query's repeated terms count once. An update re-reads the kept chunks' texts to rebuild the postings (vectors are copied, never embedded again).
  • The lock file holds the writer's process id only (no start time); a lock whose process is gone is taken over.
  • Directories are walked with entries starting with . left out; --exclude GLOB is not there yet.
  • oklm rag search defaults to hybrid when the index has postings.

What K4 and K5 build on. A reranker takes the hits of oklm_rag_query() (k_rerank of them, chunk texts from oklm_rag_index_chunk(), which points into the mapping) and reorders them; oklm_rag_query()'s candidates and the fusion functions are the retriever stage of retrieve.h's contract. K5 reads the chunk's source, lines and heading for citations from oklm_rag_chunk, and its hash to tell a changed source.

Implemented: K4

What K4 built (user guide: rag.md) and where it differs from the plan above.

Headers and files. include/oklm/rag/rerank.h (oklm_reranker_new, oklm_rerank, oklm_rag_rerank_hits, oklm_rerank_order, oklm_rerank_probability) and src/rag/rerank.c, in place of an oklm_rerank() on the embedder in embed.h: one object covers both kinds (cross-encoder, decide.c), chosen from the model. embed.h gains oklm_embed_pairs() and oklm_embed_pair_tokens(); model.h gains oklm_eval_batch_types() (the first token of type 1 per item of an embedding context's batch), oklm_model_n_cls_out() and oklm_model_classify(). oklm_seq_tokens is unchanged (a new field would have touched every positional initializer). /v1/rerank is srv_rerank() in src/server/embed.c; the decide.c fallback lives in rerank.c (the retrieve.h stage contracts and pipeline.c come with K5).

The head on the host. cls and cls.output are read as F32 host copies at load (any type oklm_get_rows() reads) and applied to each text's [CLS] row after the forward pass, in double precision: a few hundred thousand multiply-adds a pair, nothing for the device. Token type 1 is a second F32 row beside type 0, placed on the device with it.

llama.cpp's converter drops BERT's pooler. For BertForSequenceClassification (the MiniLM cross-encoders) llama.cpp's conversion/bert.py skips pooler.dense (it keeps only the classifier as cls.output), so its GGUFs score classifier(h[CLS]) instead of classifier(tanh(pooler(h[CLS]))): GGUF-my-repo's F16 file gives 0.08 / −0.22 on the model card's Berlin pair (8.85 / −11.25 from the checkpoint) and falls on SciFact. OKLM maps the pooler to cls for that class only (an embedding checkpoint's pooler stays skipped, as llama.cpp does), oklm convert writes it, and llama.cpp runs such a file correctly (its RANK head applies cls with tanh). A GGUF without cls runs as llama.cpp runs it, and so does one with cls alone (the head is then its tanh).

The WordPiece separator. llama.cpp frames every WordPiece text [CLS] ... [SEP] whatever add_eos_token / add_sep_token say; those only choose what llama-embedding puts between a pair's texts. OKLM's GGUF reader no longer lets add_sep_token = false drop the [SEP] (GGUF-my-repo's files say so), and oklm convert writes add_sep_token = false for a BERT vocabulary as llama.cpp's converter does, so llama-embedding joins a pair with one [SEP], not two.

The llama.cpp proof is 1.3e-3, not 1e-4. ggml's CPU GELU rounds its input to F16 and reads an F16 table, about 1e-3 relative a value; the cross-encoder's logits (−11 to +2) carry that. Against the F32 GGUF with OKLM's tanh and type-0 switches the worst difference over 990 pairs is 1.3e-3 (mean 2.3e-4; 1.9e-3 with llama.cpp's flash attention on, which rounds keys and values to F16); the numpy reference given the same F16 GELU (embed_ref.py --gelu-f16) comes to 4.0e-4 of llama.cpp (mean 6e-5), the rest being F16 roundings that flip on tiny differences upstream. The numpy float64 reference, the correctness proof, is met with room (4.0e-6).

Not done. bge-reranker-base / -v2-m3 (the XLM-R path) were not downloaded or checked (1.1 GB and 2.3 GB; the classes are accepted and the classifier.dense / out_proj names mapped, untested). The SciFact measure is tests/rag/bench_rerank.c (a C program over the library: the index with bge-small, oklm_rag_query() hybrid, oklm_rag_rerank_hits()) rather than oklm rag eval, written at the same time elsewhere.

Implemented: K6

Tool selection by retrieval as planned above, with these differences (the user guide and the measures are in rag.md):

  • One switch, not a mode. toolSearch.mode (keyword, semantic, hybrid) and toolSearch.prefetch became --tool-rag K / toolSearch.topK (0, off): when on, ToolSearch's word queries are ranked by the index and each user message gets its K best entries; when off nothing changes (the prompt's bytes checked in tests/harness/test_tool_index.c). The ranking is hybrid when an embedding model is loaded, BM25 alone when not (toolSearch.model none, or the default model not found); the keyword score is no longer a tie-break (the measure shows it below BM25 on every column).
  • The cache is a RAG index file in the state directory (~/.local/state/oklm/rag/tools-HASH.gguf, HASH of the project's path), not .oklm/rag/tools.gguf: the project's tree stays as it was, and two projects with other skills or servers keep their own. It is built by oklm_rag_index_build_sources(), one source an entry recorded as TYPE/SERVICE/NAME, so an unchanged description keeps its vector across commits and restarts. The set is staged and committed lazily before each retrieval: a set equal to the last one costs a hash, not a build, which covers MCP list_changed, a skills reload and plugins without hooks of their own.
  • Grouping is recorded (Type, Service, Tool) and used by the prompt (each MCP server's tools counted) and the logs; the grouped ranking (services by their best entry, a service's entries together) is an option of oklm_tool_index_params left off: it gave the same top-1 and a lower recall@5 than the flat ranking on the tool set.
  • What K counts. The K best entries of the whole ranking, core tools included; only the deferred tools among them are given to the model, and only when their cosine with the query reaches toolSearch.minScore (0.62): a request served by the core tools adds nothing. Skills and sub-agent types are ranked and logged but not suggested (suggesting skills did not change the agent evals' pass rate, and a request that needs no deferred tool then sees the prompt it has without retrieval); ToolSearch still finds skills. The query is the user's message and the previous one (2000 bytes at most).
  • The in-process model gets the retrieved definitions in a reminder after the user's message (as a ToolSearch result would read), the backend in its tools array; sub-agents do not retrieve (the index is used from the user's agent's thread only).
  • The log is a new agent event, OKLM_AGENT_EV_LOG, written to the debug log by the telemetry recorder and ignored by the other consumers.

Risks

  • Index drift. Files change under an index: updates are by hash at oklm rag index and before an agent Retrieve; search results carry their source's hash and the agent re-reads a file before quoting it.
  • Cost on small machines. Embedding a large repository with a 568 M model takes long on the CPU: the default model is the 33 M one, the index build reports its rate and can be interrupted and resumed (chunks already embedded are kept by hash on the next run).
  • Reference gaps. llama.cpp differs on GELU and token types; the numpy reference covers what it does not.
  • License of the public set. SciFact is non-commercial: it stays a local download for measurement.

Decisions confirmed on review

Asked after K0 and answered on 2026-11-08:

  1. No PyTorch in build/venv. ADR-0015 stays as it is: the references are llama.cpp's llama-embedding and the numpy float64 encoder (tools/embed_ref.py); embedding GGUFs come from published repositories or oklm convert, not from llama.cpp's converter.
  2. BEIR SciFact as the public set, downloaded to data/ for local measurement only, never committed or redistributed (CC BY-NC 2.0).
  3. The exact (erf) GELU by default, as the checkpoints are trained; a test-only switch to llama.cpp's tanh approximation for the comparison with llama-embedding.

As built in K1

K1 follows the design, with these differences (each kept for a reason):

  • The embedding context. oklm_context_params.embeddings keeps every row's final state of an oklm_eval_batch() batch (oklm_get_hidden() by batch row) and restarts each item's sequence at position 0; an encoder's context holds one unused cache cell a layer, and its logits buffer one row. The encoder's per-layer norms are new fields (attn_out_norm, out_norm), the embedding norm and token type row 0 model fields (tok_norm, type0); the cls / cls_out heads wait for K4. bert's learned positions cap the context at the training length.
  • Pooling NONE and RANK are refused by oklm_embedder_new() (one vector a text is the K1 contract; RANK comes with K4), so oklm embed has no --pooling none yet. A file saying NONE gets the family's default.
  • Prefixes. query_prefix / passage_prefix NULL means none, not "the model's": a GGUF carries no instruction, and guessing one from the model's name would be wrong for fine-tunes. The evaluation passes bge's instruction explicitly.
  • The server takes the embedding model as --embed-model beside the generative one; a server made of an encoder alone (no scheduler) is not offered yet. Embedding requests wait for each other on a lock rather than on the scheduler's queue (no 503 from them).
  • Fixtures come from their own generator, tools/gen_embed_fixtures.py, into tests/rag/fixtures/ (the design named tools/gen_model_fixtures.py, whose shared random stream would have rewritten every other fixture).
  • The RAG evaluation in K1 is tools/rag_eval.py and tools/embed_compare.py (Python, numpy): SciFact nDCG@10 and the reference comparisons. src/rag/eval.c, oklm rag eval and the documentation question set move to K2, where the index they score exists; the metrics are computed the same way (BEIR's nDCG@10).
  • Conversion. oklm convert writes bert and nomic-bert GGUFs (the keys and names llama.cpp's converter writes) but refuses RoBERTa / XLM-R checkpoints (their position offset is applied at load only); OKLM loads those directories directly.
  • OpenCL: the device runs products, norms and activations; bidirectional attention falls back to the CPU per layer (kept, measured in rag.md).

Implemented: models in oklm rag, and the evaluation

The pieces K1 and K2 left to join (2026-11-09; user guide: rag.md), and where they differ from the plan above.

  • The encoder hook is a library call rather than code in src/rag/cli.c: oklm_rag_encoder_open(path, embed_params, &enc) loads the model and its embedder and fills the identity as planned ("Implemented: K2 and K3"), oklm_rag_encoder_close() frees them, oklm_rag_encoder_from_spec() takes --model's forms (hash, hash:DIM, a path). general.name is read from the GGUF header (the model API has no metadata accessor); a checkpoint directory is named after itself. The GGUF file and its checkpoint share one oklm_model_id(), so either searches an index built with the other.
  • What an index supplies. Beyond reloading rag.embed.file in oklm rag search, both search and an index update take from an existing index whatever is not given: the model file, the prefixes, the pooling and the normalization; so a prefix needs giving once, at build time. --rrf-k and --weights were added to search for the fusion measures.
  • The evaluation is include/oklm/rag/eval.h and src/rag/eval.c, reading any BEIR-format directory rather than the planned docs_questions.jsonl with answer facts: the generation metrics (answer facts, abstention, faithfulness) belong to K5, which will add them to the set. Documents are ranked where their best chunk ranks; enough chunks are asked for --depth documents (100), which in hybrid mode also raises each retriever's candidates to that number. nDCG uses trec_eval's linear gain (the grade), as BEIR and MTEB do; tools/rag_eval.py was moved to it from 2^grade - 1 (the same on SciFact's binary grades).
  • The documentation set labels sections, not spans: its corpus is the docs cut at headings of levels 1 to 3 into PATH#ANCHOR documents (tools/rag_docs_set.py, read from the pinned revision of the repository, into build/rag-eval/docs/), which BEIR's format scores directly; the "half of a chunk inside a labelled section" rule is not needed, since chunks never cross a section. 120 answerable questions (70 lookups, 30 how-to, 20 two-section), no unanswerable ones and no dev / test split yet (both serve K5's thresholds). The --check is the Python script's, not oklm rag eval's.
  • Fusion defaults stay as decided (RRF k = 60, weights 0.5 / 0.5, 50 candidates): the measure in rag.md shows hybrid at least each retriever alone on both sets, weighted fusion ahead of RRF, and no tuned setting clearly better on both. The default fusion became the weighted one afterwards (2026-11-09), RRF kept as --fusion rrf.

Implemented: retrieval in generation

What K5 built (2026-11-09; user guide: rag.md) and where it differs from the stage contracts above.

  • Header and files. include/oklm/rag/rag.h; src/rag/rag.c (the pipeline, the context and prompt, citations, the log line, counters, the retriever of an index), src/rag/rag_model.c (the generator as judge and rewriter on a context, and the prompts the server shares), src/server/rag.c (the server's stages through its scheduler); tests/rag/test_rag.c and test_rag in tests/server/test_server.c.
  • One run function, no pipeline object. oklm_rag_run(params, stages, query, result) replaces oklm_rag_pipeline_new() / _retrieve() / _answer(): the stages are a plain struct of pointers (oklm_rag_stages), so a surface builds them per call (the server per request, with the request's adapter). The generation is not a stage: oklm_rag_prompt() gives the user message and each surface generates as it already does (its sampler, grammar, streaming, tool calls), so the planned generator.answer callback is not there; its rewrite half is the rewriter stage (oklm_rag_rewriter), and a web search stage (oklm_rag_web) is added for the agent.
  • Passages, not hits. Stages exchange oklm_rag_passage (source, heading, lines, text owned) rather than chunk ids, so a web result and an index chunk go through the same judge and context; the trace is oklm_rag_result itself (rounds of passages with scores, verdicts and their numbers in the prompt) and oklm_rag_result_json() its log line.
  • Evaluator output. The judge returns a probability per passage; the pipeline applies the thresholds (relevant at 0.5, irrelevant under 0.2, as decided). They were not tuned on a dev split: the documentation set has none, and the measure below did not call for it.
  • The judge's prompt puts the question before the passage, so every passage of a query shares the prompt up to the passage (decide.h's prefix rule); the yes / no labels are oklm_decide_yes_no()'s.
  • Fallback order. A rewrite equal to a query already searched is not searched; the web is tried with the original question. A failing evaluator turns the run into naive retrieval, a failing rewriter or web search is skipped, both noted in the result's error.
  • Reranker. The hook is oklm_rag_reranker (the contract's score function); candidates (20) passages go to it. Wiring the cross-encoder of K4 into --rag is left to that step (no --rag-rerank flag yet).
  • Surfaces. --rag-tokens and --rag-rerank of the plan are not flags (the budget is max_tokens of the server's field and the library parameter); --rag-corrective became --rag-crag on|off, on by default. The agent gets a Retrieve tool now (K7 planned it with the index kept by the agent; here it searches the index given with --rag), its judge on a context of its own; with a served model (--url) the passages are not judged. The server answers the run in a rag member of the whole answer, with the cited texts.

Implemented: K7

Agentic, adaptive and conversational retrieval (2026-11-09; user guide: rag.md, Conversational retrieval, Agentic retrieval), and where it differs from the plan above.

  • No pipeline.c. The router, the decomposer and the condenser are three more stages of oklm_rag_stages (oklm_rag_router, oklm_rag_decomposer, oklm_rag_condenser; the generator's on a context in rag_model.c, the server's through its scheduler in src/server/rag.c), run by oklm_rag_run_turn(params, stages, history, query, result); oklm_rag_run() is that call without a history. The planned generator.rewrite(query, history) became the condenser, kept apart from K5's rewriter (which rewrites a query that found nothing, without a conversation). The history is text (oklm_rag_history_text(): six messages, 1200 bytes each, 4000 in all), so every surface builds it from its own message list.
  • Multi-hop in one result. A multi-hop run keeps its rounds in the same oklm_rag_result (each round's hop; OKLM_RAG_MAX_ROUNDS grew to three sub-questions of five rounds), the sub-questions in hops, and one context merged rank by rank, so the prompt, the citations, the log line, the counters and the server's sources needed no change of shape. The route none is a result without rounds whose prompt is the question (oklm_rag_skipped()).
  • The router is off by default (--rag-route, "route", --rag-inject route), as decided before measuring it: it did not measure better (it chose "one" for 38 of 38 questions, rag.md). Conversational rewriting is on by default: it only acts on a follow-up (a request with earlier turns) and measured better than searching the follow-up alone (recall@5 70.3 % against 32.8 %; joining the previous question, 79.2 %, did better still on that set, rag.md).
  • The agent's index lives in the state directory (~/.local/state/oklm/rag/project-HASH.gguf) like K6's tool index, not .oklm/rag/project.gguf, and rag.autoIndex / /rag index are not there: turning retrieval on (rag.enabled, --rag-project) is the consent decision 4 asked for, the index is built at the first search and refreshed before each one by a stamp of the files' names, sizes and times (an unchanged project costs a listing), changed files re-embedded by content hash. The files are those Glob lists (.gitignore, the Read deny rules) that rag.include selects, the documentation by default: the agent greps code well, and indexing a large tree on the CPU costs minutes. /rag status is the debug log's retrieval index: line.
  • Retrieve is a core built-in, not deferred: dispatched by the tool table of tools.c (oklm_tool_ctx.retrieve, the program's search), its definition offered only when that search is set (prompt bytes unchanged otherwise), read-only for the permission modes (Read's class, its argument a query, not a path). The K5 custom tool asked a question per call in the default mode; this one does not.
  • Session memory is a scope of Retrieve (project, memory, all) rather than a second tool, over an index of its own (memory-HASH.gguf): the transcripts as session/ID (user and assistant texts, reminders out, the current session out) and the memory files.
  • Fixed retrieval in the agent (--rag-inject always) was not in the plan: the agent had only the tool. It was needed to compare the agentic tool with K5's fixed retrieval on the same tasks, and it gives the conversational rewrite its place in the agent (oklm_agent_params. rag_context, a reminder after the user's message).
  • K5 leftovers. --rag-rerank MODEL [--rag-rerank-top N] wires a rerank.h reranker into the reranker hook (oklm_rag_cross_reranker()) on run, chat, serve and the agent, off by default. The server's rag now rides on streams (the last chunk, the one with the finish_reason) and on /v1/messages (the message's top level, the stream's message_delta).

Implemented: K8

Graph retrieval, self-critique and figures (2026-11-09; user guide: rag.md, rag.md, rag.md; format: ADR-0022, minor 1), and where it differs from the plan above.

  • Files. include/oklm/rag/graph.h and src/rag/graph.c (names, the extraction's parser and prompt, the tables, the walk), include/oklm/rag/caption.h and src/rag/caption.c (a document's figures, the caption chunk), src/rag/rag_gen.c (the generator and the vision model of an index build, each on a context of its own), include/oklm/rag/critique.h and src/rag/critique.c (the critique's state machine, its record), the generator as critic in src/rag/rag_model.c, the server's critic and retry in src/server/rag.c; oklm_ingest_pdf_images() in src/mm/pdf.c. Tests: tests/rag/test_graph.c (parsing scripted extractions, the graph's round trip and updates, an index without a graph, the walk and its fusion, the judge's rule, a PDF page's image, captions through a stub), tests/rag/test_critique.c (the state machine with a scripted critic). The sets: tests/rag/multihop_set/, tests/rag/figures_set/ with their generators tools/rag_multihop_set.py and tools/rag_figures_set.py; tools/rag_set_eval.py scores both; tools/rag_gen_eval.py gains the critique mode.
  • Graph, stored by chunk. The plan named "entities, relations, their chunks"; the tables are kept in chunk order (mentions, relations) with a sorted entity table, and the reverse indexes are built when the file is opened, so an update copies a kept chunk's facts as it copies its vector and never extracts it again (unless the generator changed). Linking is by normalized name only; embedding similarity was not needed on the measured set (the first stage's hits seed the walk, which links a question worded otherwise).
  • The walk is a list fused, not a retriever stage. It lives in oklm_rag_query() (oklm_rag_search_params.graph), seeded by the question's entities and the first stage's three best hits, so every retriever of a store (run, chat, serve, the agent's Retrieve) has it with --rag-graph on, and oklm rag search measures it.
  • The judge keeps the graph's passages. Measured, corrective retrieval dropped the second fact of every chain (it does not answer the question by itself): 6 of 30 right against 23 naive. A passage the graph found is now kept as ambiguous beside a relevant passage.
  • Self-critique after the answer, not reflection tokens. No model OKLM runs writes Self-RAG's tokens; the support and usefulness questions go through decide.c after the generation, with one retry and a flag. The retry is the surface's callback: oklm run generates without streaming when it is on, the server retries whole answers only (the retry is the request's prompt with the critique after its user message, found in the rendered prompt); oklm chat and the agent do not have it.
  • Figures as captions, no joint encoder. Captions are indexed as text chunks of their document (chunk.image names the figure). The SigLIP text tower was to come only if captions are not found: on the figure set every caption was in the top 5, and the misses are what the 256 M captioner fails to read, which a joint encoder would not change.