OKLMdocumentation v1.0.3
On this page

ADR-0022: The RAG index file

Records how a retrieval index is stored: the container, what identifies the embedding model, the chunks, the vectors, the BM25 postings, how the file is updated, and what later steps may add.

Status: accepted (2026-11-08), with the RAG specification's K0 (rag-design.md); implemented in K2 (src/rag/index.c; the lock file holds the writer's process id only); minor 1 (the graph and the figures) added in K8 (2026-11-09).

Context

rag_spec.md K2 adds oklm rag index, which embeds the chunks of documents and keeps them for search; K3 adds keyword search over the same chunks; K7 lets the agent keep an index of its project up to date; K8 adds a graph. The index is read by oklm rag search, by --rag on run, chat and serve, and by the agent, on Linux and Windows. Its layout is a convention between the builder, the searchers and the tests, so it is settled before any code.

What it must hold:

  • The embedding model's identity: vectors of another model (another quantization of it, another pooling, other prefixes) are meaningless and must be refused, not searched.
  • Per chunk: its source, where it lies (bytes in the extracted text, lines, PDF page), its token count, its heading or symbol, its text (to put it in a prompt without reading the source again, which may have changed) and a hash of that text.
  • Per source: its path, size, modification time and content hash, for incremental updates.
  • The vectors, F32, F16 or Q8_0.
  • From K3, BM25: terms, postings, chunk lengths.

What OKLM already has: GGUF read through memory maps (oklm_gguf, oklm_file_map), aligned tensors of any OKLM type, string arrays and scalars as keys; a writer that takes tensors from memory or from a callback as it writes (oklm_gguf_writer_add_tensor_cb), into a temporary file renamed over the target (gguf_write.c:507-516, oklm_file_replace, which also works on Windows); oklm_model_id() (kv_file.c), a model's hash over its architecture, sizes, tensor table and sampled weights, already used to refuse a KV file of another model; and oklm_matmul() / oklm_topk(), which take a [dim, n] tensor of F32, F16 or Q8_0 rows as their weights.

Options considered:

  1. A GGUF file: the container OKLM reads and writes everywhere, gguf-dump and gguf-py read it for tests and debugging, the vector tensor maps straight into the search kernels with no copy.
  2. A format of its own (header, sections, offsets): the same content with a reader, a writer, alignment and fuzzing to write again.
  3. A directory of segment files with a manifest (append-only segments merged later): cheaper updates of very large indexes, at the cost of a compaction policy and several files to keep consistent.
  4. SQLite or another store: a dependency, ruled out (architecture.md).

Decision

  • One GGUF file per index (option 1), version 3 of GGUF, the writer's default alignment (32). The default name is INDEX.gguf; general.type is rag-index and there is no general.architecture, so the model loader refuses it with a clear message and oklm rag info recognizes it.
  • Keys (all under rag.):
    • rag.version (u32, 1) and rag.minor (u32, 0): a reader refuses a higher version and ignores tensors and keys of a higher minor it does not know.
    • The model: rag.embed.model_id (u64, oklm_model_id()), rag.embed.name (general.name of the model), rag.embed.arch, rag.embed.file (the path it was built with, a hint only), rag.embed.dim (u32), rag.embed.pooling (i32, llama.cpp's numbering), rag.embed.normalize (bool), rag.embed.query_prefix, rag.embed.passage_prefix (strings), rag.embed.max_tokens (u32). Opening for search or update with a model whose id, pooling, normalization or prefixes differ fails (OKLM_ERR_STATE, the message naming both models); oklm rag info works without a model.
    • The chunker: rag.chunk.kind, rag.chunk.size, rag.chunk.overlap, rag.chunk.min, rag.chunk.max, rag.chunk.breakpoint. An update with other chunker settings rebuilds every source (and says so).
    • rag.root (the directory sources are relative to, as given), rag.n_chunks (u64), rag.n_sources (u32), rag.created, rag.updated (i64, Unix seconds), rag.vec_type (the vectors' type name, also the tensor's type).
    • BM25 (K3): rag.bm25.k1, rag.bm25.b (f32), rag.bm25.n_terms (u64), rag.bm25.avg_len (f32), rag.bm25.analyzer (string: the analyzer's name and version, unicode-lower-1).
  • Tensors. n chunks, s sources, t terms:
    • vectors [dim, n], F32, F16 or Q8_0 (Q8_0 needs dim % 32 == 0, true of every model in rag-design.md), rows normalized when rag.embed.normalize, so a dot product is the cosine.
    • chunk.source I32 [n]; chunk.span I64 [6, n] (start byte, end byte, first line, last line, page, tokens); chunk.hash I64 [n] (64-bit hash of the chunk's text, the same function as the KV files'); chunk.text I8 [bytes] (every chunk's UTF-8 text, one after the other, no terminator) with chunk.text_off I64 [n + 1]; chunk.heading I8 and chunk.heading_off I64 [n + 1] likewise.
    • source.path I8 with source.path_off I64 [s + 1] (UTF-8, / as the separator on every platform); source.meta I64 [4, s] (size, modification time, content hash, document kind); source.chunk0 I32 [s + 1]: the chunks of a source are contiguous, source i owning chunks chunk0[i] to chunk0[i + 1] - 1.
    • BM25 (K3): bm25.term I8 with bm25.term_off I64 [t + 1], terms sorted by bytes (binary search); bm25.post_off I64 [t + 1]; bm25.post_chunk I32 [postings] (ascending within a term); bm25.post_tf I16 [postings]; bm25.len I32 [n] (terms per chunk).
  • Updates rewrite the file. oklm rag index on an existing index hashes each source; unchanged sources keep their chunks, vectors and postings (copied, never re-embedded), changed ones are chunked and embedded again, missing ones dropped; the new file is written through the writer's callback tensors (nothing holds the whole index twice in memory) to the temporary file and renamed over the old one, so a reader sees the old index or the new one, never half of each, and an interrupted build leaves the old index intact. Embedding dominates the cost (milliseconds per chunk against a sequential write of the file); a rewrite of a million-chunk F16 index of 384 dimensions writes about 1.5 GB, a few seconds.
  • Concurrent writers take an exclusive lock file INDEX.gguf.lock (created with O_EXCL, holding the process id and start time, a stale one taken over as the job manager does with its journals); readers take no lock.
  • Reserved for later minors: hnsw.* tensors (an approximate graph over vectors, only if K2's measure asks for it), graph.* (K8: entities, relations, their chunks), chunk.image_* (K8: captions' sources). A reader of minor 0 ignores them.

Minor 1: the graph and the figures (2026-11-09)

Added with graph retrieval and captioned figures (rag.md, rag.md). An index that has neither is written as before, rag.minor 0, byte for byte; one with either is rag.minor 1. Every addition is optional: a reader of minor 0 ignores it (the chunks of figures are then ordinary text chunks), and this reader opens minor 0 files as before.

  • Graph (built with --graph). Keys rag.graph.model (the extractor's name, the generator's file name), rag.graph.extraction (triples-1: the prompt and the JSON schema of graph.h), rag.graph.n_entities, rag.graph.n_mentions, rag.graph.n_relations (u64). Tensors, e entities, m mentions, r relations:
    • graph.entity I8 with graph.entity_off I64 [e + 1]: the normalized names (oklm_rag_entity_norm()), sorted by bytes and distinct, so a name is found by binary search and two chunks naming the same thing share one entity;
    • graph.mention I32 [2, m]: (chunk, entity) pairs, sorted by chunk then entity: the entities each chunk names (a relation's two ends included);
    • graph.relation I32 [3, r]: (chunk, subject, object), in chunk order, with graph.label I8 and graph.label_off I64 [r + 1] (the relation's words, lowercased). Everything is stored by chunk, so an update copies a kept chunk's facts as it copies its vector; the reverse indexes (an entity's chunks, its neighbors) are built in memory when the file is opened. A kept chunk keeps its facts only when rag.graph.model is the extractor of the update; with another one every chunk is extracted again (the vectors are still kept).
  • Figures (built with --caption-model). Key rag.caption.model (the captioner's name). Tensors chunk.image I8 with chunk.image_off I64 [n + 1]: per chunk the figure it captions (FILE, FILE#page=N&image=NAME for a PDF's, FILE#MEMBER for a DOCX's), empty for a text chunk. A caption chunk's span is the line where its document shows it (start and end bytes 0), its page the PDF page. The caption model is part of what makes an index reusable: an update with another captioner, or without one, chunks every source again. Key rag.caption.prompt (added 2026-11-09, still minor 1, optional): the captioner's question when it is not the default (--caption-prompt); absent for the default. It is compared like the model: an update with another prompt, or back to the default, captions every figure again.

Consequences

  • Search maps the file and runs oklm_matmul() on the vectors tensor as it lies in the map, then oklm_topk(): no loading step, no second copy, the page cache shared between processes searching the same index.
  • gguf-py reads an index, so the tests check the layout with an independent reader, and gguf-dump shows its keys.
  • An update costs a full rewrite. For the sizes targeted (up to a million chunks) this is seconds; if the agent's incremental refreshes of a large project show it in K7, segment files (option 3) become a minor version: a base file plus small delta files merged on the next full write.
  • The chunk texts are duplicated from the sources (an index of docs/ is about the size of the text plus the vectors), which is what lets a prompt cite exactly what was retrieved even after the file changed.
  • A model upgrade (another quantization of the same model included) requires a rebuild; oklm rag info names the model the index needs.