OKLMdocumentation v1.0.3
On this page

Architecture

How OKLM is layered, which module owns what, and the rules every module follows. The core layer exists today; the layers above it are designed here and built in the order given in roadmap.md.

Status: core, I/O, model, training, harness, server and client layers implemented, with the native window; multimodal designed.

Layers

┌──────────────────────────────────────────────────────────────────────┐
│ Clients                                                              │
│  oklm-gui (Allegro 5 + n_gui)   web dashboard (HTML/JS over HTTPS)   │
│  oklm (CLI/REPL)                any OpenAI-compatible client         │
├──────────────────────────────────────────────────────────────────────┤
│ oklm-serve  (n_http_server, TLS, SSE)  /v1/chat/completions ...      │
├──────────────────────────────────────────────────────────────────────┤
│ Harness: agent loop, tools, skills, agents, plans, permissions, jobs │
├──────────────────────────────────────────────────────────────────────┤
│ Multimodal: ingest (text/json/html/docx), image, audio, video        │
├──────────────────────────────────────────────────────────────────────┤
│ Model: llama, gpt2; forward, KV cache, embeddings as input,          │
│        sampling, chat templates; backward, AdamW, LoRA, checkpoints  │
├──────────────────────────────────────────────────────────────────────┤
│ Tokenizer: byte-BPE, SentencePiece-BPE, trainer                      │
├──────────────────────────────────────────────────────────────────────┤
│ I/O: GGUF read/write, safetensors read, HF config, convert (exists)  │
├──────────────────────────────────────────────────────────────────────┤
│ Core: dtypes, f16, quants, tensors, arena, backends, ops   (exists)  │
│       cpu backend (scalar + AVX2, nilorea worker team)               │
│       opencl backend (runtime-loaded, own kernels)                   │
│       vulkan backend (runtime-loaded, SPIR-V built in)               │
├──────────────────────────────────────────────────────────────────────┤
│ nilorea-library (source tree)    Allegro 5 (optional)   OpenCL rt    │
└──────────────────────────────────────────────────────────────────────┘

Each layer only calls the layers below it. Every layer is a set of headers under include/oklm/<module>/ and sources under src/<module>/, linked into one static library liboklm.a together with the nilorea modules it needs. Applications under apps/ are thin main() functions over the library.

Core layer

See core.md for the API. In short:

  • dtype: element types numbered like ggml so GGUF type ids map 1:1.
  • f16: half and bfloat16 conversions, bit-exact.
  • quant, quant_k: block quants with ggml-identical layouts and reference algorithms, plus dot-product kernels.
  • arena: a bump allocator; models size it once at load time.
  • tensor: up to 4-D tensors with byte strides and views.
  • backend: the backend interface, a registry, and selection.
  • ops: operation descriptors and convenience functions.
  • cpu/: the CPU backend and its kernels.

I/O layer

See convert.md, formats/gguf.md and formats/safetensors.md. In short:

  • file (oklm/io/file.h): the platform layer under the streaming code, with no #ifdef outside it: read-only file mapping (mmap / Windows file mappings) and its advice (oklm_file_advise_willneed() / oklm_file_advise_dontneed(), madvise), positioned reads from any thread (oklm_file_open_read() / oklm_file_pread(), pread / overlapped ReadFile, optionally unbuffered: O_DIRECT, F_NOCACHE, FILE_FLAG_NO_BUFFERING), the page cache's read-ahead (oklm_file_prefetch(), posix_fadvise), aligned allocation (oklm_file_alloc_aligned()), mkdir -p (oklm_file_make_parent_dirs()), the atomic replace (oklm_file_tmp_name() then oklm_file_replace(): rename / MoveFileEx). io/progress.h is the JSON-lines progress stream of the long commands (--progress-json of oklm train, rl, abliterate, refusals: training.md). io/sys.h adds the available and total RAM (/proc/meminfo, GlobalMemoryStatusEx, host_statistics64) and the free disk space. Only POSIX calls and Win32 ones are used; the macOS branches (F_NOCACHE, host_statistics64, sysctl) are written to the documented API but not compiled here (no Apple machine).
  • gguf: reader (bounds-checked), sharded sets, writer (gguf-py layout, streaming producers).
  • safetensors: reader and sharded repositories.
  • hf: hyper-parameters from config.json or GGUF, name tables per family, Q/K permutation, conversion and requantization.
  • model_check: the compatibility verdict of oklm info.
  • layout (oklm/model/layout.h, in the model module since it places layers): a model's blocks with their tensors by role from the tensor table, the placement of the layers for a projection or a context, and their JSON (oklm info --json, the placement of oklm plan --json, the server's /api/model).

Training

See training.md. The train module (include/oklm/train/, src/train/) sits on the model layer: it reads a loaded model's weights and tokenizer, runs its own F32 forward and backward kernels of GPT-2 and Llama blocks on the CPU worker team, and writes checkpoints that the model layer loads as models (adr/0008-training-checkpoints.md).

Harness

See harness.md. The harness module (include/oklm/harness/, src/harness/) decides whether each tool call runs, from Tool(pattern) rules of several sources and a mode, and records every decision in an append-only audit log (adr/0010-permission-rules.md); tools, the agent loop, jobs and the sandbox build on it.

Server

See server.md. The server module (include/oklm/server/, src/server/) puts a model behind OpenAI-compatible HTTP endpoints on nilorea's HTTP/1.1 server (TLS, streamed bodies), with a dashboard and the access rules of adr/0018-server-api.md. A scheduler thread shares one context of several sequences between the requests, one batched forward pass per step. The client module (include/oklm/client/, src/client/) is the API's HTTP client, plain or TLS through nilorea; oklm-gui (gui.md) is a native window on it, with nilorea's n_gui.

Multimodal

See multimodal.md. The mm module (include/oklm/mm/, src/mm/) decodes media (PNM and WAV itself, the rest through Allegro's addons; Ogg Theora with libtheora; other videos through ffmpeg when allowed, ADR-0007), runs the encoders of llama.cpp mmproj files (SigLIP / idefics3 vision, the Whisper encoder with Ultravox's projector) on the core ops, and evaluates their embeddings in a prompt through oklm_eval_embd(). Whisper speech recognition loads Hugging Face checkpoints. Speech synthesis (speech.md) pairs a language model run by the model module with a vocoder of its own (src/mm/vocoder.c, src/mm/speech.c). Documents become text with nilorea's n_zip and n_html.

Design rules

  • One backend interface. A backend implements buffer management, host↔︎device copies, supports(op), compute(op) and synchronize(). The CPU backend is always registered; the OpenCL backend registers itself when libOpenCL can be loaded at run time and a device exists, and the Vulkan backend when libvulkan can (asked for by name, not by auto yet). Selection: OKLM_BACKEND=auto|cpu|opencl|vulkan. Every op has a CPU implementation that is the reference for the others.
  • Static compute. No dynamic graph: models call layer functions in order over preallocated buffers, the way llm.c does. Training adds explicit backward functions; a light tape is added only where LoRA and gradient checkpointing need it.
  • Caller-allocated outputs. Ops never allocate tensors; the destination is created by the caller with the right shape (usually from an arena).
  • Memory. Weights are memory-mapped from GGUF when their type can be used in place; activations and gradients live in one arena sized at load; the KV cache is preallocated for the context length.
  • Threads. The CPU backend runs parallel loops on a worker team whose threads stay alive between ops and claim each loop's chunks one at a time, so a preempted thread does not stall the loop (see adr/0016-cpu-worker-team.md). Several application threads may run ops at once; their loops share the team. Several contexts of one model may each run in a thread of their own: the weights are read-only, activations and caches belong to the context, and an OpenCL instance takes a lock around each call that enqueues work, so one call's kernels stay together on its in-order queue.
  • Determinism. Every output element of an op is computed by one thread in a fixed order, so CPU results do not depend on the thread count. The quantized SIMD kernels are bit-identical to the scalar ones; the float SIMD kernels sum in a different order and may differ by rounding.
  • Errors. Library functions return int status codes (oklm_status) or NULL, log through n_log, and never call exit().
  • Global state. Only the backend registry (mutex-guarded), the CPU worker team and the detected CPU features are global.