Architecture
How OKLM is layered, which module owns what, and the rules every module follows. The core layer exists today; the layers above it are designed here and built in the order given in roadmap.md.
Status: core, I/O, model, training, harness, server and client layers implemented, with the native window; multimodal designed.
Layers
┌──────────────────────────────────────────────────────────────────────┐
│ Clients │
│ oklm-gui (Allegro 5 + n_gui) web dashboard (HTML/JS over HTTPS) │
│ oklm (CLI/REPL) any OpenAI-compatible client │
├──────────────────────────────────────────────────────────────────────┤
│ oklm-serve (n_http_server, TLS, SSE) /v1/chat/completions ... │
├──────────────────────────────────────────────────────────────────────┤
│ Harness: agent loop, tools, skills, agents, plans, permissions, jobs │
├──────────────────────────────────────────────────────────────────────┤
│ Multimodal: ingest (text/json/html/docx), image, audio, video │
├──────────────────────────────────────────────────────────────────────┤
│ Model: llama, gpt2; forward, KV cache, embeddings as input, │
│ sampling, chat templates; backward, AdamW, LoRA, checkpoints │
├──────────────────────────────────────────────────────────────────────┤
│ Tokenizer: byte-BPE, SentencePiece-BPE, trainer │
├──────────────────────────────────────────────────────────────────────┤
│ I/O: GGUF read/write, safetensors read, HF config, convert (exists) │
├──────────────────────────────────────────────────────────────────────┤
│ Core: dtypes, f16, quants, tensors, arena, backends, ops (exists) │
│ cpu backend (scalar + AVX2, nilorea worker team) │
│ opencl backend (runtime-loaded, own kernels) │
│ vulkan backend (runtime-loaded, SPIR-V built in) │
├──────────────────────────────────────────────────────────────────────┤
│ nilorea-library (source tree) Allegro 5 (optional) OpenCL rt │
└──────────────────────────────────────────────────────────────────────┘
Each layer only calls the layers below it. Every layer is a set of
headers under include/oklm/<module>/ and sources
under src/<module>/, linked into one static library
liboklm.a together with the nilorea modules it needs.
Applications under apps/ are thin main()
functions over the library.
Core layer
See core.md for the API. In short:
dtype: element types numbered like ggml so GGUF type ids map 1:1.f16: half and bfloat16 conversions, bit-exact.quant,quant_k: block quants with ggml-identical layouts and reference algorithms, plus dot-product kernels.arena: a bump allocator; models size it once at load time.tensor: up to 4-D tensors with byte strides and views.backend: the backend interface, a registry, and selection.ops: operation descriptors and convenience functions.cpu/: the CPU backend and its kernels.
I/O layer
See convert.md, formats/gguf.md and formats/safetensors.md. In short:
file(oklm/io/file.h): the platform layer under the streaming code, with no#ifdefoutside it: read-only file mapping (mmap / Windows file mappings) and its advice (oklm_file_advise_willneed()/oklm_file_advise_dontneed(),madvise), positioned reads from any thread (oklm_file_open_read()/oklm_file_pread(),pread/ overlappedReadFile, optionally unbuffered:O_DIRECT,F_NOCACHE,FILE_FLAG_NO_BUFFERING), the page cache's read-ahead (oklm_file_prefetch(),posix_fadvise), aligned allocation (oklm_file_alloc_aligned()),mkdir -p(oklm_file_make_parent_dirs()), the atomic replace (oklm_file_tmp_name()thenoklm_file_replace():rename/MoveFileEx).io/progress.his the JSON-lines progress stream of the long commands (--progress-jsonofoklm train,rl,abliterate,refusals: training.md).io/sys.hadds the available and total RAM (/proc/meminfo,GlobalMemoryStatusEx,host_statistics64) and the free disk space. Only POSIX calls and Win32 ones are used; the macOS branches (F_NOCACHE,host_statistics64,sysctl) are written to the documented API but not compiled here (no Apple machine).gguf: reader (bounds-checked), sharded sets, writer (gguf-py layout, streaming producers).safetensors: reader and sharded repositories.hf: hyper-parameters from config.json or GGUF, name tables per family, Q/K permutation, conversion and requantization.model_check: the compatibility verdict ofoklm info.layout(oklm/model/layout.h, in the model module since it places layers): a model's blocks with their tensors by role from the tensor table, the placement of the layers for a projection or a context, and their JSON (oklm info --json, the placement ofoklm plan --json, the server's/api/model).
Training
See training.md. The train
module (include/oklm/train/, src/train/) sits
on the model layer: it reads a loaded model's weights and tokenizer,
runs its own F32 forward and backward kernels of GPT-2 and Llama blocks
on the CPU worker team, and writes checkpoints that the model layer
loads as models (adr/0008-training-checkpoints.md).
Harness
See harness.md. The harness
module (include/oklm/harness/, src/harness/)
decides whether each tool call runs, from Tool(pattern)
rules of several sources and a mode, and records every decision in an
append-only audit log (adr/0010-permission-rules.md);
tools, the agent loop, jobs and the sandbox build on it.
Server
See server.md. The server
module (include/oklm/server/, src/server/)
puts a model behind OpenAI-compatible HTTP endpoints on nilorea's
HTTP/1.1 server (TLS, streamed bodies), with a dashboard and the access
rules of adr/0018-server-api.md.
A scheduler thread shares one context of several sequences between the
requests, one batched forward pass per step. The client
module (include/oklm/client/, src/client/) is
the API's HTTP client, plain or TLS through nilorea;
oklm-gui (gui.md) is a native window
on it, with nilorea's n_gui.
Multimodal
See multimodal.md. The mm
module (include/oklm/mm/, src/mm/) decodes
media (PNM and WAV itself, the rest through Allegro's addons; Ogg Theora
with libtheora; other videos through ffmpeg when allowed, ADR-0007),
runs the encoders of llama.cpp mmproj files (SigLIP / idefics3 vision,
the Whisper encoder with Ultravox's projector) on the core ops, and
evaluates their embeddings in a prompt through
oklm_eval_embd(). Whisper speech recognition loads Hugging
Face checkpoints. Speech synthesis (speech.md)
pairs a language model run by the model module with a vocoder of its own
(src/mm/vocoder.c, src/mm/speech.c). Documents
become text with nilorea's n_zip and n_html.
Design rules
- One backend interface. A backend implements buffer
management, host↔︎device copies,
supports(op),compute(op)andsynchronize(). The CPU backend is always registered; the OpenCL backend registers itself whenlibOpenCLcan be loaded at run time and a device exists, and the Vulkan backend whenlibvulkancan (asked for by name, not byautoyet). Selection:OKLM_BACKEND=auto|cpu|opencl|vulkan. Every op has a CPU implementation that is the reference for the others. - Static compute. No dynamic graph: models call layer functions in order over preallocated buffers, the way llm.c does. Training adds explicit backward functions; a light tape is added only where LoRA and gradient checkpointing need it.
- Caller-allocated outputs. Ops never allocate tensors; the destination is created by the caller with the right shape (usually from an arena).
- Memory. Weights are memory-mapped from GGUF when their type can be used in place; activations and gradients live in one arena sized at load; the KV cache is preallocated for the context length.
- Threads. The CPU backend runs parallel loops on a worker team whose threads stay alive between ops and claim each loop's chunks one at a time, so a preempted thread does not stall the loop (see adr/0016-cpu-worker-team.md). Several application threads may run ops at once; their loops share the team. Several contexts of one model may each run in a thread of their own: the weights are read-only, activations and caches belong to the context, and an OpenCL instance takes a lock around each call that enqueues work, so one call's kernels stay together on its in-order queue.
- Determinism. Every output element of an op is computed by one thread in a fixed order, so CPU results do not depend on the thread count. The quantized SIMD kernels are bit-identical to the scalar ones; the float SIMD kernels sum in a different order and may differ by rounding.
- Errors. Library functions return
intstatus codes (oklm_status) orNULL, log throughn_log, and never callexit(). - Global state. Only the backend registry (mutex-guarded), the CPU worker team and the detected CPU features are global.