OKLMdocumentation v1.0.3
On this page

ADR-0003: JSON on the model path

Records how OKLM parses the JSON it meets while loading models (safetensors headers, config.json, tokenizer.json, index files).

Status: accepted (2026-10-29).

Context

safetensors headers and Hugging Face configuration files are JSON. The options were nilorea's n_json wrapper over its vendored cJSON (nilorea is the mandated base library), or a small in-tree parser. The default considered first was cJSON through n_json, isolated behind an oklm_json_* layer so it could be swapped.

Decision

  • Use nilorea's n_json directly. It already is an isolation layer: an opaque N_JSON type and accessor functions, with cJSON hidden behind it. A second wrapper in OKLM would only duplicate it; swapping the parser means reimplementing n_json.
  • n_json lacked what model files need, so it was extended in nilorea: linear iteration (n_json_first() / n_json_next(); per-index access is quadratic on cJSON's linked lists, which matters for the 100 000+ entries of tokenizer merges), member names (n_json_key()), doubles (n_json_number(), n_json_as_number()), type tests, and a length-bounded parse (n_json_parse_len()) for headers inside mapped files.
  • cJSON stores numbers as doubles: integers above 2^53 are not exact. Callers check that offsets and sizes are integral and below 2^53, which covers files up to 8 PiB.
  • Inputs are bounded before parsing: safetensors headers at 100 MB, config.json at 1 MB, index files at 64 MB.

Consequences

The JSON parser is third-party code (vendored in nilorea, MIT), so it is covered by OKLM's fuzzers only through the safetensors target. Replacing it later is a nilorea change with no OKLM API impact.