OKLMdocumentation v1.0.3
On this page

GGUF

Byte-level layout of GGUF files as OKLM reads and writes them, the validation rules the reader applies, and how sharded models are handled. API: include/oklm/io/gguf.h.

Status: implemented (read v2/v3, write v3, sharded sets).

Specification: ggml docs/gguf.md (https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). Writer layout checked against gguf-py 0.19.0 (the Python package published from llama.cpp) and against files written by llama.cpp's C++ quantizer.

Layout

All integers are little-endian. A GGUF string is a uint64 byte length followed by that many UTF-8 bytes, without terminator.

Offset Field Type
0 magic GGUF (0x46554747 as uint32) uint32
4 version (2 or 3) uint32
8 tensor count uint64
16 metadata entry count uint64
24 metadata entries see below
... tensor infos see below
... zero padding to general.alignment
data tensor data, each tensor padded to the alignment

Metadata entry: key (string), value type (uint32), value. Value types:

Id Type Encoding
0 / 1 uint8 / int8 1 byte
2 / 3 uint16 / int16 2 bytes
4 / 5 uint32 / int32 4 bytes
6 float32 4 bytes
7 bool 1 byte, 0 or 1
8 string uint64 length + bytes
9 array uint32 element type, uint64 count, elements (arrays may nest)
10 / 11 uint64 / int64 8 bytes
12 float64 8 bytes

Tensor info: name (string), uint32 n_dims, uint64 ne[n_dims] (innermost first, the same order as oklm_tensor.ne), uint32 type (ggml type id, see quant.md), uint64 offset relative to the start of the data section. The data section starts at the end of the tensor infos rounded up to general.alignment (default 32).

Worked example

tests/fixtures/gguf/kitchen.gguf (written by gguf-py) starts with 47 47 55 46 (magic), 03 00 00 00 (version 3), 05 00.. (5 tensors), 14 00.. (20 entries), then the first entry: key length 14 00.. and general.architecture, type 08 00 00 00 (string), length 04 00.., test. Its first tensor, f32 of shape 5 x 3, has offset 0 and its second, f16, offset 64: 60 bytes of F32 data padded to 64.

Reader validation

The reader (oklm_gguf_open(), oklm_gguf_open_memory()) maps the file and rejects, with OKLM_ERR_FORMAT and a logged reason:

  • a bad magic, an unknown version (1 and big-endian files give OKLM_ERR_UNSUPPORTED with a specific message);
  • counts that cannot fit in the bytes left (every entry needs at least 12 bytes, every tensor info 32), checked before allocating anything;
  • any string, array or value running past the end of the file; bools other than 0 or 1; unknown value types; arrays nested deeper than 8;
  • empty keys, keys with NUL bytes, duplicate keys;
  • general.alignment that is not a power-of-two uint32;
  • tensor names empty, longer than 63 bytes, with NUL bytes, or duplicated;
  • 0 or more than 4 dimensions, retired or unknown types, rows that are not a whole number of blocks, sizes that overflow;
  • tensor offsets not aligned, data outside the file, or two tensors whose data overlap.

The reader is more lenient than ggml in one respect: it does not require tensors to be stored in the order of their infos, only aligned, inside the file and non-overlapping.

Values are exposed as parsed oklm_gguf_values. Arrays of scalars point at the raw elements in the mapping (read them with oklm_gguf_arr_int() / oklm_gguf_arr_float(), which handle unaligned little-endian data); arrays of strings and arrays of arrays have parsed items. Typed getters (oklm_gguf_get_int(), _float(), _bool(), _str(), _arr()) accept any integer width, since files disagree on uint32 vs int32 for the same key.

The parser is fuzzed (make fuzz-run, fuzz/fuzz_gguf.c), every truncation of a fixture is tested, and 20 000 random header corruptions run under ASan in tests/io/test_gguf.c.

Writer

oklm_gguf_writer reproduces gguf-py's GGUFWriter layout exactly:

  1. header (version 3);
  2. metadata in insertion order (setting an existing key replaces its value in place);
  3. tensor infos in insertion order, offsets accumulated with each tensor's size padded to the alignment;
  4. zero padding to the alignment;
  5. each tensor's data followed by zero padding to the alignment, the last tensor included.

oklm_gguf_writer_set_alignment() also writes general.alignment, and copying a general.alignment key from a reader changes the data alignment, so a file copied key by key keeps its layout. Tensors can be given as borrowed memory or as a producer callback called during the write, which keeps conversions at one tensor in memory. Files are written to path.tmp and renamed.

A file read and written back with every key and tensor copied in order is byte-identical: checked on gguf-py fixtures (all value types, nested arrays, custom alignment, shards) and on the published SmolLM2 Q4_K_M, Q8_0 and F16 files written by llama.cpp.

Sharded models

llama.cpp's gguf-split and gguf-py write shards named <prefix>-00001-of-0000N.gguf, each with split.no (uint16, 0-based), split.count (uint16) and split.tensors.count (int32); metadata other than split.* is only in shard 0. oklm_gguf_set_open() opens the model from any shard: it derives the sibling names, checks every shard's split.no/split.count, that tensor names are unique across shards and that their total equals split.tensors.count, and presents one tensor list. A single-file model is a set of one.

oklm_gguf_split() / oklm split write shards as gguf-split --split does: by maximum size (each tensor counted rounded up to 32 bytes) or by tensor count; shard 0 gets every key of the input followed by split.no, split.tensors.count and split.count, the other shards only those three keys. oklm_gguf_merge() / oklm merge write one file from the shards, keeping shard 0's keys with split.count set to 0, as gguf-split --merge. Both outputs are identical byte for byte to llama.cpp's (tools/check_split.sh, part of make test-public).