OKLMdocumentation v1.0.3
On this page

safetensors

Layout of safetensors files, the validation OKLM applies, and how sharded Hugging Face repositories are opened. API: include/oklm/io/safetensors.h.

Status: implemented (read only).

Specification: huggingface/safetensors README, "Format" (https://github.com/huggingface/safetensors). Fixtures are written by the reference safetensors Python package.

Layout

Bytes Content
8 uint64 little-endian N, the header size
N UTF-8 JSON object (the reference pads it with spaces to a multiple of 8)
rest the byte buffer

The header maps each tensor name to {"dtype": "BF16", "shape": [49152, 576], "data_offsets": [begin, end]}, offsets relative to the start of the buffer. An optional "__metadata__" object holds string-to-string pairs ({"format": "pt"} for PyTorch checkpoints). Shapes are in PyTorch order, outermost first: a Hugging Face linear weight [out, in] is GGUF ne = {in, out} with the same bytes.

Dtypes: BOOL, U8, I8, U16, I16, F16, BF16, U32, I32, F32, U64, I64, F64, F8_E4M3, F8_E5M2, all little-endian.

Validation

The rules of the reference implementation, each tested by a crafted file in tests/io/test_safetensors.c:

  • the header length fits in the file and is at most 100 MB;
  • the header starts with { and is a JSON object;
  • every entry is an object with a known dtype, a shape of non-negative integers (at most 8 dimensions, product without overflow) and two integer offsets begin <= end <= buffer size;
  • end - begin equals the element count times the dtype size;
  • sorted by offset, the tensors tile the buffer exactly: no hole, no overlap, nothing after the last tensor;
  • __metadata__ values are strings; tensor names are unique.

JSON is parsed with nilorea's n_json (vendored cJSON) from the length-bounded header; numbers are doubles there, so offsets and sizes are also checked to be integral and below 2^53 (see ../adr/0003-json-on-the-model-path.md).

Conversions

oklm_st_to_f32() converts any tensor to floats. F8 formats:

Format Layout Notes
F8_E4M3 sign, 4 exponent bits (bias 7), 3 mantissa bits OCP "fn" variant: no infinities, 0x7F/0xFF are NaN, max 448
F8_E5M2 sign, 5 exponent bits (bias 15), 2 mantissa bits the top byte of an IEEE half: has infinities and NaNs

Repositories

oklm_st_set_open() accepts a .safetensors file, a model.safetensors.index.json, or a directory holding either the index or model.safetensors. With an index, the distinct files of weight_map are opened (names containing / or \ are rejected), every mapped tensor must be in the file the map names, and tensor names must be unique across files.