safetensors
Layout of safetensors files, the validation OKLM applies, and how
sharded Hugging Face repositories are opened. API:
include/oklm/io/safetensors.h.
Status: implemented (read only).
Specification: huggingface/safetensors README, "Format" (https://github.com/huggingface/safetensors).
Fixtures are written by the reference safetensors Python
package.
Layout
| Bytes | Content |
|---|---|
| 8 | uint64 little-endian N, the header size |
| N | UTF-8 JSON object (the reference pads it with spaces to a multiple of 8) |
| rest | the byte buffer |
The header maps each tensor name to
{"dtype": "BF16", "shape": [49152, 576], "data_offsets": [begin, end]},
offsets relative to the start of the buffer. An optional
"__metadata__" object holds string-to-string pairs
({"format": "pt"} for PyTorch checkpoints). Shapes are in
PyTorch order, outermost first: a Hugging Face linear weight
[out, in] is GGUF ne = {in, out} with the same
bytes.
Dtypes: BOOL, U8, I8,
U16, I16, F16, BF16,
U32, I32, F32, U64,
I64, F64, F8_E4M3,
F8_E5M2, all little-endian.
Validation
The rules of the reference implementation, each tested by a crafted
file in tests/io/test_safetensors.c:
- the header length fits in the file and is at most 100 MB;
- the header starts with
{and is a JSON object; - every entry is an object with a known dtype, a shape of non-negative
integers (at most 8 dimensions, product without overflow) and two
integer offsets
begin <= end <= buffer size; end - beginequals the element count times the dtype size;- sorted by offset, the tensors tile the buffer exactly: no hole, no overlap, nothing after the last tensor;
__metadata__values are strings; tensor names are unique.
JSON is parsed with nilorea's n_json (vendored cJSON)
from the length-bounded header; numbers are doubles there, so offsets
and sizes are also checked to be integral and below 2^53 (see ../adr/0003-json-on-the-model-path.md).
Conversions
oklm_st_to_f32() converts any tensor to floats. F8
formats:
| Format | Layout | Notes |
|---|---|---|
| F8_E4M3 | sign, 4 exponent bits (bias 7), 3 mantissa bits | OCP "fn" variant: no infinities, 0x7F/0xFF
are NaN, max 448 |
| F8_E5M2 | sign, 5 exponent bits (bias 15), 2 mantissa bits | the top byte of an IEEE half: has infinities and NaNs |
Repositories
oklm_st_set_open() accepts a .safetensors
file, a model.safetensors.index.json, or a directory
holding either the index or model.safetensors. With an
index, the distinct files of weight_map are opened (names
containing / or \ are rejected), every mapped
tensor must be in the file the map names, and tensor names must be
unique across files.