![]() |
Nilorea Library
C utilities for networking, threading, graphics
|
Reading the tensors of a PyTorch checkpoint (.pth, .pt) without Python. More...
#include <stddef.h>#include <stdint.h>Go to the source code of this file.
Data Structures | |
| struct | N_PTH |
| A checkpoint's tensors. More... | |
| struct | N_PTH_TENSOR |
| One tensor of a checkpoint. More... | |
Macros | |
| #define | N_PTH_MAX_DIMS 8 |
| Most dimensions a tensor read may have. | |
Functions | |
| const N_PTH_TENSOR * | n_pth_find (const N_PTH *pth, const char *name) |
| find a tensor by its path | |
| void | n_pth_free (N_PTH **pth) |
| release a checkpoint's tensors | |
| N_PTH * | n_pth_load (const char *path) |
| read every tensor of a checkpoint | |
Reading the tensors of a PyTorch checkpoint (.pth, .pt) without Python.
A checkpoint saved by torch.save() since PyTorch 1.6 is a ZIP archive: a pickle, data.pkl, that describes the saved object, and one file a storage holding the raw numbers. This reads the pickle with a small interpreter that builds data only: dictionaries, lists, tuples, numbers, strings, and the three constructors a state dictionary uses (collections.OrderedDict, torch._utils._rebuild_tensor_v2 and _rebuild_parameter). Nothing named in the file is ever called; any other constructor becomes an opaque value, so a checkpoint's extra metadata does not stop its tensors being read.
Every tensor reachable through dictionaries is returned by its path, the keys joined by dots ("state_dict.backbone.0.0.weight"), as floats: float32, float16, bfloat16 and float64 storages are read and converted. Strided views are gathered. Little endian storages only, which is what every platform PyTorch runs on writes.
Every length, index and count in the file is checked against the data actually there: a damaged or hostile file is refused, not trusted.
Definition in file n_pth.h.