Converting and inspecting models
How oklm info, oklm convert and
oklm quantize work, what they produce, and how their output
is checked against llama.cpp. API: include/oklm/io/hf.h,
include/oklm/io/model_check.h,
include/oklm/model/layout.h (the JSON form of
oklm info). The same commands in a window:
oklm-gui --tool convert (gui.md).
Status: implemented for Llama, Mistral, Qwen2, Qwen3 and GPT-2 checkpoints; tokenizer metadata is added by the tokenizer module (not yet implemented).
oklm info
oklm info [-t] [--json] model.gguf # any shard of a split model opens the whole modelPrints the format version and alignment, every metadata key (arrays
shortened), tensor count, parameters, size, tensors per type
(-t lists every tensor), then a verdict:
- loadable: every tensor type and the architecture are supported;
- loadable with warnings: usable, with caveats (no tokenizer, no tensors);
- unsupported: the reasons are listed, for example a tensor type not implemented yet or an architecture whose inference does not exist yet.
The architecture table is in src/io/model_check.c; it is
the single place to update when a family gains inference.
JSON output
oklm info --json model.gguf prints one JSON object on
one line, for programs (oklm_model_info_json(),
include/oklm/model/layout.h): what the text form says, the
general metadata picked out, and the model's layout:
each block's tensors grouped by role, with the embeddings and the head.
Only the headers are read (the files are mapped without read-ahead). The
text form is unchanged without --json. Sizes are bytes of
tensor data in the file; params counts elements.
| Member | Type | What |
|---|---|---|
file |
string | the first file's path |
shards |
number | files of the model (1 unless split) |
file_size |
number | bytes on disk, every shard |
format |
object | version (GGUF version), alignment,
metadata_entries |
arch |
string or null | general.architecture |
name |
string or null | general.name |
n_params |
number | elements over every tensor |
context_length |
number or null | <arch>.context_length, the trained context |
vocab_size |
number or null | the tokenizer's token count; without a vocabulary
<arch>.vocab_size, else the rows of
token_embd.weight |
quantization |
string or null | the name of general.file_type (Q4_K_M,
Q8_0, F16...) |
hparams |
object | every scalar or string key <arch>.* of the file,
the prefix removed: block_count,
embedding_length, feed_forward_length,
attention.head_count, attention.head_count_kv,
rope.freq_base, expert_count,
expert_used_count... as the file names them |
tokenizer |
object | kind (tokenizer.ggml.model:
gpt2, llama, bert..., null
without a tokenizer), pre (tokenizer.ggml.pre
or null), tokens (the vocabulary's size, 0 without one),
chat_template (true when the file has one) |
totals |
object | tensors, params, bytes, and
types: tensors per element type
({"F32":61,"Q8_0":211}) |
verdict |
string | loadable, warnings or
unsupported |
verdict_text |
string | the verdict as the text form prints it
(loadable with warnings) |
arch_support |
string | supported, planned or unknown
(the architecture table) |
notes |
array of strings | the reasons the text form lists under the verdict |
layout |
object | the layout, below |
metadata |
object | every key of the first file with its value: numbers, booleans and
strings as they are (a number that is not finite is null); an array as
{"type": T, "count": N}, with "values" when it
has 32 elements at most |
tensors |
array | with -t only: every tensor, in the file's order:
name, type, shape (innermost
first, as GGUF stores it), bytes, block (its
block's index, null outside the blocks), role (below) |
layout:
| Member | Type | What |
|---|---|---|
n_layer |
number | blocks of the main stack (<arch>.block_count less
<arch>.nextn_predict_layers) |
n_blocks |
number | blocks the file holds: those of the main stack, then the multi-token prediction ones |
tensors, bytes, params |
number | the model's totals (those of totals) |
blocks |
array | one part per block, with its index and mtp
(true for a multi-token prediction block) |
embeddings |
part | outside the blocks: token_embd.*,
position_embd.*, token_types.* |
head |
part | outside the blocks: output_norm.*,
output.*, rope_* (what goes to a device with
the last layers; a model with tied embeddings has no output matrix
here) |
other |
part | outside the blocks: anything else (a reranker's classification head...) |
A part is
{"tensors": N, "bytes": N, "params": N, "roles": {...}};
roles has a member per role present in the part, each
{"tensors": N, "bytes": N, "params": N, "types": {"Q4_K": 3, ...}}.
The roles are the kinds oklm plan counts (memory-plan.md), from the tensors'
names:
| Role | Tensors |
|---|---|
embeddings |
token, position and token-type embeddings, the output matrix |
attention |
blk.N.attn_* (and a hybrid stack's
ssm_*) |
dense_ffn |
blk.N.ffn_* of a dense feed-forward |
shared_experts |
blk.N.*_shexp* |
routed_experts |
blk.N.*_exps*, the stacked routed experts of a
mixture-of-experts block |
routers |
blk.N.ffn_gate_inp*,
blk.N.exp_probs_b* |
norms |
any tensor whose name has norm |
mtp |
every tensor of a block past the main stack |
other |
the rest |
The parts are disjoint and cover the file: the bytes of
blocks, embeddings, head and
other add up to layout.bytes and
totals.bytes, a part's roles to the part (checked on every
fixture by test_layout in
tests/model/test_mem_plan.c, on files that are not models
by test_info_json in tests/io/test_gguf.c). A
file that is not a model has no block; a file whose architecture OKLM
does not know is laid out all the same.
A dense model (SmolLM2-135M Q8_0; the 29 other blocks and 32 of the 37 metadata keys cut, lines wrapped):
{
"file":"models/smollm2-135m/SmolLM2-135M-Instruct-Q8_0.gguf",
"shards":1,
"file_size":144811360,
"format":{"version":3,"alignment":32,"metadata_entries":37},
"arch":"llama",
"name":"Smollm2 135M 8k Lc100K Mix1 Ep2",
"n_params":134515008,
"context_length":8192,
"vocab_size":49152,
"quantization":"Q8_0",
"hparams":{"block_count":30,"context_length":8192,"embedding_length":576,"feed_forward_length":1536,
"attention.head_count":9,"attention.head_count_kv":3,"rope.freq_base":100000,
"attention.layer_norm_rms_epsilon":9.99999975e-06,"vocab_size":49152,"rope.dimension_count":64},
"tokenizer":{"kind":"gpt2","pre":"smollm","tokens":49152,"chat_template":true},
"totals":{"tensors":272,"params":134515008,"bytes":143025408,"types":{"F32":61,"Q8_0":211}},
"verdict":"loadable",
"verdict_text":"loadable",
"arch_support":"supported",
"notes":[],
"layout":{"n_layer":30,"n_blocks":30,"tensors":272,"bytes":143025408,"params":134515008,"embeddings":{"tensors":1,
"bytes":30081024,"params":28311552,"roles":{"embeddings":{"tensors":1,"bytes":30081024,"params":28311552,
"types":{"Q8_0":1}}}},"blocks":[{"index":0,"mtp":false,"tensors":9,"bytes":3764736,"params":3540096,
"roles":{"attention":{"tensors":4,"bytes":940032,"params":884736,"types":{"Q8_0":4}},"dense_ffn":{"tensors":3,
"bytes":2820096,"params":2654208,"types":{"Q8_0":3}},"norms":{"tensors":2,"bytes":4608,"params":1152,
"types":{"F32":2}}}}],"head":{"tensors":1,"bytes":2304,"params":576,"roles":{"norms":{"tensors":1,"bytes":2304,
"params":576,"types":{"F32":1}}}},"other":{"tensors":0,"bytes":0,"params":0,"roles":{}}},
"metadata":{"general.architecture":"llama","general.name":"Smollm2 135M 8k Lc100K Mix1 Ep2","general.file_type":7,
"tokenizer.ggml.tokens":{"type":"string","count":49152},"tokenizer.ggml.bos_token_id":1}
}A mixture-of-experts model with -t (Granite 3.1 1B
A400M, 32 experts a block; blocks, metadata and tensors cut):
{
"file":"models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf",
"shards":1,
"file_size":821847360,
"format":{"version":3,"alignment":32,"metadata_entries":46},
"arch":"granitemoe",
"name":"Granite 3.1 1b A400M Instruct",
"n_params":1334628352,
"context_length":131072,
"vocab_size":49155,
"quantization":"Q4_K_M",
"hparams":{"block_count":24,"context_length":131072,"embedding_length":1024,"feed_forward_length":512,
"attention.head_count":16,"attention.head_count_kv":8,"rope.freq_base":1500000,
"attention.layer_norm_rms_epsilon":9.99999997e-07,"expert_count":32,"expert_used_count":8,"vocab_size":49155,
"rope.dimension_count":64,"attention.scale":0.015625,"embedding_scale":12,"residual_scale":0.219999999,
"logit_scale":6},
"tokenizer":{"kind":"gpt2","pre":"refact","tokens":49155,"chat_template":true},
"totals":{"tensors":242,"params":1334628352,"bytes":820107736,"types":{"F32":73,"Q4_K":144,"Q6_K":25}},
"verdict":"loadable",
"verdict_text":"loadable",
"arch_support":"supported",
"notes":[],
"layout":{"n_layer":24,"n_blocks":24,"tensors":242,"bytes":820107736,"params":1334628352,"embeddings":{"tensors":1,
"bytes":41290200,"params":50334720,"roles":{"embeddings":{"tensors":1,"bytes":41290200,"params":50334720,
"types":{"Q6_K":1}}}},"blocks":[{"index":0,"mtp":false,"tensors":10,"bytes":34680832,"params":53512192,
"roles":{"attention":{"tensors":4,"bytes":1904640,"params":3145728,"types":{"Q4_K":3,"Q6_K":1}},
"routed_experts":{"tensors":3,"bytes":32636928,"params":50331648,"types":{"Q4_K":2,"Q6_K":1}},
"routers":{"tensors":1,"bytes":131072,"params":32768,"types":{"F32":1}},"norms":{"tensors":2,"bytes":8192,
"params":2048,"types":{"F32":2}}}}],"head":{"tensors":1,"bytes":4096,"params":1024,"roles":{"norms":{"tensors":1,
"bytes":4096,"params":1024,"types":{"F32":1}}}},"other":{"tensors":0,"bytes":0,"params":0,"roles":{}}},
"metadata":{"general.architecture":"granitemoe","general.name":"Granite 3.1 1b A400M Instruct",
"general.file_type":15,"tokenizer.ggml.tokens":{"type":"string","count":49155},"tokenizer.ggml.bos_token_id":0},
"tensors":[{"name":"output_norm.weight","type":"F32","shape":[1024],"bytes":4096,"block":null,"role":"norms"},
{"name":"token_embd.weight","type":"Q6_K","shape":[1024,49155],"bytes":41290200,"block":null,"role":"embeddings"}]
}oklm convert
oklm convert <hf_dir> --out model.gguf [--type f16] [--name NAME] [-v] [--dry-run] [--progress-json FILE|-]Reads config.json and the safetensors weights (single
file or sharded with an index) and writes a GGUF file. Target types:
f32, f16, bf16,
q8_0, q4_0, q4_1,
q5_0, q5_1. Converting is not needed to run a
checkpoint stored in F32, F16 or BF16: oklm run and
oklm chat load it directly (see inference.md); conversion gives a single file,
quantization and faster loading.
Hugging Face architectures |
GGUF architecture | Q/K permutation | Notes |
|---|---|---|---|
| LlamaForCausalLM | llama | yes | Llama 3 rope scaling adds rope_freqs.weight |
| MistralForCausalLM | llama | yes | sliding window kept |
| Qwen2ForCausalLM | qwen2 | no | QKV biases |
| Qwen3ForCausalLM | qwen3 | no | Q/K norms |
| GPT2LMHeadModel, GPT2Model | gpt2 | no | Conv1D weights transposed, output head duplicated from
token_embd |
| GemmaForCausalLM, Gemma2ForCausalLM, Gemma3ForCausalLM | gemma, gemma2, gemma3 | no | norm weights stored plus one (rounded to the checkpoint's type, as
llama.cpp's converter); Gemma 2 / 3's pre / post FFN norms; soft-caps,
window, local RoPE base and attention scale from
config.json |
| Phi3ForCausalLM | phi3 | no | fused qkv_proj / gate_up_proj; LongRoPE
long_factor / short_factor written as
rope_factors_long / _short, attention factor
sqrt(1 + ln(scale) / ln(original)); partial rotation |
| GraniteForCausalLM | granite | yes | embedding, attention, residual multipliers and
logits_scaling |
| GraniteMoeForCausalLM | granitemoe | yes | as Granite; input_linear [experts, 2 ff, embd] split
into ffn_gate_exps / ffn_up_exps,
output_linear kept as ffn_down_exps,
router.layer as ffn_gate_inp |
| Qwen2MoeForCausalLM, Qwen3MoeForCausalLM | qwen2moe, qwen3moe | no | mlp.experts.E.*_proj stacked into
ffn_*_exps; mlp.gate as
ffn_gate_inp; Qwen2-MoE's shared expert and its [1, embd]
gate (stored as a vector); norm_topk_prob kept as
expert_weights_norm when it differs from the family's
default; dense layers between MoE layers
(decoder_sparse_step, mlp_only_layers)
refused |
What the conversion does, following llama.cpp's
convert_hf_to_gguf.py:
- Names: one table per family
(
src/io/names_llama.c,src/io/names_gpt2.c),{N}standing for the block number; rotary caches and attention-mask buffers are skipped; a tiedlm_headis omitted. - Shapes: GGUF
neis the PyTorch shape reversed; the bytes do not move. - Q/K permutation (llama, mistral): Hugging Face
stores each head's rotary pairs as two halves, GGUF Llama weights as
adjacent pairs, so within each head row
j * head_dim/2 + ibecomes row2i + j(K uses the KV head count; biases are permuted the same way). This is why GGUF Llama models use the normal RoPE layout and Qwen/Phi/Gemma the NeoX one. - Types: 1-D tensors,
*_norm.weightandposition_embd.weightstay F32; tensors whose rows are not a whole number of blocks fall back to F16. - Metadata:
general.architecture,general.name,general.type,general.file_type(llama.cpp'sllama_ftypenumbering),general.quantization_version= 2, and the<arch>.*hyper-parameters (context_length,embedding_length,block_count,feed_forward_length,attention.head_count,attention.head_count_kv,rope.freq_base,attention.layer_norm_rms_epsilonorattention.layer_norm_epsilon,vocab_size,rope.dimension_count,attention.key_length/value_lengthwhen the head size is notn_embd / n_head,attention.sliding_window,rope.scaling.*). - Memory: each tensor is read, converted and written in turn through the writer's producer callback.
oklm quantize
oklm quantize in.gguf out.gguf Q4_K_M # or Q8_0, Q5_K_M, Q3_K_L, F16, ...
oklm quantize --pure in.gguf out.gguf Q4_K # every matrix in one type
oklm quantize --override-kv general.name=str:OKLM-Micro in.gguf out.gguf F16
oklm quantize --dry-run in.gguf out.gguf Q4_K_M # the size out.gguf would have
oklm quantize --progress-json - in.gguf out.gguf Q4_K_M # the write as JSON lines--override-kv KEY=TYPE:VALUE (TYPE str,
int, float or bool, repeatable)
sets a metadata key in the output, replacing the source's in place, as
llama-quantize's option of the same name.
Takes llama-quantize's file types (F32, F16, BF16, Q8_0, Q4_0, Q4_1,
Q5_0, Q5_1, Q2_K, Q2_K_S, Q3_K_S, Q3_K_M, Q3_K_L, Q4_K_S, Q4_K_M,
Q5_K_S, Q5_K_M, Q6_K, IQ4_NL, IQ4_XS, TQ1_0, TQ2_0, MXFP4_MOE, NVFP4,
Q1_0, Q2_0, IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ2_M, IQ3_XXS,
IQ3_XS, IQ3_S, IQ3_M) and their per-tensor rules: the output matrix and
tied token embeddings in Q6_K, attention V and some FFN down matrices
one or two steps up depending on their layer, rows that are not a whole
number of blocks in the fallback type (Q4_K -> Q5_0, Q6_K -> Q8_0,
... then F16). Norms, position embeddings, MoE routers and a few small
special tensors are copied; split.* keys are dropped (the
output is one file) and general.quantization_version /
general.file_type set. Any shard of a split model can be
the input; quantized inputs are dequantized first.
The output is identical byte for byte to llama-quantize's for every
file type, with or without an importance matrix
(tools/check_quantize.sh, adr/0013-k-quant-write-path.md),
except the matrices in the grid IQ types (IQ1_, IQ2_, IQ3_*):
the per-tensor types are llama-quantize's, the blocks come from OKLM's
own codebook search (src/core/quant_iq_grid.c; ggml's
quantizers are not copied). Measured with llama.cpp's
llama-perplexity (Qwen3-0.6B Q8_0 requantized, one
llama-imatrix file for both tools, 16 chunks of 512 tokens of
tinyshakespeare):
| file type | llama-quantize | oklm quantize |
|---|---|---|
| IQ1_M | 2001 | 1761 |
| IQ2_XXS | 511 | 399 |
| IQ2_M | 156 | 144 |
| IQ3_XXS | 103 | 98 |
| IQ3_S, no importance matrix | 449 | 111 |
The quantizers work without an importance matrix (llama-quantize
refuses the 1- and 2-bit types then; oklm quantize warns),
though the 1-bit types are unusable that way.
Importance matrices
oklm imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix.gguf # [-c 512] [--chunks N]
oklm quantize --imatrix model.imatrix.gguf model-f16.gguf model-Q4_K_M.gguf Q4_K_Moklm imatrix runs the model on the CPU over chunks of
-c tokens (each from an empty cache, BOS first) and sums,
for every blk.* matrix, the squares of each input column
with the number of rows, as llama-imatrix does; the file is in
llama-imatrix's GGUF format (<name>.in_sum2,
<name>.counts, <name>.in_sum,
imatrix.datasets, imatrix.chunk_count,
imatrix.chunk_size). On SmolLM2-135M and 60 kB of text, its
210 entries are those of llama-imatrix, with the same counts and means
within 0.6 %; Q2_K perplexity 34.85 with either (39.90 without, 28.74 in
F16).
oklm quantize --imatrix reads that format and
llama-imatrix's legacy binary one, divides the sums by their counts (per
expert), weights the quantizers of Q4_0, Q4_1, Q5_0, Q5_1, Q2_K .. Q6_K,
IQ4_NL, IQ4_XS and the grid IQ types with them
(oklm_quantize_row_weighted()), applies the rules that
depend on an importance matrix, and records
quantize.imatrix.file, .dataset,
.entries_count and .chunks_count.
The size before writing
oklm convert --dry-run and
oklm quantize --dry-run do everything but write: the
source's headers are read, the metadata and the tokenizer are prepared,
each tensor gets the type the command would give it (the per-tensor
rules, the fallbacks, the importance matrix's rules), and the size of
the file is added up from the header and the tensors' sizes. No tensor
is read or converted and the output is not touched. The figure is the
size the command writes, to the byte:
$ oklm quantize --dry-run SmolLM2-135M-Instruct-Q8_0.gguf out.gguf Q4_K_M 2>/dev/null
out.gguf: 272 tensors, 105454432 bytes (100.57 MiB), not written (--dry-run)
which is the size of the Q4_K_M file llama-quantize published
(tests/harness/test_cli_progress.c compares the dry run
with the file written, for both commands; test_write_opts
in tests/io/test_gguf.c does it for the library). The
warnings of the real run (a row size a type does not take, a type meant
for an importance matrix) are printed too. In the library:
oklm_gguf_write_opts (dry_run, and
size and tensors received), the
write member of oklm_quantize_params and
oklm_convert_params;
oklm_gguf_writer_file_size() for a writer.
Progress as JSON lines
--progress-json FILE (- for standard
output) on both commands writes the run as JSON lines a program can
follow, in the format of the other long commands (training.md):
start once the file's header is ready (the tensor count and
the size the file will have), a tensor line after each
tensor written (its index, its name, the bytes written so far), and
end with ok, the size and the file. A dry run
writes end alone, with "dry_run": true and the
size. In the library: the progress callback of
oklm_gguf_write_opts.
$ oklm quantize --progress-json - tiny.gguf tiny-q8_0.gguf Q8_0 2>/dev/null
{"type":"start","elapsed":0.000,"command":"quantize","in":"tiny.gguf","out":"tiny-q8_0.gguf","ftype":"Q8_0","tensors":20,"bytes":37248}
{"type":"tensor","elapsed":0.000,"index":1,"total":20,"name":"output_norm.weight","bytes_done":6944,"bytes_total":37248}
...
{"type":"end","elapsed":0.001,"ok":true,"dry_run":false,"tensors":20,"bytes":37248,"files":["tiny-q8_0.gguf"]}
oklm split, oklm merge
oklm split model.gguf out/model --split-max-size 2G # or --split-max-tensors 128
oklm merge out/model-00001-of-00003.gguf model.ggufShards and merged files are identical byte for byte to llama.cpp's
gguf-split (formats/gguf.md); every
oklm command opens a model from any of its shards.
Verification against llama.cpp
make test-public downloads SmolLM2-135M-Instruct (the
Hugging Face checkpoint and bartowski's GGUFs made with llama.cpp) and
checks:
| Check | Result |
|---|---|
oklm convert of the BF16 checkpoint to F16 vs the
published F16 GGUF |
272/272 tensors byte-identical, same order |
oklm convert --type q8_0 of the checkpoint vs the
published Q8_0 GGUF |
272/272 tensors byte-identical |
oklm quantize of the published F16 GGUF to Q8_0 vs the
published Q8_0 GGUF |
272/272 tensors byte-identical |
| every tensor of the Q4_K_M, Q8_0 and F16 files dequantized vs gguf-py | bit-identical (Q4_K, Q5_0, Q6_K, Q8_0, F16, F32) |
| copying every key and tensor of each published file | byte-identical file |
The same comparison is available as a script:
build/venv/bin/python tools/hf_to_gguf_check.py ours.gguf reference.gguf.