OKLMdocumentation v1.0.3
On this page

Converting and inspecting models

How oklm info, oklm convert and oklm quantize work, what they produce, and how their output is checked against llama.cpp. API: include/oklm/io/hf.h, include/oklm/io/model_check.h, include/oklm/model/layout.h (the JSON form of oklm info). The same commands in a window: oklm-gui --tool convert (gui.md).

Status: implemented for Llama, Mistral, Qwen2, Qwen3 and GPT-2 checkpoints; tokenizer metadata is added by the tokenizer module (not yet implemented).

oklm info

oklm info [-t] [--json] model.gguf      # any shard of a split model opens the whole model

Prints the format version and alignment, every metadata key (arrays shortened), tensor count, parameters, size, tensors per type (-t lists every tensor), then a verdict:

  • loadable: every tensor type and the architecture are supported;
  • loadable with warnings: usable, with caveats (no tokenizer, no tensors);
  • unsupported: the reasons are listed, for example a tensor type not implemented yet or an architecture whose inference does not exist yet.

The architecture table is in src/io/model_check.c; it is the single place to update when a family gains inference.

JSON output

oklm info --json model.gguf prints one JSON object on one line, for programs (oklm_model_info_json(), include/oklm/model/layout.h): what the text form says, the general metadata picked out, and the model's layout: each block's tensors grouped by role, with the embeddings and the head. Only the headers are read (the files are mapped without read-ahead). The text form is unchanged without --json. Sizes are bytes of tensor data in the file; params counts elements.

Member Type What
file string the first file's path
shards number files of the model (1 unless split)
file_size number bytes on disk, every shard
format object version (GGUF version), alignment, metadata_entries
arch string or null general.architecture
name string or null general.name
n_params number elements over every tensor
context_length number or null <arch>.context_length, the trained context
vocab_size number or null the tokenizer's token count; without a vocabulary <arch>.vocab_size, else the rows of token_embd.weight
quantization string or null the name of general.file_type (Q4_K_M, Q8_0, F16...)
hparams object every scalar or string key <arch>.* of the file, the prefix removed: block_count, embedding_length, feed_forward_length, attention.head_count, attention.head_count_kv, rope.freq_base, expert_count, expert_used_count... as the file names them
tokenizer object kind (tokenizer.ggml.model: gpt2, llama, bert..., null without a tokenizer), pre (tokenizer.ggml.pre or null), tokens (the vocabulary's size, 0 without one), chat_template (true when the file has one)
totals object tensors, params, bytes, and types: tensors per element type ({"F32":61,"Q8_0":211})
verdict string loadable, warnings or unsupported
verdict_text string the verdict as the text form prints it (loadable with warnings)
arch_support string supported, planned or unknown (the architecture table)
notes array of strings the reasons the text form lists under the verdict
layout object the layout, below
metadata object every key of the first file with its value: numbers, booleans and strings as they are (a number that is not finite is null); an array as {"type": T, "count": N}, with "values" when it has 32 elements at most
tensors array with -t only: every tensor, in the file's order: name, type, shape (innermost first, as GGUF stores it), bytes, block (its block's index, null outside the blocks), role (below)

layout:

Member Type What
n_layer number blocks of the main stack (<arch>.block_count less <arch>.nextn_predict_layers)
n_blocks number blocks the file holds: those of the main stack, then the multi-token prediction ones
tensors, bytes, params number the model's totals (those of totals)
blocks array one part per block, with its index and mtp (true for a multi-token prediction block)
embeddings part outside the blocks: token_embd.*, position_embd.*, token_types.*
head part outside the blocks: output_norm.*, output.*, rope_* (what goes to a device with the last layers; a model with tied embeddings has no output matrix here)
other part outside the blocks: anything else (a reranker's classification head...)

A part is {"tensors": N, "bytes": N, "params": N, "roles": {...}}; roles has a member per role present in the part, each {"tensors": N, "bytes": N, "params": N, "types": {"Q4_K": 3, ...}}. The roles are the kinds oklm plan counts (memory-plan.md), from the tensors' names:

Role Tensors
embeddings token, position and token-type embeddings, the output matrix
attention blk.N.attn_* (and a hybrid stack's ssm_*)
dense_ffn blk.N.ffn_* of a dense feed-forward
shared_experts blk.N.*_shexp*
routed_experts blk.N.*_exps*, the stacked routed experts of a mixture-of-experts block
routers blk.N.ffn_gate_inp*, blk.N.exp_probs_b*
norms any tensor whose name has norm
mtp every tensor of a block past the main stack
other the rest

The parts are disjoint and cover the file: the bytes of blocks, embeddings, head and other add up to layout.bytes and totals.bytes, a part's roles to the part (checked on every fixture by test_layout in tests/model/test_mem_plan.c, on files that are not models by test_info_json in tests/io/test_gguf.c). A file that is not a model has no block; a file whose architecture OKLM does not know is laid out all the same.

A dense model (SmolLM2-135M Q8_0; the 29 other blocks and 32 of the 37 metadata keys cut, lines wrapped):

{
 "file":"models/smollm2-135m/SmolLM2-135M-Instruct-Q8_0.gguf",
 "shards":1,
 "file_size":144811360,
 "format":{"version":3,"alignment":32,"metadata_entries":37},
 "arch":"llama",
 "name":"Smollm2 135M 8k Lc100K Mix1 Ep2",
 "n_params":134515008,
 "context_length":8192,
 "vocab_size":49152,
 "quantization":"Q8_0",
 "hparams":{"block_count":30,"context_length":8192,"embedding_length":576,"feed_forward_length":1536,
  "attention.head_count":9,"attention.head_count_kv":3,"rope.freq_base":100000,
  "attention.layer_norm_rms_epsilon":9.99999975e-06,"vocab_size":49152,"rope.dimension_count":64},
 "tokenizer":{"kind":"gpt2","pre":"smollm","tokens":49152,"chat_template":true},
 "totals":{"tensors":272,"params":134515008,"bytes":143025408,"types":{"F32":61,"Q8_0":211}},
 "verdict":"loadable",
 "verdict_text":"loadable",
 "arch_support":"supported",
 "notes":[],
 "layout":{"n_layer":30,"n_blocks":30,"tensors":272,"bytes":143025408,"params":134515008,"embeddings":{"tensors":1,
  "bytes":30081024,"params":28311552,"roles":{"embeddings":{"tensors":1,"bytes":30081024,"params":28311552,
  "types":{"Q8_0":1}}}},"blocks":[{"index":0,"mtp":false,"tensors":9,"bytes":3764736,"params":3540096,
  "roles":{"attention":{"tensors":4,"bytes":940032,"params":884736,"types":{"Q8_0":4}},"dense_ffn":{"tensors":3,
  "bytes":2820096,"params":2654208,"types":{"Q8_0":3}},"norms":{"tensors":2,"bytes":4608,"params":1152,
  "types":{"F32":2}}}}],"head":{"tensors":1,"bytes":2304,"params":576,"roles":{"norms":{"tensors":1,"bytes":2304,
  "params":576,"types":{"F32":1}}}},"other":{"tensors":0,"bytes":0,"params":0,"roles":{}}},
 "metadata":{"general.architecture":"llama","general.name":"Smollm2 135M 8k Lc100K Mix1 Ep2","general.file_type":7,
  "tokenizer.ggml.tokens":{"type":"string","count":49152},"tokenizer.ggml.bos_token_id":1}
}

A mixture-of-experts model with -t (Granite 3.1 1B A400M, 32 experts a block; blocks, metadata and tensors cut):

{
 "file":"models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf",
 "shards":1,
 "file_size":821847360,
 "format":{"version":3,"alignment":32,"metadata_entries":46},
 "arch":"granitemoe",
 "name":"Granite 3.1 1b A400M Instruct",
 "n_params":1334628352,
 "context_length":131072,
 "vocab_size":49155,
 "quantization":"Q4_K_M",
 "hparams":{"block_count":24,"context_length":131072,"embedding_length":1024,"feed_forward_length":512,
  "attention.head_count":16,"attention.head_count_kv":8,"rope.freq_base":1500000,
  "attention.layer_norm_rms_epsilon":9.99999997e-07,"expert_count":32,"expert_used_count":8,"vocab_size":49155,
  "rope.dimension_count":64,"attention.scale":0.015625,"embedding_scale":12,"residual_scale":0.219999999,
  "logit_scale":6},
 "tokenizer":{"kind":"gpt2","pre":"refact","tokens":49155,"chat_template":true},
 "totals":{"tensors":242,"params":1334628352,"bytes":820107736,"types":{"F32":73,"Q4_K":144,"Q6_K":25}},
 "verdict":"loadable",
 "verdict_text":"loadable",
 "arch_support":"supported",
 "notes":[],
 "layout":{"n_layer":24,"n_blocks":24,"tensors":242,"bytes":820107736,"params":1334628352,"embeddings":{"tensors":1,
  "bytes":41290200,"params":50334720,"roles":{"embeddings":{"tensors":1,"bytes":41290200,"params":50334720,
  "types":{"Q6_K":1}}}},"blocks":[{"index":0,"mtp":false,"tensors":10,"bytes":34680832,"params":53512192,
  "roles":{"attention":{"tensors":4,"bytes":1904640,"params":3145728,"types":{"Q4_K":3,"Q6_K":1}},
  "routed_experts":{"tensors":3,"bytes":32636928,"params":50331648,"types":{"Q4_K":2,"Q6_K":1}},
  "routers":{"tensors":1,"bytes":131072,"params":32768,"types":{"F32":1}},"norms":{"tensors":2,"bytes":8192,
  "params":2048,"types":{"F32":2}}}}],"head":{"tensors":1,"bytes":4096,"params":1024,"roles":{"norms":{"tensors":1,
  "bytes":4096,"params":1024,"types":{"F32":1}}}},"other":{"tensors":0,"bytes":0,"params":0,"roles":{}}},
 "metadata":{"general.architecture":"granitemoe","general.name":"Granite 3.1 1b A400M Instruct",
  "general.file_type":15,"tokenizer.ggml.tokens":{"type":"string","count":49155},"tokenizer.ggml.bos_token_id":0},
 "tensors":[{"name":"output_norm.weight","type":"F32","shape":[1024],"bytes":4096,"block":null,"role":"norms"},
  {"name":"token_embd.weight","type":"Q6_K","shape":[1024,49155],"bytes":41290200,"block":null,"role":"embeddings"}]
}

oklm convert

oklm convert <hf_dir> --out model.gguf [--type f16] [--name NAME] [-v] [--dry-run] [--progress-json FILE|-]

Reads config.json and the safetensors weights (single file or sharded with an index) and writes a GGUF file. Target types: f32, f16, bf16, q8_0, q4_0, q4_1, q5_0, q5_1. Converting is not needed to run a checkpoint stored in F32, F16 or BF16: oklm run and oklm chat load it directly (see inference.md); conversion gives a single file, quantization and faster loading.

Hugging Face architectures GGUF architecture Q/K permutation Notes
LlamaForCausalLM llama yes Llama 3 rope scaling adds rope_freqs.weight
MistralForCausalLM llama yes sliding window kept
Qwen2ForCausalLM qwen2 no QKV biases
Qwen3ForCausalLM qwen3 no Q/K norms
GPT2LMHeadModel, GPT2Model gpt2 no Conv1D weights transposed, output head duplicated from token_embd
GemmaForCausalLM, Gemma2ForCausalLM, Gemma3ForCausalLM gemma, gemma2, gemma3 no norm weights stored plus one (rounded to the checkpoint's type, as llama.cpp's converter); Gemma 2 / 3's pre / post FFN norms; soft-caps, window, local RoPE base and attention scale from config.json
Phi3ForCausalLM phi3 no fused qkv_proj / gate_up_proj; LongRoPE long_factor / short_factor written as rope_factors_long / _short, attention factor sqrt(1 + ln(scale) / ln(original)); partial rotation
GraniteForCausalLM granite yes embedding, attention, residual multipliers and logits_scaling
GraniteMoeForCausalLM granitemoe yes as Granite; input_linear [experts, 2 ff, embd] split into ffn_gate_exps / ffn_up_exps, output_linear kept as ffn_down_exps, router.layer as ffn_gate_inp
Qwen2MoeForCausalLM, Qwen3MoeForCausalLM qwen2moe, qwen3moe no mlp.experts.E.*_proj stacked into ffn_*_exps; mlp.gate as ffn_gate_inp; Qwen2-MoE's shared expert and its [1, embd] gate (stored as a vector); norm_topk_prob kept as expert_weights_norm when it differs from the family's default; dense layers between MoE layers (decoder_sparse_step, mlp_only_layers) refused

What the conversion does, following llama.cpp's convert_hf_to_gguf.py:

  • Names: one table per family (src/io/names_llama.c, src/io/names_gpt2.c), {N} standing for the block number; rotary caches and attention-mask buffers are skipped; a tied lm_head is omitted.
  • Shapes: GGUF ne is the PyTorch shape reversed; the bytes do not move.
  • Q/K permutation (llama, mistral): Hugging Face stores each head's rotary pairs as two halves, GGUF Llama weights as adjacent pairs, so within each head row j * head_dim/2 + i becomes row 2i + j (K uses the KV head count; biases are permuted the same way). This is why GGUF Llama models use the normal RoPE layout and Qwen/Phi/Gemma the NeoX one.
  • Types: 1-D tensors, *_norm.weight and position_embd.weight stay F32; tensors whose rows are not a whole number of blocks fall back to F16.
  • Metadata: general.architecture, general.name, general.type, general.file_type (llama.cpp's llama_ftype numbering), general.quantization_version = 2, and the <arch>.* hyper-parameters (context_length, embedding_length, block_count, feed_forward_length, attention.head_count, attention.head_count_kv, rope.freq_base, attention.layer_norm_rms_epsilon or attention.layer_norm_epsilon, vocab_size, rope.dimension_count, attention.key_length / value_length when the head size is not n_embd / n_head, attention.sliding_window, rope.scaling.*).
  • Memory: each tensor is read, converted and written in turn through the writer's producer callback.

oklm quantize

oklm quantize in.gguf out.gguf Q4_K_M        # or Q8_0, Q5_K_M, Q3_K_L, F16, ...
oklm quantize --pure in.gguf out.gguf Q4_K   # every matrix in one type
oklm quantize --override-kv general.name=str:OKLM-Micro in.gguf out.gguf F16
oklm quantize --dry-run in.gguf out.gguf Q4_K_M           # the size out.gguf would have
oklm quantize --progress-json - in.gguf out.gguf Q4_K_M   # the write as JSON lines

--override-kv KEY=TYPE:VALUE (TYPE str, int, float or bool, repeatable) sets a metadata key in the output, replacing the source's in place, as llama-quantize's option of the same name.

Takes llama-quantize's file types (F32, F16, BF16, Q8_0, Q4_0, Q4_1, Q5_0, Q5_1, Q2_K, Q2_K_S, Q3_K_S, Q3_K_M, Q3_K_L, Q4_K_S, Q4_K_M, Q5_K_S, Q5_K_M, Q6_K, IQ4_NL, IQ4_XS, TQ1_0, TQ2_0, MXFP4_MOE, NVFP4, Q1_0, Q2_0, IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ2_M, IQ3_XXS, IQ3_XS, IQ3_S, IQ3_M) and their per-tensor rules: the output matrix and tied token embeddings in Q6_K, attention V and some FFN down matrices one or two steps up depending on their layer, rows that are not a whole number of blocks in the fallback type (Q4_K -> Q5_0, Q6_K -> Q8_0, ... then F16). Norms, position embeddings, MoE routers and a few small special tensors are copied; split.* keys are dropped (the output is one file) and general.quantization_version / general.file_type set. Any shard of a split model can be the input; quantized inputs are dequantized first.

The output is identical byte for byte to llama-quantize's for every file type, with or without an importance matrix (tools/check_quantize.sh, adr/0013-k-quant-write-path.md), except the matrices in the grid IQ types (IQ1_, IQ2_, IQ3_*): the per-tensor types are llama-quantize's, the blocks come from OKLM's own codebook search (src/core/quant_iq_grid.c; ggml's quantizers are not copied). Measured with llama.cpp's llama-perplexity (Qwen3-0.6B Q8_0 requantized, one llama-imatrix file for both tools, 16 chunks of 512 tokens of tinyshakespeare):

file type llama-quantize oklm quantize
IQ1_M 2001 1761
IQ2_XXS 511 399
IQ2_M 156 144
IQ3_XXS 103 98
IQ3_S, no importance matrix 449 111

The quantizers work without an importance matrix (llama-quantize refuses the 1- and 2-bit types then; oklm quantize warns), though the 1-bit types are unusable that way.

Importance matrices

oklm imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix.gguf   # [-c 512] [--chunks N]
oklm quantize --imatrix model.imatrix.gguf model-f16.gguf model-Q4_K_M.gguf Q4_K_M

oklm imatrix runs the model on the CPU over chunks of -c tokens (each from an empty cache, BOS first) and sums, for every blk.* matrix, the squares of each input column with the number of rows, as llama-imatrix does; the file is in llama-imatrix's GGUF format (<name>.in_sum2, <name>.counts, <name>.in_sum, imatrix.datasets, imatrix.chunk_count, imatrix.chunk_size). On SmolLM2-135M and 60 kB of text, its 210 entries are those of llama-imatrix, with the same counts and means within 0.6 %; Q2_K perplexity 34.85 with either (39.90 without, 28.74 in F16).

oklm quantize --imatrix reads that format and llama-imatrix's legacy binary one, divides the sums by their counts (per expert), weights the quantizers of Q4_0, Q4_1, Q5_0, Q5_1, Q2_K .. Q6_K, IQ4_NL, IQ4_XS and the grid IQ types with them (oklm_quantize_row_weighted()), applies the rules that depend on an importance matrix, and records quantize.imatrix.file, .dataset, .entries_count and .chunks_count.

The size before writing

oklm convert --dry-run and oklm quantize --dry-run do everything but write: the source's headers are read, the metadata and the tokenizer are prepared, each tensor gets the type the command would give it (the per-tensor rules, the fallbacks, the importance matrix's rules), and the size of the file is added up from the header and the tensors' sizes. No tensor is read or converted and the output is not touched. The figure is the size the command writes, to the byte:

$ oklm quantize --dry-run SmolLM2-135M-Instruct-Q8_0.gguf out.gguf Q4_K_M 2>/dev/null
out.gguf: 272 tensors, 105454432 bytes (100.57 MiB), not written (--dry-run)

which is the size of the Q4_K_M file llama-quantize published (tests/harness/test_cli_progress.c compares the dry run with the file written, for both commands; test_write_opts in tests/io/test_gguf.c does it for the library). The warnings of the real run (a row size a type does not take, a type meant for an importance matrix) are printed too. In the library: oklm_gguf_write_opts (dry_run, and size and tensors received), the write member of oklm_quantize_params and oklm_convert_params; oklm_gguf_writer_file_size() for a writer.

Progress as JSON lines

--progress-json FILE (- for standard output) on both commands writes the run as JSON lines a program can follow, in the format of the other long commands (training.md): start once the file's header is ready (the tensor count and the size the file will have), a tensor line after each tensor written (its index, its name, the bytes written so far), and end with ok, the size and the file. A dry run writes end alone, with "dry_run": true and the size. In the library: the progress callback of oklm_gguf_write_opts.

$ oklm quantize --progress-json - tiny.gguf tiny-q8_0.gguf Q8_0 2>/dev/null
{"type":"start","elapsed":0.000,"command":"quantize","in":"tiny.gguf","out":"tiny-q8_0.gguf","ftype":"Q8_0","tensors":20,"bytes":37248}
{"type":"tensor","elapsed":0.000,"index":1,"total":20,"name":"output_norm.weight","bytes_done":6944,"bytes_total":37248}
...
{"type":"end","elapsed":0.001,"ok":true,"dry_run":false,"tensors":20,"bytes":37248,"files":["tiny-q8_0.gguf"]}

oklm split, oklm merge

oklm split model.gguf out/model --split-max-size 2G     # or --split-max-tensors 128
oklm merge out/model-00001-of-00003.gguf model.gguf

Shards and merged files are identical byte for byte to llama.cpp's gguf-split (formats/gguf.md); every oklm command opens a model from any of its shards.

Verification against llama.cpp

make test-public downloads SmolLM2-135M-Instruct (the Hugging Face checkpoint and bartowski's GGUFs made with llama.cpp) and checks:

Check Result
oklm convert of the BF16 checkpoint to F16 vs the published F16 GGUF 272/272 tensors byte-identical, same order
oklm convert --type q8_0 of the checkpoint vs the published Q8_0 GGUF 272/272 tensors byte-identical
oklm quantize of the published F16 GGUF to Q8_0 vs the published Q8_0 GGUF 272/272 tensors byte-identical
every tensor of the Q4_K_M, Q8_0 and F16 files dequantized vs gguf-py bit-identical (Q4_K, Q5_0, Q6_K, Q8_0, F16, F32)
copying every key and tensor of each published file byte-identical file

The same comparison is available as a script: build/venv/bin/python tools/hf_to_gguf_check.py ours.gguf reference.gguf.