ADR-0013: K-quant write path
Records how OKLM writes K-quants and picks per-tensor types, so its quantized models are interchangeable with llama.cpp's.
Status: accepted (2026-10-30).
Context
Published GGUF models are mostly Q4_K_M, Q5_K_M, Q6_K and Q8_0 files
made by llama-quantize. A file type such as Q4_K_M is not one tensor
type: the output matrix, attention V and some FFN down matrices get more
bits depending on their layer, the GQA ratio and the architecture. Users
expect oklm quantize to produce the same files as
llama-quantize, and the test suite can only check that byte for byte if
the algorithms are the same.
Decision
The K-quant quantizers (Q2_K .. Q6_K) re-derive ggml's reference algorithms: candidate-scale searches, least-squares fits of scale and min, requantization against the rounded sub-block scales, with the same float operations in the same order. No ggml code is included (NOTICE).
oklm quantizetakes llama-quantize's file type names (Q4_K_M, Q3_K_L, Q8_0, ...) and applies its per-tensor rules: categories from tensor names, layer counters, the "more bits" layers, the 70B and 8-expert special cases, falcon's variants, and the fallback types for rows that are not a whole number of blocks. Tensors are visited in llama-quantize's order (by layer, then by name), which the layer counters depend on, and written in that order;general.quantization_versionandgeneral.file_typeare written last.--puredisables the rules.The result is checked against llama-quantize byte for byte by
tools/check_quantize.sh(16 file types on an F16 model, 4 on each of four other architectures by requantization), and per type by hashes of ggml's output intests/core/test_quant_k.c.IQ types and Q2_K_S need an importance matrix to be useful; llama-quantize refuses Q2_K_S without one,
oklm quantizewarns.Importance matrices: both llama-imatrix formats are read and normalized as llama-quantize does; the weighted quantizer variants (
quantize_row_*_impl, the weighted IQ4 path) are re-derived the same way, sooklm quantize --imatrixwrites llama-quantize's files byte for byte.oklm imatrixcomputes them with llama-imatrix's procedure and file format; its values cannot be byte-identical (they depend on the float evaluation of the model) and are checked within a tolerance and by the perplexity of the models they produce.
Consequences
Any model OKLM quantizes without an importance matrix is the file
llama-quantize would have written. The rules are a copy of a moving
target: a llama.cpp change to them shows up as a
check_quantize.sh difference and is ported then.