OKLMdocumentation v1.0.3
On this page

ADR-0013: K-quant write path

Records how OKLM writes K-quants and picks per-tensor types, so its quantized models are interchangeable with llama.cpp's.

Status: accepted (2026-10-30).

Context

Published GGUF models are mostly Q4_K_M, Q5_K_M, Q6_K and Q8_0 files made by llama-quantize. A file type such as Q4_K_M is not one tensor type: the output matrix, attention V and some FFN down matrices get more bits depending on their layer, the GQA ratio and the architecture. Users expect oklm quantize to produce the same files as llama-quantize, and the test suite can only check that byte for byte if the algorithms are the same.

Decision

  • The K-quant quantizers (Q2_K .. Q6_K) re-derive ggml's reference algorithms: candidate-scale searches, least-squares fits of scale and min, requantization against the rounded sub-block scales, with the same float operations in the same order. No ggml code is included (NOTICE).

  • oklm quantize takes llama-quantize's file type names (Q4_K_M, Q3_K_L, Q8_0, ...) and applies its per-tensor rules: categories from tensor names, layer counters, the "more bits" layers, the 70B and 8-expert special cases, falcon's variants, and the fallback types for rows that are not a whole number of blocks. Tensors are visited in llama-quantize's order (by layer, then by name), which the layer counters depend on, and written in that order; general.quantization_version and general.file_type are written last. --pure disables the rules.

  • The result is checked against llama-quantize byte for byte by tools/check_quantize.sh (16 file types on an F16 model, 4 on each of four other architectures by requantization), and per type by hashes of ggml's output in tests/core/test_quant_k.c.

  • IQ types and Q2_K_S need an importance matrix to be useful; llama-quantize refuses Q2_K_S without one, oklm quantize warns.

  • Importance matrices: both llama-imatrix formats are read and normalized as llama-quantize does; the weighted quantizer variants (quantize_row_*_impl, the weighted IQ4 path) are re-derived the same way, so oklm quantize --imatrix writes llama-quantize's files byte for byte. oklm imatrix computes them with llama-imatrix's procedure and file format; its values cannot be byte-identical (they depend on the float evaluation of the model) and are checked within a tolerance and by the perplexity of the models they produce.

Consequences

Any model OKLM quantizes without an importance matrix is the file llama-quantize would have written. The rules are a copy of a moving target: a llama.cpp change to them shows up as a check_quantize.sh difference and is ported then.