OKLMdocumentation v1.0.3
On this page

Block quantization formats

Byte-level layout and decoding rule of every block-quantized type OKLM reads or writes. These are ggml's formats (ggml-common.h for the layouts, ggml-quants.c for the reference algorithms), re-derived; a block written by OKLM is byte-identical to the block ggml's reference quantizer writes for the same input.

Status: legacy quants, K-quants, IQ4_NL / IQ4_XS, MXFP4 / NVFP4, TQ1_0 / TQ2_0 and Q1_0 / Q2_0 implemented (read, compute, write), and PQ2_0 / PTQ1_0, which are a fork's and not ggml's; the grid IQ types (IQ1 to IQ3) read and computed, not written.

Upstream revision: the layouts below have been stable in ggml since the K-quants landed (2023). No ggml checkout was pinned when this code was written; the fixture-based checks against public GGUF files pin it.

All multi-byte fields are little-endian. f16 is IEEE-754 binary16. QK = 32, QK_K = 256.

Legacy quants (32-element blocks)

Type id Bytes Fields (in order) Value of element j
Q4_0 2 18 f16 d; u8 qs[16] d · (q_j − 8)
Q4_1 3 20 f16 d; f16 m; u8 qs[16] d · q_j + m
Q5_0 6 22 f16 d; u8 qh[4]; u8 qs[16] d · (q_j − 16)
Q5_1 7 24 f16 d; f16 m; u8 qh[4]; u8 qs[16] d · q_j + m
Q8_0 8 34 f16 d; i8 qs[32] d · qs[j]
Q8_1 9 36 f16 d; f16 s; i8 qs[32] d · qs[j], s = d · Σ qs

Nibble order (Q4_, and the low 4 bits of Q5_): element j < 16 is the low nibble of qs[j], element j ≥ 16 the high nibble of qs[j − 16]. The fifth bit of Q5_* element j is bit j of qh read as a little-endian uint32.

Reference quantizers

Type Scale Code
Q4_0 max = the value of largest magnitude (sign kept); d = max / −8 min(15, (int8)(x/d + 8.5))
Q4_1 d = (max − min) / 15, m = min min(15, (int8)((x − min)/d + 0.5))
Q5_0 d = max / −16 min(31, (int8)(x/d + 16.5))
Q5_1 d = (max − min) / 31, m = min (uint8)((x − min)/d + 0.5)
Q8_0 d = amax / 127 roundf(x/d)
Q8_1 as Q8_0 as Q8_0; s = f16(Σ q · d) with the unrounded d

1/d is used (0 when d = 0), and d is stored rounded to f16 after it was used unrounded for the codes. Negative scales are intentional: with d = max / −8, the extreme value maps exactly to code 0 and the other sign gets codes up to 15 (clipped).

Worked example

Q4_0 of x_j = j − 16, j = 0..31: the largest magnitude is x_0 = −16, so d = −16 / −8 = 2 (f16 0x4000) and 1/d = 0.5. x_0 → (int8)(−8 + 8.5) = 0, x_16 → (int8)(0 + 8.5) = 8, x_31 → (int8)(7.5 + 8.5) = 16 → 15. Byte qs[0] holds x_0 (low) and x_16 (high): 0x80. Decoding gives 2 · (0 − 8) = −16 and 2 · (8 − 8) = 0; x_31 decodes to 2 · 7 = 14 (clipping error 1 = one step). tests/core/test_quant.c checks every byte of this block.

K-quants (256-element super-blocks)

Type id Bytes Fields (in order)
Q2_K 10 84 u8 scales[16]; u8 qs[64]; f16 d; f16 dmin
Q3_K 11 110 u8 hmask[32]; u8 qs[64]; u8 scales[12]; f16 d
Q4_K 12 144 f16 d; f16 dmin; u8 scales[12]; u8 qs[128]
Q5_K 13 176 f16 d; f16 dmin; u8 scales[12]; u8 qh[32]; u8 qs[128]
Q6_K 14 210 u8 ql[128]; u8 qh[64]; i8 scales[16]; f16 d
Q8_K 15 292 f32 d; i8 qs[256]; i16 bsums[16]

Q2_K

16 groups of 16. Element i: half n = i / 128, shift index j = (i mod 128) / 32, lane l = i mod 32. Code q = (qs[32n + l] >> 2j) & 3; group byte sc = scales[8n + 2j + l/16]. Value: (d · (sc & 15)) · q − dmin · (sc >> 4).

Q3_K

16 groups of 16, same n, j, l as Q2_K. Low bits (qs[32n + l] >> 2j) & 3, high bit (hmask[l] >> (4n + j)) & 1; the code is low − 4 when the high bit is 0, low when it is 1. Group k = 8n + 2j + l/16 has a 6-bit scale: low 4 bits from scales[k] (low nibble, k < 8) or scales[k − 8] (high nibble, k ≥ 8), high 2 bits from scales[8 + k mod 4] >> 2(k/4). Value: (d · (scale − 32)) · code.

Q4_K and Q5_K

8 sub-blocks of 32. Element i: chunk j = i / 64, upper half u = (i mod 64) ≥ 32, lane l = i mod 32. Low 4 bits: low nibble of qs[32j + l] (u = 0) or high nibble (u = 1). Q5_K adds 16 when bit 2j + u of qh[l] is set. Sub-block s = i / 32 has a 6-bit scale and a 6-bit min packed in the 12 bytes:

  • s < 4: scale scales[s] & 63, min scales[s + 4] & 63
  • s ≥ 4: scale (scales[s + 4] & 15) | (scales[s − 4] >> 6) << 4, min (scales[s + 4] >> 4) | (scales[s] >> 6) << 4

Value: (d · scale) · code − dmin · min.

Q6_K

16 groups of 16. Element i: half n = i / 128, row r = (i mod 128) / 32, lane l = i mod 32. Low 4 bits from ql[64n + 32(r & 1) + l] (low nibble for r < 2, high nibble for r ≥ 2), high 2 bits (qh[32n + l] >> 2r) & 3. Group scale scales[8n + 2r + l/16] (signed). Value: (d · scale) · (code − 32).

Q8_K

Activation format of the K-quant dot products. max is the value of largest magnitude, iscale = −127 / max, q = min(127, nearest_int(iscale · x)) (round half to even), d = 1 / iscale; bsums[g] is the sum of the 16 codes of group g, which lets the dot products apply the mins without touching the weights. An all-zero block has d = 0.

Non-linear 4-bit: IQ4_NL, IQ4_XS

Codes index a fixed 16-value grid kvalues = {−127, −104, −83, −65, −49, −35, −22, −10, 1, 13, 25, 38, 53, 69, 89, 113}.

  • IQ4_NL (20, 32-element blocks, 18 B): half d, uchar qs[16], codes ordered as Q4_0 (low nibbles are elements 0..15). Value d · kvalues[code].
  • IQ4_XS (23, 256-element super-blocks, 136 B): half d, ushort scales_h, uchar scales_l[4], uchar qs[128]. Group g of 32 has the 6-bit scale ls = ((scales_l[g/2] >> 4(g mod 2)) & 15) | (((scales_h >> 2g) & 3) << 4), value (d · (ls − 32)) · kvalues[code], codes of group g in qs[16g ..] ordered as Q4_0.

Quantization (as llama-quantize, without importance matrix): per group, the weights are x², 15 candidate scales (t + kvalues[0]) / max for t = −7 .. 7 plus a first guess −max / kvalues[0] are each fitted by least squares on the nearest grid codes, the best kept; IQ4_XS then rounds the group scales to 6 bits against d = −max_scale / 32 and requantizes.

FP4: MXFP4, NVFP4

E2M1 codes, values doubled: {0, 1, 2, 3, 4, 6, 8, 12, 0, −1, −2, −3, −4, −6, −8, −12}.

  • MXFP4 (39, 32-element blocks, 17 B): uchar e, uchar qs[16] (Q4_0 order). Scale 2^(e − 128) (half the E8M0 value, as the codes are doubled; subnormal for e < 2). Quantization: e = floor(log2(amax)) − 2 + 127, each code the closest value.
  • NVFP4 (40, 64-element blocks, 36 B): uchar d[4], uchar qs[32]; group s of 16 uses bytes 8s .. 8s + 7 (low nibbles first) and the UE4M3 scale d[s] (unsigned, 4 exponent bits with bias 7, 3 mantissa bits; 0 and 0x7F mean 0), halved. Quantization: d[s] = ue4m3(amax / 6).

Ternary: TQ1_0, TQ2_0

Values {−1, 0, 1} · d, d = amax of the super-block (f16).

  • TQ1_0 (34, 256 elements, 54 B): uchar qs[48], uchar qh[4], half d. A byte holds 5 base-3 digits as a fixed-point fraction: digit n of byte q is ((q · 3^n mod 256) · 3) >> 8, minus one. Elements 0..159 are digit e / 32 of qs[e mod 32], 160..239 digit (e − 160) / 16 of qs[32 + (e mod 16)], 240..255 digit (e − 240) / 4 of qh[e mod 4] (4 digits, the last slot unused). Encoding rounds the fraction up: byte = ceil(v · 256 / 243).
  • TQ2_0 (35, 256 elements, 66 B): uchar qs[64], half d; for each 32 bytes, bit pair l of byte m is element 128 c + 32 l + m, value code − 1.

Low-bit: Q1_0, Q2_0

  • Q1_0 (41, 128-element blocks, 18 B): half d, uchar qs[16]; element j is +d when bit j mod 8 of qs[j / 8] is set, −d otherwise; d is the mean of |x|.
  • Q2_0 (42, 64-element blocks, 18 B): half d, uchar qs[16]; element j is bits 2 (j mod 4) of qs[j / 4], value (code − 1) · d, d = amax, codes round(x / d) + 1 clamped to 0..3.

Ternary with a scale per 128: PQ2_0, PTQ1_0

Two types that are not ggml's: PrismML's llama.cpp fork defines them (ggml-common.h, ggml-quants.c; read at its commit 6447906b) and numbers them above ggml's, so that both sets fit one table. The Ternary Bonsai 2 files use them. Blocks of 128 elements, d = amax of the block (f16).

  • PQ2_0 (142, 34 B): half d, uchar qs[32]: Q2_0's code, element j at bits 2 (j mod 4) of qs[j / 4], value (code - 1) · d.
  • PTQ1_0 (143, 28 B): uchar qs[24], uchar qh[2], half d: TQ1_0's base-3 digits. Elements 0..79 are digit e / 16 of qs[e mod 16], 80..119 digit (e - 80) / 8 of qs[16 + (e mod 8)], 120..127 digit (e - 120) / 2 of qh[e mod 2] (4 digits, the last slot unused). 1.75 bits per weight, and exact for weights that are already ternary with one magnitude per 128, which TQ1_0's scale per 256 is not.

Decoding and quantization are checked bit for bit against the fork's dequantize_row_* and quantize_row_*_ref (tests/core/test_quant_p128.c: hashes of its output for the same random blocks and the same row). OKLM_DTYPE_COUNT is 144; the ids between ggml's last type and these are not valid. The OpenCL and Vulkan backends decode them in their matrix products as they decode Q2_0 and TQ1_0, 32 weights at a time (opencl.md, vulkan.md); oklm quantize does not offer them as a file type.

Grid IQ types (read and computed)

256-element super-blocks whose groups of 8 (IQ2, IQ1) or 4 (IQ3) values are indices into codebooks of magnitudes, ggml's tables copied as data into src/core/iq_grids.inc (see NOTICE), with separate sign bits:

  • IQ2_XXS (16, 66 B): half d, per 32 values two uint32: four 8-bit indices into iq2xxs_grid (256 entries of 8 bytes), then four 7-bit sign codes (ksigns_iq2xs gives the 8th, even-parity bit) and a 4-bit scale s on top; value d · (0.5 + s) / 4 · grid · sign.
  • IQ2_XS (17, 74 B): half d, ushort qs[32], uchar scales[8]; each ushort is a 9-bit index into iq2xs_grid (512) and a 7-bit sign code; two 4-bit scales per 32 values.
  • IQ2_S (22, 82 B): half d, uchar qs[32], uchar signs[32], uchar qh[8], uchar scales[8]; 10-bit indices into iq2s_grid (1024), the high bits in qh, a whole sign byte per 8 values.
  • IQ3_XXS (18, 98 B): half d, uchar qs[64] (8-bit indices into iq3xxs_grid, 256 entries of 4 bytes), then per 32 values a uint32 of four 7-bit sign codes and a 4-bit scale; value d · (0.5 + s) / 2 · grid · sign.
  • IQ3_S (21, 110 B): half d, uchar qs[64], uchar qh[8], uchar signs[32], uchar scales[4]; 9-bit indices into iq3s_grid (512), value d · (1 + 2 s) · grid · sign.
  • IQ1_S (19, 50 B): half d, uchar qs[32], ushort qh[8]; 11-bit indices into iq1s_grid (2048 entries of eight values in {−1, 0, 1}), per 32 values a 3-bit scale and a delta sign; value d · (2 s + 1) · (grid ± 1/8).
  • IQ1_M (29, 56 B): uchar qs[32], uchar qh[16], ushort scales[4]; as IQ1_S with a delta sign per 8 values and a 3-bit scale per 16; the f16 super-block scale is spread over the top nibbles of the four scale words.

Dot products take Q8_K activations and accumulate ggml's integers in its order (IQ1 deltas through the activation block sums), so they equal its reference kernels; dequantization is bit-identical to ggml's (checked against gguf-py on fixed blocks and on llama-quantize's files). OKLM does not write these types: llama-quantize does, with an importance matrix.

Dot products

Weight type Activation type Integer part per block Float part
Q4_0, Q5_0 Q8_0 Σ (q − offset) · a sumi · d_w · d_a
Q8_0 Q8_0 Σ q · a sumi · (d_w · d_a)
Q4_1, Q5_1 Q8_1 Σ q · a d_w · d_a · sumi + m_w · s_a
Q2_K, Q4_K, Q5_K Q8_K Σ_sub scale · Σ q · a, mins via bsums d_w · d_a · isum − dmin_w · d_a · summs
Q3_K, Q6_K Q8_K Σ_group scale · Σ (q − offset) · a d_w · d_a · isum
IQ4_NL, MXFP4 Q8_0 Σ value · a (d_a · d_w) · sumi
IQ4_XS Q8_K per group Σ value · a (d_w · d_a · (ls − 32)) · sumi, per group
NVFP4 Q8_0 per group of 16 Σ value · a d_a · d_s · sumi, per group
TQ1_0, TQ2_0 Q8_K Σ (code − 1) · a sumi · (d_w · d_a)
Q1_0, Q2_0 Q8_0 per activation block Σ ±a / Σ (code − 1) · a d_w · Σ_blocks d_a · sumi
PQ2_0, PTQ1_0 Q8_0 per activation block Σ (code − 1) · a (four blocks per weight block) d_w · Σ_blocks d_a · sumi

The integer parts are exact in every kernel; the scalar and AVX2 kernels then perform the same float operations in the same order, so they return the same bits.