Block quantization formats
Byte-level layout and decoding rule of every block-quantized type
OKLM reads or writes. These are ggml's formats
(ggml-common.h for the layouts, ggml-quants.c
for the reference algorithms), re-derived; a block written by OKLM is
byte-identical to the block ggml's reference quantizer writes for the
same input.
Status: legacy quants, K-quants, IQ4_NL / IQ4_XS, MXFP4 / NVFP4, TQ1_0 / TQ2_0 and Q1_0 / Q2_0 implemented (read, compute, write), and PQ2_0 / PTQ1_0, which are a fork's and not ggml's; the grid IQ types (IQ1 to IQ3) read and computed, not written.
Upstream revision: the layouts below have been stable in ggml since the K-quants landed (2023). No ggml checkout was pinned when this code was written; the fixture-based checks against public GGUF files pin it.
All multi-byte fields are little-endian. f16 is IEEE-754
binary16. QK = 32, QK_K = 256.
Legacy quants (32-element blocks)
| Type | id | Bytes | Fields (in order) | Value of element j |
|---|---|---|---|---|
| Q4_0 | 2 | 18 | f16 d; u8 qs[16] |
d · (q_j − 8) |
| Q4_1 | 3 | 20 | f16 d; f16 m; u8 qs[16] |
d · q_j + m |
| Q5_0 | 6 | 22 | f16 d; u8 qh[4]; u8 qs[16] |
d · (q_j − 16) |
| Q5_1 | 7 | 24 | f16 d; f16 m; u8 qh[4]; u8 qs[16] |
d · q_j + m |
| Q8_0 | 8 | 34 | f16 d; i8 qs[32] |
d · qs[j] |
| Q8_1 | 9 | 36 | f16 d; f16 s; i8 qs[32] |
d · qs[j], s = d · Σ qs |
Nibble order (Q4_, and the low 4 bits of Q5_): element
j < 16 is the low nibble of qs[j], element
j ≥ 16 the high nibble of qs[j − 16]. The
fifth bit of Q5_* element j is bit j of
qh read as a little-endian uint32.
Reference quantizers
| Type | Scale | Code |
|---|---|---|
| Q4_0 | max = the value of largest magnitude (sign kept);
d = max / −8 |
min(15, (int8)(x/d + 8.5)) |
| Q4_1 | d = (max − min) / 15, m = min |
min(15, (int8)((x − min)/d + 0.5)) |
| Q5_0 | d = max / −16 |
min(31, (int8)(x/d + 16.5)) |
| Q5_1 | d = (max − min) / 31, m = min |
(uint8)((x − min)/d + 0.5) |
| Q8_0 | d = amax / 127 |
roundf(x/d) |
| Q8_1 | as Q8_0 | as Q8_0; s = f16(Σ q · d) with the unrounded
d |
1/d is used (0 when d = 0), and
d is stored rounded to f16 after it was used unrounded for
the codes. Negative scales are intentional: with
d = max / −8, the extreme value maps exactly to code 0 and
the other sign gets codes up to 15 (clipped).
Worked example
Q4_0 of x_j = j − 16, j = 0..31: the largest magnitude
is x_0 = −16, so d = −16 / −8 = 2 (f16
0x4000) and 1/d = 0.5.
x_0 → (int8)(−8 + 8.5) = 0,
x_16 → (int8)(0 + 8.5) = 8,
x_31 → (int8)(7.5 + 8.5) = 16 → 15. Byte qs[0]
holds x_0 (low) and x_16 (high):
0x80. Decoding gives 2 · (0 − 8) = −16 and
2 · (8 − 8) = 0; x_31 decodes to
2 · 7 = 14 (clipping error 1 = one step).
tests/core/test_quant.c checks every byte of this
block.
K-quants (256-element super-blocks)
| Type | id | Bytes | Fields (in order) |
|---|---|---|---|
| Q2_K | 10 | 84 | u8 scales[16]; u8 qs[64]; f16 d; f16 dmin |
| Q3_K | 11 | 110 | u8 hmask[32]; u8 qs[64]; u8 scales[12]; f16 d |
| Q4_K | 12 | 144 | f16 d; f16 dmin; u8 scales[12]; u8 qs[128] |
| Q5_K | 13 | 176 | f16 d; f16 dmin; u8 scales[12]; u8 qh[32]; u8 qs[128] |
| Q6_K | 14 | 210 | u8 ql[128]; u8 qh[64]; i8 scales[16]; f16 d |
| Q8_K | 15 | 292 | f32 d; i8 qs[256]; i16 bsums[16] |
Q2_K
16 groups of 16. Element i: half
n = i / 128, shift index j = (i mod 128) / 32,
lane l = i mod 32. Code
q = (qs[32n + l] >> 2j) & 3; group byte
sc = scales[8n + 2j + l/16]. Value:
(d · (sc & 15)) · q − dmin · (sc >> 4).
Q3_K
16 groups of 16, same n, j, l as Q2_K. Low bits
(qs[32n + l] >> 2j) & 3, high bit
(hmask[l] >> (4n + j)) & 1; the code is
low − 4 when the high bit is 0, low when it is
1. Group k = 8n + 2j + l/16 has a 6-bit scale: low 4 bits
from scales[k] (low nibble, k < 8) or
scales[k − 8] (high nibble, k ≥ 8), high 2 bits from
scales[8 + k mod 4] >> 2(k/4). Value:
(d · (scale − 32)) · code.
Q4_K and Q5_K
8 sub-blocks of 32. Element i: chunk
j = i / 64, upper half u = (i mod 64) ≥ 32,
lane l = i mod 32. Low 4 bits: low nibble of
qs[32j + l] (u = 0) or high nibble (u = 1). Q5_K adds 16
when bit 2j + u of qh[l] is set. Sub-block
s = i / 32 has a 6-bit scale and a 6-bit min packed in the
12 bytes:
s < 4: scalescales[s] & 63, minscales[s + 4] & 63s ≥ 4: scale(scales[s + 4] & 15) | (scales[s − 4] >> 6) << 4, min(scales[s + 4] >> 4) | (scales[s] >> 6) << 4
Value: (d · scale) · code − dmin · min.
Q6_K
16 groups of 16. Element i: half
n = i / 128, row r = (i mod 128) / 32, lane
l = i mod 32. Low 4 bits from
ql[64n + 32(r & 1) + l] (low nibble for r < 2, high
nibble for r ≥ 2), high 2 bits
(qh[32n + l] >> 2r) & 3. Group scale
scales[8n + 2r + l/16] (signed). Value:
(d · scale) · (code − 32).
Q8_K
Activation format of the K-quant dot products. max is
the value of largest magnitude, iscale = −127 / max,
q = min(127, nearest_int(iscale · x)) (round half to even),
d = 1 / iscale; bsums[g] is the sum of the 16
codes of group g, which lets the dot products apply the
mins without touching the weights. An all-zero block has
d = 0.
Non-linear 4-bit: IQ4_NL, IQ4_XS
Codes index a fixed 16-value grid
kvalues = {−127, −104, −83, −65, −49, −35, −22, −10, 1, 13, 25, 38, 53, 69, 89, 113}.
- IQ4_NL (20, 32-element blocks, 18 B):
half d, uchar qs[16], codes ordered as Q4_0 (low nibbles are elements 0..15). Valued · kvalues[code]. - IQ4_XS (23, 256-element super-blocks, 136 B):
half d, ushort scales_h, uchar scales_l[4], uchar qs[128]. Groupgof 32 has the 6-bit scalels = ((scales_l[g/2] >> 4(g mod 2)) & 15) | (((scales_h >> 2g) & 3) << 4), value(d · (ls − 32)) · kvalues[code], codes of groupginqs[16g ..]ordered as Q4_0.
Quantization (as llama-quantize, without importance matrix): per
group, the weights are x², 15 candidate scales
(t + kvalues[0]) / max for t = −7 .. 7 plus a
first guess −max / kvalues[0] are each fitted by least
squares on the nearest grid codes, the best kept; IQ4_XS then rounds the
group scales to 6 bits against d = −max_scale / 32 and
requantizes.
FP4: MXFP4, NVFP4
E2M1 codes, values doubled:
{0, 1, 2, 3, 4, 6, 8, 12, 0, −1, −2, −3, −4, −6, −8, −12}.
- MXFP4 (39, 32-element blocks, 17 B):
uchar e, uchar qs[16](Q4_0 order). Scale2^(e − 128)(half the E8M0 value, as the codes are doubled; subnormal fore < 2). Quantization:e = floor(log2(amax)) − 2 + 127, each code the closest value. - NVFP4 (40, 64-element blocks, 36 B):
uchar d[4], uchar qs[32]; groupsof 16 uses bytes8s .. 8s + 7(low nibbles first) and the UE4M3 scaled[s](unsigned, 4 exponent bits with bias 7, 3 mantissa bits; 0 and 0x7F mean 0), halved. Quantization:d[s] = ue4m3(amax / 6).
Ternary: TQ1_0, TQ2_0
Values {−1, 0, 1} · d, d = amax of the
super-block (f16).
- TQ1_0 (34, 256 elements, 54 B):
uchar qs[48], uchar qh[4], half d. A byte holds 5 base-3 digits as a fixed-point fraction: digitnof byteqis((q · 3^n mod 256) · 3) >> 8, minus one. Elements 0..159 are digite / 32ofqs[e mod 32], 160..239 digit(e − 160) / 16ofqs[32 + (e mod 16)], 240..255 digit(e − 240) / 4ofqh[e mod 4](4 digits, the last slot unused). Encoding rounds the fraction up:byte = ceil(v · 256 / 243). - TQ2_0 (35, 256 elements, 66 B):
uchar qs[64], half d; for each 32 bytes, bit pairlof bytemis element128 c + 32 l + m, valuecode − 1.
Low-bit: Q1_0, Q2_0
- Q1_0 (41, 128-element blocks, 18 B):
half d, uchar qs[16]; elementjis+dwhen bitj mod 8ofqs[j / 8]is set,−dotherwise;dis the mean of|x|. - Q2_0 (42, 64-element blocks, 18 B):
half d, uchar qs[16]; elementjis bits2 (j mod 4)ofqs[j / 4], value(code − 1) · d,d = amax, codesround(x / d) + 1clamped to 0..3.
Ternary with a scale per 128: PQ2_0, PTQ1_0
Two types that are not ggml's: PrismML's llama.cpp fork defines them
(ggml-common.h, ggml-quants.c; read at its
commit 6447906b) and numbers them above ggml's, so that
both sets fit one table. The Ternary Bonsai 2 files use them. Blocks of
128 elements, d = amax of the block (f16).
- PQ2_0 (142, 34 B):
half d, uchar qs[32]: Q2_0's code, elementjat bits2 (j mod 4)ofqs[j / 4], value(code - 1) · d. - PTQ1_0 (143, 28 B):
uchar qs[24], uchar qh[2], half d: TQ1_0's base-3 digits. Elements 0..79 are digite / 16ofqs[e mod 16], 80..119 digit(e - 80) / 8ofqs[16 + (e mod 8)], 120..127 digit(e - 120) / 2ofqh[e mod 2](4 digits, the last slot unused). 1.75 bits per weight, and exact for weights that are already ternary with one magnitude per 128, which TQ1_0's scale per 256 is not.
Decoding and quantization are checked bit for bit against the fork's
dequantize_row_* and quantize_row_*_ref
(tests/core/test_quant_p128.c: hashes of its output for the
same random blocks and the same row). OKLM_DTYPE_COUNT is
144; the ids between ggml's last type and these are not valid. The
OpenCL and Vulkan backends decode them in their matrix products as they
decode Q2_0 and TQ1_0, 32 weights at a time (opencl.md, vulkan.md); oklm quantize does
not offer them as a file type.
Grid IQ types (read and computed)
256-element super-blocks whose groups of 8 (IQ2, IQ1) or 4 (IQ3)
values are indices into codebooks of magnitudes, ggml's tables copied as
data into src/core/iq_grids.inc (see NOTICE), with separate
sign bits:
- IQ2_XXS (16, 66 B):
half d, per 32 values two uint32: four 8-bit indices intoiq2xxs_grid(256 entries of 8 bytes), then four 7-bit sign codes (ksigns_iq2xsgives the 8th, even-parity bit) and a 4-bit scaleson top; valued · (0.5 + s) / 4 · grid · sign. - IQ2_XS (17, 74 B):
half d, ushort qs[32], uchar scales[8]; each ushort is a 9-bit index intoiq2xs_grid(512) and a 7-bit sign code; two 4-bit scales per 32 values. - IQ2_S (22, 82 B):
half d, uchar qs[32], uchar signs[32], uchar qh[8], uchar scales[8]; 10-bit indices intoiq2s_grid(1024), the high bits inqh, a whole sign byte per 8 values. - IQ3_XXS (18, 98 B):
half d, uchar qs[64](8-bit indices intoiq3xxs_grid, 256 entries of 4 bytes), then per 32 values a uint32 of four 7-bit sign codes and a 4-bit scale; valued · (0.5 + s) / 2 · grid · sign. - IQ3_S (21, 110 B):
half d, uchar qs[64], uchar qh[8], uchar signs[32], uchar scales[4]; 9-bit indices intoiq3s_grid(512), valued · (1 + 2 s) · grid · sign. - IQ1_S (19, 50 B):
half d, uchar qs[32], ushort qh[8]; 11-bit indices intoiq1s_grid(2048 entries of eight values in {−1, 0, 1}), per 32 values a 3-bit scale and a delta sign; valued · (2 s + 1) · (grid ± 1/8). - IQ1_M (29, 56 B):
uchar qs[32], uchar qh[16], ushort scales[4]; as IQ1_S with a delta sign per 8 values and a 3-bit scale per 16; the f16 super-block scale is spread over the top nibbles of the four scale words.
Dot products take Q8_K activations and accumulate ggml's integers in its order (IQ1 deltas through the activation block sums), so they equal its reference kernels; dequantization is bit-identical to ggml's (checked against gguf-py on fixed blocks and on llama-quantize's files). OKLM does not write these types: llama-quantize does, with an importance matrix.
Dot products
| Weight type | Activation type | Integer part per block | Float part |
|---|---|---|---|
| Q4_0, Q5_0 | Q8_0 | Σ (q − offset) · a |
sumi · d_w · d_a |
| Q8_0 | Q8_0 | Σ q · a |
sumi · (d_w · d_a) |
| Q4_1, Q5_1 | Q8_1 | Σ q · a |
d_w · d_a · sumi + m_w · s_a |
| Q2_K, Q4_K, Q5_K | Q8_K | Σ_sub scale · Σ q · a, mins via bsums |
d_w · d_a · isum − dmin_w · d_a · summs |
| Q3_K, Q6_K | Q8_K | Σ_group scale · Σ (q − offset) · a |
d_w · d_a · isum |
| IQ4_NL, MXFP4 | Q8_0 | Σ value · a |
(d_a · d_w) · sumi |
| IQ4_XS | Q8_K | per group Σ value · a |
(d_w · d_a · (ls − 32)) · sumi, per group |
| NVFP4 | Q8_0 | per group of 16 Σ value · a |
d_a · d_s · sumi, per group |
| TQ1_0, TQ2_0 | Q8_K | Σ (code − 1) · a |
sumi · (d_w · d_a) |
| Q1_0, Q2_0 | Q8_0 | per activation block Σ ±a /
Σ (code − 1) · a |
d_w · Σ_blocks d_a · sumi |
| PQ2_0, PTQ1_0 | Q8_0 | per activation block Σ (code − 1) · a (four blocks per
weight block) |
d_w · Σ_blocks d_a · sumi |
The integer parts are exact in every kernel; the scalar and AVX2 kernels then perform the same float operations in the same order, so they return the same bits.