OKLMdocumentation v1.0.3
On this page

ADR-0005: KV cache precision

Records which types the KV cache can hold and how a quantized cache is kept accurate.

Status: accepted (2026-10-30).

Context

The KV cache grows with the context: 2 x layers x KV heads x head size per token, 32 KiB per token in F16 for Llama-3.2-1B, 128 KiB for an 8B model. llama.cpp lets users pick the key and value cache types (-ctk / -ctv: f32, f16, bf16, q8_0, q4_0, q4_1, q5_0, q5_1, iq4_nl), and rotates the heads of a quantized cache with a Walsh-Hadamard matrix, which spreads the few large components of keys and values over the group and makes block quantization much less lossy. Measured on Llama-3.2-1B-Q4_K_M, 512-token windows of tinyshakespeare (60 kB): llama.cpp F16 39.96, Q8_0 39.98, Q4_0 42.20 with the rotation and 45.90 without.

Decision

  • F16 stays the default. The key and value types can be chosen separately among llama.cpp's cache types (oklm_context_params.kv_type / v_type, -ctk / -ctv); the block quants need head_dim % 32 == 0.
  • A quantized key cache stores keys rotated by the orthonormal Sylvester Hadamard matrix over groups of the largest power of two dividing head_dim (at most 256); queries are rotated the same way, so the scores do not change. A quantized value cache stores values rotated over groups of 64, and the attention output is rotated back (the matrix is its own inverse). This is llama.cpp's scheme; OKLM_KV_ROTATION=0 turns it off. The rotation is an op (oklm_hadamard, a fast Walsh-Hadamard transform) on the CPU and OpenCL.
  • On the CPU, attention converts the query to the key type's activation type and uses the quantized dot products of the matrix products; values are dequantized row by row. On OpenCL, cache writes quantize on the device (Q8_0, Q4_0, Q4_1, Q5_0, Q5_1; IQ4_NL writes run on the CPU) and attention dequantizes keys and values as it reads them.

Consequences

OKLM's perplexities with quantized caches follow llama.cpp's (Q8_0 40.06, Q4_0 42.12 on the measurement above, against 40.12 for OKLM's F16 cache). On the OpenCL device a quantized cache costs about 10 % of decode speed on Llama-3.2-1B; attention reads less memory, but the rotations and the dequantization are extra work. Cache types stay fixed for the life of a context.