Core: types, tensors, backends and CPU ops
The core layer is the tensor engine everything else runs on: element
types, half-precision conversion, block quantization, tensors and views,
the arena allocator, the backend interface and the CPU backend with its
kernels. Headers: include/oklm/oklm.h and
include/oklm/core/*.h.
Status: implemented (CPU backend; the OpenCL backend comes later).
Initialization
#include "oklm/oklm.h"
oklm_init(NULL); /* logging, CPU features, worker team, backends */
...
oklm_shutdown();oklm_init() is reference counted.
oklm_init_params overrides the environment
(OKLM_THREADS, OKLM_BACKEND,
OKLM_SIMD, OKLM_LOG_LEVEL,
OKLM_LOG_FILE). Status codes are negative
oklm_status values; oklm_strerror() turns them
into text.
Element types (dtype.h)
oklm_dtype uses ggml's numbering, so the type id stored
in a GGUF tensor info is used as is. Retired ggml ids are gaps that
oklm_dtype_is_valid() rejects.
oklm_dtype_get_traits() gives the block size, block byte
size and what OKLM can do with the type:
| Types | Dequantize | Quantize | Matmul |
|---|---|---|---|
| F32, F16, BF16 | yes | yes | yes |
| Q4_0, Q4_1, Q5_0, Q5_1, Q8_0 | yes | yes (ggml reference) | yes |
| Q8_1, Q8_K | yes | yes | activation types |
| Q2_K, Q3_K, Q4_K, Q5_K, Q6_K | yes | yes (ggml reference) | yes |
| IQ4_NL, IQ4_XS, MXFP4, NVFP4, TQ1_0, TQ2_0, Q1_0, Q2_0 | yes | yes (ggml reference) | yes |
| IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S | yes (ggml reference, its codebook tables) | yes (own codebook search, lower perplexity than llama-quantize's) | yes |
| I8, I16, I32, I64, F64 | element access | no | no |
oklm types prints this table from the library
itself.
Half precision (f16.h)
Branch-free scalar conversions, bit-identical with the F16C instructions for every non-NaN input (verified exhaustively for half→float and on 16.7 million float patterns for float→half): round to nearest even, subnormals kept, overflow to infinity. bfloat16 rounds to nearest even and keeps NaNs quiet. Row conversions use F16C when available.
Quantization (quant.h)
Block layouts are in formats/quant.md. The quantizers are the
ggml reference algorithms, re-derived: same scale selection, same
truncating conversions, same rounding, so rows are byte-identical to
ggml's. Where ggml calls roundf (Q8_0, Q8_1, TQ1_0, TQ2_0,
Q2_0) the quantizers compute it themselves (oklm_roundf(),
src/core/quant_internal.h): the roundf of
mingw-w64, linked into the Windows build, returns 1 for the float just
below a half, which put a level one too high in a Windows build's rows
now and then (2 activation rows in 20000 of a SmolLM2-135M prompt).
Dot products follow ggml's scheme: the activation row is quantized to
the weight type's vec_dot type
(oklm_vec_dot_type()), then
oklm_vec_dot_get(type)(n, w, a) returns the dot product.
The kernel set follows the selected SIMD level. Each quantized kernel
computes exact integer sums per block, then applies the scale in a fixed
order: the AVX2 and AVX-512 kernels return the same bits as the scalar
ones (for activation rows written by the quantizers: Q8_0 activation
bytes are in -127..127, which the Q8_0 kernels rely on).
oklm_vec_dot4_get(type) gives, where one exists, a
kernel computing the dot products of one weight row with four activation
rows, unpacking each weight block once; each of its four results is
bit-identical to the one-row kernel's. The CPU matrix product uses it
for prompts. oklm_vec_dot8_get(type) is the same for eight
rows. F16 weights have both: four rows on AVX2 (two at a time, the
weight row converted once for the two) and eight on AVX-512, where a
row's two 512-bit accumulators are the four 256-bit ones of the one-row
kernel, added in its order. A product of many rows against F16 weights
(an image encoder's: its projector files are F16) then converts each
weight block once for eight rows instead of once a row, and keeps eight
or sixteen sums going where one row has four: 89 to 96 GFlop/s against
48 on one thread of a Core i7-11700K (1 280 x 1 280, 1 008 rows), the
same bits. From 256 activation rows such a product is also cut by its
activation rows instead of its weight rows
(matmul_row_blocks()): a thread keeps a block of them (384
KiB, 64 rows at most) in its cache and passes it over every weight row,
a tile of 64 KiB at a time, where a tile of weight rows had every
activation row read from memory once a tile; the weights are what the
threads then share, in the last-level cache when they fit. The rows are
grouped as before and each output is one call of the same kernels, so
the bits are the same (test_many_rows). On eight threads, 1
280 x 1 280 over 4 032 rows: 0.044 s with one row a dot product, 0.031 s
with eight, 0.026 s by row blocks; 1 280 x 5 120 over 16 060 rows: 0.82
s, 0.57 s and 0.30 s. Rows too long for eight of them to stay in the
first-level cache (5 120 elements, the second product of an encoder's
MLP) gain less: 1.63 s, 0.64 s and 0.49 s at 16 060 rows, 0.18 s, 0.16 s
and 0.16 s at 4 032; their inner dimension cannot be cut into blocks
without carrying each output's sums from one block to the next.
Arena (arena.h)
oklm_arena_new(capacity) allocates one aligned buffer;
oklm_arena_alloc() bumps a cursor (64-byte alignment by
default); oklm_arena_reset() frees everything at once and
zero-fills what was used; oklm_arena_peak() tells how much
a dry run needed. Not thread-safe: one arena per thread.
Tensors (tensor.h)
Same conventions as ggml: ne[0] innermost,
nb[i] byte strides, at most 4 dimensions. Tensors come from
an arena or the heap (oklm_tensor_new*() with a NULL arena,
released by oklm_tensor_free()), or wrap external memory
(oklm_tensor_init_extern(), for mapped files). Views
(oklm_tensor_view(), oklm_tensor_reshape(),
oklm_tensor_permute(),
oklm_tensor_transpose()) share memory, are bounds-checked
against their source and remember the root tensor and offset. A
quantized tensor's dimension 0 cannot be permuted, since its blocks span
it.
Backends (backend.h)
typedef struct oklm_backend_iface {
void* (*buffer_alloc)(oklm_backend*, size_t size, size_t align);
void (*buffer_free)(oklm_backend*, void*);
int (*upload)(oklm_backend*, oklm_tensor*, const void*, size_t off, size_t n);
int (*download)(oklm_backend*, const oklm_tensor*, void*, size_t off, size_t n);
int (*supports)(oklm_backend*, const oklm_op_desc*);
int (*compute)(oklm_backend*, const oklm_op_desc*);
int (*synchronize)(oklm_backend*);
void (*release)(oklm_backend*);
} oklm_backend_iface;oklm_backend_register() adds a backend;
oklm_backend_select("auto") returns the one of highest
priority. An op runs on its destination tensor's backend
(tensor->backend, NULL meaning host memory and the CPU
backend).
Operations (ops.h)
Every op is an oklm_op_desc (op id, destination, up to
four sources, eight float and eight integer parameters) plus a
convenience function. Outputs are always preallocated by the caller.
| Function | Semantics |
|---|---|
oklm_cpy |
copy/convert any strided source into a destination of any quantizable type |
oklm_add, oklm_mul |
element-wise with broadcasting (sizes of b divide those of a) |
oklm_scale |
multiply by a constant |
oklm_silu, oklm_gelu,
oklm_gelu_erf, oklm_swiglu,
oklm_geglu |
activations |
oklm_sigglu |
the gated unit of gpt-oss's experts:
g · sigmoid(1.702 g) · (clamp(u, -c, c) + 1) with
g = min(gate, c) |
oklm_softcap |
c · tanh(x / c) (Gemma 2 logits and scores) |
oklm_hadamard |
each group of n elements of a row (n up to 4096) times the orthonormal Walsh-Hadamard matrix (its own inverse): rotation of a quantized KV cache |
oklm_hadamard_fold |
the same with a sign per element, before the rotation or after it, and a row's heads reordered first: the input of a weight stored in a rotated basis and a rotated embedding row turned back (Ternary Bonsai 2) on a device |
oklm_rms_norm, oklm_layer_norm |
per-row normalization, optional weight and bias, double accumulation |
oklm_softmax |
per-row, scaled, optional additive mask, fully masked rows give zeros |
oklm_rope |
rotary embedding: normal or NeoX pairs, partial rotation, linear scaling, YaRN, frequency factors |
oklm_get_rows |
embedding lookup from any dequantizable table |
oklm_matmul |
dst = x · wᵀ, weights of any computable type, batch
broadcasting |
oklm_matmul_acc |
dst += x · wᵀ in one pass (residual connections); same
values as a product then oklm_add |
oklm_matmul_dense |
oklm_matmul for many activation rows against F32 or F16
weights (an encoder's products), through a blocked kernel on the CPU;
another order of the sums, so the last bits differ from
oklm_matmul |
oklm_matmul_wide |
oklm_matmul for activations that need F32's precision
(an encoder's blocks on a device): Vulkan's cooperative matrices, which
take a prompt's activations rounded to F16, then compute it without that
rounding (vulkan.md); the
CPU's and OpenCL's products are oklm_matmul's |
oklm_attention |
causal attention over a KV cache, grouped-query heads, sliding window, soft-capping, attention sinks (a score per head that joins the softmax without a value, gpt-oss; a tensor in the memory the op runs in); F32, F16, BF16 or block-quant (Q8_0, Q4_0, Q4_1, Q5_0, Q5_1, IQ4_NL) caches, the query converted to the key type's dot product type |
oklm_attention_spans |
bidirectional attention where each query sees a span of the keys of
its own, given as a table of first key and one past the last (the
windowed blocks of an image encoder: a window's rows see each other).
The CPU attends each run of queries with one span on views, the bits of
oklm_attention over them; a device attends every query in
one launch on its tiles (opencl.md, vulkan.md) and returns
OKLM_ERR_UNSUPPORTED for operands they do not take, the
caller then attending span by span |
oklm_cross_entropy |
per-row loss against target ids |
oklm_argmax, oklm_topk |
selection, lower index first on ties |
oklm_ssm_conv |
a Gated DeltaNet block's short causal depthwise conv over time (+ SiLU), the inputs before the first token read from a per-sequence ring that the op then refills, so the next call continues the sequence |
oklm_gated_delta |
the Gated DeltaNet recurrence of every value head over the tokens (q
and k L2-normalized per key group, decay and write strength through a
sigmoid), its state updated in place; on OpenCL one work-group per value
head (heads of up to 128 values, keys up to 256), the tokens in order.
oklm_gated_delta_tok is the same recurrence with a decay
per token (exp(factor * softplus(projection + bias))), the
key heads repeated over the value heads and the query divided by
sqrt(dk): the published qwen35 layout, on the
CPU, on OpenCL and on Vulkan by the same kernels |
Matmul
dst[m][n] = Σ_k w[n][k] · x[m][k]: weights are stored
one output row per contiguous (quantized) row, exactly as in GGUF.
Quantized weights first quantize all activation rows to the vec_dot
type, in parallel. The output is then cut into tiles of 16 weight rows
per batch index; inside a tile each activation row streams once against
the 16 weight rows. With one activation row (decode) this parallelizes
over weight rows, which suits a memory-bound matrix-vector product.
oklm_matmul_n() runs several products of the same
activations (a layer's Q, K and V, its gate and up) as one: on the CPU
the rows are quantized once and the tiles of every product make one loop
of the worker team, so a decode step pays one team start and end where
it paid three or two (SmolLM2-135M from 211 loops of products a token to
121: decoding 4 to 8 % faster from four threads). The values are those
of one oklm_matmul() per weight; products whose kernels
want other activation rows, and device tensors, run one after the
other.
A convolution over rows has no op of its own. The encoders and the
vocoders compute one as products: on the CPU a single product over
windows unfolded in host memory (src/mm/audio.c,
src/mm/vocoder.c); on a device one product a position of
the kernel, over a view of the rows that position reads (every stride-th
row from the position's own; a view is a tensor with a byte offset and a
row stride, which every backend's products and element-wise ops take),
the first written and the others added with the accumulating form. A
depthwise convolution is the same with a product by a vector
(oklm_mul) and oklm_add in place of the matrix
product. Both forms are checked on each backend against the arithmetic
they stand for (test_conv_views in
tests/core/test_opencl.c and
tests/core/test_vulkan_gemm.c).
oklm_matmul_dense() is the product of an encoder:
hundreds or thousands of activation rows (the 1500 positions of a
Whisper window) against float weights. oklm_matmul()
computes such a product one dot product at a time, each one reading a
whole weight row and a whole activation row; the blocked kernel
(src/core/cpu/gemm_cpu.c, AVX2) cuts the weight rows into
tiles of 64 (one task of the worker team each) and the inner dimension
into blocks of 512 elements, copies the tile's block eight rows side by
side (F16 widened to F32 once, there), and runs twelve activation rows
against eight weight rows at a time: twelve accumulators, one load of
weights and twelve broadcast multiply-adds per inner element. Subnormal
operands are read as zeros: each one costs a microcode assist per
multiply-add otherwise (activations of which a third are subnormal made
the product sixty times slower; Whisper's hold none). Measured on 1500
rows against a 1280 x 5120 F16 matrix (Whisper large-v3's MLP), on a
machine other jobs shared: 41 to 44 thousand million multiply-adds a
second on one thread against 15 to 17 for oklm_matmul(), 72
to 112 on eight threads against 56 to 82. Its sums run in another order
than oklm_matmul()'s, so the two differ in their last bits:
a model whose rows must give the same values alone and in a batch (every
language model: decode against prompt) keeps oklm_matmul().
A product the blocked kernel does not take (a device tensor, quantized
weights, fewer than 24 rows, no AVX2) is
oklm_matmul()'s.
The indexed product (oklm_matmul_id, mixture-of-experts
layers) groups its output rows by expert: each tile of an expert's
weight rows is read once for up to 64 of the rows routed to it, with the
four- and eight-row kernels, and threads claim the (expert, tile, rows)
items one at a time so a skewed routing stays balanced. Every element is
the same one-row dot product as before, bit for bit. With the experts
repacked (above) the items run the packed kernels, eight expert rows a
call: MoE prompts 1.4 to 1.5 times faster (benchmarks.md).
For a Qwen3-30B-A3B-like layer (Q4_K, 8 threads) this is 3 to 3.8 times
faster for 128 to 512 tokens, 1.2 times for 16 and as fast for one (benchmarks.md).
RoPE
The angle of dimension pair i at position p
is p · base^(−2i/n_dims) / freq_factor[i], then scaled by
freq_scale; with YaRN (ext_factor ≠ 0)
interpolated and extrapolated angles are mixed over the correction range
computed from n_ctx_orig, beta_fast and
beta_slow, and the magnitude is multiplied by
1 + 0.1 ln(1/freq_scale). These are ggml's
rope_ext formulas, so imported models get the same
angles.
Attention
For query head h and token t (absolute
position pos0 + t), keys 0 .. pos0 + t of KV
head h / (n_head / n_head_kv) are visible (restricted to
the last window positions when set). Scores are scaled,
optionally soft-capped, soft-maxed with the maximum subtracted and the
denominator in double.
On the CPU (src/core/cpu/attention_cpu.c, AVX2) a
prompt's rows are computed in tiles of up to 32 query rows of one KV
head, the keys and the values a tile sees gathered from the cache 64 at
a time; with too few tiles for the threads (a decode step) the same
arithmetic is split by keys, rows and columns. Bidirectional attention
of many rows over float keys and values (an image encoder's blocks: from
512 keys and 32 rows a thread) runs head by head instead: a head's keys
are copied once, eight side by side, and its values row after row, for
all of its rows, and every thread works on that head, whose two copies
stay in the last-level cache; the scores of eight rows are then written
in the keys' order by one kernel over 256 keys at a time, and a tile
takes fewer rows (eight at least) when its scores over every key would
pass 512 KiB. Each score is one chain of fused multiply-adds in the
order of the head's elements and each output one chain in the order of
the keys on every path, so a row has the same bits whatever its tile,
the thread count and the path (test_bidirectional). One
block of Qwen2-VL (16 heads of 80) on eight threads of a Core i7-11700K:
4 032 rows in 0.14 to 0.17 s where the tiles took 0.41 s, 2 048 of 16
060 rows in 0.44 s where they took 1.0 s.
CPU backend
- Kernels: portable C; AVX2+FMA+F16C dot products for every computed
type (F32, F16, BF16, the legacy and K quants) and half row conversions;
four-row AVX2 kernels for Q8_0, Q4_0, Q5_0, Q4_K, Q5_K, Q6_K and F16
(and an eight-row AVX-512 one for F16); AVX-512 (F, BW, VL, VNNI)
kernels for Q6_K, the other types falling back to AVX2 at that level.
Chosen at run time (
OKLM_SIMD:scalar,avx2,avx512). NEON is not planned yet. - Threads: a worker team (
src/core/cpu/team_cpu.c) sized byOKLM_THREADS(default: the number of physical cores;oklm_cpu_set_threads()), kept alive between ops. Each parallel loop is cut into contiguous chunks, at most four per thread, whose bounds depend only on the range, the grain and the thread count; the calling thread and the workers claim them one at a time from one atomic counter, and the loop ends when every chunk is done, not when every thread has checked in. A thread the kernel deschedules (other processes on the cores, more threads than cores) holds a loop up by at most the chunk it runs, and not at all when it holds none: the others take its share. Matrix-vector products (decode) run on fewer threads than the team: bound by memory, they read it fastest on half the physical cores with the light kernels of the packed copies (OKLM_MATVEC_THREADS), and on as many when their weights are under 4 MiB (a loop that short does not pay for more threads joining it), on one thread per core with the file layout's above that; they are cut into four tiles per thread they run on, for the same reason as the chunks. A loop takes at most that many participants, never more than it has chunks: the caller and the workers of the lowest indices, each knowing its index, so they join without contending for seats. Waiting workers poll a generation word on its own cache line (the claims do not take it from them) for a few microseconds, then alternatesched_yield()and short polls for about a millisecond (on a busy core a thread with work runs instead of the poller), then sleep on a condition variable; loops a worker is left out of do not restart that wait, so the workers no loop of a decoding step takes sleep and leave their cores, or the other hardware thread of a core, to those that run, and a loop wakes the sleepers only when it takes one of them. The caller waits for the last chunks the same way. Two limits for a busy machine (2026-11-11): a row loop's slices take at least 16384 values, so a decode step's small ops (the query rotation, 32 heads of 64 values; the cache copies, 8 heads) run on the calling thread without waking the team (about a third of a Llama-3.2-1B token's loops); and a loop takes no more participants than the cores other programs leave free: on Linux, the threads other programs keep runnable (/proc/stat'sprocs_runningless this process's running threads), averaged over samples taken at most every 100 ms by the thread starting a loop (OKLM_CPU_FREE_CAP=0turns it off). Counting runnable threads, not busy time, keeps the measure independent of OKLM's own threads, which take the others' time while they run but not their place in the queue. The kernels whose split or layout depends on the thread count (the decode tiles, the packed copy one thread reads, attention's split) go by the threads available now (oklm_cpu_threads_now()); no result depends on either. - Repacked weights: the matrices of the layers on the CPU get a copy
of eight interleaved rows per block (
cpu_pack.h), read by the products of two or more activation rows; one-row products (decode) read it with one thread, at any thread count when it is no larger than the file layout (Q8_0 and Q5_0, whose copies keep the file's half-precision scales and Q5_0's nibbles beside a word of fifth bits per row), and with two when it is at most 10 % larger (Q4_K): one thread waits on each row's chain of block sums on the file layout, while the packed kernels run eight rows side by side; more threads are bound by memory, where a larger copy loses. Every layout gives the same bits. Latent attention's per-head matrices (a stack[K, N, heads]read by batched products) get one copy per head, counted in place of the mapped stack (GLM-4.7-Flash prompts 73 to 78 t/s at four threads). A MoE layer's routed experts (a stack[K, N, E]) get one copy per expert when the model repacks them (OKLM_CPU_PACK_EXPERTS, by default when the copies fit in memory beside the mapping, never with an expert cache): the indexed product reads it for prompts and for decode steps with one or two threads. Repacking runs on the worker team (GLM-4.7-Flash's 17 GiB of experts in a few seconds). On Linux the copies are cut one after the other from chunks aligned on 2 MiB and advised as huge pages (2 to 32 MiB a chunk, a larger copy its own;OKLM_CPU_PACK_HUGE=0gives each copy its own allocation): a decoding step reads every copy once, and small mappings of 4 KiB pages cost SmolLM2-135M 3 to 5 %. - A quantized product rounds its activation rows to Q8_0 or Q8_K
blocks first (
oklm_cpu_quantize_act()): with AVX2 at the AVX2 and AVX-512 levels, the bits ofoklm_quantize_row(). - Profiling:
OKLM_CPU_PROFILE=1logs the time spent per op (and per weight type for matrix products) at shutdown. - Determinism: each output element is computed by one thread, by the same code whatever chunk holds it and whichever thread claims that chunk; results depend neither on the thread count nor on the timing.
Performance numbers are in benchmarks.md.