Mixture-of-experts models
How OKLM runs a mixture-of-experts (MoE) model, whose feed-forward
layers route each token to a few of many experts: the grouped expert
products, the expert cache that streams routed experts from the disk on
the CPU, --n-cpu-moe, which keeps experts in host memory
beside a device, and the device expert tier, with their measurements.
The rest of inference is in inference.md;
which MoE families load is in compatibility.md.
Status: maintained.
Grouped expert products
MoE layers run on either backend, routing included
(oklm_moe_select(), oklm_moe_weights()). On
the CPU every batch, decode and prompts alike, multiplies each (token,
rank) pair by its own expert through an indexed product
(oklm_matmul_id()), which groups its rows by expert so a
batch reads each chosen expert once, and nothing is read back: 1.2 times
faster for 16 tokens on a Qwen3-30B-A3B-like layer, prompts of 128 to
2048 tokens 1.2 to 1.4 times faster end to end on Granite-3.1-1B-A400M
than one product per expert (see benchmarks.md). On a device the indexed
product does the same: with few rows (n x experts per token <= expert
count: decode, speculative verification) one work-group per row and
expert (mv_<type>); with more, a kernel groups the
rows by expert on the device (moe_group, no read-back) and
one launch of the grouped tile kernels (gi_<type>,
wi_<type> with the matrix units) multiplies each
expert's weights by its rows, each weight tile read once for up to 64 of
them (opencl.md); Vulkan sends the
grouped rows to its prompt products' tiles from 16 rows an expert (vulkan.md). Until 2026-11-08
larger device batches read the chosen experts back and ran one product
per expert: OLMoE-1B-7B prompts went from 4175 to 8614 t/s, Granite from
5585 to 15341 (matrix units). On the CPU both ways give the same bits
(each output element is the same one-row dot product of the same
quantized rows); OKLM_MOE_EXPERT_LOOP=1 runs CPU batches
one expert at a time, for comparison.
On the CPU the routed experts also get a repacked copy, eight rows of
an expert interleaved per block (cpu_pack.h), as the dense
matrices do, when the copies and the mapped experts fit in the available
memory with the memory plan's margin and no expert cache is asked for
(the cache's slots hold the file layout, read from disk). The grouped
product then runs the packed kernels for prompts (OLMoE-1B-7B 56 to 84
t/s at one thread, 202 to 289 at four; Granite-3.1-1B-A400M 139 to 195
and 490 to 685) and for decode steps with one or two threads (OLMoE 21.1
to 25.5 t/s at one); from three threads decode steps read the file
layout, smaller, which measured 2 to 6 % faster. The copies take about
as much as the experts again (GLM-4.7-Flash: 17.4 GiB on top of 15.6 GiB
mapped), counted so by oklm plan.
OKLM_CPU_PACK_EXPERTS=0 turns them off, =1
makes them whatever the memory says. The bits are the file layout's
(test_ops checks the grouped product on both).
Expert cache
oklm run|chat|ppl|serve -m moe.gguf --backend cpu --expert-cache GB|auto [--pin-gb GB] [--expert-usage FILE]
[--[no-]expert-direct] [--expert-stats] [--expert-loaders N] [--expert-predict off|last|router]
[--expert-pack FILE] [--expert-mirror FILE]
oklm expert-pack -m moe.gguf [-o FILE]
On the CPU, a MoE model's routed experts (ffn_gate_exps,
ffn_up_exps, ffn_down_exps) can stay on disk:
with --expert-cache they are no longer read through the
mapping but read into a cache of the given size (GiB; auto
takes the largest cache oklm plan finds beside the dense
part, at most every routed expert). The dense tensors (attention, shared
experts, embeddings, norms, routers) stay mapped as before. In the
library this is oklm_context_params::expert_cache (bytes,
-1 for auto) with expert_pin, expert_usage,
expert_direct, expert_loaders and
expert_predict, and
oklm_expert_cache_stats_get() /
oklm_expert_cache_residency() /
oklm_expert_cache_save_usage().
How it works:
- At context creation each MoE layer's three stacked expert tensors
are located in the GGUF file(s): file, offset of expert 0, bytes per
expert. A GGUF stores gate, up and down apart, each expert contiguous
inside each, so an expert is three
preads (oklm_file_pread(), the platform layer of architecture.md: the cache has no system call of its own). The model is loaded without read-ahead (OKLM_MODEL_LOAD_NO_READAHEAD, asoklm plandoes) and the experts' mapped pages are dropped (MADV_DONTNEED), so the mapping does not hold the experts beside the cache. - A slot is one aligned buffer holding one expert's three matrices.
Each MoE layer gets an equal share of the cache
(
GB / (MoE layers x expert bytes)slots), at least one, at most one per expert. - In a MoE layer the router runs as before; its chosen experts are in
host memory. The experts the batch needs (the union over its tokens) run
in groups that fit the layer's slots: the resident ones first, then
missing ones read (in parallel over the experts) into free or least
recently used slots the group does not use. Each group runs the grouped
indexed products with a table expert -> slot address
(
oklm_matmul_id_ex(), thew_expertsfield of the op descriptor), which writes only that group's rows; then the next group. With enough slots for the batch there is one group; with fewer (one slot and eight experts per token, a long prompt and its every expert) the batch is cut by expert, not grown. The products read the same bytes with the same kernels, so the logits are the same bit for bit as the resident run (test_model: a 160-token prompt and 24 greedy decode steps on the nine tiny MoE fixtures, with one slot per layer and direct reads, and with half the experts). - Routing counts (per layer and expert) accumulate in a usage file,
--expert-usage FILEor~/.cache/oklm/usage/<model hash>.jsonby default (the hash is of the tensor table), read at creation (a first run has none, which is not an error and logs nothing) and written at the context's end and every minute.--pin-gb GBpins the most routed experts of that file, read at creation and never evicted, on top of--expert-cache. --expert-directreads withO_DIRECT(F_NOCACHE on macOS, FILE_FLAG_NO_BUFFERING on Windows): the page cache is bypassed, each read covering the 4 KiB-aligned range around the expert. Where the file system refuses, buffered reads are used with a warning.--expert-statsprints at exit the hits (experts a batch needed that were in memory), misses (read), bytes read and the time spent reading, in total and per layer;oklm serveadds anexpert_cacheobject to/api/stats.- Loader threads (
--expert-loaders N, default 4 with the cache or whatoklm tune --expert-cachekept, below, 0 for the reads made by the forward pass itself as above): the missing experts' reads are queued, one job per matrix, and the layer consumes its slots as they fill. The experts in memory compute first while the loaders read the others; the next round waits for every read in flight (a round of products has a fixed cost) unless experts wait for a slot, in which case it computes what is ready, which frees those slots for the next reads. Each output row is still written once by the same kernel: the logits do not change. - Read-ahead (
--expert-predict last|router, with loaders): once a layer's routing is known, the experts the next MoE layer is predicted to choose are queued behind the layer's own reads, into that layer's free or least recently used slots (the predicted ones in memory are kept from eviction).lastpredicts the experts that layer chose in the last batch: decoding reuses routing a lot, but the LRU slots already hold them, so it reads little.routerapplies the next layer's router to this layer's FFN input, rescaled from this layer's norm weights to the next one's, and takes each token's top experts (votes over a batch, at most the layer's slots): the router look-ahead.--expert-statsreports the time the forward pass waited for reads (beside the loaders' read time, which overlaps), the experts read ahead and used, and the prediction's recall (experts predicted that the batch chose, over the experts it chose). - Off by default. On a device backend it covers the layers whose
routed experts the CPU computes: the CPU layers
(
--gpu-layersbelow the layer count) and those of--n-cpu-moe(below); with neither it is off, with a warning.OKLM_MOE_EXPERT_LOOPis ignored with the cache.
Granite-3.1-1B-A400M Q4_K_M (24 MoE layers of 32 experts, 8 per
token; 697.5 MiB of routed experts, 930 KiB each), CPU, the page cache
warm (dropping it needs root) except with --expert-direct;
the output identical in every row:
| cache (of 697.5 MiB) | slots / layer | prompt t/s | decode t/s | hit rate | read | read time |
|---|---|---|---|---|---|---|
| off (mapped, resident) | - | 234 | 42.0 | - | - | - |
| 100 % (0.68 GiB) | 31 | 221 | 37.9 | 93.4 % | 836 MiB | 0.14 s |
| 50 % | 15 | 208 | 33.6 | 69.1 % | 3.8 GiB | 0.45 s |
| 25 % | 7 | 186 | 26.0 | 39.7 % | 7.4 GiB | 0.86 s |
| 10 % | 3 | 144 | 21.0 | 10.9 % | 10.9 GiB | 1.17 s |
100 %, --expert-direct |
31 | 158 | 37.2 | 93.4 % | 836 MiB | 1.33 s |
| 50 %, direct | 14 | 153 | 8.7 | 67.7 % | 4.0 GiB | 6.4 s |
| 25 %, direct | 7 | 132 | 4.8 | 39.7 % | 7.4 GiB | 12.1 s |
| 10 %, direct | 3 | 89 | 2.7 | 10.9 % | 10.9 GiB | 20.5 s |
The prompt is 731 tokens (two batches) then 64 greedy tokens, 4
threads (the machine shared with other jobs: 8 threads oversubscribed
it), best of 3. The hit rate counts every (batch, layer, expert) use: a
prompt batch needs nearly every expert of every layer, so with 3 slots a
layer runs in about eleven groups, each reading its experts; the decoded
tokens then miss on most of their 8 experts. With the page cache warm
the reads are memory copies (9 GB/s) and the cost is mostly the
products' smaller groups; with O_DIRECT they come from the
NVMe at about 550-600 MiB/s (three reads of about 300 KiB per expert,
one expert per thread), which is what a model larger than RAM sees:
there decode is bound by the reads, the case for read-ahead (roadmap H4)
and the most used experts pinned. At 100 % a layer holds 31 of its 32
experts (the 64-byte alignment of the three matrices takes the last
slot's room), hence the misses beyond the first fill. Direct slots keep
room for the 4 KiB alignment, one slot fewer per layer at 50 %. (That
table was measured with the reads made by the forward pass itself,
--expert-loaders 0 today.)
Expert pack
A GGUF keeps a layer's gate, up and down experts in three tensors, so
an expert is three reads of a third of its bytes each.
oklm expert-pack -m moe.gguf [-o FILE]
(oklm_expert_pack_write()) writes the routed experts again,
each expert's three matrices one after the other (64-byte aligned) in
one record of a multiple of 4 KiB: --expert-pack FILE
(oklm_context_params::expert_pack) then reads a missing
expert in one read, direct or not, straight into a slot of the same
layout (one loader job instead of three). The pack holds the bytes the
products read, so the logits are the same bit for bit
(check_expert_pack in test_model, every MoE
fixture); it takes as much disk again as the routed experts (Granite:
698 MiB beside the 784 MiB model), and the model's file is still needed
for the dense part.
The file: a 48-byte header (OKLMEXP1, version 1 and the
record alignment 4096 as two 32-bit values, the model's hash (that of
its tensor table, as the usage file uses), the layer count with the
prediction blocks, the expert count, the layers packed), then 72 bytes a
packed layer (layer, offset of expert 0's record, record size, the three
matrices' sizes, their offsets in the record), all little-endian 64-bit,
padded to 4 KiB; then each layer's records, expert by expert. A pack of
another model, or whose sizes differ from the model's, is refused with a
warning and the experts are read from the model's file.
--expert-mirror FILE (expert_mirror) names
a copy of the file the experts come from (the pack with
--expert-pack, else the model when it is one file) on
another drive: every other read goes to it (--expert-stats
counts the read calls and those of the mirror; the test checks half go
there and the logits do not change). It must have the same size and
first 4 KiB, else it is ignored with a warning.
Measured (2026-11-11), Granite-3.1-1B-A400M Q4_K_M, CPU, 4 threads,
--expert-direct (the reads from the NVMe, not the page
cache), a 33-token prompt then 64 greedy tokens, no read-ahead, best of
3 (the three runs within 1 %):
| cache | loaders | from | prompt t/s | decode t/s | read calls | waited for reads |
|---|---|---|---|---|---|---|
| 10 % (0.07 GiB, 3 slots a layer) | 0 | the model's file | 32.2 | 4.2 | 33882 | 15.3 s |
| 10 % | 0 | the pack | 68.1 | 9.6 | 11294 | 6.1 s |
| 10 % | 4 | the model's file | 47.8 | 5.8 | 33882 | 10.2 s |
| 10 % | 4 | the pack | 97.4 | 12.9 | 11294 | 4.1 s |
| 25 % (0.17 GiB, 7 slots) | 0 | the model's file | 39.0 | 6.3 | 23673 | 10.2 s |
| 25 % | 0 | the pack | 83.8 | 16.4 | 7891 | 3.4 s |
| 25 % | 4 | the model's file | 47.5 | 7.9 | 23871 | 7.7 s |
| 25 % | 4 | the pack | 130.3 | 17.0 | 7950 | 3.0 s |
The same experts are read (the hits and misses equal), in a third of the calls: without loaders the reads ran at 1670 to 2090 MiB/s from the pack against 670 to 710 MiB/s from the model's file (one 1 MiB read against three of a third of it, plus the aligned ranges direct reads of the file add around each matrix), so decoding is 2.1 to 2.6 times faster. The mirror is not measured: this machine has one NVMe (the other drives are hard disks, slower than it).
Loader threads and read-ahead, the same model with
--expert-direct (the reads from the NVMe), a 979-token
prompt then 64 greedy tokens, 4 compute threads, best of 3 (the machine
shared with other jobs at a load of 20 to 45 on 16 cores, so the
throughputs scatter; the time the forward pass waited for reads is the
steadier figure); the output identical in every row:
| cache, reads | prompt t/s | decode t/s | hit rate | read | waited | read ahead used | recall |
|---|---|---|---|---|---|---|---|
50 %, inline (--expert-loaders 0) |
135 | 2.9 | 64.8 % | 4.3 GiB | 7.9 s | - | - |
| 50 %, 2 loaders | 131 | 2.4 | 64.9 % | 4.3 GiB | 4.4 s | - | - |
| 50 %, 4 loaders | 124 | 2.2 | 64.9 % | 4.3 GiB | 2.5 s | - | - |
50 %, 2 loaders, last |
112 | 2.0 | 64.7 % | 4.8 GiB | 3.4 s | 79 % of 568 | 45.2 % |
50 %, 2 loaders, router |
143 | 3.7 | 87.0 % | 5.5 GiB | 3.6 s | 89 % of 4377 | 84.9 % |
50 %, 4 loaders, router |
151 | 4.5 | 87.0 % | 5.5 GiB | 2.0 s | 89 % of 4377 | 84.9 % |
| 25 %, inline | 111 | 1.7 | 37.9 % | 7.6 GiB | 13.6 s | - | - |
| 25 %, 2 loaders | 136 | 1.9 | 37.4 % | 7.7 GiB | 8.1 s | - | - |
| 25 %, 4 loaders | 171 | 5.1 | 37.4 % | 7.7 GiB | 7.8 s | - | - |
25 %, 2 loaders, last |
175 | 2.6 | 38.2 % | 8.5 GiB | 11.1 s | 58 % of 1044 | 41.3 % |
25 %, 2 loaders, router |
144 | 3.2 | 75.3 % | 9.3 GiB | 7.7 s | 89 % of 6977 | 77.7 % |
25 %, 4 loaders, router |
196 | 7.0 | 75.3 % | 9.3 GiB | 6.3 s | 90 % of 7055 | 77.7 % |
| 10 %, inline | 97 | 1.7 | 10.8 % | 11.0 GiB | 23.6 s | - | - |
| 10 %, 2 loaders | 84 | 2.8 | 10.8 % | 11.0 GiB | 14.1 s | - | - |
| 10 %, 4 loaders | 80 | 4.9 | 10.8 % | 11.0 GiB | 9.8 s | - | - |
10 %, 2 loaders, last |
82 | 3.4 | 14.7 % | 13.6 GiB | 15.2 s | 53 % of 3612 | 15.9 % |
10 %, 2 loaders, router |
84 | 3.7 | 25.3 % | 12.2 GiB | 13.2 s | 91 % of 3440 | 26.2 % |
10 %, 4 loaders, router |
95 | 4.6 | 25.3 % | 12.4 GiB | 10.3 s | 90 % of 3623 | 26.2 % |
The loaders hide part of the reads behind the products (the forward
pass waits 45 % less with 2, 60 to 70 % less with 4: the NVMe serves
several reads at once, the inline reads use one compute thread per
expert). The last prediction is right only 41 to 45 % of
the time on this model and, the LRU slots holding most of those experts
already, reads little. The router look-ahead predicts 78 to 85 % of the
next layer's experts (colibrì measured 71.6 % on GLM-5.2) and nine reads
ahead in ten are used: the hit rate goes from 65 to 87 % at 50 % and
from 37 to 75 % at 25 %, for 25 % more bytes read. At 10 % (3 slots for
8 experts a token) a prediction is capped at the layer's 3 slots, hence
its 26 % recall (precision 90 %).
Decode alone, measured again with the configurations interleaved (an 18-token prompt then 64 greedy tokens, best of 5, direct reads):
| cache | inline | 2 loaders | 2 loaders, router |
4 loaders, router |
|---|---|---|---|---|
| 50 % | 8.6 t/s | 8.6 t/s | 8.8 t/s | 13.0 t/s (1.5x) |
| 25 % | 5.7 t/s | 5.8 t/s | 5.4 t/s | 7.9 t/s (1.4x) |
Two loaders keep about as many reads in flight as the inline reads did (one compute thread per expert), so they hide the reads behind the products without reading faster; the reads ahead then compete with the batch's own reads for the two loaders. With 4 loaders the NVMe serves the batch's misses and the reads ahead together, and decode gains 40 to 50 %. Hence 4 loaders are the default with the cache.
OLMoE-1B-7B-0125-Instruct Q4_K_M (16 MoE layers of 64 experts, 8 per
token; 3.63 GiB of routed experts, 3.6 MiB each),
--expert-direct, a 1000-token prompt then 62 greedy tokens,
4 compute threads, best of 3 (the machine's load lower than for Granite,
5 to 15); the output identical in every row:
| cache, reads | prompt t/s | decode t/s | hit rate | read | waited | read ahead used | recall |
|---|---|---|---|---|---|---|---|
| 50 %, inline | 117 | 10.8 | 71.1 % | 10.2 GiB | 6.4 s | - | - |
| 50 %, 2 loaders | 134 | 12.1 | 71.0 % | 10.3 GiB | 4.0 s | - | - |
| 50 %, 4 loaders | 142 | 12.9 | 71.1 % | 10.2 GiB | 3.5 s | - | - |
50 %, 2 loaders, router |
133 | 12.6 | 85.3 % | 13.7 GiB | 4.1 s | 90 % of 2393 | 79.1 % |
50 %, 4 loaders, router |
137 | 13.9 | 85.3 % | 13.7 GiB | 3.5 s | 90 % of 2393 | 79.1 % |
| 25 %, inline | 110 | 6.2 | 47.1 % | 18.7 GiB | 10.8 s | - | - |
| 25 %, 2 loaders | 122 | 6.2 | 47.0 % | 18.7 GiB | 7.8 s | - | - |
| 25 %, 4 loaders | 128 | 7.0 | 47.1 % | 18.7 GiB | 6.9 s | - | - |
25 %, 2 loaders, router |
112 | 6.5 | 75.5 % | 22.4 GiB | 7.1 s | 86 % of 3876 | 73.9 % |
25 %, 4 loaders, router |
84 | 5.3 | 75.5 % | 22.5 GiB | 3.5 s | 86 % of 3883 | 73.9 % |
On OLMoE the router look-ahead predicts 74 to 79 % of the next layer's experts and lifts the hit rate from 71 to 85 % (50 %) and from 47 to 76 % (25 %); 4 loaders with it decode 1.3x the inline reads at 50 %. At 25 % its wait for reads is a third of the inline one, but its throughput fell in all three runs of the last row while the prompt rate fell with it (84 t/s against 110 to 128): the machine's other jobs, not the reads, which the waits measure; the row is kept as measured.
OLMoE-1B-7B-0125-Instruct Q4_K_M (16 MoE layers of 64 experts, 8 per token; 3.63 GiB of routed experts, 3.6 MiB each), CPU, 4 threads on a machine shared with other jobs (load 20 to 35), page cache warm, a 682-token prompt then 64 greedy tokens; the output identical in every row:
| cache (of 3.63 GiB) | slots / layer | prompt t/s | decode t/s | hit rate | read | read time |
|---|---|---|---|---|---|---|
| off (mapped, resident) | - | 97 | 13.3 | - | - | - |
| 50 % (1.815 GiB) | 31.5 (504 in all) | 80 | 8.9 | 68.5 % | 11.4 GiB | 1.73 s |
| 25 % (0.908 GiB) | 15.5 (248 in all) | 82 | 9.8 | 48.2 % | 18.7 GiB | 2.39 s |
The speeds are within the noise of the shared machine (the 25 % decode above the 50 % one); the hit rates and reads are exact.
Tuning the streaming settings
oklm tune -m moe.gguf --backend cpu --expert-cache GB|auto [--streaming] [-n 256] [--gen 64] [--budget 300] [--force] [--cache-dir DIR]
Which loaders, read-ahead and read mode are fastest depends on the
disk, the RAM (whether the page cache holds the file) and the model, so
oklm tune with --expert-cache (or
--streaming, the auto size) measures them for
this machine, this model and that cache size
(oklm_tune_stream()): --expert-loaders 0, 2,
4, 8, --expert-predict off, last,
router and --expert-direct off, on, one at a
time from the defaults (4 loaders, no read-ahead, buffered reads) until
no change helps or the budget is spent. A run is a context of its own (a
cold cache), a 256-token prompt (the tokenizer's text, repeated) and 64
greedy tokens, timed without the context's creation; a candidate's time
is the median of three runs (one when it is 1.5 times the best so far),
and its output is the hash of every logit and token of the run, which
must equal the defaults' or the candidate is rejected. A setting kept
must be 3 % faster; it is then timed against the defaults once more,
their runs interleaved so the machine's load weighs on both alike, and
kept only when still faster with the same output. The result,
stream-<key>.json beside the kernel tuning
(~/.cache/oklm/tuning/), is keyed by the model's shapes and
types, the CPU, its threads, the cache size in bytes and the disk
holding the model (st_dev and statfs type of
its file), not the backend, and kept 90 days.
Contexts with that cache size apply it to the settings not given:
oklm_context_params::expert_loaders,
expert_predict and expert_direct default to -1
(the tuned value, else 4 / off / off;
oklm_tune_stream_resolve()), a flag given always wins
(--no-expert-direct gives buffered reads explicitly), a
tuned read-ahead is dropped with --expert-loaders 0, and
OKLM_TUNE=0 turns it off with the kernel tuning. The cache
size is never tuned: it is the user's, and part of the key. The settings
are the host-side cache's, so a device context applies the same entry to
the expert cache that serves its CPU experts (--n-cpu-moe)
or the misses of the device expert tier (--expert-gpu with
--expert-cache).
Measured on this machine (NVMe, 124 GiB of RAM so the page cache holds the files, 4 threads; the machine shared with other jobs at a load of 10 to 21 on 16 cores), with the times of the defaults' run (prompt and decode) and what was kept:
| model, cache | defaults | best other candidate | --expert-direct |
kept |
|---|---|---|---|---|
| Granite-3.1-1B-A400M, 50 % (0.34 GiB) | 1.7-3.4 s | within the noise (0, 2, 8 loaders) | 3.6x, 2.0x slower | defaults |
| Granite-3.1-1B-A400M, 25 % (0.17 GiB) | 4.7 s interleaved | 2 loaders, 5.1 s interleaved | 2.4x slower | defaults |
| OLMoE-1B-7B, 50 % (1.8 GiB) | 6.9 s interleaved | 2 loaders, 6.9 s interleaved | 1.3x slower | defaults |
| OLMoE-1B-7B, 25 % (0.9 GiB) | 8.0 s | router, 10.6 s |
2.5x slower | defaults |
The defaults stay: with the file in the page cache a buffered read is
a memory copy, so the loaders and the read-ahead have little to hide and
O_DIRECT (which goes to the NVMe each time) is 1.3 to 3.6
times slower; the gain over the defaults is 1.00x on all four. Twice a
candidate first timed faster (8 loaders 1.23x at Granite 25 %,
last 1.05x at OLMoE 50 %, before the interleaved check
existed) and twice the interleaved check rejected a win of the search (2
loaders, 1.23x at Granite 25 % and 1.5x at OLMoE 50 %): on a loaded
machine one sequence of medians is not enough, hence that check. A model
larger than RAM, or a machine whose page cache cannot hold the experts,
is where direct reads, 4 loaders and the router look-ahead win (the
tables above, measured with --expert-direct), and where the
tuner is meant to find them.
Experts on the CPU
oklm run|chat|ppl|serve|plan -m moe.gguf --backend opencl --n-cpu-moe N|all [--gpu-layers N] [--expert-cache GB|auto]
On a device backend, --n-cpu-moe N (as llama.cpp's flag;
alias -ncmoe) keeps the routed experts
(ffn_gate_exps, ffn_up_exps,
ffn_down_exps) of the first N layers in host memory and
runs them on the CPU, while everything else of those layers (attention,
norms, the router, the shared experts) goes to the device with the other
device layers; all does it for every layer. Experts are
most of a MoE model's weights but each token reads a few of them, so a
MoE model larger than the device keeps its attention on the device
rather than whole layers on the CPU (--gpu-layers alone).
In the library it is oklm_context_params::n_cpu_moe (-1 for
all), oklm_n_cpu_moe() and
oklm_context_mem::n_cpu_moe.
How it works:
- Placement: the device copy of such a layer gets every tensor but the
three expert stacks, whose headers stay the mapped host tensors
(
dev_place_layer()). With--gpu-layersleft out, the count of layers that fit uses those slimmer layers, so more fit; with it, the N layers are counted from the first, whether on the device or not (a CPU layer runs its experts on the CPU anyway). One placement per model: the model's first device context fixes it. - A device layer whose experts are on the host crosses devices in
moe_ffn(): the router, the expert choice and the weights run on the device as before; then the chosen experts and their weights (n x Keach) and the normalized rows (n_embd x n) read back in one round trip (the backend'sdownload_many: OpenCL queues the first two reads without waiting and blocks on the last), the shared expert queued on the device and started (clFlush, the backend's newflushentry) so it runs while the CPU works, the CPU's grouped indexed products over the batch (oklm_matmul_id, gate, up, SwiGLU, down: each expert's rows together, each expert read once per batch), the weighted sum of each token's ranks on the CPU, and one write of that sum (n_embd x n), which the in-order queue runs after the shared expert, which is then added on the device. Per layer and step: about4 n (n_embd + 2 K)bytes down and4 n n_embdup (Granite: 8 KiB for one token). --expert-cachecomposes: on a device context the cache covers the layers whose routed experts the CPU computes (the CPU layers and these), the read-back routing feeding it as on the CPU, the loaders and--expert-predicttoo. The router look-ahead of such a layer runs on the device when the next MoE layer's router is there (2026-11-11): the layer's normalized rows multiplied by the ratio of the two layers' norm weights (uploaded once per layer), the next router applied, and the logits (n_expert xthe batch) read back in the same round trip as the routing, into the host scratch the look-ahead votes on; otherwise it runs the next router on the host as on a CPU context.--expert-stats' counters say how many look-aheads the device computed (ahead_device). Granite with--n-cpu-moe all, a 25 % cache, 4 loaders, direct reads: prompts 513 -> 526 t/s (the host product per layer and batch gone), decoding 8.1 t/s either way (bound by the reads); OLMoE 52.7 / 3.4 t/s either way.oklm plan --n-cpu-moe Ncounts those experts in host memory and not on the device (the device line says how many layers' experts are on the CPU), plus the host buffers the crossing uses.- Checked in
make test(test_model) on the eight tiny MoE fixtures (glm4moe included), with the experts of the first layer and of every layer on the CPU, every layer on the device or one on the CPU: the references' greedy tokens and logits within 1e-4 (3e-5 at worst), a 160-token prompt and 24 greedy steps giving the device-only run's tokens (logits within 1e-5), the expert cache (one slot per layer) giving the same bits as without it;test_multi_devicewith the layers spread over two device instances;test_mem_planthe projection.
Granite-3.1-1B-A400M Q4_K_M (24 MoE layers, 32 experts of 930 KiB, 8
per token; 697.5 MiB of its 782 MiB are routed experts) on the AMD
gfx1100 with the i7-11700K; an 836-token prompt (two batches of 512 and
324) then 64 greedy tokens, tokens per second, best of 3. The machine
was shared with other jobs throughout (load average 30 to 35 on 16
hardware threads), which hurts the CPU side most: these runs use 2 CPU
threads, the fastest setting under that load (4 and 8 threads were
slower and erratic). The greedy text follows the device-only run's for
its first dozen tokens, then takes another wording, as the CPU backend's
and the --gpu-layers 3 run's do at the same place: the
CPU's products round the activations to 8 bits, the device's do not.
| placement | layers on the device | prompt t/s | decode t/s |
|---|---|---|---|
| CPU backend | 0 | 153 | 40.0 |
| device | 24 | 2 499 | 135.9 |
--n-cpu-moe 6 |
24 | 682 | 46.6 |
--n-cpu-moe 12 |
24 | 398 | 31.4 |
--n-cpu-moe 18 |
24 | 280 | 30.5 |
--n-cpu-moe all |
24 | 221 | 27.5 |
640 MiB device (OKLM_GPU_MEM_MB=640),
--gpu-layers fit |
3 | 182 | 44.3 |
640 MiB device, --n-cpu-moe all |
16 | 193 | 26.7 |
640 MiB device, --n-cpu-moe 12 |
3 | 154 | 34.7 |
Per crossing, timed in the code (since removed) over a short prompt
and 64 decode steps: the two further reads 0.13 ms and the write of the
sum 0.24 ms, against 0.77 ms of CPU expert products, all inflated by the
load (an idle bus round trip is tens of microseconds). On this model the
crossing does not pay: Granite fits the device whole, its experts are
small (each token reads 7.3 MiB of them per layer, which the CPU reads
in well under a millisecond), and its dense part is mostly the 49k-row
tied head and the logits buffers, so on the 640 MiB device only 16 of
the slimmed layers fit beside them; the model it is for is one whose
experts exceed the device by far while its attention and dense weights
fit (Qwen3-30B-A3B, DeepSeek-Coder-V2-Lite at larger quantizations),
which --gpu-layers alone would leave mostly on the CPU. No
such model is in models/ at the time of writing (only
Granite among the MoE files), so that measurement is still to make.
Two follow-ups measured on 2026-11-11 (Granite, quiet machine, 8 CPU threads, a 948-token prompt, best of 3):
- One round trip for the routing and the rows instead of three
(
download_many): decoding with every layer's experts on the CPU 72.2 t/s before, 72.8 after, within the runs' spread: on an idle bus a read costs tens of microseconds, against the CPU expert products' 0.77 ms. Kept (one wait instead of three, and the backend's read errors now reported:cl_download_locked()dropped its read's status). - Streamed CPU layers (
--gpu-stream) still carry their routed experts to the device, even within--n-cpu-moe. Leaving those experts in host memory (a streamed layer then crossing as a device layer does) shrank the two slots from 33.1 to 1.9 MiB but cut prompts with--gpu-layers 12 --gpu-stream 32 --n-cpu-moe 12from 4 767 to 2 855 t/s: for a batch of hundreds of tokens, copying a layer's experts to the device beats computing them on the CPU, and on a larger MoE more so (more tokens per expert). Not kept; streamed layers stay whole.
Device expert tier
oklm run|chat|ppl|serve -m moe.gguf --backend opencl|vulkan --expert-gpu GB|auto [--n-cpu-moe N|all] [--expert-usage FILE] [--expert-cache GB|auto] [--expert-stats]
On Vulkan the tier works the same way, its refills on the instance's transfer queue (vulkan.md).
--expert-gpu GB keeps the hottest routed experts of
every layer whose experts are in host memory (--n-cpu-moe;
every layer's when --expert-gpu is given without it) in
device memory, an equal share of GB per layer; auto takes
the device's free memory once the context is created (the dense part,
the KV cache and the buffers placed), less the layer fit's margin (8 %,
at least 256 MiB); with OKLM_GPU_MEM_MB the memory in use
is taken from that size. The rest stay where they are: resident in RAM,
or read from disk through --expert-cache. In each step the
device computes the (token, rank) pairs of the experts it holds while
the CPU computes the others, so experts are placed by heat rather than
by layer (--n-cpu-moe all places them by layer: all on the
CPU). In the library it is oklm_context_params::expert_gpu
(-1 for auto), oklm_expert_gpu_stats_get() and
oklm_expert_gpu_residency(); --expert-stats
prints the tier's counters and /api/stats has them under
expert_gpu.
How it works (src/model/expert_gpu.c):
- Slots: per layer, device stacks of the three expert matrices (gate, up, down) with S slots, plus one slot of zeros while S is below the expert count. A slot holds one expert's bytes as the file stores them, so the device's products read the weights a device copy would.
- Heat: the routing counts of the usage file (the expert cache's,
~/.cache/oklm/usage/<hash>.jsonor--expert-usage), then every routing the tier sees, halved at each refill so recent routing weighs more. With counts the hottest experts are uploaded at creation; without them the tier starts empty and fills after 4 steps. Without--expert-cachethe tier writes the counts back at the end, so the next run starts filled. - A step, per such layer: the router, the expert choice and the
weights run on the device and are read back with the normalized rows (as
--n-cpu-moedoes). Each pair whose expert the device holds is mapped to its slot, the others to the slot of zeros (their products are exact zeros); the mapped ids go up and the device runs its indexed products over the slots and their weighted sum (a batch with more pairs than experts sorts the held pairs by slot instead, one product per slot, as device batches do), then the shared expert, and starts them. Meanwhile the CPU runs the grouped indexed products over the other pairs only (a table without the held experts' addresses, or the expert cache told to skip them: its loaders never read them and its read-ahead never predicts them) and their weighted sum, which is written up and added to the device's. A step whose pairs are all held costs the CPU nothing; one with none held runs no device product. - A layer whose tier holds all its experts places expert e in slot e and runs as a device layer: nothing is read back.
- Refill: every 32 steps, or after 8 when the hit rate fell below 80 %
of the previous period's, each layer, after the CPU's share, moves the
hottest host experts into free slots and swaps the coldest held ones for
host experts at least 1.25 times as hot, at most an eighth of the slots
per refill. The uploads are queued after the layer's device work, so a
step never sees a slot change under it; on the device's transfer queue
when it has one (OpenCL: a second command queue; Vulkan: a copy engine
or a second compute queue, with timeline semaphores) and the context
streams no weights (
OKLM_EXPERT_GPU_XFER=0for the device's queue): the copies wait for the work queued before them, run while the device computes the layers that follow, and the layer's next products wait for them. - Long batches (2026-11-11): a batch of 32 tokens or more
(
OKLM_EXPERT_GPU_STREAM=Nsets the threshold, 0 turns it off) runs every pair on the device. The pairs are sorted by expert; a held expert's rows multiply its slot, a missing one is first copied into one of four staging slots (the largest expert of each matrix, taken in turn; the in-order queue keeps a slot from being written while a product still reads it), so the CPU computes nothing for the layer.--expert-statscounts those pairs and copies. Not with--expert-cache, whose reads of the host experts it would bypass: the misses stay on the CPU there. - Checked in
make test(test_model,test_expert_gpu) on the nine tiny MoE fixtures: a 160-token prompt and 24 greedy steps with a tier holding no expert, half of them (filled from the first steps' routing, then a second pass that refills), all of them, half filled at creation from a usage file, and half with the misses read through an expert cache of one slot per layer (without loaders, with 4, with 4 and either read-ahead: the same bits each time) give the device-only run's greedy tokens, logits within 1e-5 (the CPU-vs-OpenCL tolerance is 1e-4).
Measured on the AMD gfx1100 (20 GB) with the i7-11700K, 4 CPU
threads, oklm run of an 821- to 850-token prompt then 64
greedy tokens, tokens per second, best of the rounds. The machine was
shared with other jobs throughout (load average 15 to 46 on 16 hardware
threads): the CPU side varied by 2x to 5x from one round to the next, so
the placements were run interleaved, round after round, and only large
differences mean something; 8 CPU threads were far slower under that
load (decode 1 to 4 tokens/s where 4 threads gave 8 to 24). "Cold": no
usage file, the tier fills from the routing of the first 4 steps (the
prompt's), "warm": filled at creation from the usage file of a run on
another text (README.md's first 2.6 kB). The hit rate counts the
prompt's pairs, most of the total (a cold tier holds nothing while the
prompt runs).
OLMoE-1B-7B-0125-Instruct Q4_K_M (16 layers, 64 experts of 3.6 MiB, 8
per token; 4.0 GiB), on a simulated 2 GiB device
(OKLM_GPU_MEM_MB=2048): the tier gets 519 MiB, 8 slots per
layer (12.5 % of the experts), 3 rounds:
| placement | prompt t/s | decode t/s | tier hits | uploads after creation |
|---|---|---|---|---|
| 20 GB device, the whole model | 2 166 | 227 | ||
2 GiB, --gpu-layers fit |
120 | 14.4 | ||
2 GiB, --n-cpu-moe all |
127 | 15.4 | ||
2 GiB, --n-cpu-moe all --expert-gpu auto, cold |
127 | 18.4 | 3.0 % | 128 (1.9 per step) |
| 2 GiB, the same, warm | 189 | 14.9 | 45 % | 17 (0.26 per step) |
GLM-4.7-Flash Q4_K_M (47 layers, 46 of them MoE with 64 experts of 5.4 MiB, 4 per token and a shared expert; 17 GiB), on a simulated 8 GiB device: the tier gets 5.0 GiB, 19 or 20 slots per layer (31 % of the experts), 2 rounds:
| placement | prompt t/s | decode t/s | tier hits | uploads after creation |
|---|---|---|---|---|
| 20 GB device, the whole model | 509 | 55.1 | ||
8 GiB, --gpu-layers fit |
36.5 | 6.8 | ||
8 GiB, --n-cpu-moe all |
60.8 | 4.9 | ||
8 GiB, --n-cpu-moe all --expert-gpu auto, cold |
46.8 | 9.4 | 5.8 % | 902 (13.7 per step) |
| 8 GiB, the same, warm | 93.5 | 12.5 | 63 % | 179 (2.7 per step) |
Granite-3.1-1B-A400M Q4_K_M (24 layers, 32 experts of 930 KiB), on the whole device with a usage file from another text, 3 rounds:
| placement | prompt t/s | decode t/s | tier hits |
|---|---|---|---|
| device | 3 884 | 200 | |
--n-cpu-moe all |
251 | 24.1 | |
--expert-gpu 0.15 (132 slots, 17 %) |
486 | 16.9 | 38 % |
--expert-gpu 0.3 (312 slots, 41 %) |
672 | 19.6 | 71 % |
--expert-gpu auto (every expert: device layers) |
2 110 | 142 | 100 % |
What the numbers say: prompts gain most, as the CPU's share of a
batch shrinks with the hit rate (GLM 1.5x, OLMoE 1.5x, Granite 2.7x over
--n-cpu-moe all when warm); decode gains when experts are
large and the hit rate high (GLM 2.5x over --n-cpu-moe all,
1.8x over --gpu-layers), and not with Granite's small
experts, where a step's fixed costs (the blocking transfers per layer,
one more than --n-cpu-moe: the mapped ids) outweigh the few
hundred kilobytes of expert reads saved, and the load swamped the
difference. A cold tier's first fill uploads every slot at once (GLM:
4.9 GiB in 2 to 9 s inside the fifth step), so a usage file from an
earlier run is worth keeping; the tier writes it back itself. Transfers
per decode step besides refills: the routing and rows down and the CPU's
sum up, as --n-cpu-moe, plus 4 n K bytes of
mapped ids.
Measured on 2026-11-11 (quiet machine, 8 CPU threads, no usage file, a 948-token prompt for Granite and 847 for OLMoE, then 64 greedy tokens; 256 for the decode rows; three runs each, the means):
| before | after | |
|---|---|---|
Granite, --expert-gpu 0.35 (half the experts),
prompt |
1 063 t/s | 2 060 t/s (the misses streamed) |
| Granite, the same, decode | 113 t/s | 119 t/s |
OLMoE-1B-7B, --expert-gpu 1.9 (half), prompt |
466 t/s | 512 t/s |
| OLMoE-1B-7B, the same, decode | 63 t/s | 73 t/s |
| Granite, decode, refills on the device's queue / transfer queue | 127.9 t/s | 129.1 t/s |
| OLMoE-1B-7B, decode, the same | 78.5 t/s | 80.2 t/s |
Streaming a long batch's missing experts doubles Granite's prompts (its experts are small: copying 30 MiB a layer beats computing 948 tokens' rows on the CPU) and adds 10 % to OLMoE's (experts four times larger). The decode rows of the first table rose too (the decode path itself did not change; not looked into further). Measured and not done: the mapped ids computed on the device (their write takes 12 us a layer, 3.5 % of a Granite decode step at most, against a gather launch of its own), and a per-layer share of the slots by heat (from the routing counts of a 256-token run, holding the hottest experts of the whole model instead of an equal share per layer would cover 49.1 instead of 48.8 % of Granite's routings at a quarter of the experts, 77.2 / 76.7 % at half, 82.3 / 81.9 % for OLMoE at half: under a point, measured on the very counts it would be chosen from).
Measured on this machine
One table per model, every streaming mode, on the machine's NVMe (CPU i7-11700K, 4 threads for the CPU rows, the AMD gfx1100 for the device rows, 124 GiB of RAM; 2026-11-08, shared with other jobs, so the throughputs have a few percent of noise and the hit rates and bytes are the exact figures). A 295-token prompt (one sentence repeated, so its routing is more concentrated than natural text's: the hit rates are higher than a real document would give) then 64 greedy tokens, best of 2 (the two runs agree to 2 % except where said). The text generated is byte for byte the same in every row of a model, the device tier's included.
What is emulated: no model larger than RAM is at hand, and the page
cache cannot be dropped without root (dd iflag=nocache left
the mapped runs as fast as the warm ones: the "cold" rows are not cold
and are not reported). So every streaming row uses
--expert-direct (O_DIRECT: each expert read
comes from the NVMe, never the page cache) with an expert cache of 25 %
or 50 % of the experts: the experts do not fit in the memory given to
them, which is the situation of a model larger than RAM, the dense part
staying mapped. The device rows set OKLM_GPU_MEM_MB (2 GiB
for OLMoE, 8 GiB for GLM) so the experts exceed the device, with
--n-cpu-moe all --expert-gpu auto. "Read" is the bytes the
experts' reads moved over the run, "waited" the seconds the forward pass
spent waiting for them out of the run's wall time.
OLMoE-1B-7B-0125-Instruct Q4_K_M (16 MoE layers of 64 experts, 8 per token; 3.63 GiB of experts):
| mode | prompt t/s | decode t/s | hit rate | read | waited / wall | bottleneck |
|---|---|---|---|---|---|---|
| plain mapping, page cache warm | 609 | 33.9 | - | - | - | the kernels |
| cache 50 % (1.8 GiB), direct, 4 loaders | 128 | 9.9 | 70.0 % | 9.8 GiB | 5.5 / 8.8 s | disk |
| cache 25 % (0.9 GiB) | 138 | 5.5 | 44.9 % | 17.8 GiB | 10.1 / 13.7 s | disk (410 MiB/s) |
25 % and --pin-gb 0.3 |
148 | 6.2 | 51.8 % | 15.9 GiB | 8.9 / 12.4 s | disk |
25 % and --expert-predict router |
90 | 3.9 | 77.4 % | 22.4 GiB | 15.2 / 19.7 s | disk (see below) |
25 %, pinned and router |
158 | 6.1 | 80.2 % | 20.0 GiB | 8.9 / 12.3 s | disk |
| 25 % and the device tier (2 GiB device, 519 MiB of tier) | 164 | 6.9 | 45.5 % cache, 17.9 % tier | 13.9 GiB | 8.1 / 11.0 s | disk |
| device tier alone (experts in RAM, no cache) | 495 | 45.8 | 18.8 % tier | - | - | the CPU's share of the experts |
GLM-4.7-Flash Q4_K_M (46 MoE layers of 64 experts, 4 per token and a shared expert; 15.6 GiB of experts), cache 25 % (3.78 GiB, 716 slots):
| mode | prompt t/s | decode t/s | hit rate | read | waited / wall | bottleneck |
|---|---|---|---|---|---|---|
| plain mapping, page cache warm | 132 | 10.4 | - | - | - | the kernels |
| cache 25 %, direct, 4 loaders | 36.1 | 6.3 | 76.0 % | 17.7 GiB | 6.0 / 18.4 s | the kernels, then disk |
25 %, --pin-gb 1.2 and router |
48.0 | 7.4 | 86.6 % | 17.9 GiB | 4.6 / 14.8 s | the kernels, then disk |
| 25 %, the device tier (8 GiB device, 5.0 GiB of tier) | 68 | 17.1 | 55.0 % cache, 81 % tier | 8.7 GiB | 3.4 / 8.2 s | the kernels |
Reading them:
- Streaming costs on OLMoE. Its experts are 3.6 MiB and 8 of 64 are used per token, so a cache of a quarter of them misses over half the time and decode falls from 34 to 5.5 t/s, 74 % of the wall time spent waiting for the reads at about 410 MiB/s (the NVMe's sustained rate for these 1 MiB-class reads, one per matrix; the loaders keep four in flight). Prompts fall less (609 to 138 t/s): a prompt batch needs most experts of a layer, which the cache serves in groups.
- The hit rate is bought with bytes. Pinning the most routed experts (from the usage file of earlier runs) lifts the hit rate by 7 points and cuts the bytes read by 11 %. The router look-ahead lifts it from 45 to 77 % (80 % with pinning) for 26 % more bytes read, predicting 73 % of the routed experts with 80 % of the reads ahead used. Both together decode 11 % faster than the cache alone, on the throughput; the waits do not fall in proportion because the loaders are already the NVMe's queue and the prompt's batches still read what they need.
- Read-ahead alone measured slower (3.9 against 5.5 t/s) and moved bytes at 254-298 MiB/s against 410-430 in every other row: the disk shared with other jobs at that moment (the same row repeated with pinning ran at 433 MiB/s). Taken as noise from the shared machine, not as a result; read it as no gain, not as a loss.
- GLM has more, smaller-per-token reads over 46 layers but only 4 experts per token and a high hit rate with this text (76 % at 25 %), so it is not disk-bound here: the waits are a third of the wall time and the rest is the CPU's products (decode 10.4 t/s mapped, 6.3 with the cache, which also runs without the repacked expert copies). Pinning and read-ahead take the waits from 6.0 to 4.6 s and decode from 6.3 to 7.4 t/s. On a text that routes less repetitively the hit rate would be lower and the disk would take over, as OLMoE shows.
- The device tier is the largest gain: on a device too small for the model, 5.0 GiB of tier on GLM holds 81 % of the routed pairs (warm from the usage file) and decodes 17.1 t/s, 2.7 times the cache alone and 1.6 times plain mapping, since the device's share is computed at device speed and the CPU is left a fifth of the pairs and a smaller read stream (8.7 GiB, half). On OLMoE the tier holds only 12.5 % of the experts (519 MiB): 18 % of the pairs, a 25 % gain over the cache alone while the disk is still the bottleneck; with the experts in RAM it decodes 45.8 t/s against 33.9 on the CPU.
- Where it does not apply: with the page cache warm (the experts fit in RAM), plain mapping is fastest and every streaming row is slower (OLMoE 609 against 138 t/s). The streaming modes are for a model that does not fit, which the rows above emulate with direct reads; a real model larger than RAM, with 8 to 16 times the experts here, would read more per token than any row above, which is what pinning, the router look-ahead and the device tier are for.