OKLMdocumentation v1.0.3
On this page

Mixture-of-experts models

How OKLM runs a mixture-of-experts (MoE) model, whose feed-forward layers route each token to a few of many experts: the grouped expert products, the expert cache that streams routed experts from the disk on the CPU, --n-cpu-moe, which keeps experts in host memory beside a device, and the device expert tier, with their measurements. The rest of inference is in inference.md; which MoE families load is in compatibility.md.

Status: maintained.

Grouped expert products

MoE layers run on either backend, routing included (oklm_moe_select(), oklm_moe_weights()). On the CPU every batch, decode and prompts alike, multiplies each (token, rank) pair by its own expert through an indexed product (oklm_matmul_id()), which groups its rows by expert so a batch reads each chosen expert once, and nothing is read back: 1.2 times faster for 16 tokens on a Qwen3-30B-A3B-like layer, prompts of 128 to 2048 tokens 1.2 to 1.4 times faster end to end on Granite-3.1-1B-A400M than one product per expert (see benchmarks.md). On a device the indexed product does the same: with few rows (n x experts per token <= expert count: decode, speculative verification) one work-group per row and expert (mv_<type>); with more, a kernel groups the rows by expert on the device (moe_group, no read-back) and one launch of the grouped tile kernels (gi_<type>, wi_<type> with the matrix units) multiplies each expert's weights by its rows, each weight tile read once for up to 64 of them (opencl.md); Vulkan sends the grouped rows to its prompt products' tiles from 16 rows an expert (vulkan.md). Until 2026-11-08 larger device batches read the chosen experts back and ran one product per expert: OLMoE-1B-7B prompts went from 4175 to 8614 t/s, Granite from 5585 to 15341 (matrix units). On the CPU both ways give the same bits (each output element is the same one-row dot product of the same quantized rows); OKLM_MOE_EXPERT_LOOP=1 runs CPU batches one expert at a time, for comparison.

On the CPU the routed experts also get a repacked copy, eight rows of an expert interleaved per block (cpu_pack.h), as the dense matrices do, when the copies and the mapped experts fit in the available memory with the memory plan's margin and no expert cache is asked for (the cache's slots hold the file layout, read from disk). The grouped product then runs the packed kernels for prompts (OLMoE-1B-7B 56 to 84 t/s at one thread, 202 to 289 at four; Granite-3.1-1B-A400M 139 to 195 and 490 to 685) and for decode steps with one or two threads (OLMoE 21.1 to 25.5 t/s at one); from three threads decode steps read the file layout, smaller, which measured 2 to 6 % faster. The copies take about as much as the experts again (GLM-4.7-Flash: 17.4 GiB on top of 15.6 GiB mapped), counted so by oklm plan. OKLM_CPU_PACK_EXPERTS=0 turns them off, =1 makes them whatever the memory says. The bits are the file layout's (test_ops checks the grouped product on both).

Expert cache

oklm run|chat|ppl|serve -m moe.gguf --backend cpu --expert-cache GB|auto [--pin-gb GB] [--expert-usage FILE]
                        [--[no-]expert-direct] [--expert-stats] [--expert-loaders N] [--expert-predict off|last|router]
                        [--expert-pack FILE] [--expert-mirror FILE]
oklm expert-pack -m moe.gguf [-o FILE]

On the CPU, a MoE model's routed experts (ffn_gate_exps, ffn_up_exps, ffn_down_exps) can stay on disk: with --expert-cache they are no longer read through the mapping but read into a cache of the given size (GiB; auto takes the largest cache oklm plan finds beside the dense part, at most every routed expert). The dense tensors (attention, shared experts, embeddings, norms, routers) stay mapped as before. In the library this is oklm_context_params::expert_cache (bytes, -1 for auto) with expert_pin, expert_usage, expert_direct, expert_loaders and expert_predict, and oklm_expert_cache_stats_get() / oklm_expert_cache_residency() / oklm_expert_cache_save_usage().

How it works:

  • At context creation each MoE layer's three stacked expert tensors are located in the GGUF file(s): file, offset of expert 0, bytes per expert. A GGUF stores gate, up and down apart, each expert contiguous inside each, so an expert is three preads (oklm_file_pread(), the platform layer of architecture.md: the cache has no system call of its own). The model is loaded without read-ahead (OKLM_MODEL_LOAD_NO_READAHEAD, as oklm plan does) and the experts' mapped pages are dropped (MADV_DONTNEED), so the mapping does not hold the experts beside the cache.
  • A slot is one aligned buffer holding one expert's three matrices. Each MoE layer gets an equal share of the cache (GB / (MoE layers x expert bytes) slots), at least one, at most one per expert.
  • In a MoE layer the router runs as before; its chosen experts are in host memory. The experts the batch needs (the union over its tokens) run in groups that fit the layer's slots: the resident ones first, then missing ones read (in parallel over the experts) into free or least recently used slots the group does not use. Each group runs the grouped indexed products with a table expert -> slot address (oklm_matmul_id_ex(), the w_experts field of the op descriptor), which writes only that group's rows; then the next group. With enough slots for the batch there is one group; with fewer (one slot and eight experts per token, a long prompt and its every expert) the batch is cut by expert, not grown. The products read the same bytes with the same kernels, so the logits are the same bit for bit as the resident run (test_model: a 160-token prompt and 24 greedy decode steps on the nine tiny MoE fixtures, with one slot per layer and direct reads, and with half the experts).
  • Routing counts (per layer and expert) accumulate in a usage file, --expert-usage FILE or ~/.cache/oklm/usage/<model hash>.json by default (the hash is of the tensor table), read at creation (a first run has none, which is not an error and logs nothing) and written at the context's end and every minute. --pin-gb GB pins the most routed experts of that file, read at creation and never evicted, on top of --expert-cache.
  • --expert-direct reads with O_DIRECT (F_NOCACHE on macOS, FILE_FLAG_NO_BUFFERING on Windows): the page cache is bypassed, each read covering the 4 KiB-aligned range around the expert. Where the file system refuses, buffered reads are used with a warning.
  • --expert-stats prints at exit the hits (experts a batch needed that were in memory), misses (read), bytes read and the time spent reading, in total and per layer; oklm serve adds an expert_cache object to /api/stats.
  • Loader threads (--expert-loaders N, default 4 with the cache or what oklm tune --expert-cache kept, below, 0 for the reads made by the forward pass itself as above): the missing experts' reads are queued, one job per matrix, and the layer consumes its slots as they fill. The experts in memory compute first while the loaders read the others; the next round waits for every read in flight (a round of products has a fixed cost) unless experts wait for a slot, in which case it computes what is ready, which frees those slots for the next reads. Each output row is still written once by the same kernel: the logits do not change.
  • Read-ahead (--expert-predict last|router, with loaders): once a layer's routing is known, the experts the next MoE layer is predicted to choose are queued behind the layer's own reads, into that layer's free or least recently used slots (the predicted ones in memory are kept from eviction). last predicts the experts that layer chose in the last batch: decoding reuses routing a lot, but the LRU slots already hold them, so it reads little. router applies the next layer's router to this layer's FFN input, rescaled from this layer's norm weights to the next one's, and takes each token's top experts (votes over a batch, at most the layer's slots): the router look-ahead. --expert-stats reports the time the forward pass waited for reads (beside the loaders' read time, which overlaps), the experts read ahead and used, and the prediction's recall (experts predicted that the batch chose, over the experts it chose).
  • Off by default. On a device backend it covers the layers whose routed experts the CPU computes: the CPU layers (--gpu-layers below the layer count) and those of --n-cpu-moe (below); with neither it is off, with a warning. OKLM_MOE_EXPERT_LOOP is ignored with the cache.

Granite-3.1-1B-A400M Q4_K_M (24 MoE layers of 32 experts, 8 per token; 697.5 MiB of routed experts, 930 KiB each), CPU, the page cache warm (dropping it needs root) except with --expert-direct; the output identical in every row:

cache (of 697.5 MiB) slots / layer prompt t/s decode t/s hit rate read read time
off (mapped, resident) - 234 42.0 - - -
100 % (0.68 GiB) 31 221 37.9 93.4 % 836 MiB 0.14 s
50 % 15 208 33.6 69.1 % 3.8 GiB 0.45 s
25 % 7 186 26.0 39.7 % 7.4 GiB 0.86 s
10 % 3 144 21.0 10.9 % 10.9 GiB 1.17 s
100 %, --expert-direct 31 158 37.2 93.4 % 836 MiB 1.33 s
50 %, direct 14 153 8.7 67.7 % 4.0 GiB 6.4 s
25 %, direct 7 132 4.8 39.7 % 7.4 GiB 12.1 s
10 %, direct 3 89 2.7 10.9 % 10.9 GiB 20.5 s

The prompt is 731 tokens (two batches) then 64 greedy tokens, 4 threads (the machine shared with other jobs: 8 threads oversubscribed it), best of 3. The hit rate counts every (batch, layer, expert) use: a prompt batch needs nearly every expert of every layer, so with 3 slots a layer runs in about eleven groups, each reading its experts; the decoded tokens then miss on most of their 8 experts. With the page cache warm the reads are memory copies (9 GB/s) and the cost is mostly the products' smaller groups; with O_DIRECT they come from the NVMe at about 550-600 MiB/s (three reads of about 300 KiB per expert, one expert per thread), which is what a model larger than RAM sees: there decode is bound by the reads, the case for read-ahead (roadmap H4) and the most used experts pinned. At 100 % a layer holds 31 of its 32 experts (the 64-byte alignment of the three matrices takes the last slot's room), hence the misses beyond the first fill. Direct slots keep room for the 4 KiB alignment, one slot fewer per layer at 50 %. (That table was measured with the reads made by the forward pass itself, --expert-loaders 0 today.)

Expert pack

A GGUF keeps a layer's gate, up and down experts in three tensors, so an expert is three reads of a third of its bytes each. oklm expert-pack -m moe.gguf [-o FILE] (oklm_expert_pack_write()) writes the routed experts again, each expert's three matrices one after the other (64-byte aligned) in one record of a multiple of 4 KiB: --expert-pack FILE (oklm_context_params::expert_pack) then reads a missing expert in one read, direct or not, straight into a slot of the same layout (one loader job instead of three). The pack holds the bytes the products read, so the logits are the same bit for bit (check_expert_pack in test_model, every MoE fixture); it takes as much disk again as the routed experts (Granite: 698 MiB beside the 784 MiB model), and the model's file is still needed for the dense part.

The file: a 48-byte header (OKLMEXP1, version 1 and the record alignment 4096 as two 32-bit values, the model's hash (that of its tensor table, as the usage file uses), the layer count with the prediction blocks, the expert count, the layers packed), then 72 bytes a packed layer (layer, offset of expert 0's record, record size, the three matrices' sizes, their offsets in the record), all little-endian 64-bit, padded to 4 KiB; then each layer's records, expert by expert. A pack of another model, or whose sizes differ from the model's, is refused with a warning and the experts are read from the model's file.

--expert-mirror FILE (expert_mirror) names a copy of the file the experts come from (the pack with --expert-pack, else the model when it is one file) on another drive: every other read goes to it (--expert-stats counts the read calls and those of the mirror; the test checks half go there and the logits do not change). It must have the same size and first 4 KiB, else it is ignored with a warning.

Measured (2026-11-11), Granite-3.1-1B-A400M Q4_K_M, CPU, 4 threads, --expert-direct (the reads from the NVMe, not the page cache), a 33-token prompt then 64 greedy tokens, no read-ahead, best of 3 (the three runs within 1 %):

cache loaders from prompt t/s decode t/s read calls waited for reads
10 % (0.07 GiB, 3 slots a layer) 0 the model's file 32.2 4.2 33882 15.3 s
10 % 0 the pack 68.1 9.6 11294 6.1 s
10 % 4 the model's file 47.8 5.8 33882 10.2 s
10 % 4 the pack 97.4 12.9 11294 4.1 s
25 % (0.17 GiB, 7 slots) 0 the model's file 39.0 6.3 23673 10.2 s
25 % 0 the pack 83.8 16.4 7891 3.4 s
25 % 4 the model's file 47.5 7.9 23871 7.7 s
25 % 4 the pack 130.3 17.0 7950 3.0 s

The same experts are read (the hits and misses equal), in a third of the calls: without loaders the reads ran at 1670 to 2090 MiB/s from the pack against 670 to 710 MiB/s from the model's file (one 1 MiB read against three of a third of it, plus the aligned ranges direct reads of the file add around each matrix), so decoding is 2.1 to 2.6 times faster. The mirror is not measured: this machine has one NVMe (the other drives are hard disks, slower than it).

Loader threads and read-ahead, the same model with --expert-direct (the reads from the NVMe), a 979-token prompt then 64 greedy tokens, 4 compute threads, best of 3 (the machine shared with other jobs at a load of 20 to 45 on 16 cores, so the throughputs scatter; the time the forward pass waited for reads is the steadier figure); the output identical in every row:

cache, reads prompt t/s decode t/s hit rate read waited read ahead used recall
50 %, inline (--expert-loaders 0) 135 2.9 64.8 % 4.3 GiB 7.9 s - -
50 %, 2 loaders 131 2.4 64.9 % 4.3 GiB 4.4 s - -
50 %, 4 loaders 124 2.2 64.9 % 4.3 GiB 2.5 s - -
50 %, 2 loaders, last 112 2.0 64.7 % 4.8 GiB 3.4 s 79 % of 568 45.2 %
50 %, 2 loaders, router 143 3.7 87.0 % 5.5 GiB 3.6 s 89 % of 4377 84.9 %
50 %, 4 loaders, router 151 4.5 87.0 % 5.5 GiB 2.0 s 89 % of 4377 84.9 %
25 %, inline 111 1.7 37.9 % 7.6 GiB 13.6 s - -
25 %, 2 loaders 136 1.9 37.4 % 7.7 GiB 8.1 s - -
25 %, 4 loaders 171 5.1 37.4 % 7.7 GiB 7.8 s - -
25 %, 2 loaders, last 175 2.6 38.2 % 8.5 GiB 11.1 s 58 % of 1044 41.3 %
25 %, 2 loaders, router 144 3.2 75.3 % 9.3 GiB 7.7 s 89 % of 6977 77.7 %
25 %, 4 loaders, router 196 7.0 75.3 % 9.3 GiB 6.3 s 90 % of 7055 77.7 %
10 %, inline 97 1.7 10.8 % 11.0 GiB 23.6 s - -
10 %, 2 loaders 84 2.8 10.8 % 11.0 GiB 14.1 s - -
10 %, 4 loaders 80 4.9 10.8 % 11.0 GiB 9.8 s - -
10 %, 2 loaders, last 82 3.4 14.7 % 13.6 GiB 15.2 s 53 % of 3612 15.9 %
10 %, 2 loaders, router 84 3.7 25.3 % 12.2 GiB 13.2 s 91 % of 3440 26.2 %
10 %, 4 loaders, router 95 4.6 25.3 % 12.4 GiB 10.3 s 90 % of 3623 26.2 %

The loaders hide part of the reads behind the products (the forward pass waits 45 % less with 2, 60 to 70 % less with 4: the NVMe serves several reads at once, the inline reads use one compute thread per expert). The last prediction is right only 41 to 45 % of the time on this model and, the LRU slots holding most of those experts already, reads little. The router look-ahead predicts 78 to 85 % of the next layer's experts (colibrì measured 71.6 % on GLM-5.2) and nine reads ahead in ten are used: the hit rate goes from 65 to 87 % at 50 % and from 37 to 75 % at 25 %, for 25 % more bytes read. At 10 % (3 slots for 8 experts a token) a prediction is capped at the layer's 3 slots, hence its 26 % recall (precision 90 %).

Decode alone, measured again with the configurations interleaved (an 18-token prompt then 64 greedy tokens, best of 5, direct reads):

cache inline 2 loaders 2 loaders, router 4 loaders, router
50 % 8.6 t/s 8.6 t/s 8.8 t/s 13.0 t/s (1.5x)
25 % 5.7 t/s 5.8 t/s 5.4 t/s 7.9 t/s (1.4x)

Two loaders keep about as many reads in flight as the inline reads did (one compute thread per expert), so they hide the reads behind the products without reading faster; the reads ahead then compete with the batch's own reads for the two loaders. With 4 loaders the NVMe serves the batch's misses and the reads ahead together, and decode gains 40 to 50 %. Hence 4 loaders are the default with the cache.

OLMoE-1B-7B-0125-Instruct Q4_K_M (16 MoE layers of 64 experts, 8 per token; 3.63 GiB of routed experts, 3.6 MiB each), --expert-direct, a 1000-token prompt then 62 greedy tokens, 4 compute threads, best of 3 (the machine's load lower than for Granite, 5 to 15); the output identical in every row:

cache, reads prompt t/s decode t/s hit rate read waited read ahead used recall
50 %, inline 117 10.8 71.1 % 10.2 GiB 6.4 s - -
50 %, 2 loaders 134 12.1 71.0 % 10.3 GiB 4.0 s - -
50 %, 4 loaders 142 12.9 71.1 % 10.2 GiB 3.5 s - -
50 %, 2 loaders, router 133 12.6 85.3 % 13.7 GiB 4.1 s 90 % of 2393 79.1 %
50 %, 4 loaders, router 137 13.9 85.3 % 13.7 GiB 3.5 s 90 % of 2393 79.1 %
25 %, inline 110 6.2 47.1 % 18.7 GiB 10.8 s - -
25 %, 2 loaders 122 6.2 47.0 % 18.7 GiB 7.8 s - -
25 %, 4 loaders 128 7.0 47.1 % 18.7 GiB 6.9 s - -
25 %, 2 loaders, router 112 6.5 75.5 % 22.4 GiB 7.1 s 86 % of 3876 73.9 %
25 %, 4 loaders, router 84 5.3 75.5 % 22.5 GiB 3.5 s 86 % of 3883 73.9 %

On OLMoE the router look-ahead predicts 74 to 79 % of the next layer's experts and lifts the hit rate from 71 to 85 % (50 %) and from 47 to 76 % (25 %); 4 loaders with it decode 1.3x the inline reads at 50 %. At 25 % its wait for reads is a third of the inline one, but its throughput fell in all three runs of the last row while the prompt rate fell with it (84 t/s against 110 to 128): the machine's other jobs, not the reads, which the waits measure; the row is kept as measured.

OLMoE-1B-7B-0125-Instruct Q4_K_M (16 MoE layers of 64 experts, 8 per token; 3.63 GiB of routed experts, 3.6 MiB each), CPU, 4 threads on a machine shared with other jobs (load 20 to 35), page cache warm, a 682-token prompt then 64 greedy tokens; the output identical in every row:

cache (of 3.63 GiB) slots / layer prompt t/s decode t/s hit rate read read time
off (mapped, resident) - 97 13.3 - - -
50 % (1.815 GiB) 31.5 (504 in all) 80 8.9 68.5 % 11.4 GiB 1.73 s
25 % (0.908 GiB) 15.5 (248 in all) 82 9.8 48.2 % 18.7 GiB 2.39 s

The speeds are within the noise of the shared machine (the 25 % decode above the 50 % one); the hit rates and reads are exact.

Tuning the streaming settings

oklm tune -m moe.gguf --backend cpu --expert-cache GB|auto [--streaming] [-n 256] [--gen 64] [--budget 300] [--force] [--cache-dir DIR]

Which loaders, read-ahead and read mode are fastest depends on the disk, the RAM (whether the page cache holds the file) and the model, so oklm tune with --expert-cache (or --streaming, the auto size) measures them for this machine, this model and that cache size (oklm_tune_stream()): --expert-loaders 0, 2, 4, 8, --expert-predict off, last, router and --expert-direct off, on, one at a time from the defaults (4 loaders, no read-ahead, buffered reads) until no change helps or the budget is spent. A run is a context of its own (a cold cache), a 256-token prompt (the tokenizer's text, repeated) and 64 greedy tokens, timed without the context's creation; a candidate's time is the median of three runs (one when it is 1.5 times the best so far), and its output is the hash of every logit and token of the run, which must equal the defaults' or the candidate is rejected. A setting kept must be 3 % faster; it is then timed against the defaults once more, their runs interleaved so the machine's load weighs on both alike, and kept only when still faster with the same output. The result, stream-<key>.json beside the kernel tuning (~/.cache/oklm/tuning/), is keyed by the model's shapes and types, the CPU, its threads, the cache size in bytes and the disk holding the model (st_dev and statfs type of its file), not the backend, and kept 90 days.

Contexts with that cache size apply it to the settings not given: oklm_context_params::expert_loaders, expert_predict and expert_direct default to -1 (the tuned value, else 4 / off / off; oklm_tune_stream_resolve()), a flag given always wins (--no-expert-direct gives buffered reads explicitly), a tuned read-ahead is dropped with --expert-loaders 0, and OKLM_TUNE=0 turns it off with the kernel tuning. The cache size is never tuned: it is the user's, and part of the key. The settings are the host-side cache's, so a device context applies the same entry to the expert cache that serves its CPU experts (--n-cpu-moe) or the misses of the device expert tier (--expert-gpu with --expert-cache).

Measured on this machine (NVMe, 124 GiB of RAM so the page cache holds the files, 4 threads; the machine shared with other jobs at a load of 10 to 21 on 16 cores), with the times of the defaults' run (prompt and decode) and what was kept:

model, cache defaults best other candidate --expert-direct kept
Granite-3.1-1B-A400M, 50 % (0.34 GiB) 1.7-3.4 s within the noise (0, 2, 8 loaders) 3.6x, 2.0x slower defaults
Granite-3.1-1B-A400M, 25 % (0.17 GiB) 4.7 s interleaved 2 loaders, 5.1 s interleaved 2.4x slower defaults
OLMoE-1B-7B, 50 % (1.8 GiB) 6.9 s interleaved 2 loaders, 6.9 s interleaved 1.3x slower defaults
OLMoE-1B-7B, 25 % (0.9 GiB) 8.0 s router, 10.6 s 2.5x slower defaults

The defaults stay: with the file in the page cache a buffered read is a memory copy, so the loaders and the read-ahead have little to hide and O_DIRECT (which goes to the NVMe each time) is 1.3 to 3.6 times slower; the gain over the defaults is 1.00x on all four. Twice a candidate first timed faster (8 loaders 1.23x at Granite 25 %, last 1.05x at OLMoE 50 %, before the interleaved check existed) and twice the interleaved check rejected a win of the search (2 loaders, 1.23x at Granite 25 % and 1.5x at OLMoE 50 %): on a loaded machine one sequence of medians is not enough, hence that check. A model larger than RAM, or a machine whose page cache cannot hold the experts, is where direct reads, 4 loaders and the router look-ahead win (the tables above, measured with --expert-direct), and where the tuner is meant to find them.

Experts on the CPU

oklm run|chat|ppl|serve|plan -m moe.gguf --backend opencl --n-cpu-moe N|all [--gpu-layers N] [--expert-cache GB|auto]

On a device backend, --n-cpu-moe N (as llama.cpp's flag; alias -ncmoe) keeps the routed experts (ffn_gate_exps, ffn_up_exps, ffn_down_exps) of the first N layers in host memory and runs them on the CPU, while everything else of those layers (attention, norms, the router, the shared experts) goes to the device with the other device layers; all does it for every layer. Experts are most of a MoE model's weights but each token reads a few of them, so a MoE model larger than the device keeps its attention on the device rather than whole layers on the CPU (--gpu-layers alone). In the library it is oklm_context_params::n_cpu_moe (-1 for all), oklm_n_cpu_moe() and oklm_context_mem::n_cpu_moe.

How it works:

  • Placement: the device copy of such a layer gets every tensor but the three expert stacks, whose headers stay the mapped host tensors (dev_place_layer()). With --gpu-layers left out, the count of layers that fit uses those slimmer layers, so more fit; with it, the N layers are counted from the first, whether on the device or not (a CPU layer runs its experts on the CPU anyway). One placement per model: the model's first device context fixes it.
  • A device layer whose experts are on the host crosses devices in moe_ffn(): the router, the expert choice and the weights run on the device as before; then the chosen experts and their weights (n x K each) and the normalized rows (n_embd x n) read back in one round trip (the backend's download_many: OpenCL queues the first two reads without waiting and blocks on the last), the shared expert queued on the device and started (clFlush, the backend's new flush entry) so it runs while the CPU works, the CPU's grouped indexed products over the batch (oklm_matmul_id, gate, up, SwiGLU, down: each expert's rows together, each expert read once per batch), the weighted sum of each token's ranks on the CPU, and one write of that sum (n_embd x n), which the in-order queue runs after the shared expert, which is then added on the device. Per layer and step: about 4 n (n_embd + 2 K) bytes down and 4 n n_embd up (Granite: 8 KiB for one token).
  • --expert-cache composes: on a device context the cache covers the layers whose routed experts the CPU computes (the CPU layers and these), the read-back routing feeding it as on the CPU, the loaders and --expert-predict too. The router look-ahead of such a layer runs on the device when the next MoE layer's router is there (2026-11-11): the layer's normalized rows multiplied by the ratio of the two layers' norm weights (uploaded once per layer), the next router applied, and the logits (n_expert x the batch) read back in the same round trip as the routing, into the host scratch the look-ahead votes on; otherwise it runs the next router on the host as on a CPU context. --expert-stats' counters say how many look-aheads the device computed (ahead_device). Granite with --n-cpu-moe all, a 25 % cache, 4 loaders, direct reads: prompts 513 -> 526 t/s (the host product per layer and batch gone), decoding 8.1 t/s either way (bound by the reads); OLMoE 52.7 / 3.4 t/s either way.
  • oklm plan --n-cpu-moe N counts those experts in host memory and not on the device (the device line says how many layers' experts are on the CPU), plus the host buffers the crossing uses.
  • Checked in make test (test_model) on the eight tiny MoE fixtures (glm4moe included), with the experts of the first layer and of every layer on the CPU, every layer on the device or one on the CPU: the references' greedy tokens and logits within 1e-4 (3e-5 at worst), a 160-token prompt and 24 greedy steps giving the device-only run's tokens (logits within 1e-5), the expert cache (one slot per layer) giving the same bits as without it; test_multi_device with the layers spread over two device instances; test_mem_plan the projection.

Granite-3.1-1B-A400M Q4_K_M (24 MoE layers, 32 experts of 930 KiB, 8 per token; 697.5 MiB of its 782 MiB are routed experts) on the AMD gfx1100 with the i7-11700K; an 836-token prompt (two batches of 512 and 324) then 64 greedy tokens, tokens per second, best of 3. The machine was shared with other jobs throughout (load average 30 to 35 on 16 hardware threads), which hurts the CPU side most: these runs use 2 CPU threads, the fastest setting under that load (4 and 8 threads were slower and erratic). The greedy text follows the device-only run's for its first dozen tokens, then takes another wording, as the CPU backend's and the --gpu-layers 3 run's do at the same place: the CPU's products round the activations to 8 bits, the device's do not.

placement layers on the device prompt t/s decode t/s
CPU backend 0 153 40.0
device 24 2 499 135.9
--n-cpu-moe 6 24 682 46.6
--n-cpu-moe 12 24 398 31.4
--n-cpu-moe 18 24 280 30.5
--n-cpu-moe all 24 221 27.5
640 MiB device (OKLM_GPU_MEM_MB=640), --gpu-layers fit 3 182 44.3
640 MiB device, --n-cpu-moe all 16 193 26.7
640 MiB device, --n-cpu-moe 12 3 154 34.7

Per crossing, timed in the code (since removed) over a short prompt and 64 decode steps: the two further reads 0.13 ms and the write of the sum 0.24 ms, against 0.77 ms of CPU expert products, all inflated by the load (an idle bus round trip is tens of microseconds). On this model the crossing does not pay: Granite fits the device whole, its experts are small (each token reads 7.3 MiB of them per layer, which the CPU reads in well under a millisecond), and its dense part is mostly the 49k-row tied head and the logits buffers, so on the 640 MiB device only 16 of the slimmed layers fit beside them; the model it is for is one whose experts exceed the device by far while its attention and dense weights fit (Qwen3-30B-A3B, DeepSeek-Coder-V2-Lite at larger quantizations), which --gpu-layers alone would leave mostly on the CPU. No such model is in models/ at the time of writing (only Granite among the MoE files), so that measurement is still to make.

Two follow-ups measured on 2026-11-11 (Granite, quiet machine, 8 CPU threads, a 948-token prompt, best of 3):

  • One round trip for the routing and the rows instead of three (download_many): decoding with every layer's experts on the CPU 72.2 t/s before, 72.8 after, within the runs' spread: on an idle bus a read costs tens of microseconds, against the CPU expert products' 0.77 ms. Kept (one wait instead of three, and the backend's read errors now reported: cl_download_locked() dropped its read's status).
  • Streamed CPU layers (--gpu-stream) still carry their routed experts to the device, even within --n-cpu-moe. Leaving those experts in host memory (a streamed layer then crossing as a device layer does) shrank the two slots from 33.1 to 1.9 MiB but cut prompts with --gpu-layers 12 --gpu-stream 32 --n-cpu-moe 12 from 4 767 to 2 855 t/s: for a batch of hundreds of tokens, copying a layer's experts to the device beats computing them on the CPU, and on a larger MoE more so (more tokens per expert). Not kept; streamed layers stay whole.

Device expert tier

oklm run|chat|ppl|serve -m moe.gguf --backend opencl|vulkan --expert-gpu GB|auto [--n-cpu-moe N|all] [--expert-usage FILE] [--expert-cache GB|auto] [--expert-stats]

On Vulkan the tier works the same way, its refills on the instance's transfer queue (vulkan.md).

--expert-gpu GB keeps the hottest routed experts of every layer whose experts are in host memory (--n-cpu-moe; every layer's when --expert-gpu is given without it) in device memory, an equal share of GB per layer; auto takes the device's free memory once the context is created (the dense part, the KV cache and the buffers placed), less the layer fit's margin (8 %, at least 256 MiB); with OKLM_GPU_MEM_MB the memory in use is taken from that size. The rest stay where they are: resident in RAM, or read from disk through --expert-cache. In each step the device computes the (token, rank) pairs of the experts it holds while the CPU computes the others, so experts are placed by heat rather than by layer (--n-cpu-moe all places them by layer: all on the CPU). In the library it is oklm_context_params::expert_gpu (-1 for auto), oklm_expert_gpu_stats_get() and oklm_expert_gpu_residency(); --expert-stats prints the tier's counters and /api/stats has them under expert_gpu.

How it works (src/model/expert_gpu.c):

  • Slots: per layer, device stacks of the three expert matrices (gate, up, down) with S slots, plus one slot of zeros while S is below the expert count. A slot holds one expert's bytes as the file stores them, so the device's products read the weights a device copy would.
  • Heat: the routing counts of the usage file (the expert cache's, ~/.cache/oklm/usage/<hash>.json or --expert-usage), then every routing the tier sees, halved at each refill so recent routing weighs more. With counts the hottest experts are uploaded at creation; without them the tier starts empty and fills after 4 steps. Without --expert-cache the tier writes the counts back at the end, so the next run starts filled.
  • A step, per such layer: the router, the expert choice and the weights run on the device and are read back with the normalized rows (as --n-cpu-moe does). Each pair whose expert the device holds is mapped to its slot, the others to the slot of zeros (their products are exact zeros); the mapped ids go up and the device runs its indexed products over the slots and their weighted sum (a batch with more pairs than experts sorts the held pairs by slot instead, one product per slot, as device batches do), then the shared expert, and starts them. Meanwhile the CPU runs the grouped indexed products over the other pairs only (a table without the held experts' addresses, or the expert cache told to skip them: its loaders never read them and its read-ahead never predicts them) and their weighted sum, which is written up and added to the device's. A step whose pairs are all held costs the CPU nothing; one with none held runs no device product.
  • A layer whose tier holds all its experts places expert e in slot e and runs as a device layer: nothing is read back.
  • Refill: every 32 steps, or after 8 when the hit rate fell below 80 % of the previous period's, each layer, after the CPU's share, moves the hottest host experts into free slots and swaps the coldest held ones for host experts at least 1.25 times as hot, at most an eighth of the slots per refill. The uploads are queued after the layer's device work, so a step never sees a slot change under it; on the device's transfer queue when it has one (OpenCL: a second command queue; Vulkan: a copy engine or a second compute queue, with timeline semaphores) and the context streams no weights (OKLM_EXPERT_GPU_XFER=0 for the device's queue): the copies wait for the work queued before them, run while the device computes the layers that follow, and the layer's next products wait for them.
  • Long batches (2026-11-11): a batch of 32 tokens or more (OKLM_EXPERT_GPU_STREAM=N sets the threshold, 0 turns it off) runs every pair on the device. The pairs are sorted by expert; a held expert's rows multiply its slot, a missing one is first copied into one of four staging slots (the largest expert of each matrix, taken in turn; the in-order queue keeps a slot from being written while a product still reads it), so the CPU computes nothing for the layer. --expert-stats counts those pairs and copies. Not with --expert-cache, whose reads of the host experts it would bypass: the misses stay on the CPU there.
  • Checked in make test (test_model, test_expert_gpu) on the nine tiny MoE fixtures: a 160-token prompt and 24 greedy steps with a tier holding no expert, half of them (filled from the first steps' routing, then a second pass that refills), all of them, half filled at creation from a usage file, and half with the misses read through an expert cache of one slot per layer (without loaders, with 4, with 4 and either read-ahead: the same bits each time) give the device-only run's greedy tokens, logits within 1e-5 (the CPU-vs-OpenCL tolerance is 1e-4).

Measured on the AMD gfx1100 (20 GB) with the i7-11700K, 4 CPU threads, oklm run of an 821- to 850-token prompt then 64 greedy tokens, tokens per second, best of the rounds. The machine was shared with other jobs throughout (load average 15 to 46 on 16 hardware threads): the CPU side varied by 2x to 5x from one round to the next, so the placements were run interleaved, round after round, and only large differences mean something; 8 CPU threads were far slower under that load (decode 1 to 4 tokens/s where 4 threads gave 8 to 24). "Cold": no usage file, the tier fills from the routing of the first 4 steps (the prompt's), "warm": filled at creation from the usage file of a run on another text (README.md's first 2.6 kB). The hit rate counts the prompt's pairs, most of the total (a cold tier holds nothing while the prompt runs).

OLMoE-1B-7B-0125-Instruct Q4_K_M (16 layers, 64 experts of 3.6 MiB, 8 per token; 4.0 GiB), on a simulated 2 GiB device (OKLM_GPU_MEM_MB=2048): the tier gets 519 MiB, 8 slots per layer (12.5 % of the experts), 3 rounds:

placement prompt t/s decode t/s tier hits uploads after creation
20 GB device, the whole model 2 166 227
2 GiB, --gpu-layers fit 120 14.4
2 GiB, --n-cpu-moe all 127 15.4
2 GiB, --n-cpu-moe all --expert-gpu auto, cold 127 18.4 3.0 % 128 (1.9 per step)
2 GiB, the same, warm 189 14.9 45 % 17 (0.26 per step)

GLM-4.7-Flash Q4_K_M (47 layers, 46 of them MoE with 64 experts of 5.4 MiB, 4 per token and a shared expert; 17 GiB), on a simulated 8 GiB device: the tier gets 5.0 GiB, 19 or 20 slots per layer (31 % of the experts), 2 rounds:

placement prompt t/s decode t/s tier hits uploads after creation
20 GB device, the whole model 509 55.1
8 GiB, --gpu-layers fit 36.5 6.8
8 GiB, --n-cpu-moe all 60.8 4.9
8 GiB, --n-cpu-moe all --expert-gpu auto, cold 46.8 9.4 5.8 % 902 (13.7 per step)
8 GiB, the same, warm 93.5 12.5 63 % 179 (2.7 per step)

Granite-3.1-1B-A400M Q4_K_M (24 layers, 32 experts of 930 KiB), on the whole device with a usage file from another text, 3 rounds:

placement prompt t/s decode t/s tier hits
device 3 884 200
--n-cpu-moe all 251 24.1
--expert-gpu 0.15 (132 slots, 17 %) 486 16.9 38 %
--expert-gpu 0.3 (312 slots, 41 %) 672 19.6 71 %
--expert-gpu auto (every expert: device layers) 2 110 142 100 %

What the numbers say: prompts gain most, as the CPU's share of a batch shrinks with the hit rate (GLM 1.5x, OLMoE 1.5x, Granite 2.7x over --n-cpu-moe all when warm); decode gains when experts are large and the hit rate high (GLM 2.5x over --n-cpu-moe all, 1.8x over --gpu-layers), and not with Granite's small experts, where a step's fixed costs (the blocking transfers per layer, one more than --n-cpu-moe: the mapped ids) outweigh the few hundred kilobytes of expert reads saved, and the load swamped the difference. A cold tier's first fill uploads every slot at once (GLM: 4.9 GiB in 2 to 9 s inside the fifth step), so a usage file from an earlier run is worth keeping; the tier writes it back itself. Transfers per decode step besides refills: the routing and rows down and the CPU's sum up, as --n-cpu-moe, plus 4 n K bytes of mapped ids.

Measured on 2026-11-11 (quiet machine, 8 CPU threads, no usage file, a 948-token prompt for Granite and 847 for OLMoE, then 64 greedy tokens; 256 for the decode rows; three runs each, the means):

before after
Granite, --expert-gpu 0.35 (half the experts), prompt 1 063 t/s 2 060 t/s (the misses streamed)
Granite, the same, decode 113 t/s 119 t/s
OLMoE-1B-7B, --expert-gpu 1.9 (half), prompt 466 t/s 512 t/s
OLMoE-1B-7B, the same, decode 63 t/s 73 t/s
Granite, decode, refills on the device's queue / transfer queue 127.9 t/s 129.1 t/s
OLMoE-1B-7B, decode, the same 78.5 t/s 80.2 t/s

Streaming a long batch's missing experts doubles Granite's prompts (its experts are small: copying 30 MiB a layer beats computing 948 tokens' rows on the CPU) and adds 10 % to OLMoE's (experts four times larger). The decode rows of the first table rose too (the decode path itself did not change; not looked into further). Measured and not done: the mapped ids computed on the device (their write takes 12 us a layer, 3.5 % of a Granite decode step at most, against a gather launch of its own), and a per-layer share of the slots by heat (from the routing counts of a 256-token run, holding the hottest experts of the whole model instead of an equal share per layer would cover 49.1 instead of 48.8 % of Granite's routings at a quarter of the experts, 77.2 / 76.7 % at half, 82.3 / 81.9 % for OLMoE at half: under a point, measured on the very counts it would be chosen from).

Measured on this machine

One table per model, every streaming mode, on the machine's NVMe (CPU i7-11700K, 4 threads for the CPU rows, the AMD gfx1100 for the device rows, 124 GiB of RAM; 2026-11-08, shared with other jobs, so the throughputs have a few percent of noise and the hit rates and bytes are the exact figures). A 295-token prompt (one sentence repeated, so its routing is more concentrated than natural text's: the hit rates are higher than a real document would give) then 64 greedy tokens, best of 2 (the two runs agree to 2 % except where said). The text generated is byte for byte the same in every row of a model, the device tier's included.

What is emulated: no model larger than RAM is at hand, and the page cache cannot be dropped without root (dd iflag=nocache left the mapped runs as fast as the warm ones: the "cold" rows are not cold and are not reported). So every streaming row uses --expert-direct (O_DIRECT: each expert read comes from the NVMe, never the page cache) with an expert cache of 25 % or 50 % of the experts: the experts do not fit in the memory given to them, which is the situation of a model larger than RAM, the dense part staying mapped. The device rows set OKLM_GPU_MEM_MB (2 GiB for OLMoE, 8 GiB for GLM) so the experts exceed the device, with --n-cpu-moe all --expert-gpu auto. "Read" is the bytes the experts' reads moved over the run, "waited" the seconds the forward pass spent waiting for them out of the run's wall time.

OLMoE-1B-7B-0125-Instruct Q4_K_M (16 MoE layers of 64 experts, 8 per token; 3.63 GiB of experts):

mode prompt t/s decode t/s hit rate read waited / wall bottleneck
plain mapping, page cache warm 609 33.9 - - - the kernels
cache 50 % (1.8 GiB), direct, 4 loaders 128 9.9 70.0 % 9.8 GiB 5.5 / 8.8 s disk
cache 25 % (0.9 GiB) 138 5.5 44.9 % 17.8 GiB 10.1 / 13.7 s disk (410 MiB/s)
25 % and --pin-gb 0.3 148 6.2 51.8 % 15.9 GiB 8.9 / 12.4 s disk
25 % and --expert-predict router 90 3.9 77.4 % 22.4 GiB 15.2 / 19.7 s disk (see below)
25 %, pinned and router 158 6.1 80.2 % 20.0 GiB 8.9 / 12.3 s disk
25 % and the device tier (2 GiB device, 519 MiB of tier) 164 6.9 45.5 % cache, 17.9 % tier 13.9 GiB 8.1 / 11.0 s disk
device tier alone (experts in RAM, no cache) 495 45.8 18.8 % tier - - the CPU's share of the experts

GLM-4.7-Flash Q4_K_M (46 MoE layers of 64 experts, 4 per token and a shared expert; 15.6 GiB of experts), cache 25 % (3.78 GiB, 716 slots):

mode prompt t/s decode t/s hit rate read waited / wall bottleneck
plain mapping, page cache warm 132 10.4 - - - the kernels
cache 25 %, direct, 4 loaders 36.1 6.3 76.0 % 17.7 GiB 6.0 / 18.4 s the kernels, then disk
25 %, --pin-gb 1.2 and router 48.0 7.4 86.6 % 17.9 GiB 4.6 / 14.8 s the kernels, then disk
25 %, the device tier (8 GiB device, 5.0 GiB of tier) 68 17.1 55.0 % cache, 81 % tier 8.7 GiB 3.4 / 8.2 s the kernels

Reading them:

  • Streaming costs on OLMoE. Its experts are 3.6 MiB and 8 of 64 are used per token, so a cache of a quarter of them misses over half the time and decode falls from 34 to 5.5 t/s, 74 % of the wall time spent waiting for the reads at about 410 MiB/s (the NVMe's sustained rate for these 1 MiB-class reads, one per matrix; the loaders keep four in flight). Prompts fall less (609 to 138 t/s): a prompt batch needs most experts of a layer, which the cache serves in groups.
  • The hit rate is bought with bytes. Pinning the most routed experts (from the usage file of earlier runs) lifts the hit rate by 7 points and cuts the bytes read by 11 %. The router look-ahead lifts it from 45 to 77 % (80 % with pinning) for 26 % more bytes read, predicting 73 % of the routed experts with 80 % of the reads ahead used. Both together decode 11 % faster than the cache alone, on the throughput; the waits do not fall in proportion because the loaders are already the NVMe's queue and the prompt's batches still read what they need.
  • Read-ahead alone measured slower (3.9 against 5.5 t/s) and moved bytes at 254-298 MiB/s against 410-430 in every other row: the disk shared with other jobs at that moment (the same row repeated with pinning ran at 433 MiB/s). Taken as noise from the shared machine, not as a result; read it as no gain, not as a loss.
  • GLM has more, smaller-per-token reads over 46 layers but only 4 experts per token and a high hit rate with this text (76 % at 25 %), so it is not disk-bound here: the waits are a third of the wall time and the rest is the CPU's products (decode 10.4 t/s mapped, 6.3 with the cache, which also runs without the repacked expert copies). Pinning and read-ahead take the waits from 6.0 to 4.6 s and decode from 6.3 to 7.4 t/s. On a text that routes less repetitively the hit rate would be lower and the disk would take over, as OLMoE shows.
  • The device tier is the largest gain: on a device too small for the model, 5.0 GiB of tier on GLM holds 81 % of the routed pairs (warm from the usage file) and decodes 17.1 t/s, 2.7 times the cache alone and 1.6 times plain mapping, since the device's share is computed at device speed and the CPU is left a fifth of the pairs and a smaller read stream (8.7 GiB, half). On OLMoE the tier holds only 12.5 % of the experts (519 MiB): 18 % of the pairs, a 25 % gain over the cache alone while the disk is still the bottleneck; with the experts in RAM it decodes 45.8 t/s against 33.9 on the CPU.
  • Where it does not apply: with the page cache warm (the experts fit in RAM), plain mapping is fastest and every streaming row is slower (OLMoE 609 against 138 t/s). The streaming modes are for a model that does not fit, which the rows above emulate with direct reads; a real model larger than RAM, with 8 to 16 times the experts here, would read more per token than any row above, which is what pinning, the router look-ahead and the device tier are for.