Memory plan
What a model would take before anything loads: oklm plan
for one model or every model of a directory, and the warning
run, chat and serve print when a
model will not fit. Inference itself is in inference.md; the expert cache the plan sizes
for a MoE model in moe.md.
Status: maintained.
oklm plan -m model.gguf [-c CTX] [--parallel N] [-ctk TYPE] [-ctv TYPE] [--gpu-layers N] [--backend cpu|opencl|vulkan|auto]
[--draft small.gguf] [--mmproj mmproj.gguf] [--json]
[--serve [--kv-mem MB] [--tiles SIDE] [--transcribe-model DIR [--transcribe-backend B]]
[--speak-model FILE [--speak-vocoder FILE] [--speak-backend B]]
[--embed-model FILE [--embed-backend B]] [--rerank-model FILE [--rerank-backend B]]]
oklm plan --list [DIR] [--recommend] [-c CTX] [--backend cpu|opencl] [--json]
plan takes GGUF files; a Hugging Face directory is
refused ("convert a Hugging Face checkpoint first"): loading one stacks
per-expert matrices, widens FP8 ones and transposes GPT-2's, all of
which read the weights the plan is meant not to touch.
plan says what a model would take before anything loads.
It reads the GGUF metadata and tensor table only (any shard opens the
whole model, the files mapped without read-ahead, no tensor data
touched) and prints:
- the weights by kind (embeddings and output, attention, dense FFN,
shared experts, routed experts, routers, norms, multi-token prediction
blocks) with their tensor counts and sizes, as
oklm infocounts the file; - the host memory: the resident weights (the repacked copies the CPU
matrix product makes counted in place of the mapped matrices they stand
for; the routed experts' copies, made when they fit beside the mapping
and no expert cache is asked for, counted on top of it, since decode
steps from three threads still read the file layout, but for Q8_0 and
Q5_0 experts, whose copies are the file's size), the KV cache of
-ctokens x--parallelsequences in the-ctk/-ctvtypes (MLA's compressed cache for DeepSeek V2 / V3), a hybrid stack's recurrent state (with the states and the journal kept for cut sequences, hybrid-layers-design.md), the activation buffers and the scratch, a draft model's weights and context, a projector's weights. These come fromoklm_context_mem_estimate(), which sizes them with the functionsoklm_context_new()allocates with: on the CPU the figures are the context's own (checked on every tiny model intest_mem_plan); - with
--backend opencl(orautofinding a device), the device's layers (--gpu-layers, or what fits its memory), their weights, cache and buffers, against its free memory (AMD's query) or its size less 8 %. The default backend iscpu: no device is opened; - the available RAM (
MemAvailableon Linux,GlobalMemoryStatusExon Windows,host_statistics64on macOS), less a margin of 3 % (at least 256 MiB), and the verdict: fits resident, fits with N layers on the device, or does not fit (exit status 3); - for a MoE model, the routed experts' share, what stays resident if they were streamed from disk (the dense part, its cache and buffers), and the largest expert cache that would fit beside it.
--json prints the same as one JSON object, with each
layer's placement (JSON output).
--list [DIR] finds the GGUF models under
models/ (or DIR, walked recursively, hidden
entries left out; a split model's shards counted once, as their first)
and projects each from its headers alone, with a context of 8192 tokens
capped at the model's trained context (-c sets another,
uncapped). A row each, by size on disk, the smallest first: the size,
the need, the verdict, the file. The need is what the
model takes resident (its weights, cache and buffers); for a MoE model,
the part that stays resident when the routed experts are left in the
file (the dense part, its cache and buffers) plus the smallest expert
cache:
min expert cache = expert size (gate, up and down of one expert) x experts per token x MoE layers x 2
(the experts one token uses in every layer, twice: the token's and the next one's while the router's choice moves; never more than every routed expert). The verdict, the easiest first:
- fits resident on the CPU: every weight resident, with the margin;
- fits on the GPU with N layers (only with
--backend opencl, which asks the device; the default backend iscpuand no device is opened): every layer when they all fit, else the most layers with which the device and the host both have room, when the model does not fit resident on the CPU; - fits only with the expert cache: a MoE model whose
need fits where the whole model does not (
--expert-cache, below); - does not fit.
A MoE row adds its resident figure and the two terms of its need.
Files that are GGUF but not a model OKLM loads (a projector, an
importance matrix, a vocabulary) are listed apart with the reason.
--recommend (which implies --list) names the
largest model that fits resident and the largest that runs only with the
expert cache. --json prints the list as one object:
models (each with size, resident,
dense, min_expert_cache, need,
fit: cpu, gpu,
expert_cache or none,
gpu_layers), skipped, the memory, and
recommend.resident and recommend.expert_cache
(paths or null). On this machine (125 GiB, no device asked; rows
cut):
$ oklm plan --list --recommend
size need verdict model
...
136.4 MiB 440.0 MiB fits resident on the CPU models/smollm2-135m/SmolLM2-135M-Instruct-Q8_0.gguf
782.1 MiB 1.32 GiB fits resident on the CPU models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf [MoE 8/32: resident 1.32 GiB, with the expert cache 656.4 MiB + 348.8 MiB]
3.92 GiB 4.74 GiB fits resident on the CPU models/olmoe/OLMoE-1B-7B-0125-Instruct-Q4_K_M.gguf [MoE 8/64: resident 4.74 GiB, with the expert cache 1.11 GiB + 930.0 MiB]
4.68 GiB 6.78 GiB fits resident on the CPU models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf
15.26 GiB 16.80 GiB fits resident on the CPU models/qwen3-8b/Qwen3-8B-BF16.gguf
17.05 GiB 18.21 GiB fits resident on the CPU models/glm-4.7-flash/GLM-4.7-Flash-Q4_K_M.gguf [MoE 4/64: resident 18.21 GiB, with the expert cache 2.59 GiB + 1.95 GiB]
RAM: 99.45 GiB available of 124.64 GiB, 3.78 GiB kept free; context 8192 tokens unless the model's is shorter; backend cpu
left out: 3 files
models/qwen3-8b/Qwen3-8B.imatrix.gguf: not a model (a projector, an importance matrix, a vocabulary...)
models/smolvlm-256m/mmproj-SmolVLM-256M-Instruct-f16.gguf: not a model (a projector, an importance matrix, a vocabulary...)
models/ultravox-1b/mmproj-ultravox-v0_5-llama-3_2-1b-f16.gguf: not a model (a projector, an importance matrix, a vocabulary...)
recommended, resident: models/glm-4.7-flash/GLM-4.7-Flash-Q4_K_M.gguf (fits resident on the CPU)
recommended, with the expert cache: none (no MoE model here needs it to fit)
JSON output
oklm plan -m model.gguf --json prints one JSON object on
one line (oklm_mem_plan_layout_json()): every figure of the
text form, then placement, where each layer would be. Sizes
are bytes. The exit status is the text form's (3 when the model does not
fit).
| Member | Type | What |
|---|---|---|
model |
string | the path given |
arch |
string | architecture |
n_layer |
number | blocks of the main stack |
n_expert, n_expert_used |
number | experts per MoE layer (0 for a dense model), experts a token goes to |
files |
number | files of the model (shards) |
backend |
string | the backend the context would run on (cpu,
opencl, vulkan) |
cache_type_k, cache_type_v |
string | the key and value cache's types |
weights |
object | per kind (embeddings, attention,
dense_ffn, shared_experts,
routed_experts, routers, norms,
mtp, other)
{"tensors": N, "bytes": N}, the file's; then
total (every tensor), host (resident in host
memory: the CPU layers' and the embeddings', the repacked matrices at
their copies' size), device (file bytes of the weights
placed on the device), routed_experts_host (the routed
experts among host) |
context |
object | the context's projection, below |
draft |
object | with --draft only: weights (resident),
cache_room (with --serve: the device bytes its
context takes of the cache's room first) and context (the
same members as context) |
mmproj |
number | the projector's weights (0 without --mmproj, and with
--serve, where the projector is in
beside) |
beside |
object | what loads beside the model (as a
server): device, the bytes on the model's device, and
host, those in host memory (the weights of what computes on
the CPU and what it works with there); both 0 without
--serve |
host_need |
number | the projected host peak: weights, cache, state, activations, scratch, draft, projector and what loads beside the model |
device_need |
number | the projected device memory (0 on the CPU) |
device_capacity |
number | what the device can take: its free memory when it tells, else its size less 8 % (0 on the CPU) |
ram_available, ram_total |
number | available and installed physical memory |
margin |
number | the headroom kept free |
verdict |
string | fits, fits_partial (some layers on the
device, the others on the CPU) or no_fit |
verdict_text |
string | the verdict's sentence of the text form
(fits with 10 of 24 layers on the device) |
n_gpu_layers |
number | layers on the device |
experts |
object | a MoE model with routed experts in host memory only:
routed_host, count (routed experts over every
layer), expert_bytes (one expert, on average),
dense_need (the host need with the routed experts streamed
from disk), cache (the largest expert cache that fits
beside it), cached (experts it holds) |
placement |
object | each layer's placement, below |
The verdict holds when
host_need + margin <= ram_available and
device_need <= device_capacity.
context:
| Member | What |
|---|---|
n_ctx, n_seq, n_batch |
context length, sequences, tokens a forward pass |
n_gpu_layers |
layers on the device (the last ones, with the head) |
n_cpu_moe |
the first layers whose routed experts stay in host memory
(--n-cpu-moe, resolved) |
kv_layer |
one layer's key/value cache (the widest layer's when they differ: Gemma 4) |
kv |
host: the cache of the layers on the CPU, each at its own width,
none for a block that reads another's (Gemma 4 E2B, E4B) nor for a block
without attention (the linear blocks of qwen35,
qwen35moe and qwen3next, the convolution
blocks of lfm2 and lfm2moe) |
state |
host: a hybrid stack's recurrent state, and the states and journals kept for cut sequences (every slot counted; the states alone, without the journals, when a linear block runs on a device) |
activations |
host: the activation buffers |
scratch |
host: the rest of the context's allocations |
repacked, repacked_from |
host: the CPU matrix product's repacked copies, and the mapped bytes they stand for |
repacked_experts |
the part of repacked that is the routed experts'
copies, on top of their mapping |
expert_cache |
host: the routed-expert cache (0 without one) |
device_weights, device_kv,
device_buffers |
device: its layers' weights as the device holds them, their cache (none for a block without attention, nor for a block that reads another block's), the activation buffers (with the embeddings per layer of a batch for Gemma 4 E2B and E4B) |
device_total, device_free |
the device's memory and what is free now (0 when unknown or on the CPU) |
placement (oklm_layout_place_json()), from
the tensor table and the rule the weights are counted with: the last
n_gpu_layers layers on the device with the head, the others
on the CPU with the embeddings, a device layer's routed experts in host
memory when it is among the first n_cpu_moe:
| Member | Type | What |
|---|---|---|
n_gpu_layers, n_cpu_moe |
number | as in context |
devices |
array | each device holding layers, in layer order: name (the
backend), first_layer, layers (their count),
head (true for the one that holds the head). Empty on the
CPU |
layers |
array | one entry per layer of the main stack, below |
embeddings, head, other |
object | the parts outside the blocks (as oklm info --json lays
them out, convert.md):
on (cpu or device),
device (its name or null), bytes |
mtp_bytes |
number | the multi-token prediction blocks: left in the file, read by
--mtp only |
host_bytes, device_bytes |
number | the weights' file bytes on either side, over the layers and the parts |
kv_layer_bytes |
number | one layer's cache (the widest layer's when they differ) |
kv_host_bytes, kv_device_bytes |
number | the cache of the CPU layers, of the device layers |
A layer:
| Member | Type | What |
|---|---|---|
index |
number | the layer |
on |
string | cpu or device |
device |
string or null | the device's name |
bytes |
number | its tensors' file bytes |
host_bytes, device_bytes |
number | of them, those in host memory and those on the device |
kv_bytes |
number | its own key/value cache, on the side the layer is on: 0 for a block that keeps no rows (no attention, or another block's cache read), a narrower block's own size |
experts_on |
string | a layer with routed experts only: cpu or
device |
experts_bytes |
number | a layer with routed experts only: their file bytes |
The figures add up:
placement.host_bytes + device_bytes + mtp_bytes is
weights.total; placement.device_bytes is
weights.device;
placement.host_bytes - context.repacked_from + context.repacked
is weights.host; the layers' kv_bytes on the
CPU add up to context.kv, those on the device to
context.device_kv (test_placement and
test_rows_of_attention_blocks in
tests/model/test_mem_plan.c, on every fixture: the two
widths and the shared caches of Gemma 4, the blocks without attention of
the hybrid stacks). The text form says how many layers keep rows when
some do not: KV cache 256.0 MiB (8 of 32 layers x 32.0 MiB)
for Qwen3.5-9B at 8192 tokens. The buffers are not split by layer:
context.activations and context.scratch serve
every CPU layer, context.device_buffers every device layer.
context.device_weights is what the device holds for
weights.device bytes of the file (a matrix it stores in
another layout takes more). The head's tensors are the final norm, the
output matrix, the rotary factors and the norm named
token_embd_norm (the final norm of an lfm2
file, the norm of an encoder's summed embeddings). An output matrix that
is the token embeddings (a tied head) is one tensor of the file: its
bytes are counted once, with the embeddings the host looks up, and the
copy the device holds for the output product is in
context.device_weights alone.
A dense model on the CPU (SmolLM2-135M Q8_0; the layers between the first and the last cut, lines wrapped):
{
"model":"models/smollm2-135m/SmolLM2-135M-Instruct-Q8_0.gguf",
"arch":"llama",
"n_layer":30,
"n_expert":0,
"n_expert_used":0,
"files":1,
"backend":"cpu",
"cache_type_k":"F16",
"cache_type_v":"F16",
"weights":{"embeddings":{"tensors":1,"bytes":30081024},"attention":{"tensors":120,"bytes":28200960},
"dense_ffn":{"tensors":90,"bytes":84602880},"shared_experts":{"tensors":0,"bytes":0},"routed_experts":{"tensors":0,
"bytes":0},"routers":{"tensors":0,"bytes":0},"norms":{"tensors":61,"bytes":140544},"mtp":{"tensors":0,"bytes":0},
"other":{"tensors":0,"bytes":0},"total":143025408,"host":143025408,"device":0,"routed_experts_host":0},
"context":{"n_ctx":4096,"n_seq":1,"n_batch":512,"n_gpu_layers":0,"n_cpu_moe":0,"kv_layer":3145728,"kv":94371840,
"state":0,"activations":119539712,"scratch":1671168,"repacked":142884864,"repacked_from":142884864,
"repacked_experts":0,"expert_cache":0,"device_weights":0,"device_kv":0,"device_buffers":0,"device_total":0,
"device_free":0},
"mmproj":0,"beside":{"device":0,"host":0},
"host_need":358608128,
"device_need":0,
"device_capacity":0,
"ram_available":95372947456,
"ram_total":133827751936,
"margin":4055386422,
"verdict":"fits",
"n_gpu_layers":0,
"verdict_text":"fits resident",
"placement":{"n_gpu_layers":0,"n_cpu_moe":0,"devices":[],"layers":[{"index":0,"on":"cpu","device":null,
"bytes":3764736,"host_bytes":3764736,"device_bytes":0,"kv_bytes":3145728},{"index":29,"on":"cpu","device":null,
"bytes":3764736,"host_bytes":3764736,"device_bytes":0,"kv_bytes":3145728}],"embeddings":{"on":"cpu","device":null,
"bytes":30081024},"head":{"on":"cpu","device":null,"bytes":2304},"other":{"on":"cpu","device":null,"bytes":0},
"mtp_bytes":0,"host_bytes":143025408,"device_bytes":0,"kv_layer_bytes":3145728,"kv_host_bytes":94371840,
"kv_device_bytes":0}
}A mixture-of-experts model with ten layers on a device and the routed
experts of the first eighteen layers in host memory (Granite 3.1 1B
A400M, --backend opencl --gpu-layers 10 --n-cpu-moe 18;
layers 14 to 17 are on the device with their experts on the CPU):
{
"model":"models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf",
"arch":"granitemoe",
"n_layer":24,
"n_expert":32,
"n_expert_used":8,
"files":1,
"backend":"opencl",
"cache_type_k":"F16",
"cache_type_v":"F16",
"weights":{"embeddings":{"tensors":1,"bytes":41290200},"attention":{"tensors":96,"bytes":44089344},
"dense_ffn":{"tensors":0,"bytes":0},"shared_experts":{"tensors":0,"bytes":0},"routed_experts":{"tensors":72,
"bytes":731381760},"routers":{"tensors":24,"bytes":3145728},"norms":{"tensors":49,"bytes":200704},
"mtp":{"tensors":0,"bytes":0},"other":{"tensors":0,"bytes":0},"total":820107736,"host":1082264024,
"device":207073280,"routed_experts_host":544210944},
"context":{"n_ctx":4096,"n_seq":1,"n_batch":512,"n_gpu_layers":10,"n_cpu_moe":18,"kv_layer":8388608,"kv":117440512,
"state":0,"activations":189937664,"scratch":105941760,"repacked":494813184,"repacked_from":25583616,
"repacked_experts":467140608,"expert_cache":0,"device_weights":248363520,"device_kv":83886080,
"device_buffers":189943808,"device_total":21458059264,"device_free":21244149760},
"mmproj":0,"beside":{"device":0,"host":0},
"host_need":1495583960,
"device_need":522193408,
"device_capacity":19669887659,
"ram_available":95466070016,
"ram_total":133827751936,
"margin":4055386422,
"verdict":"fits_partial",
"n_gpu_layers":10,
"experts":{"routed_host":544210944,"count":774,"expert_bytes":944810,"dense_need":484232408,"cache":544210944,
"cached":576},
"verdict_text":"fits with 10 of 24 layers on the device",
"placement":{"n_gpu_layers":10,"n_cpu_moe":18,"devices":[{"name":"opencl","first_layer":14,"layers":10,"head":true}],
"layers":[{"index":0,"on":"cpu","device":null,"bytes":34680832,"host_bytes":34680832,"device_bytes":0,
"kv_bytes":8388608,"experts_on":"cpu","experts_bytes":32636928},{"index":23,"on":"device","device":"opencl",
"bytes":34680832,"host_bytes":0,"device_bytes":34680832,"kv_bytes":8388608,"experts_on":"device",
"experts_bytes":32636928}],"embeddings":{"on":"cpu","device":null,"bytes":41290200},"head":{"on":"device",
"device":"opencl","bytes":4096},"other":{"on":"cpu","device":null,"bytes":0},"mtp_bytes":0,"host_bytes":613034456,
"device_bytes":207073280,"kv_layer_bytes":8388608,"kv_host_bytes":117440512,"kv_device_bytes":83886080}
}As a server
oklm plan projects a cache of -c tokens for
each of --parallel sequences, which is what
oklm run and oklm chat make.
oklm serve does not: its requests share one paged cache,
whose size is what is left beside the model on a device, four contexts
on the CPU, or --kv-mem. --serve projects
that:
$ oklm plan -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --serve -c 40960 --backend vulkan
context : 40960 tokens x 512 sequences, batch 512, KV F16 / F16, backend vulkan
cache : a server's, shared by its requests: 97664 tokens, 2.4 contexts, 13.41 GiB (what is left beside the model, where four contexts are kept when there is room)
...
device : 36 of 36 layers, weights 4.35 GiB, KV 13.41 GiB, buffers 428.8 MiB: 18.18 GiB of 18.32 GiB usable (19.81 GiB free now)
verdict : fits resident (every layer on the device), with a smaller cache: 2.4 contexts
The cache's blocks come from oklm_kv_cache_rule(), the
function oklm serve sizes its cache with (over
oklm_kv_blocks_fit(), the room a device has beside the
model), so the plan and the server cannot drift: the weights go first,
then the cache takes what is left, one block at least; with
--kv-mem MB it is that size, whatever the room (a size
above what a device has left places fewer layers there). The sequences
are --parallel, else the server's own count: as many as a
step has rows and the cache has blocks. --kv-mem -1 is the
contiguous cache, the whole context for each sequence, four without a
count.
A model larger than its device. When the device's
budget does not hold every layer's weights, the activations and a block
of cache, and no size is given, the cache is not the one block that is
left: the rule sizes it for the layers that do go on the device. Those
are the layers --gpu-layers asks for, else the most that
leave the device one context of their cache (with the draft model and
what loads later); the cache is then what the device has beside them,
four contexts at most, and the other layers keep theirs in host memory,
as a model split by --gpu-layers does.
oklm_kv_cache_room() finds the layers and the room
(oklm_kv_cache_ask::part, part_layers,
part_fit) for the server and for the plan, and
oklm_kv_cache_rule() gives the blocks and the layers the
context is made with (oklm_kv_cache_size::part,
gpu_layers):
$ OKLM_GPU_MEM_MB=4096 oklm plan -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --serve -c 4096 --backend vulkan
cache : a server's, shared by its requests: 4736 tokens, 1.2 contexts, 666.0 MiB (the model is larger than its device: 407.0 MiB on the device with 22 of 36 layers, 259.0 MiB in host memory)
...
device : 22 of 36 layers, weights 2.84 GiB, KV 407.0 MiB, buffers 428.8 MiB: 3.66 GiB of 3.67 GiB usable
verdict : fits with 22 of 36 layers on the device, with a smaller cache: 1.2 contexts
A server started so tells the same 22 layers and 4736 tokens
(GET /api/context) and the same 3 929 860 096 bytes for the
model on the device (device_memory.model of
GET /api/stats), on Vulkan and on OpenCL, and answers at
11.7 to 11.9 tokens a second where the whole model on the device gives
93 (2026-11-16, RX 7900 XT held to 4 GiB). --json says it
in serve.part. A size given with --kv-mem and
every layer asked for with --gpu-layers keep what they had.
With OKLM_GPU_MEM_MB the plan's device is that size, and
what it has free is not asked, as for the layers and the cache.
A model the device holds with less than one context of
cache. A device that holds the whole weights may have less than
one context left beside them: Qwen3-8B Q4_K_M at 131072 tokens on a card
of 20 GB has room for 0.7 contexts. With no --gpu-layers
and no --kv-mem, the rule then makes the choice it makes
for a model larger than its device: the most layers that leave the
device one context of their cache go there, the other layers and their
cache stay in host memory (oklm_kv_cache_ask::part_cache,
oklm_kv_cache_size::part_cache;
serve.part_cache in --json, with
serve.part). The server holds a conversation that fills the
context, and runs slower than the whole model on the device would:
$ oklm plan -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --serve -c 131072 --backend vulkan
cache : a server's, shared by its requests: 133664 tokens, 1.0 contexts, 18.36 GiB (the device holds the model with less than one context beside it: 14.28 GiB on the device with 28 of 36 layers, 4.08 GiB in host memory, slower than the whole model on the device)
...
device : 28 of 36 layers, weights 3.48 GiB, KV 14.28 GiB, buffers 428.8 MiB: 18.17 GiB of 18.32 GiB usable (19.38 GiB free now)
verdict : fits with 28 of 36 layers on the device, with a smaller cache: 1.0 contexts
A server started so tells the same 28 layers and 133664 tokens
(GET /api/context, kv_blocks 8354 in
GET /api/stats), on OpenCL and on Vulkan, and answers 128
tokens in 7.1 and 7.6 s where the whole model on the device
(--gpu-layers 36, whose cache then holds 97664 tokens, 0.7
contexts) takes 1.5 and 1.3 s (2026-11-16, RX 7900 XT). What does not
change:
- a device that has one context or more beside the whole model keeps every layer there and gives the cache what is left, to the block (97664 tokens for Qwen3-8B at 40960);
--gpu-layerskeeps the layers it asks for and--kv-memthe size it asks for: every layer asked for at 131072 tokens is stilldoes not fit: the cache holds less than one context;- when no count of layers leaves the device one context, the whole
model stays there and the plan says the same
does not fit; - host memory must hold the layers and the cache that stay there: when
it does not, the verdict is
does not fit, by the host's figures; - a server that runs keeps its layers where they are: a larger context
asked of it (
POST /api/context) is sized beside the layers in place, and a server started with fewer layers keeps them at a smaller context.
The verdict has one case more and one reason more. A cache that holds
one context at least and less than four still fits,
with a smaller cache: the server serves, and keeps fewer
conversations of that length at once. A cache that holds less than one
context does not fit
(does not fit: the cache holds less than one context, exit
status 3): a conversation filling the context could not be kept.
--json gains
"serve": {"cache_tokens", "cache_bytes", "cache_want", "cache"},
cache being full, smaller or
none, cache_want the tokens of four contexts,
or of the size asked for, and part true when the cache was
sized for a part of the layers (a model larger
than its device); --json then has
"serve": {..., "part": true}, and
"part_cache": true beside it when the device does hold the
whole weights and the layers were taken off it for the cache's room.
Without --serve, oklm plan means what it
meant: at 40960 tokens and --parallel 4 it says
fits with 23 of 36 layers on the device, the plan of a run
that reserves four whole contexts, where the server keeps every layer on
the device with a cache of 2.4. The graphical tools' memory line is the
plan with --serve (gui.md), and counts as free what the
caller frees first: device memory
(oklm_mem_plan_params::dev_reclaim) and host memory
(ram_reclaim), which a server that is stopped and started
again gives back.
What shares the memory. With --serve,
what the server loads beside the model is counted where the server
places it, and what goes on the model's device comes out of the cache's
room before the cache is sized, through the same rule
(oklm_kv_cache_ask::draft and pending):
$ oklm plan -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --serve -c 8192 --backend vulkan --draft models/qwen3-0.6b/Qwen3-0.6B-Q8_0.gguf
cache : a server's, shared by its requests: 59072 tokens, 7.2 contexts, 8.11 GiB (four contexts or more: what is left beside the model)
...
draft model 458.0 MiB (weights 157.6 MiB, cache and buffers 300.3 MiB)
device : 36 of 36 layers, weights 4.35 GiB, KV 8.11 GiB, buffers 428.8 MiB: 18.18 GiB of 18.32 GiB usable (19.81 GiB free now)
of which the draft model 5.30 GiB
Without --draft the same command plans 97664 tokens; a
server started with that draft model tells 59072 in
cache_tokens, on Vulkan and on OpenCL (2026-11-16, RX 7900
XT).
| Option | Counted |
|---|---|
--draft MODEL |
the context the server gives a draft model
(oklm_serve_draft_params()): the model's backend and
length, a paged cache of five contexts, one sequence; on the device with
oklm_context_device_bytes(), the rest in host memory |
--mmproj FILE, --tiles SIDE |
the encoder on the model's device (its weights and the activations
of the largest picture the server accepts); on the CPU backend, the
file's weights in host memory and the arena its activations are computed
in (oklm_vision_host_bytes_at(): a tile's, or the largest
picture's for an encoder at the picture's own size;
oklm_audio_host_bytes() for a sound's encoder) |
--transcribe-model DIR,
--transcribe-backend B |
on the model's device when B is that device; else (cpu,
the default) in host memory: the checkpoint's weights, the recognizer's
two arenas and the copies it makes of its weights
(oklm_whisper_host_bytes()) |
--speak-model FILE, --speak-vocoder FILE,
--speak-backend B |
likewise: the language model's weights, cache and activations and
the vocoder's network on the device; or in host memory the language
model's weights with the context the synthesizer opens, the vocoder's
weights with its copies and the longest call of its network, of as many
codes as the language model's context holds tokens
(oklm_speech_host_bytes(),
oklm_vocoder_host_call()). A catalogue model's vocoder is
found without --speak-vocoder |
--embed-model FILE, --embed-backend B,
--rerank-model FILE, --rerank-backend B |
the context the embedder or the reranker makes
(oklm_embed_context_params(),
oklm_rerank_context_params()): on the model's device when B
is that device (oklm_context_device_bytes(), what the
server tells as device_memory.embed and
rerank), with what it keeps in host memory; else
(cpu, the default) its weights, cache, activations and
scratch in host memory |
When an encoder or a sound model shares the device, the backend's own
buffers are left their reserve too. The figures are those of
oklm_server_beside_plan()
(oklm/server/server.h), which sizes each part from its file
by the function the server sizes it with once loaded (server.md); the plan takes
their sums (oklm_mem_plan_params::beside_dev,
beside_held, beside_host). What the server
loads after it sized its cache (an encoder, the sound models) comes out
of the cache's room; what it loads before (an embedding or a reranking
model on the device, beside_held) comes out of the device's
budget and of its free memory, as the server finds them taken
(oklm_kv_blocks_fit_beside()). The table shows them as
beside the model under the host memory and
of which what loads beside the model under the device. What
computes on the CPU is counted with what it works with there, sized by
what sizes it when it is loaded: bge-small-en-v1.5 Q8_0 (a file of 35
MiB) 112 MiB, MiniLM-L6 F16 (44 MiB) 120 MiB, Whisper base (277 MiB) 609
MiB, OuteTTS 0.2 with its vocoder (636 MiB) 1.3 GiB, SmolVLM-256M's
projector on the CPU (181 MiB) 284 MiB. The vocoder's call is the
longest it can be given, a whole prompt's codes (4 096 for OuteTTS 0.2's
context): it was counted at 2 048 frames before.
A part on a device that is not the model's (the
model on vulkan and --embed-backend vulkan1;
or the model on the CPU and a side model on a device) takes nothing of
the model's cache. It is counted on its own device by the same
functions, with what it keeps in host memory, and that device is judged
for it: its need against what the device can take
(oklm_server_beside::other, a device each). The plan prints
a line for each and ends with status 3 when one is short:
$ OKLM_VK_DEVICES=all oklm plan -m models/qwen3-0.6b/Qwen3-0.6B-Q8_0.gguf --serve -c 8192 --backend vulkan \
--embed-model models/bge-small-en-v1.5/bge-small-en-v1.5-q8_0.gguf --embed-backend vulkan1
...
verdict : fits resident (every layer on the device)
on vulkan1 : 94.1 MiB for the embedding model, of 84.13 GiB usable there
A part that cannot be sized is said on the error stream and not
counted
(oklm: not counted: the speech model (no vocoder named)): a
file that is not read, a backend that is not one here. These options are
refused without --serve, where --draft and
--mmproj count as they did: the draft model with the
model's own context, the projector's weights in host memory.
The warning of run, chat and serve
run, chat and serve make the
same projection once the model's headers are read, before the context is
created, and warn on stderr (the numbers, and a pointer to
oklm plan) when the weights, the cache and the buffers
exceed the available RAM plus the device memory the context would use;
serve counts its --parallel sequences (four by
default) and a --draft model. Nothing is refused: mapped
weights can still page from the disk, only slowly.