OKLMdocumentation v1.0.3
On this page

Memory plan

What a model would take before anything loads: oklm plan for one model or every model of a directory, and the warning run, chat and serve print when a model will not fit. Inference itself is in inference.md; the expert cache the plan sizes for a MoE model in moe.md.

Status: maintained.

oklm plan -m model.gguf [-c CTX] [--parallel N] [-ctk TYPE] [-ctv TYPE] [--gpu-layers N] [--backend cpu|opencl|vulkan|auto]
          [--draft small.gguf] [--mmproj mmproj.gguf] [--json]
          [--serve [--kv-mem MB] [--tiles SIDE] [--transcribe-model DIR [--transcribe-backend B]]
           [--speak-model FILE [--speak-vocoder FILE] [--speak-backend B]]
           [--embed-model FILE [--embed-backend B]] [--rerank-model FILE [--rerank-backend B]]]
oklm plan --list [DIR] [--recommend] [-c CTX] [--backend cpu|opencl] [--json]

plan takes GGUF files; a Hugging Face directory is refused ("convert a Hugging Face checkpoint first"): loading one stacks per-expert matrices, widens FP8 ones and transposes GPT-2's, all of which read the weights the plan is meant not to touch.

plan says what a model would take before anything loads. It reads the GGUF metadata and tensor table only (any shard opens the whole model, the files mapped without read-ahead, no tensor data touched) and prints:

  • the weights by kind (embeddings and output, attention, dense FFN, shared experts, routed experts, routers, norms, multi-token prediction blocks) with their tensor counts and sizes, as oklm info counts the file;
  • the host memory: the resident weights (the repacked copies the CPU matrix product makes counted in place of the mapped matrices they stand for; the routed experts' copies, made when they fit beside the mapping and no expert cache is asked for, counted on top of it, since decode steps from three threads still read the file layout, but for Q8_0 and Q5_0 experts, whose copies are the file's size), the KV cache of -c tokens x --parallel sequences in the -ctk / -ctv types (MLA's compressed cache for DeepSeek V2 / V3), a hybrid stack's recurrent state (with the states and the journal kept for cut sequences, hybrid-layers-design.md), the activation buffers and the scratch, a draft model's weights and context, a projector's weights. These come from oklm_context_mem_estimate(), which sizes them with the functions oklm_context_new() allocates with: on the CPU the figures are the context's own (checked on every tiny model in test_mem_plan);
  • with --backend opencl (or auto finding a device), the device's layers (--gpu-layers, or what fits its memory), their weights, cache and buffers, against its free memory (AMD's query) or its size less 8 %. The default backend is cpu: no device is opened;
  • the available RAM (MemAvailable on Linux, GlobalMemoryStatusEx on Windows, host_statistics64 on macOS), less a margin of 3 % (at least 256 MiB), and the verdict: fits resident, fits with N layers on the device, or does not fit (exit status 3);
  • for a MoE model, the routed experts' share, what stays resident if they were streamed from disk (the dense part, its cache and buffers), and the largest expert cache that would fit beside it.

--json prints the same as one JSON object, with each layer's placement (JSON output).

--list [DIR] finds the GGUF models under models/ (or DIR, walked recursively, hidden entries left out; a split model's shards counted once, as their first) and projects each from its headers alone, with a context of 8192 tokens capped at the model's trained context (-c sets another, uncapped). A row each, by size on disk, the smallest first: the size, the need, the verdict, the file. The need is what the model takes resident (its weights, cache and buffers); for a MoE model, the part that stays resident when the routed experts are left in the file (the dense part, its cache and buffers) plus the smallest expert cache:

min expert cache = expert size (gate, up and down of one expert) x experts per token x MoE layers x 2

(the experts one token uses in every layer, twice: the token's and the next one's while the router's choice moves; never more than every routed expert). The verdict, the easiest first:

  • fits resident on the CPU: every weight resident, with the margin;
  • fits on the GPU with N layers (only with --backend opencl, which asks the device; the default backend is cpu and no device is opened): every layer when they all fit, else the most layers with which the device and the host both have room, when the model does not fit resident on the CPU;
  • fits only with the expert cache: a MoE model whose need fits where the whole model does not (--expert-cache, below);
  • does not fit.

A MoE row adds its resident figure and the two terms of its need. Files that are GGUF but not a model OKLM loads (a projector, an importance matrix, a vocabulary) are listed apart with the reason. --recommend (which implies --list) names the largest model that fits resident and the largest that runs only with the expert cache. --json prints the list as one object: models (each with size, resident, dense, min_expert_cache, need, fit: cpu, gpu, expert_cache or none, gpu_layers), skipped, the memory, and recommend.resident and recommend.expert_cache (paths or null). On this machine (125 GiB, no device asked; rows cut):

$ oklm plan --list --recommend
      size       need  verdict                                  model
...
 136.4 MiB  440.0 MiB  fits resident on the CPU                 models/smollm2-135m/SmolLM2-135M-Instruct-Q8_0.gguf
 782.1 MiB   1.32 GiB  fits resident on the CPU                 models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf [MoE 8/32: resident 1.32 GiB, with the expert cache 656.4 MiB + 348.8 MiB]
  3.92 GiB   4.74 GiB  fits resident on the CPU                 models/olmoe/OLMoE-1B-7B-0125-Instruct-Q4_K_M.gguf [MoE 8/64: resident 4.74 GiB, with the expert cache 1.11 GiB + 930.0 MiB]
  4.68 GiB   6.78 GiB  fits resident on the CPU                 models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf
 15.26 GiB  16.80 GiB  fits resident on the CPU                 models/qwen3-8b/Qwen3-8B-BF16.gguf
 17.05 GiB  18.21 GiB  fits resident on the CPU                 models/glm-4.7-flash/GLM-4.7-Flash-Q4_K_M.gguf [MoE 4/64: resident 18.21 GiB, with the expert cache 2.59 GiB + 1.95 GiB]
RAM: 99.45 GiB available of 124.64 GiB, 3.78 GiB kept free; context 8192 tokens unless the model's is shorter; backend cpu
left out: 3 files
  models/qwen3-8b/Qwen3-8B.imatrix.gguf: not a model (a projector, an importance matrix, a vocabulary...)
  models/smolvlm-256m/mmproj-SmolVLM-256M-Instruct-f16.gguf: not a model (a projector, an importance matrix, a vocabulary...)
  models/ultravox-1b/mmproj-ultravox-v0_5-llama-3_2-1b-f16.gguf: not a model (a projector, an importance matrix, a vocabulary...)
recommended, resident: models/glm-4.7-flash/GLM-4.7-Flash-Q4_K_M.gguf (fits resident on the CPU)
recommended, with the expert cache: none (no MoE model here needs it to fit)

JSON output

oklm plan -m model.gguf --json prints one JSON object on one line (oklm_mem_plan_layout_json()): every figure of the text form, then placement, where each layer would be. Sizes are bytes. The exit status is the text form's (3 when the model does not fit).

Member Type What
model string the path given
arch string architecture
n_layer number blocks of the main stack
n_expert, n_expert_used number experts per MoE layer (0 for a dense model), experts a token goes to
files number files of the model (shards)
backend string the backend the context would run on (cpu, opencl, vulkan)
cache_type_k, cache_type_v string the key and value cache's types
weights object per kind (embeddings, attention, dense_ffn, shared_experts, routed_experts, routers, norms, mtp, other) {"tensors": N, "bytes": N}, the file's; then total (every tensor), host (resident in host memory: the CPU layers' and the embeddings', the repacked matrices at their copies' size), device (file bytes of the weights placed on the device), routed_experts_host (the routed experts among host)
context object the context's projection, below
draft object with --draft only: weights (resident), cache_room (with --serve: the device bytes its context takes of the cache's room first) and context (the same members as context)
mmproj number the projector's weights (0 without --mmproj, and with --serve, where the projector is in beside)
beside object what loads beside the model (as a server): device, the bytes on the model's device, and host, those in host memory (the weights of what computes on the CPU and what it works with there); both 0 without --serve
host_need number the projected host peak: weights, cache, state, activations, scratch, draft, projector and what loads beside the model
device_need number the projected device memory (0 on the CPU)
device_capacity number what the device can take: its free memory when it tells, else its size less 8 % (0 on the CPU)
ram_available, ram_total number available and installed physical memory
margin number the headroom kept free
verdict string fits, fits_partial (some layers on the device, the others on the CPU) or no_fit
verdict_text string the verdict's sentence of the text form (fits with 10 of 24 layers on the device)
n_gpu_layers number layers on the device
experts object a MoE model with routed experts in host memory only: routed_host, count (routed experts over every layer), expert_bytes (one expert, on average), dense_need (the host need with the routed experts streamed from disk), cache (the largest expert cache that fits beside it), cached (experts it holds)
placement object each layer's placement, below

The verdict holds when host_need + margin <= ram_available and device_need <= device_capacity.

context:

Member What
n_ctx, n_seq, n_batch context length, sequences, tokens a forward pass
n_gpu_layers layers on the device (the last ones, with the head)
n_cpu_moe the first layers whose routed experts stay in host memory (--n-cpu-moe, resolved)
kv_layer one layer's key/value cache (the widest layer's when they differ: Gemma 4)
kv host: the cache of the layers on the CPU, each at its own width, none for a block that reads another's (Gemma 4 E2B, E4B) nor for a block without attention (the linear blocks of qwen35, qwen35moe and qwen3next, the convolution blocks of lfm2 and lfm2moe)
state host: a hybrid stack's recurrent state, and the states and journals kept for cut sequences (every slot counted; the states alone, without the journals, when a linear block runs on a device)
activations host: the activation buffers
scratch host: the rest of the context's allocations
repacked, repacked_from host: the CPU matrix product's repacked copies, and the mapped bytes they stand for
repacked_experts the part of repacked that is the routed experts' copies, on top of their mapping
expert_cache host: the routed-expert cache (0 without one)
device_weights, device_kv, device_buffers device: its layers' weights as the device holds them, their cache (none for a block without attention, nor for a block that reads another block's), the activation buffers (with the embeddings per layer of a batch for Gemma 4 E2B and E4B)
device_total, device_free the device's memory and what is free now (0 when unknown or on the CPU)

placement (oklm_layout_place_json()), from the tensor table and the rule the weights are counted with: the last n_gpu_layers layers on the device with the head, the others on the CPU with the embeddings, a device layer's routed experts in host memory when it is among the first n_cpu_moe:

Member Type What
n_gpu_layers, n_cpu_moe number as in context
devices array each device holding layers, in layer order: name (the backend), first_layer, layers (their count), head (true for the one that holds the head). Empty on the CPU
layers array one entry per layer of the main stack, below
embeddings, head, other object the parts outside the blocks (as oklm info --json lays them out, convert.md): on (cpu or device), device (its name or null), bytes
mtp_bytes number the multi-token prediction blocks: left in the file, read by --mtp only
host_bytes, device_bytes number the weights' file bytes on either side, over the layers and the parts
kv_layer_bytes number one layer's cache (the widest layer's when they differ)
kv_host_bytes, kv_device_bytes number the cache of the CPU layers, of the device layers

A layer:

Member Type What
index number the layer
on string cpu or device
device string or null the device's name
bytes number its tensors' file bytes
host_bytes, device_bytes number of them, those in host memory and those on the device
kv_bytes number its own key/value cache, on the side the layer is on: 0 for a block that keeps no rows (no attention, or another block's cache read), a narrower block's own size
experts_on string a layer with routed experts only: cpu or device
experts_bytes number a layer with routed experts only: their file bytes

The figures add up: placement.host_bytes + device_bytes + mtp_bytes is weights.total; placement.device_bytes is weights.device; placement.host_bytes - context.repacked_from + context.repacked is weights.host; the layers' kv_bytes on the CPU add up to context.kv, those on the device to context.device_kv (test_placement and test_rows_of_attention_blocks in tests/model/test_mem_plan.c, on every fixture: the two widths and the shared caches of Gemma 4, the blocks without attention of the hybrid stacks). The text form says how many layers keep rows when some do not: KV cache 256.0 MiB (8 of 32 layers x 32.0 MiB) for Qwen3.5-9B at 8192 tokens. The buffers are not split by layer: context.activations and context.scratch serve every CPU layer, context.device_buffers every device layer. context.device_weights is what the device holds for weights.device bytes of the file (a matrix it stores in another layout takes more). The head's tensors are the final norm, the output matrix, the rotary factors and the norm named token_embd_norm (the final norm of an lfm2 file, the norm of an encoder's summed embeddings). An output matrix that is the token embeddings (a tied head) is one tensor of the file: its bytes are counted once, with the embeddings the host looks up, and the copy the device holds for the output product is in context.device_weights alone.

A dense model on the CPU (SmolLM2-135M Q8_0; the layers between the first and the last cut, lines wrapped):

{
 "model":"models/smollm2-135m/SmolLM2-135M-Instruct-Q8_0.gguf",
 "arch":"llama",
 "n_layer":30,
 "n_expert":0,
 "n_expert_used":0,
 "files":1,
 "backend":"cpu",
 "cache_type_k":"F16",
 "cache_type_v":"F16",
 "weights":{"embeddings":{"tensors":1,"bytes":30081024},"attention":{"tensors":120,"bytes":28200960},
  "dense_ffn":{"tensors":90,"bytes":84602880},"shared_experts":{"tensors":0,"bytes":0},"routed_experts":{"tensors":0,
  "bytes":0},"routers":{"tensors":0,"bytes":0},"norms":{"tensors":61,"bytes":140544},"mtp":{"tensors":0,"bytes":0},
  "other":{"tensors":0,"bytes":0},"total":143025408,"host":143025408,"device":0,"routed_experts_host":0},
 "context":{"n_ctx":4096,"n_seq":1,"n_batch":512,"n_gpu_layers":0,"n_cpu_moe":0,"kv_layer":3145728,"kv":94371840,
  "state":0,"activations":119539712,"scratch":1671168,"repacked":142884864,"repacked_from":142884864,
  "repacked_experts":0,"expert_cache":0,"device_weights":0,"device_kv":0,"device_buffers":0,"device_total":0,
  "device_free":0},
 "mmproj":0,"beside":{"device":0,"host":0},
 "host_need":358608128,
 "device_need":0,
 "device_capacity":0,
 "ram_available":95372947456,
 "ram_total":133827751936,
 "margin":4055386422,
 "verdict":"fits",
 "n_gpu_layers":0,
 "verdict_text":"fits resident",
 "placement":{"n_gpu_layers":0,"n_cpu_moe":0,"devices":[],"layers":[{"index":0,"on":"cpu","device":null,
  "bytes":3764736,"host_bytes":3764736,"device_bytes":0,"kv_bytes":3145728},{"index":29,"on":"cpu","device":null,
  "bytes":3764736,"host_bytes":3764736,"device_bytes":0,"kv_bytes":3145728}],"embeddings":{"on":"cpu","device":null,
  "bytes":30081024},"head":{"on":"cpu","device":null,"bytes":2304},"other":{"on":"cpu","device":null,"bytes":0},
  "mtp_bytes":0,"host_bytes":143025408,"device_bytes":0,"kv_layer_bytes":3145728,"kv_host_bytes":94371840,
  "kv_device_bytes":0}
}

A mixture-of-experts model with ten layers on a device and the routed experts of the first eighteen layers in host memory (Granite 3.1 1B A400M, --backend opencl --gpu-layers 10 --n-cpu-moe 18; layers 14 to 17 are on the device with their experts on the CPU):

{
 "model":"models/granite-moe/granite-3.1-1b-a400m-instruct-Q4_K_M.gguf",
 "arch":"granitemoe",
 "n_layer":24,
 "n_expert":32,
 "n_expert_used":8,
 "files":1,
 "backend":"opencl",
 "cache_type_k":"F16",
 "cache_type_v":"F16",
 "weights":{"embeddings":{"tensors":1,"bytes":41290200},"attention":{"tensors":96,"bytes":44089344},
  "dense_ffn":{"tensors":0,"bytes":0},"shared_experts":{"tensors":0,"bytes":0},"routed_experts":{"tensors":72,
  "bytes":731381760},"routers":{"tensors":24,"bytes":3145728},"norms":{"tensors":49,"bytes":200704},
  "mtp":{"tensors":0,"bytes":0},"other":{"tensors":0,"bytes":0},"total":820107736,"host":1082264024,
  "device":207073280,"routed_experts_host":544210944},
 "context":{"n_ctx":4096,"n_seq":1,"n_batch":512,"n_gpu_layers":10,"n_cpu_moe":18,"kv_layer":8388608,"kv":117440512,
  "state":0,"activations":189937664,"scratch":105941760,"repacked":494813184,"repacked_from":25583616,
  "repacked_experts":467140608,"expert_cache":0,"device_weights":248363520,"device_kv":83886080,
  "device_buffers":189943808,"device_total":21458059264,"device_free":21244149760},
 "mmproj":0,"beside":{"device":0,"host":0},
 "host_need":1495583960,
 "device_need":522193408,
 "device_capacity":19669887659,
 "ram_available":95466070016,
 "ram_total":133827751936,
 "margin":4055386422,
 "verdict":"fits_partial",
 "n_gpu_layers":10,
 "experts":{"routed_host":544210944,"count":774,"expert_bytes":944810,"dense_need":484232408,"cache":544210944,
  "cached":576},
 "verdict_text":"fits with 10 of 24 layers on the device",
 "placement":{"n_gpu_layers":10,"n_cpu_moe":18,"devices":[{"name":"opencl","first_layer":14,"layers":10,"head":true}],
  "layers":[{"index":0,"on":"cpu","device":null,"bytes":34680832,"host_bytes":34680832,"device_bytes":0,
  "kv_bytes":8388608,"experts_on":"cpu","experts_bytes":32636928},{"index":23,"on":"device","device":"opencl",
  "bytes":34680832,"host_bytes":0,"device_bytes":34680832,"kv_bytes":8388608,"experts_on":"device",
  "experts_bytes":32636928}],"embeddings":{"on":"cpu","device":null,"bytes":41290200},"head":{"on":"device",
  "device":"opencl","bytes":4096},"other":{"on":"cpu","device":null,"bytes":0},"mtp_bytes":0,"host_bytes":613034456,
  "device_bytes":207073280,"kv_layer_bytes":8388608,"kv_host_bytes":117440512,"kv_device_bytes":83886080}
}

As a server

oklm plan projects a cache of -c tokens for each of --parallel sequences, which is what oklm run and oklm chat make. oklm serve does not: its requests share one paged cache, whose size is what is left beside the model on a device, four contexts on the CPU, or --kv-mem. --serve projects that:

$ oklm plan -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --serve -c 40960 --backend vulkan
context     : 40960 tokens x 512 sequences, batch 512, KV F16 / F16, backend vulkan
cache       : a server's, shared by its requests: 97664 tokens, 2.4 contexts, 13.41 GiB (what is left beside the model, where four contexts are kept when there is room)
...
device      : 36 of 36 layers, weights 4.35 GiB, KV 13.41 GiB, buffers 428.8 MiB: 18.18 GiB of 18.32 GiB usable (19.81 GiB free now)
verdict     : fits resident (every layer on the device), with a smaller cache: 2.4 contexts

The cache's blocks come from oklm_kv_cache_rule(), the function oklm serve sizes its cache with (over oklm_kv_blocks_fit(), the room a device has beside the model), so the plan and the server cannot drift: the weights go first, then the cache takes what is left, one block at least; with --kv-mem MB it is that size, whatever the room (a size above what a device has left places fewer layers there). The sequences are --parallel, else the server's own count: as many as a step has rows and the cache has blocks. --kv-mem -1 is the contiguous cache, the whole context for each sequence, four without a count.

A model larger than its device. When the device's budget does not hold every layer's weights, the activations and a block of cache, and no size is given, the cache is not the one block that is left: the rule sizes it for the layers that do go on the device. Those are the layers --gpu-layers asks for, else the most that leave the device one context of their cache (with the draft model and what loads later); the cache is then what the device has beside them, four contexts at most, and the other layers keep theirs in host memory, as a model split by --gpu-layers does. oklm_kv_cache_room() finds the layers and the room (oklm_kv_cache_ask::part, part_layers, part_fit) for the server and for the plan, and oklm_kv_cache_rule() gives the blocks and the layers the context is made with (oklm_kv_cache_size::part, gpu_layers):

$ OKLM_GPU_MEM_MB=4096 oklm plan -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --serve -c 4096 --backend vulkan
cache       : a server's, shared by its requests: 4736 tokens, 1.2 contexts, 666.0 MiB (the model is larger than its device: 407.0 MiB on the device with 22 of 36 layers, 259.0 MiB in host memory)
...
device      : 22 of 36 layers, weights 2.84 GiB, KV 407.0 MiB, buffers 428.8 MiB: 3.66 GiB of 3.67 GiB usable
verdict     : fits with 22 of 36 layers on the device, with a smaller cache: 1.2 contexts

A server started so tells the same 22 layers and 4736 tokens (GET /api/context) and the same 3 929 860 096 bytes for the model on the device (device_memory.model of GET /api/stats), on Vulkan and on OpenCL, and answers at 11.7 to 11.9 tokens a second where the whole model on the device gives 93 (2026-11-16, RX 7900 XT held to 4 GiB). --json says it in serve.part. A size given with --kv-mem and every layer asked for with --gpu-layers keep what they had. With OKLM_GPU_MEM_MB the plan's device is that size, and what it has free is not asked, as for the layers and the cache.

A model the device holds with less than one context of cache. A device that holds the whole weights may have less than one context left beside them: Qwen3-8B Q4_K_M at 131072 tokens on a card of 20 GB has room for 0.7 contexts. With no --gpu-layers and no --kv-mem, the rule then makes the choice it makes for a model larger than its device: the most layers that leave the device one context of their cache go there, the other layers and their cache stay in host memory (oklm_kv_cache_ask::part_cache, oklm_kv_cache_size::part_cache; serve.part_cache in --json, with serve.part). The server holds a conversation that fills the context, and runs slower than the whole model on the device would:

$ oklm plan -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --serve -c 131072 --backend vulkan
cache       : a server's, shared by its requests: 133664 tokens, 1.0 contexts, 18.36 GiB (the device holds the model with less than one context beside it: 14.28 GiB on the device with 28 of 36 layers, 4.08 GiB in host memory, slower than the whole model on the device)
...
device      : 28 of 36 layers, weights 3.48 GiB, KV 14.28 GiB, buffers 428.8 MiB: 18.17 GiB of 18.32 GiB usable (19.38 GiB free now)
verdict     : fits with 28 of 36 layers on the device, with a smaller cache: 1.0 contexts

A server started so tells the same 28 layers and 133664 tokens (GET /api/context, kv_blocks 8354 in GET /api/stats), on OpenCL and on Vulkan, and answers 128 tokens in 7.1 and 7.6 s where the whole model on the device (--gpu-layers 36, whose cache then holds 97664 tokens, 0.7 contexts) takes 1.5 and 1.3 s (2026-11-16, RX 7900 XT). What does not change:

  • a device that has one context or more beside the whole model keeps every layer there and gives the cache what is left, to the block (97664 tokens for Qwen3-8B at 40960);
  • --gpu-layers keeps the layers it asks for and --kv-mem the size it asks for: every layer asked for at 131072 tokens is still does not fit: the cache holds less than one context;
  • when no count of layers leaves the device one context, the whole model stays there and the plan says the same does not fit;
  • host memory must hold the layers and the cache that stay there: when it does not, the verdict is does not fit, by the host's figures;
  • a server that runs keeps its layers where they are: a larger context asked of it (POST /api/context) is sized beside the layers in place, and a server started with fewer layers keeps them at a smaller context.

The verdict has one case more and one reason more. A cache that holds one context at least and less than four still fits, with a smaller cache: the server serves, and keeps fewer conversations of that length at once. A cache that holds less than one context does not fit (does not fit: the cache holds less than one context, exit status 3): a conversation filling the context could not be kept. --json gains "serve": {"cache_tokens", "cache_bytes", "cache_want", "cache"}, cache being full, smaller or none, cache_want the tokens of four contexts, or of the size asked for, and part true when the cache was sized for a part of the layers (a model larger than its device); --json then has "serve": {..., "part": true}, and "part_cache": true beside it when the device does hold the whole weights and the layers were taken off it for the cache's room.

Without --serve, oklm plan means what it meant: at 40960 tokens and --parallel 4 it says fits with 23 of 36 layers on the device, the plan of a run that reserves four whole contexts, where the server keeps every layer on the device with a cache of 2.4. The graphical tools' memory line is the plan with --serve (gui.md), and counts as free what the caller frees first: device memory (oklm_mem_plan_params::dev_reclaim) and host memory (ram_reclaim), which a server that is stopped and started again gives back.

What shares the memory. With --serve, what the server loads beside the model is counted where the server places it, and what goes on the model's device comes out of the cache's room before the cache is sized, through the same rule (oklm_kv_cache_ask::draft and pending):

$ oklm plan -m models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf --serve -c 8192 --backend vulkan --draft models/qwen3-0.6b/Qwen3-0.6B-Q8_0.gguf
cache       : a server's, shared by its requests: 59072 tokens, 7.2 contexts, 8.11 GiB (four contexts or more: what is left beside the model)
...
  draft model                   458.0 MiB  (weights 157.6 MiB, cache and buffers 300.3 MiB)
device      : 36 of 36 layers, weights 4.35 GiB, KV 8.11 GiB, buffers 428.8 MiB: 18.18 GiB of 18.32 GiB usable (19.81 GiB free now)
              of which the draft model 5.30 GiB

Without --draft the same command plans 97664 tokens; a server started with that draft model tells 59072 in cache_tokens, on Vulkan and on OpenCL (2026-11-16, RX 7900 XT).

Option Counted
--draft MODEL the context the server gives a draft model (oklm_serve_draft_params()): the model's backend and length, a paged cache of five contexts, one sequence; on the device with oklm_context_device_bytes(), the rest in host memory
--mmproj FILE, --tiles SIDE the encoder on the model's device (its weights and the activations of the largest picture the server accepts); on the CPU backend, the file's weights in host memory and the arena its activations are computed in (oklm_vision_host_bytes_at(): a tile's, or the largest picture's for an encoder at the picture's own size; oklm_audio_host_bytes() for a sound's encoder)
--transcribe-model DIR, --transcribe-backend B on the model's device when B is that device; else (cpu, the default) in host memory: the checkpoint's weights, the recognizer's two arenas and the copies it makes of its weights (oklm_whisper_host_bytes())
--speak-model FILE, --speak-vocoder FILE, --speak-backend B likewise: the language model's weights, cache and activations and the vocoder's network on the device; or in host memory the language model's weights with the context the synthesizer opens, the vocoder's weights with its copies and the longest call of its network, of as many codes as the language model's context holds tokens (oklm_speech_host_bytes(), oklm_vocoder_host_call()). A catalogue model's vocoder is found without --speak-vocoder
--embed-model FILE, --embed-backend B, --rerank-model FILE, --rerank-backend B the context the embedder or the reranker makes (oklm_embed_context_params(), oklm_rerank_context_params()): on the model's device when B is that device (oklm_context_device_bytes(), what the server tells as device_memory.embed and rerank), with what it keeps in host memory; else (cpu, the default) its weights, cache, activations and scratch in host memory

When an encoder or a sound model shares the device, the backend's own buffers are left their reserve too. The figures are those of oklm_server_beside_plan() (oklm/server/server.h), which sizes each part from its file by the function the server sizes it with once loaded (server.md); the plan takes their sums (oklm_mem_plan_params::beside_dev, beside_held, beside_host). What the server loads after it sized its cache (an encoder, the sound models) comes out of the cache's room; what it loads before (an embedding or a reranking model on the device, beside_held) comes out of the device's budget and of its free memory, as the server finds them taken (oklm_kv_blocks_fit_beside()). The table shows them as beside the model under the host memory and of which what loads beside the model under the device. What computes on the CPU is counted with what it works with there, sized by what sizes it when it is loaded: bge-small-en-v1.5 Q8_0 (a file of 35 MiB) 112 MiB, MiniLM-L6 F16 (44 MiB) 120 MiB, Whisper base (277 MiB) 609 MiB, OuteTTS 0.2 with its vocoder (636 MiB) 1.3 GiB, SmolVLM-256M's projector on the CPU (181 MiB) 284 MiB. The vocoder's call is the longest it can be given, a whole prompt's codes (4 096 for OuteTTS 0.2's context): it was counted at 2 048 frames before.

A part on a device that is not the model's (the model on vulkan and --embed-backend vulkan1; or the model on the CPU and a side model on a device) takes nothing of the model's cache. It is counted on its own device by the same functions, with what it keeps in host memory, and that device is judged for it: its need against what the device can take (oklm_server_beside::other, a device each). The plan prints a line for each and ends with status 3 when one is short:

$ OKLM_VK_DEVICES=all oklm plan -m models/qwen3-0.6b/Qwen3-0.6B-Q8_0.gguf --serve -c 8192 --backend vulkan \
    --embed-model models/bge-small-en-v1.5/bge-small-en-v1.5-q8_0.gguf --embed-backend vulkan1
...
verdict     : fits resident (every layer on the device)
on vulkan1 : 94.1 MiB for the embedding model, of 84.13 GiB usable there

A part that cannot be sized is said on the error stream and not counted (oklm: not counted: the speech model (no vocoder named)): a file that is not read, a backend that is not one here. These options are refused without --serve, where --draft and --mmproj count as they did: the draft model with the model's own context, the projector's weights in host memory.

The warning of run, chat and serve

run, chat and serve make the same projection once the model's headers are read, before the context is created, and warn on stderr (the numbers, and a pointer to oklm plan) when the weights, the cache and the buffers exceed the available RAM plus the device memory the context would use; serve counts its --parallel sequences (four by default) and a --draft model. Nothing is refused: mapped weights can still page from the disk, only slowly.