ADR-0021: adapters served beside the weights
Records how several LoRA adapters are served in the same batches (serving S8.5). It amends ADR-0020's "Applying" decision, which only merged one adapter into the weights.
Status: accepted (2026-11-04).
Context
serving_spec.md lists LoRA
multiplexing among the extensions: several adapters served in one batch.
ADR-0020 applied an adapter by merging it into a copy of the model as it
loads (--lora), which serves one adapter per server and
writes quantized weights it touches as F16. Its consequences left the
products beside the weights for when several adapters were asked
for.
Decision
- Products beside the weights. For each weight an adapter adapts, a sequence with that adapter computes W x + s B (A x): A [inputs, rank] then B [rank, outputs] after the weight's own product, added to its output (to the residual directly where the projection writes there). The weights are untouched, so a quantized model stays quantized and every sequence without an adapter computes exactly what it did before.
- Stacks per weight. A context keeps every adapter loaded into it as F32 stacks per weight, A [inputs, rank, adapters + 1] and B [rank, outputs, adapters + 1]. Slot 0 is zeros (the model alone) and the scale is folded into B. The rank is the largest of the adapters'. On a device it is padded to a multiple of 32, which the device products need: A with zero rows and B with zero columns, so the product does not change. There is one copy in host memory and one on each device the context runs layers on.
- Two ways to multiply. When the adapted rows of a
batch are one run with one adapter, they multiply that adapter's slices
(two products per weight). Otherwise every row of the batch multiplies
its own slot in a gathered product (
oklm_matmul_id(), the experts' product), which is three launches per weight however many adapters the batch mixes. - Fused groups. Where a device computes Q, K and V (or gate and up) as one product, the adapters of the group are one pair too (amended 2026-11-04): A stacks each adapted target's rank rows, B is block-diagonal over the fused output's rows (zeros for a target not adapted), so the group takes the launches of one weight. Built when at least two of the group's targets are adapted.
- One adapter per sequence, set while it is empty. An
adapter is chosen per sequence (
oklm_seq_set_adapter()) and only for an empty one, because its cached rows depend on the adapter. - Prefix cache. Each block of the paged cache records the salt of the adapter that wrote it. The salt enters the block's hash and is compared on a lookup, so a prefix computed with one adapter (or with none) is never attached to a sequence of another. With no adapter the salt is 0 and the hashes are those of before.
- Server.
oklm serve --lora-module NAME=ADAPTER.gguf(repeatable) loads adapters beside the model. A request whosemodelnames one runs with it, as vLLM does. Any other name, or none, runs with the model alone./v1/modelslists the adapters with the model as theirparent. The scheduler reuses a sequence's rows, or waits for a prefix being computed, only between requests of the same adapter. - Merging stays.
--lorastill merges one adapter forrun,chat,pplandserve. A single adapter merged costs nothing per token.
Consequences
Adapters can be mixed freely in a batch. In server.md, 64 clients on OpenCL over SmolLM2-135M with a q,v adapter run at 73% of the model's throughput, and Qwen2.5-0.5B Q4_K_M with q,k,v,o at 80%. Requests spread over the model and two adapters run as fast as one adapter. The cost is the adapters' launches, about two per adapted weight and token step, which weigh most on small models and on adapters of every projection (all seven of SmolLM2: 46%). With the fused groups (Q, K, V; gate, up), measured again on 2026-11-04 at 64 clients: all seven projections 4177 to 5223 tokens/s (58 % of the model's 9061), q and v 6984 to 7449, Qwen2.5-0.5B's q, k, v, o 4093 to 4251. The fused pairs are kept beside the separate ones (the host path and the non-fused models use those), and a block-diagonal B holds zeros: for SmolLM2's seven projections at rank 8 (32 on a device) about 29 MiB more per slot of the stacks (each adapter's and the model's zero slot).
Each loaded adapter takes its stacks' memory on every backend (a few MB at rank 8 for these models). Loading one restacks all of them, so adapters are loaded when the server starts, not per request. Expert and latent-attention weights cannot be adapted.