ADR-0009: Tool-call wire format and constrained decoding grammar
Records how OKLM finds tool calls in a model's answer, how it represents them, and how generation is constrained so the calls are well formed.
Status: accepted (2026-10-30).
Context
Chat models are trained to call tools in their own syntax: Qwen 2.5 /
3, Hermes and SmolLM3 write
<tool_call>{json}</tool_call>, Llama 3.1+
answers with a bare {"name", "parameters"} object
(sometimes after the <|python_tag|> control token),
Mistral writes [TOOL_CALLS] then a JSON array or
name[ARGS]{json}, GLM-4.5 / 4.6 / 4.7 write
<tool_call>name then one
<arg_key>k</arg_key><arg_value>v</arg_value>
pair per argument (strings raw, other values as JSON), GLM-4-0414 opens
its turn with the function's name, then a newline and the arguments as
JSON. Their chat templates read the tool definitions from a
tools variable (OpenAI style JSON schemas) and render past
calls from the assistant message's tool_calls field. Small
models often get the JSON or the argument names wrong unless sampling is
constrained. llama.cpp constrains with GBNF grammars, which users
already write and share, and converts JSON schemas to them.
Decision
- Tool calls are represented once, OpenAI style: a JSON array of
{"id", "type": "function", "function": {"name", "arguments": {...}}}, with arguments as a JSON object. That is what chat templates receive inmessages[].tool_calls, whatoklm chatprints, and what the harness will consume. Tool results go back astoolmessages withtool_call_idandname. - The model's own wire format is used on the model side: OKLM parses
the Hermes, Llama 3, Mistral, GLM-4.5 and GLM-4-0414 syntaxes
(
oklm_tool_calls_parse()), picked from the chat template's markers or by name. The template renders the calls and the tools; OKLM never invents a format a model was not trained on. A template that reads neithertool_callsnortoolmessages (GLM-4-0414's) gets the conversation in the form it was trained on: each call an assistant turn with the name in itsmetadata, each result anobservationturn. - Grammars use llama.cpp's GBNF syntax (literals, classes, groups, repetitions with counts, comments), so existing grammar files work. Matching is a set of pushdown stacks over Unicode code points with partial UTF-8 at token boundaries; overlong encodings and surrogates are refused; left recursion is rejected when parsing. Control and end-of-generation tokens are allowed only when the grammar can end.
- JSON schemas are converted to GBNF
(
oklm_json_schema_to_grammar()): types, properties with required ones first and optional ones in order, arrays with item counts, enum / const, anyOf / oneOf, local$ref, string lengths. Between tokens a single optional space is allowed, so a model cannot loop on whitespace. - Constraint is lazy for tool calls: generation is free until the
model writes its call marker (
<tool_call>,[TOOL_CALLS],<|python_tag|>), then follows the grammar of the declared tools until the call is closed.--tool-choice requiredwrites the marker for the model (or constrains the whole answer for Llama 3). - The sampler checks the token it chose against the grammar and only masks the whole vocabulary when that token is refused (one check per token in the common case). The mask advances groups of tokens that share a prefix together; its result is tested against feeding every token's text to the grammar.
Consequences
Any GBNF file or JSON schema written for llama.cpp constrains OKLM
the same way. Integer ranges (minimum, maximum
and the exclusive forms) become digit-by-digit alternatives,
format date / time / date-time / uuid and a regular
expression subset in pattern (matched as a whole) become
their grammars; ranges of number schemas become plain
decimals (no exponent, 16 integer digits at most) compared digit by
digit with the bounds' exact decimal text, so 0.1 means 0.1 and not its
binary neighbour. Running the tools, permissions and the conversation
loop belong to the harness milestone.