MCP tool-use fine-tuning design (T0 audit)
How mcp_training_spec.md maps onto OKLM: what the stack already gives for teaching the default model to drive an MCP tool server, so T1–T3 reuse rather than duplicate. What exists, with evidence, is the "MCP training" section of gap-analysis.md; the tool-call wire format and its grammar are adr/0009-tool-calls-and-grammars.md.
Status: T0 accepted (2026-11-05); T1 to T3 built and run (2026-11-07), see "T1 to T3" at the end.
Inference prompt and tool-call shape
The target model, Qwen3-8B, carries its own chat template
(tokenizer.chat_template), rendered by OKLM's Jinja engine
(oklm_chat_template_render(), src/model/chat_template.c;
ADR-0009). Qwen3's template is Hermes/ChatML: when a tools
array is passed it adds a block to the system turn and declares the call
syntax. Quoted from the model file (Qwen3-0.6B-Q8_0.gguf,
the same family and template):
<|im_start|>system
<system text>
# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type":"function","function":{"name":"Bash","description":"...","parameters":{...}}}
{"type":"function","function":{"name":"Read","description":"...","parameters":{...}}}
...
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
List the files here.<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "Bash", "arguments": {"command": "ls"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
a.txt b.txt
</tool_response><|im_end|>
<|im_start|>assistant
Two files: a.txt and b.txt.<|im_end|>
- The tools JSON is the OpenAI function shape. OKLM's
built-in tools are the literal in tools.c
tools_json_parts[](src/harness/tools.c:3358-3492); the Agent and SendMessage tools are agent.cagent_tool_json,send_message_json(src/harness/agent.c:491-509); MCP tools are added under theirmcp__server__toolnames (mcp.c).oklm_agent_tools_json()returns the array an agent offers (agent.h). - The call is parsed from the model's text by
tool_calls.c
parse_hermes()(src/model/tool_calls.c:378): one JSON object per<tool_call>…</tool_call>,{"name", "arguments"}, arguments a string or an object. The format is detected from the template (oklm_tool_format_detect(), tool_calls.c:21 —<tool_call>means Hermes) or forced with--tool-format; the opening marker (the grammar trigger) is<tool_call>(tool_calls.c:1043). - The result comes back as a user turn wrapped
<tool_response>… </tool_response>by the template; OKLM sends one result per call id (spec §3), errors included. - Over
--urlthe HTTP backend sends the OpenAI wire shape instead:messagesandtoolsas separate JSON, the call inchoices[].delta.tool_calls(agent_http.chttp_acc_feed). Training data must match the in-process rendered shape above (the tokens the model is trained and run on), not the wire JSON — decision 1. - Reasoning
(
<think>…</think>) is streamed as a separate event and kept out of the history sent back (M1); whether a training example keeps it is decision 2.
A live capture:
OKLM_LOG_LEVEL=DEBUG oklm agent -m <model> -p "…"
logs each raw model turn (agent: turn: …, agent.c:2736),
which shows the <tool_call> text verbatim.
Schema validation path
ADR-0009's grammar path both constrains and checks, so a data validator needs no new engine:
- Constrain generation.
oklm_json_schema_to_grammar()(include/oklm/model/grammar.h:160) turns a tool'sparametersJSON schema into GBNF (objects, required and optional keys, arrays with bounds, enums,anyOf/oneOf,$ref, stringminLength/maxLength/format/pattern, integer and decimalminimum/maximum). The agent builds one grammar of all offered tools (agent.cagent_grammar), so a turn's call is well-formed by construction. - Check a candidate. Build the grammar from the
tool's schema, feed a candidate call's argument text with
oklm_grammar_accept_text()(grammar.h:100) and requireoklm_grammar_complete()(grammar.h:107): a call that the grammar accepts to completion is schema-valid. This is the function a T1 data validator calls before an example enters the set. - Decimal bounds are exact (ADR-0009; no exponent, 16
digits), so a
number/integerrange is enforced to the digit.
Training mechanism (from R7)
LoRA fine-tuning is done on every backend (M15, R3–R7; ADR-0020, training.md):
oklm train --lora R [--lora-alpha A] [--lora-targets …]freezes the model and trains rank-R adapters on the chosen projections (q,k,v,oby default,all, or a list); the base may be quantized (--quantized-base, item 18) so an 8B fits a 20 GB GPU (≈18.7 GiB, training.md).- A checkpoint's weights file is a llama.cpp-format adapter
(
oklm_lora_merge(), oklm/io/lora.h);oklm run|chat|ppl|serve --lorause it,oklm serve --lora-moduleserves several (ADR-0021). - The chat-data path is built (2026-11-05):
oklm train --chat-data conv.jsonlreads the captured{text}JSONL, masks each conversation to its assistant tokens, and trains the loss on those only (training.md). So T1's capture feeds the trainer directly. - What this plan still adds, not R7 or the chat-data
path: the training data at scale (T1), the hyperparameters and the
validation loop (T2/T3). No gap in the mechanism — M15 (MCP client), R7
(LoRA) and
--chat-dataare all done, so the plan is unblocked (decision 4).
Reference datasets (format only)
Mined for layout and reasoning convention, never imported blindly (content, licence and quality are T1's to control):
- Hermes-style function-calling sets (e.g. the Glaive
/ NousResearch function-calling corpora): already the
<tools>/<tool_call>shape Qwen3 uses, so their per-turn layout maps one-to-one; mirror the turn structure, write OKLM's own tool set and results. - The agent's own eval tasks (tests/evals/,
especially
03-mcp-tool) are the most faithful source: a real run throughoklm agentagainst the mock MCP server produces exactly the tokens to train on. T1 can generate data by capturing successful runs rather than translating a foreign format.
Evaluation harness
The agent's evaluation (spec §25) already runs scripted tasks against a real model and judges by what the run left behind, and a mock MCP server exists:
tools/eval_agent.shruns each directory of tests/evals/ (setup.sh,prompt,options,check.sh) throughoklm agent --output-format jsonand writesbuild/eval/report.md(pass rate, turns, tokens, seconds). Task03-mcp-toolalready exercises a tool call end to end (--mcp-config,--allowedTools mcp__fake).- The mock MCP server is
tests/harness/test_mcp.crun as a server (built asbuild/release/bin/tests/harness/test_mcp,@FAKE@in an eval'soptions): it answerstools/listandtools/callover stdio JSON-RPC with toolsecho,add(structured content),fail(isError),big,slow(test_mcp.c:98-136). T3 drives training rounds against it (and against OKLM's built-in tools) without a real server; a held-out set uses tools the training never saw. - Metrics per round (T3): schema-valid call rate, task pass rate on held-out tools, and no regression on the agent's non-MCP evals — all computable from the existing report plus the schema check above.
Decisions
- Data format. Train on the in-process rendered
prompt — Qwen3's Hermes/ChatML with
<tools>/<tool_call>/<tool_response>— not the OpenAI wire JSON. Reason: it is the exact token stream the model is run on in-process; training the wire shape would mismatch inference and regress (spec Risks). Capture it from realoklm agentruns. - Reasoning on or off. Train with a short
<think>kept before the call (reason, then the call), matching how Qwen3 runs with thinking on, but strip long chains to a cap. Reason: deployed replies keep reasoning out of history (M1) but the model still emits it; examples with a brief rationale teach when to call, not just the call syntax. To confirm with a small A/B in T3. - Base model. Qwen3-8B (the default model), not the security base: the general model learns tool use; the security fine-tune is separate. Confirmed in T0 — the repo, architecture (dense qwen3, 36 layers, 151,936 vocab) and size (≈5 GB Q4_K_M) are the "Default models" figures, loaded and checked against llama.cpp (inference.md).
- Proceed now or wait. Proceed: both dependencies are done — the MCP client (spec §16, M15) gives the exact tool-call shape and the mock server, and LoRA (R7) gives the training mechanism.
Risks
| Risk | Signal | Mitigation |
|---|---|---|
| Training shape ≠ inference shape | low loss, calls malformed at run time | train the exact in-process rendering (decision 1); validate every example with the schema grammar before it enters the set |
| Base model identity wrong | VRAM, tokenizer or vocab off | the 8B is already loaded and checked (decision 3) |
| Overfitting to the training tools | high pass on seen tools, low on held-out | T3's held-out tools (benign tools or built-ins the training never saw) |
| Regression on normal coding | the agent's non-MCP evals drop | run tools/eval_agent.sh each round as a guard |
| Data content/licence | unusable or unsafe examples | mirror only the layout of reference sets; generate content from OKLM's own runs against the mock server |
T1 to T3
The tools, all under tools/ (Python's standard library
and the release oklm; nothing of them runs in CI):
- The tool zoo (
mcp_zoo.py): a benign MCP server over stdio in place of the test fixture's three tools. Training tools:add,multiply,echo(exact arguments),weatherandconvert(results to read; a unit enum),kv_set/kv_get/kv_list(state from call to call),search_notes(empty output when nothing matches) andfail, an unknown city or key, or a unit pair that does not convert (errors).--holdout:lookup_price,count_words,reverse, never trained on. Every answer is deterministic, so each task's result is known in advance. - Tasks and judging (
mcp_data.py tasks,check): the spec's five categories, numbers and phrasings drawn from a seed. A run passes whenout.txtholds the answer (numbers within 0.2 %), the tools it needed were called and none where none serves (a capital, an email no tool sends), it called only its tools (mcp__zoo__*, Write, Read: the base model sometimes calls Bash with an MCP tool's name), and it wrote the answer in a later turn than the calls it rests on (a result guessed in the same turn as its call is not kept, even when right). - Capture (
mcp_run.sh,make_mcp_data.sh): each task throughoklm agent -pin a fresh directory, its conversation saved with--save-training(the in-process rendering, decision 1) and its raw turns in the debug log; the runs that pass are kept (rejection sampling on the base model). - The gate (
mcp_data.py gate): every call in every kept conversation's assistant turns parses and fits the schema its own system turn offers (the<tools>block, so built-in and MCP tools alike); duplicates go (the conversation past the system turn compared); a tenth of each category is held out, appended last sooklm train --val-fracholds out exactly that part. - The loop (
mcp_loop.sh): the base measured, then rounds of a LoRA run (oklm train --chat-data --lora 16 --quantized-base --recompute) and a measure of the adapter; a round's failures on the training categories are captured again on fresh tasks of those categories with the last adapter and join the data through the gate. A measure runs the held-out tools and fresh training-category tasks withoklm agent --no-tool-grammar, so the model writes its calls unaided (oklm_agent_params.free_tool_calls); the JSON-valid and schema-valid rates come from the raw turns, completion from the checks (mcp_data.py metrics,build/mcp-loop/report.md).
Results (Qwen3-8B Q4_K_M, OpenCL, 2026-11-07)
Data. 200 tasks (40 a category) captured on the base model: 117 conversations kept after the checks and the gate (241 calls, every one parsing and fitting its schema; 13 duplicates dropped), 105 for training and 12 held out; format 18, params 28, parsing 28, multistep 19, negative 24. The runs dropped were mostly results guessed in the same turn as the call, Bash called with an MCP tool's name, and a required call skipped. Round 2 added 34 conversations from fresh tasks of the categories round 1 failed, captured with its adapter: 151 in all.
Training. LoRA rank 16, alpha 16, on the projections
q, k, v, o of the frozen Q4_K_M base
(--quantized-base --recompute), sequences of 6144 tokens (a
conversation is about 5700, mostly the system turn and its tools), batch
1 with 2 accumulated, 30 steps at 1e-4 (cosine to 1e-5): 46 s a
conversation, 46 minutes a round. --backend rocm could not
allocate the 5 GB of shared virtual memory the base needs. The held-out
loss on the assistant tokens fell from 0.085 to 0.061 (round 1) and from
0.094 to 0.058 (round 2): the data is the model's own filtered output,
so the loss starts low and what is learned is the filter (no guessed
results, no stray tools).
Measures, tool-call grammar off
(build/mcp-loop/report.md):
| Round | Training categories, fresh tasks: completed | their schema-valid calls | Held-out tools: completed | their schema-valid calls |
|---|---|---|---|---|
| base | 8 / 15 | 28 / 36 (78 %) | 7 / 9 | 19 / 22 (86 %) |
| round 1 | 11 / 15 | 30 / 32 (94 %) | 7 / 9 | 20 / 26 (77 %) |
| round 2 | 10 / 15 | 30 / 31 (97 %) | 5 / 9 | 22 / 37 (59 %) |
Every call the model wrote was valid JSON in every round, the grammar off: the format is not what it gets wrong. Round 1 helps on the trained tools (guessed results and skipped calls fewer; the calls that miss are now almost only stray built-in tools). It does not carry over to the held-out tools, and round 2, on more of the same self-captured data, adds nothing: on the held-out tools its "schema" misses are calls of Bash, Skill and ProcessOutput within two tasks that wander, not malformed arguments. The sets are small (15 and 9 tasks), so a task is 7 to 11 points. What would help next: examples written for the held-out failure (a tool the model has not seen, used directly rather than through Bash), more tasks per measure, and data not only from the model's own successes.