OKLMdocumentation v1.0.3
On this page

MCP tool-use fine-tuning design (T0 audit)

How mcp_training_spec.md maps onto OKLM: what the stack already gives for teaching the default model to drive an MCP tool server, so T1–T3 reuse rather than duplicate. What exists, with evidence, is the "MCP training" section of gap-analysis.md; the tool-call wire format and its grammar are adr/0009-tool-calls-and-grammars.md.

Status: T0 accepted (2026-11-05); T1 to T3 built and run (2026-11-07), see "T1 to T3" at the end.

Inference prompt and tool-call shape

The target model, Qwen3-8B, carries its own chat template (tokenizer.chat_template), rendered by OKLM's Jinja engine (oklm_chat_template_render(), src/model/chat_template.c; ADR-0009). Qwen3's template is Hermes/ChatML: when a tools array is passed it adds a block to the system turn and declares the call syntax. Quoted from the model file (Qwen3-0.6B-Q8_0.gguf, the same family and template):

<|im_start|>system
<system text>

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type":"function","function":{"name":"Bash","description":"...","parameters":{...}}}
{"type":"function","function":{"name":"Read","description":"...","parameters":{...}}}
...
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
List the files here.<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "Bash", "arguments": {"command": "ls"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
a.txt  b.txt
</tool_response><|im_end|>
<|im_start|>assistant
Two files: a.txt and b.txt.<|im_end|>
  • The tools JSON is the OpenAI function shape. OKLM's built-in tools are the literal in tools.c tools_json_parts[] (src/harness/tools.c:3358-3492); the Agent and SendMessage tools are agent.c agent_tool_json, send_message_json (src/harness/agent.c:491-509); MCP tools are added under their mcp__server__tool names (mcp.c). oklm_agent_tools_json() returns the array an agent offers (agent.h).
  • The call is parsed from the model's text by tool_calls.c parse_hermes() (src/model/tool_calls.c:378): one JSON object per <tool_call>…</tool_call>, {"name", "arguments"}, arguments a string or an object. The format is detected from the template (oklm_tool_format_detect(), tool_calls.c:21 — <tool_call> means Hermes) or forced with --tool-format; the opening marker (the grammar trigger) is <tool_call> (tool_calls.c:1043).
  • The result comes back as a user turn wrapped <tool_response>… </tool_response> by the template; OKLM sends one result per call id (spec §3), errors included.
  • Over --url the HTTP backend sends the OpenAI wire shape instead: messages and tools as separate JSON, the call in choices[].delta.tool_calls (agent_http.c http_acc_feed). Training data must match the in-process rendered shape above (the tokens the model is trained and run on), not the wire JSON — decision 1.
  • Reasoning (<think>…</think>) is streamed as a separate event and kept out of the history sent back (M1); whether a training example keeps it is decision 2.

A live capture: OKLM_LOG_LEVEL=DEBUG oklm agent -m <model> -p "…" logs each raw model turn (agent: turn: …, agent.c:2736), which shows the <tool_call> text verbatim.

Schema validation path

ADR-0009's grammar path both constrains and checks, so a data validator needs no new engine:

  • Constrain generation. oklm_json_schema_to_grammar() (include/oklm/model/grammar.h:160) turns a tool's parameters JSON schema into GBNF (objects, required and optional keys, arrays with bounds, enums, anyOf/oneOf, $ref, string minLength/maxLength/ format/pattern, integer and decimal minimum/maximum). The agent builds one grammar of all offered tools (agent.c agent_grammar), so a turn's call is well-formed by construction.
  • Check a candidate. Build the grammar from the tool's schema, feed a candidate call's argument text with oklm_grammar_accept_text() (grammar.h:100) and require oklm_grammar_complete() (grammar.h:107): a call that the grammar accepts to completion is schema-valid. This is the function a T1 data validator calls before an example enters the set.
  • Decimal bounds are exact (ADR-0009; no exponent, 16 digits), so a number/integer range is enforced to the digit.

Training mechanism (from R7)

LoRA fine-tuning is done on every backend (M15, R3–R7; ADR-0020, training.md):

  • oklm train --lora R [--lora-alpha A] [--lora-targets …] freezes the model and trains rank-R adapters on the chosen projections (q,k,v,o by default, all, or a list); the base may be quantized (--quantized-base, item 18) so an 8B fits a 20 GB GPU (≈18.7 GiB, training.md).
  • A checkpoint's weights file is a llama.cpp-format adapter (oklm_lora_merge(), oklm/io/lora.h); oklm run|chat|ppl|serve --lora use it, oklm serve --lora-module serves several (ADR-0021).
  • The chat-data path is built (2026-11-05): oklm train --chat-data conv.jsonl reads the captured {text} JSONL, masks each conversation to its assistant tokens, and trains the loss on those only (training.md). So T1's capture feeds the trainer directly.
  • What this plan still adds, not R7 or the chat-data path: the training data at scale (T1), the hyperparameters and the validation loop (T2/T3). No gap in the mechanism — M15 (MCP client), R7 (LoRA) and --chat-data are all done, so the plan is unblocked (decision 4).

Reference datasets (format only)

Mined for layout and reasoning convention, never imported blindly (content, licence and quality are T1's to control):

  • Hermes-style function-calling sets (e.g. the Glaive / NousResearch function-calling corpora): already the <tools>/<tool_call> shape Qwen3 uses, so their per-turn layout maps one-to-one; mirror the turn structure, write OKLM's own tool set and results.
  • The agent's own eval tasks (tests/evals/, especially 03-mcp-tool) are the most faithful source: a real run through oklm agent against the mock MCP server produces exactly the tokens to train on. T1 can generate data by capturing successful runs rather than translating a foreign format.

Evaluation harness

The agent's evaluation (spec §25) already runs scripted tasks against a real model and judges by what the run left behind, and a mock MCP server exists:

  • tools/eval_agent.sh runs each directory of tests/evals/ (setup.sh, prompt, options, check.sh) through oklm agent --output-format json and writes build/eval/report.md (pass rate, turns, tokens, seconds). Task 03-mcp-tool already exercises a tool call end to end (--mcp-config, --allowedTools mcp__fake).
  • The mock MCP server is tests/harness/test_mcp.c run as a server (built as build/release/bin/tests/harness/test_mcp, @FAKE@ in an eval's options): it answers tools/list and tools/call over stdio JSON-RPC with tools echo, add (structured content), fail (isError), big, slow (test_mcp.c:98-136). T3 drives training rounds against it (and against OKLM's built-in tools) without a real server; a held-out set uses tools the training never saw.
  • Metrics per round (T3): schema-valid call rate, task pass rate on held-out tools, and no regression on the agent's non-MCP evals — all computable from the existing report plus the schema check above.

Decisions

  1. Data format. Train on the in-process rendered prompt — Qwen3's Hermes/ChatML with <tools>/<tool_call>/<tool_response> — not the OpenAI wire JSON. Reason: it is the exact token stream the model is run on in-process; training the wire shape would mismatch inference and regress (spec Risks). Capture it from real oklm agent runs.
  2. Reasoning on or off. Train with a short <think> kept before the call (reason, then the call), matching how Qwen3 runs with thinking on, but strip long chains to a cap. Reason: deployed replies keep reasoning out of history (M1) but the model still emits it; examples with a brief rationale teach when to call, not just the call syntax. To confirm with a small A/B in T3.
  3. Base model. Qwen3-8B (the default model), not the security base: the general model learns tool use; the security fine-tune is separate. Confirmed in T0 — the repo, architecture (dense qwen3, 36 layers, 151,936 vocab) and size (≈5 GB Q4_K_M) are the "Default models" figures, loaded and checked against llama.cpp (inference.md).
  4. Proceed now or wait. Proceed: both dependencies are done — the MCP client (spec §16, M15) gives the exact tool-call shape and the mock server, and LoRA (R7) gives the training mechanism.

Risks

Risk Signal Mitigation
Training shape ≠ inference shape low loss, calls malformed at run time train the exact in-process rendering (decision 1); validate every example with the schema grammar before it enters the set
Base model identity wrong VRAM, tokenizer or vocab off the 8B is already loaded and checked (decision 3)
Overfitting to the training tools high pass on seen tools, low on held-out T3's held-out tools (benign tools or built-ins the training never saw)
Regression on normal coding the agent's non-MCP evals drop run tools/eval_agent.sh each round as a guard
Data content/licence unusable or unsafe examples mirror only the layout of reference sets; generate content from OKLM's own runs against the mock server

T1 to T3

The tools, all under tools/ (Python's standard library and the release oklm; nothing of them runs in CI):

  • The tool zoo (mcp_zoo.py): a benign MCP server over stdio in place of the test fixture's three tools. Training tools: add, multiply, echo (exact arguments), weather and convert (results to read; a unit enum), kv_set / kv_get / kv_list (state from call to call), search_notes (empty output when nothing matches) and fail, an unknown city or key, or a unit pair that does not convert (errors). --holdout: lookup_price, count_words, reverse, never trained on. Every answer is deterministic, so each task's result is known in advance.
  • Tasks and judging (mcp_data.py tasks, check): the spec's five categories, numbers and phrasings drawn from a seed. A run passes when out.txt holds the answer (numbers within 0.2 %), the tools it needed were called and none where none serves (a capital, an email no tool sends), it called only its tools (mcp__zoo__*, Write, Read: the base model sometimes calls Bash with an MCP tool's name), and it wrote the answer in a later turn than the calls it rests on (a result guessed in the same turn as its call is not kept, even when right).
  • Capture (mcp_run.sh, make_mcp_data.sh): each task through oklm agent -p in a fresh directory, its conversation saved with --save-training (the in-process rendering, decision 1) and its raw turns in the debug log; the runs that pass are kept (rejection sampling on the base model).
  • The gate (mcp_data.py gate): every call in every kept conversation's assistant turns parses and fits the schema its own system turn offers (the <tools> block, so built-in and MCP tools alike); duplicates go (the conversation past the system turn compared); a tenth of each category is held out, appended last so oklm train --val-frac holds out exactly that part.
  • The loop (mcp_loop.sh): the base measured, then rounds of a LoRA run (oklm train --chat-data --lora 16 --quantized-base --recompute) and a measure of the adapter; a round's failures on the training categories are captured again on fresh tasks of those categories with the last adapter and join the data through the gate. A measure runs the held-out tools and fresh training-category tasks with oklm agent --no-tool-grammar, so the model writes its calls unaided (oklm_agent_params.free_tool_calls); the JSON-valid and schema-valid rates come from the raw turns, completion from the checks (mcp_data.py metrics, build/mcp-loop/report.md).

Results (Qwen3-8B Q4_K_M, OpenCL, 2026-11-07)

Data. 200 tasks (40 a category) captured on the base model: 117 conversations kept after the checks and the gate (241 calls, every one parsing and fitting its schema; 13 duplicates dropped), 105 for training and 12 held out; format 18, params 28, parsing 28, multistep 19, negative 24. The runs dropped were mostly results guessed in the same turn as the call, Bash called with an MCP tool's name, and a required call skipped. Round 2 added 34 conversations from fresh tasks of the categories round 1 failed, captured with its adapter: 151 in all.

Training. LoRA rank 16, alpha 16, on the projections q, k, v, o of the frozen Q4_K_M base (--quantized-base --recompute), sequences of 6144 tokens (a conversation is about 5700, mostly the system turn and its tools), batch 1 with 2 accumulated, 30 steps at 1e-4 (cosine to 1e-5): 46 s a conversation, 46 minutes a round. --backend rocm could not allocate the 5 GB of shared virtual memory the base needs. The held-out loss on the assistant tokens fell from 0.085 to 0.061 (round 1) and from 0.094 to 0.058 (round 2): the data is the model's own filtered output, so the loss starts low and what is learned is the filter (no guessed results, no stray tools).

Measures, tool-call grammar off (build/mcp-loop/report.md):

Round Training categories, fresh tasks: completed their schema-valid calls Held-out tools: completed their schema-valid calls
base 8 / 15 28 / 36 (78 %) 7 / 9 19 / 22 (86 %)
round 1 11 / 15 30 / 32 (94 %) 7 / 9 20 / 26 (77 %)
round 2 10 / 15 30 / 31 (97 %) 5 / 9 22 / 37 (59 %)

Every call the model wrote was valid JSON in every round, the grammar off: the format is not what it gets wrong. Round 1 helps on the trained tools (guessed results and skipped calls fewer; the calls that miss are now almost only stray built-in tools). It does not carry over to the held-out tools, and round 2, on more of the same self-captured data, adds nothing: on the held-out tools its "schema" misses are calls of Bash, Skill and ProcessOutput within two tasks that wander, not malformed arguments. The sets are small (15 and 9 tasks), so a task is 7 to 11 points. What would help next: examples written for the held-out failure (a tool the model has not seen, used directly rather than through Bash), more tasks per measure, and data not only from the model's own successes.