MCP Tool-Use Fine-Tuning Plan
How to work through this plan
- The goal is a model that reliably drives an MCP tool server through OKLM's agent: given a task and the tools on offer, emit correct, schema-valid tool calls and chain them into coherent multi-step workflows (call a tool, read its result, decide the next call).
- One base model: the general default, Qwen3-8B (Qwen3.8-27B until 2026-11-04) (roadmap.md), fine-tuned with a LoRA adapter. This plan does not train the abliterated security base, and it is not tied to any offensive workflow: it teaches tool-calling itself, validated against benign tools.
- Phases are ordered by leverage. T1 (data) decides the outcome. Everything after it is plumbing.
- π marks a point to stop and confirm a decision before spending time.
- T0 (the audit) is blocking and read-only. No training code before it is done and approved.
Status: not started (roadmap M16). T0, the audit, is next. It waits on MCP client support in the agent (the agent specification's Β§16, so the exact tool-call wire format is known) and on LoRA training (rocm_spec.md R7).
Before starting (OKLM)
What this plan needs settled before its T0 can start, how its names map onto OKLM's, and what already exists. The plan was drafted for a generic PyTorch + PEFT setup; OKLM trains in pure C.
Phases. Numbered T0 to T3 here, as the other specifications number theirs (S, P, G, C, R): T0 is the audit, T1 the data, T2 the fine-tune, T3 the validation loop.
Names and tools.
| The draft said | OKLM |
|---|---|
AUDIT.md |
the "MCP training" section of gap-analysis.md and a design note,
docs/mcp-training-design.md |
| PyTorch + PEFT + TRL on ROCm | OKLM's own LoRA, oklm train --lora (rocm_spec.md R7); an external framework is a
fallback only (see "The two paths") |
bitsandbytes 4-bit NF4, paged_adamw_8bit |
OKLM's quantized frozen base (R7's decision 5) and its AdamW; no bitsandbytes |
| Unsloth | not used |
Alpaca
instruction/input/output
JSONL |
a messages-with-tool-calls JSONL in the shape the agent sends at inference (see T1) |
| schema validation of each example | OKLM's JSON-schema β grammar path (ADR-0009), which already validates a tool call against a tool's schema |
| data-prep scripts | under tools/, as tools/bench_serve.py and
tools/docs/lint.sh are; or a small C utility |
| "CI / pre-commit" | make test per commit |
What OKLM already has. T0 measures each part rather than assuming:
| The plan needs | OKLM today | Where |
|---|---|---|
| The exact tool-call wire format | per model, from its chat template: hermes, llama3, mistral
(oklm_tool_format); the parser is lenient and fuzzed |
src/model/tool_calls.c, adr/0009-tool-calls-and-grammars.md |
| Schema-valid generation and checking | a GBNF grammar per request from a tool's JSON schema, so a call can be generated schema-valid and an example checked the same way | ADR-0009, src/model/grammar*.c |
| The prompt shape at inference | the agent renders system + tools + messages through the model's Jinja template; training must match this exactly | src/harness/prompt.c, src/harness/agent.c |
| The MCP tools themselves | not yet: MCP client support is the agent specification's Β§16 (its M15). Until it lands, the tool set is OKLM's built-in tools and any mock server | agent_skill_chat_spec.md Β§16 |
| LoRA training | oklm train --lora R --lora-alpha A --lora-targets LIST
on a frozen model (CPU, opencl, rocm), adapter
files in llama.cpp's format, oklm run --lora,
oklm lora-merge; --quantized-base keeps the
frozen model in its file's types on the device (Qwen3-4B Q4_K_M: 2.6
GB) |
training.md, ADR-0020 |
| The base model | Qwen3-8B from the official BF16 safetensors, loaded directly
(sharded BF16 included), quantized by OKLM for training with
--quantized-base |
roadmap.md, inference.md |
| A fine-tune reference | the Qwen2.5-0.5B / Qwen3-0.6B fine-tunes, validated against llm.c | training.md |
| JSONL handling | the mock backend and the audit log read and write JSONL | src/harness/agent_mock.c, src/harness/audit.c |
The two paths. OKLM's aim is a self-contained C
stack, so the training path is oklm train --lora (R7,
done), on a quantized frozen base with --quantized-base
(roadmap item 18). Until it has trained the 8B base, a mature external
framework (PyTorch + PEFT on ROCm) may bootstrap a first adapter from
the same JSONL, and OKLM loads the resulting adapter with
oklm run --lora once converted to llama.cpp's GGUF LoRA
format (ADR-0020). The
external path is a fallback and a cross-check, never the goal; no
PyTorch dependency enters OKLM.
Hardware. The RX 7900 XT, gfx1100,
20 GB (measured 2026-11-02, rocm_spec.md; the draft's 24 GB was the XTX).
A frozen quantized 8B base (about 5 GB in Q4_K_M, 8.7 GB in Q8_0) plus
the adapter, optimizer state and recomputed activations fits with room
for sequences of a few thousand tokens (roadmap item 18, step 4,
measures it).
Decisions T0 takes (with the design note):
- Data format. Messages-with-tool-calls JSONL in the exact shape the agent sends (system prompt, the tools' JSON, the turns, an assistant turn whose tool calls use the model's tool format), so a low training loss means the runtime shape was learned, not a near-miss. The alternative single-prompt (Alpaca) shape is kept only if the client ever sends one prompt.
- Reasoning on or off. Whether a trained reply opens
with a
<think>block before its tool calls. OKLM already tells reasoning from the answer while streaming (think.h); the data must be built one way on purpose, not left to chance. Reasoning tends to help multi-step chaining; direct calls are cheaper. - Where data and scripts live.
tools/for the builder and the validator, the datasets under a path the training command reads; nothing committed that is large or not redistributable. - The base, confirmed. The exact Hugging Face repository, architecture and parameter count of Qwen3-8B (decided 2026-11-04 in place of the 27B, see Open decisions).
Owned elsewhere. LoRA, adapter files and
oklm train --lora are rocm_spec.md's (M15); MCP client support and
the tool-call wire format are the agent specification's (Β§16); the base
model and its loading are the roadmap's "Default models"; grammars and
schema validation are ADR-0009's. This plan owns only the data, the
recipe and the validation loop.
T0 β Audit (blocking, read-only)
Goal: a precise map of what OKLM already gives for tool-call training and validation, so T1βT3 reuse rather than duplicate. The output is the two documents above, no code.
Absolute rule: edit only the "MCP training" section
of docs/gap-analysis.md and
docs/mcp-training-design.md.
Tasks
- The inference shape. Capture one real rendered prompt from the agent with tools offered (system + tools JSON + a user turn), and one assistant tool call in the model's tool format. This is the template every training example must match.
- Schema validation. Confirm the ADR-0009 path can both constrain generation to a tool's schema and check a candidate call against it; note the function to call from a data validator.
- The training mechanism. State what R7 provides
(
oklm train --lora, targets, adapter files) and what this plan must supply (data, hyperparameters, the eval loop); list what is missing if R7 is not done. - Reference datasets. Any public function-calling / tool-use dataset worth mining for format (not content): its JSONL layout, its reasoning convention. Mirror the layout; do not import content blindly.
- Evaluation. How the agent's evaluation (spec Β§25) runs scripted tasks against a model, and how a mock MCP server (or the built-in tools) can stand in for a real one in T3.
- Decisions. Fill the four decisions above with reasons.
Output
docs/mcp-training-design.md, these sections:
## Inference prompt and tool-call shape,
## Schema validation path,
## Training mechanism (from R7),
## Reference datasets (format only),
## Evaluation harness, ## Decisions,
## Risks.
Acceptance criteria
π Decision point β stop and wait for approval
Present the four decisions. The user confirms the data format, the reasoning choice, the base model, and whether to proceed before MCP (Β§16) and R7 land or to wait for them.
T1 β Build the MCP tool-use training data (most critical)
The dataset is the asset. Base and hyperparameters matter far less than its quality and coverage.
Format
Match T0's captured inference shape exactly: messages with native tool calls in the model's tool format, the tools' JSON schemas in the system turn as the agent sends them. Carry reasoning or strip it per T0's decision 2 β consistently, in every example.
Coverage (minimum)
- Tool-call format β the exact JSON each tool expects, every tool.
- Parameter construction β correct parameters from varied phrasings of the same task.
- Result parsing β turning a tool's output into the next decision.
- Multi-step workflows β a coherent sequence of calls that passes state from one result to the next, and stops when done.
- Negative and boundary cases β when not to call a tool, a request outside the available tools, and graceful handling of empty or malformed tool output. Happy-path-only data makes a model hallucinate calls on the edges.
Quality gates
- Every example's tool calls parse and conform to the tool's schema (ADR-0009's validator, T0 task 2) before it enters the set. A broken example teaches broken output.
- Hold out ~10% as a validation split now, before training, for T2's loss tracking.
- Variety over volume: a few thousand clean, diverse examples beat tens of thousands of near-duplicates.
Acceptance criteria
T2 β Fine-tune the base with a LoRA adapter
π Feasibility check first
Before any full run, through oklm train --lora (R7):
load the base frozen and quantized (R7's decision 5), attach an adapter,
take one optimizer step on a handful of examples. If it does not fit or
does not step, fall back in order: (1) a lower rank or shorter sequence,
(2) a smaller base (Open decisions). If R7 is not ready, the external
path of "The two paths" produces a first adapter OKLM then loads.
Budget (20 GB)
Owned by R7's decision 5: a Q4_K frozen base (~16 GB) plus the adapter, AdamW state and activations, with gradient checkpointing, batch 1 and gradient accumulation for an effective larger batch. T2 only confirms the numbers hold for this data's sequence length.
Hyperparameters
| Param | Value | Why |
|---|---|---|
| LoRA rank | 16β32 | start at 16; raise if the budget allows |
| LoRA alpha | = rank | keeps the scaling at 1.0 |
| Dropout | 0.05β0.10 | β |
| Learning rate | 1e-4 to 2e-4 | a LoRA learning rate; 2e-5 is a full-fine-tune rate and would crawl |
| Precision | bf16 | OKLM reads BF16 directly; verify stability on RDNA3 |
| Optimizer | AdamW (OKLM's) | β |
| Epochs | watch, don't fix | more data needs fewer epochs; stop on the validation loss (early stopping), not a set count |
Acceptance criteria
T3 β Validation loop (more than one training run)
The loop
- Give the model a task and the tools on offer (a mock MCP server with benign tools, or OKLM's built-in tools, per T0 task 5).
- Check: valid tool-call JSON? Correct parameters? A coherent, stateful multi-step sequence?
- Record every failure β malformed JSON, wrong tool, wrong parameters, incoherent sequencing.
- Turn the failures into new examples (T1's gates apply) and run a second LoRA round.
- Repeat. Iterating beats one large run.
Metrics, per round
- JSON-valid rate (should approach 100%; the grammar makes this the floor when generation is constrained, so measure unconstrained too).
- Schema / parameter correctness rate.
- Multi-step workflow completion rate.
Acceptance criteria
Risks
| Risk | Impact | Mitigation |
|---|---|---|
| LoRA on the quantized base not ready (R7) | blocks training | gate T2 on R7; use the external fallback for a first adapter |
| Base model identity wrong | config, VRAM, tokenizer all off | confirm the exact repo, architecture and size in T0 (Open decisions) |
| The base too large to train in 20 GB | OOM, instability | Qwen3-8B, a frozen quantized base (--quantized-base),
activation recompute |
| Training format β inference format | low loss, runtime failure | train in the exact shape T0 captured |
| Happy-path-only data | hallucinated calls on edges | the negative and boundary category is mandatory |
| 2e-5 learning rate for LoRA | underfitting, wasted runs | 1e-4 to 2e-4 |
Open decisions
- The base: decided 2026-11-04. Qwen3-8B
(
Qwen/Qwen3-8B), in place of the 27B first named, which this GPU could neither train nor run with a long context; T0 still confirms the repository, the architecture and the parameter count before the budget is set. - Is 8B enough for this? For structured tool-calling, output format matters more than raw capability, which favours the 7B to 14B range; Qwen3-4B serves the quick runs.
- Reasoning on or off (T0 decision 2): whether deployed replies include a reasoning step before the tool call, and the data built to match.
Reporting format
After each phase, report only:
TN status: DONE | BLOCKED | NEEDS DECISION
Data: sample count, categories, schema-valid % (T1)
Training: final train/val loss, epochs, OOM? (T2)
Validation: JSON-valid %, param-correct %, workflow-complete % (T3)
Failures added to the next round: count
Blockers / questions: anything needing input
Next action: the single next concrete step