OKLMdocumentation v1.0.3
On this page

MCP Tool-Use Fine-Tuning Plan

How to work through this plan

  • The goal is a model that reliably drives an MCP tool server through OKLM's agent: given a task and the tools on offer, emit correct, schema-valid tool calls and chain them into coherent multi-step workflows (call a tool, read its result, decide the next call).
  • One base model: the general default, Qwen3-8B (Qwen3.8-27B until 2026-11-04) (roadmap.md), fine-tuned with a LoRA adapter. This plan does not train the abliterated security base, and it is not tied to any offensive workflow: it teaches tool-calling itself, validated against benign tools.
  • Phases are ordered by leverage. T1 (data) decides the outcome. Everything after it is plumbing.
  • πŸ›‘ marks a point to stop and confirm a decision before spending time.
  • T0 (the audit) is blocking and read-only. No training code before it is done and approved.

Status: not started (roadmap M16). T0, the audit, is next. It waits on MCP client support in the agent (the agent specification's Β§16, so the exact tool-call wire format is known) and on LoRA training (rocm_spec.md R7).


Before starting (OKLM)

What this plan needs settled before its T0 can start, how its names map onto OKLM's, and what already exists. The plan was drafted for a generic PyTorch + PEFT setup; OKLM trains in pure C.

Phases. Numbered T0 to T3 here, as the other specifications number theirs (S, P, G, C, R): T0 is the audit, T1 the data, T2 the fine-tune, T3 the validation loop.

Names and tools.

The draft said OKLM
AUDIT.md the "MCP training" section of gap-analysis.md and a design note, docs/mcp-training-design.md
PyTorch + PEFT + TRL on ROCm OKLM's own LoRA, oklm train --lora (rocm_spec.md R7); an external framework is a fallback only (see "The two paths")
bitsandbytes 4-bit NF4, paged_adamw_8bit OKLM's quantized frozen base (R7's decision 5) and its AdamW; no bitsandbytes
Unsloth not used
Alpaca instruction/input/output JSONL a messages-with-tool-calls JSONL in the shape the agent sends at inference (see T1)
schema validation of each example OKLM's JSON-schema β†’ grammar path (ADR-0009), which already validates a tool call against a tool's schema
data-prep scripts under tools/, as tools/bench_serve.py and tools/docs/lint.sh are; or a small C utility
"CI / pre-commit" make test per commit

What OKLM already has. T0 measures each part rather than assuming:

The plan needs OKLM today Where
The exact tool-call wire format per model, from its chat template: hermes, llama3, mistral (oklm_tool_format); the parser is lenient and fuzzed src/model/tool_calls.c, adr/0009-tool-calls-and-grammars.md
Schema-valid generation and checking a GBNF grammar per request from a tool's JSON schema, so a call can be generated schema-valid and an example checked the same way ADR-0009, src/model/grammar*.c
The prompt shape at inference the agent renders system + tools + messages through the model's Jinja template; training must match this exactly src/harness/prompt.c, src/harness/agent.c
The MCP tools themselves not yet: MCP client support is the agent specification's Β§16 (its M15). Until it lands, the tool set is OKLM's built-in tools and any mock server agent_skill_chat_spec.md Β§16
LoRA training oklm train --lora R --lora-alpha A --lora-targets LIST on a frozen model (CPU, opencl, rocm), adapter files in llama.cpp's format, oklm run --lora, oklm lora-merge; --quantized-base keeps the frozen model in its file's types on the device (Qwen3-4B Q4_K_M: 2.6 GB) training.md, ADR-0020
The base model Qwen3-8B from the official BF16 safetensors, loaded directly (sharded BF16 included), quantized by OKLM for training with --quantized-base roadmap.md, inference.md
A fine-tune reference the Qwen2.5-0.5B / Qwen3-0.6B fine-tunes, validated against llm.c training.md
JSONL handling the mock backend and the audit log read and write JSONL src/harness/agent_mock.c, src/harness/audit.c

The two paths. OKLM's aim is a self-contained C stack, so the training path is oklm train --lora (R7, done), on a quantized frozen base with --quantized-base (roadmap item 18). Until it has trained the 8B base, a mature external framework (PyTorch + PEFT on ROCm) may bootstrap a first adapter from the same JSONL, and OKLM loads the resulting adapter with oklm run --lora once converted to llama.cpp's GGUF LoRA format (ADR-0020). The external path is a fallback and a cross-check, never the goal; no PyTorch dependency enters OKLM.

Hardware. The RX 7900 XT, gfx1100, 20 GB (measured 2026-11-02, rocm_spec.md; the draft's 24 GB was the XTX). A frozen quantized 8B base (about 5 GB in Q4_K_M, 8.7 GB in Q8_0) plus the adapter, optimizer state and recomputed activations fits with room for sequences of a few thousand tokens (roadmap item 18, step 4, measures it).

Decisions T0 takes (with the design note):

  1. Data format. Messages-with-tool-calls JSONL in the exact shape the agent sends (system prompt, the tools' JSON, the turns, an assistant turn whose tool calls use the model's tool format), so a low training loss means the runtime shape was learned, not a near-miss. The alternative single-prompt (Alpaca) shape is kept only if the client ever sends one prompt.
  2. Reasoning on or off. Whether a trained reply opens with a <think> block before its tool calls. OKLM already tells reasoning from the answer while streaming (think.h); the data must be built one way on purpose, not left to chance. Reasoning tends to help multi-step chaining; direct calls are cheaper.
  3. Where data and scripts live. tools/ for the builder and the validator, the datasets under a path the training command reads; nothing committed that is large or not redistributable.
  4. The base, confirmed. The exact Hugging Face repository, architecture and parameter count of Qwen3-8B (decided 2026-11-04 in place of the 27B, see Open decisions).

Owned elsewhere. LoRA, adapter files and oklm train --lora are rocm_spec.md's (M15); MCP client support and the tool-call wire format are the agent specification's (Β§16); the base model and its loading are the roadmap's "Default models"; grammars and schema validation are ADR-0009's. This plan owns only the data, the recipe and the validation loop.


T0 β€” Audit (blocking, read-only)

Goal: a precise map of what OKLM already gives for tool-call training and validation, so T1–T3 reuse rather than duplicate. The output is the two documents above, no code.

Absolute rule: edit only the "MCP training" section of docs/gap-analysis.md and docs/mcp-training-design.md.

Tasks

  1. The inference shape. Capture one real rendered prompt from the agent with tools offered (system + tools JSON + a user turn), and one assistant tool call in the model's tool format. This is the template every training example must match.
  2. Schema validation. Confirm the ADR-0009 path can both constrain generation to a tool's schema and check a candidate call against it; note the function to call from a data validator.
  3. The training mechanism. State what R7 provides (oklm train --lora, targets, adapter files) and what this plan must supply (data, hyperparameters, the eval loop); list what is missing if R7 is not done.
  4. Reference datasets. Any public function-calling / tool-use dataset worth mining for format (not content): its JSONL layout, its reasoning convention. Mirror the layout; do not import content blindly.
  5. Evaluation. How the agent's evaluation (spec Β§25) runs scripted tasks against a model, and how a mock MCP server (or the built-in tools) can stand in for a real one in T3.
  6. Decisions. Fill the four decisions above with reasons.

Output

docs/mcp-training-design.md, these sections: ## Inference prompt and tool-call shape, ## Schema validation path, ## Training mechanism (from R7), ## Reference datasets (format only), ## Evaluation harness, ## Decisions, ## Risks.

Acceptance criteria

πŸ›‘ Decision point β€” stop and wait for approval

Present the four decisions. The user confirms the data format, the reasoning choice, the base model, and whether to proceed before MCP (Β§16) and R7 land or to wait for them.


T1 β€” Build the MCP tool-use training data (most critical)

The dataset is the asset. Base and hyperparameters matter far less than its quality and coverage.

Format

Match T0's captured inference shape exactly: messages with native tool calls in the model's tool format, the tools' JSON schemas in the system turn as the agent sends them. Carry reasoning or strip it per T0's decision 2 β€” consistently, in every example.

Coverage (minimum)

  1. Tool-call format β€” the exact JSON each tool expects, every tool.
  2. Parameter construction β€” correct parameters from varied phrasings of the same task.
  3. Result parsing β€” turning a tool's output into the next decision.
  4. Multi-step workflows β€” a coherent sequence of calls that passes state from one result to the next, and stops when done.
  5. Negative and boundary cases β€” when not to call a tool, a request outside the available tools, and graceful handling of empty or malformed tool output. Happy-path-only data makes a model hallucinate calls on the edges.

Quality gates

  • Every example's tool calls parse and conform to the tool's schema (ADR-0009's validator, T0 task 2) before it enters the set. A broken example teaches broken output.
  • Hold out ~10% as a validation split now, before training, for T2's loss tracking.
  • Variety over volume: a few thousand clean, diverse examples beat tens of thousands of near-duplicates.

Acceptance criteria


T2 β€” Fine-tune the base with a LoRA adapter

πŸ›‘ Feasibility check first

Before any full run, through oklm train --lora (R7): load the base frozen and quantized (R7's decision 5), attach an adapter, take one optimizer step on a handful of examples. If it does not fit or does not step, fall back in order: (1) a lower rank or shorter sequence, (2) a smaller base (Open decisions). If R7 is not ready, the external path of "The two paths" produces a first adapter OKLM then loads.

Budget (20 GB)

Owned by R7's decision 5: a Q4_K frozen base (~16 GB) plus the adapter, AdamW state and activations, with gradient checkpointing, batch 1 and gradient accumulation for an effective larger batch. T2 only confirms the numbers hold for this data's sequence length.

Hyperparameters

Param Value Why
LoRA rank 16–32 start at 16; raise if the budget allows
LoRA alpha = rank keeps the scaling at 1.0
Dropout 0.05–0.10 β€”
Learning rate 1e-4 to 2e-4 a LoRA learning rate; 2e-5 is a full-fine-tune rate and would crawl
Precision bf16 OKLM reads BF16 directly; verify stability on RDNA3
Optimizer AdamW (OKLM's) β€”
Epochs watch, don't fix more data needs fewer epochs; stop on the validation loss (early stopping), not a set count

Acceptance criteria


T3 β€” Validation loop (more than one training run)

The loop

  1. Give the model a task and the tools on offer (a mock MCP server with benign tools, or OKLM's built-in tools, per T0 task 5).
  2. Check: valid tool-call JSON? Correct parameters? A coherent, stateful multi-step sequence?
  3. Record every failure β€” malformed JSON, wrong tool, wrong parameters, incoherent sequencing.
  4. Turn the failures into new examples (T1's gates apply) and run a second LoRA round.
  5. Repeat. Iterating beats one large run.

Metrics, per round

  • JSON-valid rate (should approach 100%; the grammar makes this the floor when generation is constrained, so measure unconstrained too).
  • Schema / parameter correctness rate.
  • Multi-step workflow completion rate.

Acceptance criteria


Risks

Risk Impact Mitigation
LoRA on the quantized base not ready (R7) blocks training gate T2 on R7; use the external fallback for a first adapter
Base model identity wrong config, VRAM, tokenizer all off confirm the exact repo, architecture and size in T0 (Open decisions)
The base too large to train in 20 GB OOM, instability Qwen3-8B, a frozen quantized base (--quantized-base), activation recompute
Training format β‰  inference format low loss, runtime failure train in the exact shape T0 captured
Happy-path-only data hallucinated calls on edges the negative and boundary category is mandatory
2e-5 learning rate for LoRA underfitting, wasted runs 1e-4 to 2e-4

Open decisions

  1. The base: decided 2026-11-04. Qwen3-8B (Qwen/Qwen3-8B), in place of the 27B first named, which this GPU could neither train nor run with a long context; T0 still confirms the repository, the architecture and the parameter count before the budget is set.
  2. Is 8B enough for this? For structured tool-calling, output format matters more than raw capability, which favours the 7B to 14B range; Qwen3-4B serves the quick runs.
  3. Reasoning on or off (T0 decision 2): whether deployed replies include a reasoning step before the tool call, and the data built to match.

Reporting format

After each phase, report only:

TN status:   DONE | BLOCKED | NEEDS DECISION
Data:        sample count, categories, schema-valid %  (T1)
Training:    final train/val loss, epochs, OOM?  (T2)
Validation:  JSON-valid %, param-correct %, workflow-complete %  (T3)
Failures added to the next round: count
Blockers / questions: anything needing input
Next action: the single next concrete step