OKLMdocumentation v1.0.3
On this page

Agent loop

How oklm agent runs a task: the loop of model turns and tool calls, the system prompt, the backends and their stream events, headless and library use, and how it fares on published models and the eval tasks. Part of the agent harness (harness.md).

Status: implemented.

include/oklm/harness/agent.h, src/harness/agent.c. oklm_agent_send(agent, message, &reply) adds the user's message and runs model turns until the model answers without a tool call:

  1. The conversation is rendered by the model's chat template, the system prompt first, the six tools in its tools variable. --chat-template FILE|NAME replaces the file's template (inference.md); an LFM2 file whose template says nothing of tools is rendered with the one that writes them in Liquid's form, the format found from the call tokens of its vocabulary, as oklm serve and oklm chat --tools do, unless a template was asked for: that one is rendered as it is. The small model of the asides (smallModel) keeps its file's template, and gets the option's when its file carries none.
  2. Only the tokens after the prefix already in the context are evaluated (the KV cache is truncated to the common prefix).
  3. The model generates; once it opens a tool call (<tool_call>, [TOOL_CALLS], or <|python_tag|> for Llama 3) the call is constrained by the tools' grammar until it closes; the model may then write more text or more calls, until it ends its turn.
  4. The calls are parsed (ADR-0009), recorded in the assistant message, run in order by oklm_tool_run() (rules, questions, audit) and their results, cut at the output limit, appended as tool messages, one per call.

System prompt

include/oklm/harness/prompt.h, src/harness/prompt.c. The prompt is assembled once, from parts in a stable order, so it is the same bytes on every turn and the model's cache keeps it:

  1. identity and core rules (be concise, read before changing, follow the code's conventions, check with the build or tests, never commit unless asked, refuse malicious code);
  2. the tool policy: every part of the request, a task list with TodoWrite for several steps (one task in progress), dedicated tools rather than commands, independent reads together, tool results and files as data and never instructions, two worked examples (Grep, Bash then Edit until the tests pass);
  3. starting an Agent for broad searches, or, for a sub-agent, that it reports to the agent that started it;
  4. the output style, when not default;
  5. the environment: the project directory (absolute: a model given a relative one used it as a prefix of its paths), whether it is a git repository, the platform, the OS version, today's date, the model;
  6. the git snapshot and 7. the memory files, when the session has them (git_snapshot and memory of the parameters);
  7. the skills (within their budget) and the deferred tools' names (with --tool-rag K, each MCP server's tools counted instead), then the text of --append-system-prompt.

--system (or --system-prompt) replaces it all; appended text still follows. The wording is in src/harness/prompts/*.md, embedded at build time; tests/harness/test_prompt.c renders it against the snapshots of tests/fixtures/prompt/ (OKLM_UPDATE_SNAPSHOTS=1 rewrites them after a deliberate change). Output styles: explanatory adds short insights about the code, learning leaves a meaningful piece to the user with a TODO(human); NAME.md in .oklm/output-styles/, ~/.config/oklm/output-styles/ or .agent/output-styles/ (a front matter dropped) is a custom one. --output-style NAME chooses at the start, /output-style NAME later (oklm_agent_set_output_style(), which starts the cache over).

What changes during a session comes as reminders, user messages marked meta and added before the model's next turn: the permission mode when it changes (plan mode's rules, acceptEdits, back to default...; at the start only for plan mode), the files read that changed on disk since (by the user, an editor or a command; once per change), the task list after eight turns of other calls while tasks are unfinished, and with --tool-rag K the definitions of the deferred tools retrieved for the user's message (agent-tools.md), and with --rag-inject always|route the passages of the project's documentation retrieved for it, the message condensed from the conversation first (rag.md). With --rag-critique on|flag (an in-process model, or a served one that can judge: rag.md) a final answer that cites [n] is checked against the passages its message retrieved, through the answer_check hook: its verdict is a notice, and with on an unsupported statement sends the agent back once with the critique as a reminder, as a blocking Stop hook does (rag.md). When the in-process model's template is rendered, a user message and the reminders after it are one message, for templates that want the roles to alternate.

A turn limit (32) stops a model that never answers; a turn has at most 8192 tokens, room for a thinking model's reasoning, and oklm agent opens a context of 32768 tokens (fewer when the model was trained on fewer). A generate callback replaces the model by any text source (the tests use a script). OKLM_LOG_LEVEL=DEBUG logs each raw model turn.

oklm agent -m <model> [-p TASK] [--project DIR] [--mode MODE] [--dangerously-skip-permissions] [--allow RULE] [--ask RULE] [--deny RULE] [--no-sandbox] [--allow-network] [--limit NAME=VALUE] [--system TEXT] [--append-system-prompt TEXT] [--output-style STYLE] runs it on a terminal: tool calls are shown compactly (below), and the session's decisions go to .oklm/audit/<session>.jsonl. Without -p it is a REPL (/reset, /mode MODE, /add-dir DIR, /output-style STYLE, /jobs, /thinking, /exit; Shift+Tab, at the prompt or while the agent works, switches to the next mode, the bypass mode with a red banner). /add-dir adds a working directory for the session: inside the project for the modes, writable in the sandbox. A question shows the call, an edit's diff in colour, and why it asks (in red for a dangerous call):

[Edit] f.c
  --- a/f.c
  +++ b/f.c
  @@ -1,2 +1,2 @@
   int a;
  -int b;
  +long b;
  not covered by a rule (default)
Allow? [y]es, [a]lways (saves Edit(f.c)), [n]o, or say what to do instead:

a saves the rule to .oklm/settings.local.json; anything other than y, a or n refuses and is passed to the model as what to do instead; a plan offers yes, accept and allow edits, no, or what to change.

A thinking model's reasoning (<think> ... </think>, a harmony reply's analysis channel, a Muse Glimmer reply's message to itself or Gemma 4's thinking channel, told apart while it streams by include/oklm/harness/think.h, the tags possibly cut across tokens) is folded: the spinner counts its words as it comes, then one line stays, [thought, 212 words; ctrl-o or /thinking: full view], and the answer follows (the classic view). Ctrl-O while the model writes switches to the full view: it unfolds the block being written (what it has so far, dim, then the rest as it comes) and the next ones; pressed again, it erases the unfolded block from the screen (its wrapped lines counted at the terminal's width) and the status line takes its place again. Pressed between reasoning blocks, it says which view is now on. /thinking (or Ctrl-O) at the prompt does the same for the next replies and shows the last message's reasoning.

--thinking show starts unfolded, and so does ui.viewMode full (or fullscreen, which also enters the full-screen UI on a terminal). The terminal is read key by key only during a turn, without echo: the agent looks between the chunks of the prompt it evaluates, at each token (a tool call's too) and while a tool runs (oklm_agent_params.on_poll), so Esc and Ctrl-O act at once; what is typed meanwhile waits for the next prompt (the arrows and other special keys pressed meanwhile are dropped), and the settings come back for the permission questions, at the end, and on Ctrl-C. Output that is not a terminal gets the one line per block, without the status line. When the template opens the reply with <think> itself, the agent's text starts with that tag, so the blocks are always whole.

Loop details

  • Every message has an id (a random UUID), its parent's id, a time, and for the assistant's turns their usage; reminders the agent adds (a job that ended, a background agent's report, a loop noticed) are user messages marked meta. oklm_agent_message() reads them.
  • Every call gets exactly one result: a call that cannot run gets "Error: the call failed (...)", one interrupted "Interrupted by the user".
  • Read-only calls in a row (Read, Glob, Grep, JobOutput) that run without asking run at once, up to 8; their events and results stay in order.
  • The same failing call made a third time in a row earns a reminder to change approach; invalid arguments to one tool three times in a row stop the turn and tell the user.
  • oklm_agent_steer() sends the agent a message while it works: it is added before the model's next turn, and a turn that was about to answer goes on to read it. On the terminal, a line typed with Enter while the agent works is sent so ([sent to the agent: ...]).
  • The default system prompt carries worked examples (finding where a name is defined with Grep, building and fixing until the tests pass), as spec §27 advises for local models.

On Qwen3-4B (Q4_K_M, OpenCL) in this repository, "Where is X defined? Give the file and the line number." is answered right for four functions of five (oklm_agent_steer, tr_head_norm_forward, oklm_ingest_pdf_pages); for an OpenCL kernel the model filtered by a C type, which leaves .cl files out, and gave the line where the kernel's name is listed. Before the worked example and the hint after no match, it searched for def name (assuming Python) and reasoned until the turn's limit.

Backends, stream events and interrupts

The agent talks to the in-process model (oklm agent -m), or to a backend (oklm_agent_params.backend): it receives the conversation as OpenAI chat messages (the system prompt first, an assistant's calls with their arguments as JSON strings and an id each, tool messages with their tool_call_id) and the tools, streams pieces back and returns the answer's text and its calls. Two come with the library:

  • oklm_agent_http_new(), oklm agent --url URL [--api-key KEY] [--remote-model NAME]: an OpenAI-compatible server's streamed chat completions, oklm serve or another (its retrieval stages ask the same server: rag.md). Content and reasoning deltas (reasoning_content) stream as they come, a <think> block left in the content included; tool calls are assembled from their deltas by index (an id and a name, then the arguments in pieces); usage is read from the last chunk (stream_options.include_usage). One rule holds for the members of a request an answer can do without, stream_options, chat_template_kwargs and the sampling members a chat adds (oklm_agent_http_set_extra()): a server that answers 400 or 422 naming one of them is asked again without it at once, and no later request carries it (oklm_request_refused_member() and oklm_request_without_member() in src/client/request_members.c, which the chat window uses too; oklm_agent_http_refused() lists what was left out; without stream_options the counts are estimates). A refusal that names none of the request's members, or one of model, messages, stream, max_tokens, temperature or tools, stays the turn's error. The template variables (--template-vars, /effort's enable_thinking) go in each request as chat_template_kwargs (the backend's turn_vars). A turn's reasoning is kept apart from its answer (the reply and the content sent back never hold it) and sent back as the message's reasoning_content only when the agent keeps it (oklm_agent_params.keep_reasoning, the reasoning.keepInHistory setting; by default when the template variables ask for preserved thinking: clear_thinking false or preserve_thinking true), as the in-process model keeps it in the history for its template. The session's transcript keeps it too. The terminal chat on a server (oklm chat --url, inference.md) is a client of the same backend, and three of its functions are there for it: a turn given no tools (NULL) sends no tools member; oklm_agent_http_set_temperature() and oklm_agent_http_set_extra() change the temperature and add members to each request's body (top_p, top_k, min_p, seed); oklm_agent_http_last_error() gives the status and the reason of the answer the last turn failed on (401 for a refused key), which the turn's result (OKLM_ERR_IO) does not tell. The agent's own requests are what they were.
  • oklm_agent_mock_new(), oklm agent --mock FIXTURE.jsonl: a fixture replayed, one turn a line: {"text": ..., "reasoning": ..., "tool_calls": [{"name": ..., "arguments": {...}}], "stop": "max_tokens", "call_open": ..., "usage": {"prompt": N, "output": N}} (call_open with "stop": "max_tokens": the text of a tool call the turn is cut inside), the text streamed a word at a time; each request is recorded (oklm_agent_mock_request()). The tests script the agent with it, and it shows the terminal without a model.

Background sub-agents run with the in-process model, a generate callback, or a backend whose clone gives each one a copy of its own (released with the sub-agent, which SendMessage may keep): oklm agent --url gives each its own connection to the server, with the same key, model and retries. A backend without clone serves one turn at a time; with it, Agent runs in the foreground and a background call is refused.

on_event receives what happens as it happens: TEXT and REASONING pieces (the <think> blocks told apart, whichever the model or backend), each TOOL_CALL before it runs and its TOOL_RESULT, the USAGE of each model turn (prompt tokens, those already in the cache, tokens written), and its STOP: end of turn, tool use, the token limit, cancelled, or an error. on_text and on_tool still receive the raw text (with its <think> tags) and the results. Reasoning is not sent back to a backend (a server may leave it in the content); the in-process model's templates drop it themselves.

oklm_agent_cancel(), from any thread or a signal handler, interrupts the work: the model's turn stops within a token (its text so far is kept in the history), a running command is killed (its process group, as at a timeout; its output so far ends with [interrupted by the user]), the calls not run yet get "Interrupted by the user" as their result, so every call has one, a foreground sub-agent stops too, and oklm_agent_send() returns with [interrupted]. The next message starts afresh. On the terminal, Esc or Ctrl-C while the agent works interrupts it (a second Ctrl-C before it has stopped ends the program); at the prompt Ctrl-C clears the line, and twice on an empty line leaves.

Headless use

oklm agent -p TASK answers and exits. What a script pipes in comes first (cat log.txt | oklm agent -p "explain the error"; -p - takes standard input alone); a pipe that stays silent for two seconds is taken as empty, so a caller that leaves it open does not hang the agent.

  • --output-format text (the default) prints the answer as on a terminal. json prints one object at the end; stream-json prints an object per line as the work goes (oklm_headless_* in headless.h):

    {"type":"system","subtype":"init","cwd":...,"model":...,"permissionMode":"default","tools":["Read",...],"session_id":...}
    {"type":"assistant","message":{"id":"msg_1","role":"assistant","content":[{"type":"thinking","thinking":...},{"type":"text","text":...}],"stop_reason":"tool_use"},"session_id":...}
    {"type":"assistant","message":{"id":"msg_1",...,"content":[{"type":"tool_use","id":"call_0","name":"Read","input":{"file_path":"notes.txt"}}]},...}
    {"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"call_0","content":...,"is_error":false}]},...}
    {"type":"result","subtype":"success","is_error":false,"result":"plum","num_turns":2,"duration_ms":5210,"usage":{"input_tokens":...,"cache_read_input_tokens":...,"output_tokens":...},"session_id":...}

    The result's subtype is success, error_max_turns or error_during_execution; its result is the answer without the reasoning. With a JSON output nothing else reaches standard output (the agent's own lines go to standard error) and nobody is asked: a call the rules do not allow is refused, so a script allows what it needs.

  • --input-format stream-json (with --output-format stream-json) reads user messages from standard input, a JSON line each ({"type":"user","message":{"role":"user","content":"..."}}, the content a string or text parts), and writes a result after each: one conversation over several requests.

  • Prompt commands work (-p "/review src/x.c"); built-in commands answer with an error in the JSON formats (their output is for a terminal).

  • --save-training FILE appends the run's whole conversation, rendered through the model's chat template with no trailing generation prompt, to FILE as one JSONL line {"model", "text"}: the exact in-process shape a turn is trained on (oklm_agent_training_text()). It needs an in-process model (a backend has no template). Running tool-use tasks this way captures fine-tuning data; tools/make_mcp_data.sh does it against the tool zoo (tools/mcp_zoo.py) and keeps the successful runs (mcp-training-design.md).

  • --no-tool-grammar leaves the model's tool calls unconstrained (they are constrained by the tools' grammar from the call's opening marker on otherwise; oklm_agent_params.free_tool_calls): how well a model writes its calls unaided, which tools/mcp_loop.sh measures.

  • The exit status is 0 when the request (each request) succeeded, 1 when one ended in an error, 2 for a usage error. A message that was not answered because the context had no room for it is an error, in the text output as in the JSON ones (agent-sessions.md); an answer cut by --max-tokens is not.

  • The spec's flag names are read as OKLM's: --model (-m), --permission-mode (--mode), --system-prompt (--system), --append-system-prompt, --add-dir DIR, --resume [ID], --verbose (accepted), --mcp-config FILE and --strict-mcp-config (agent-extensions.md).

  • --allowedTools RULES and --disallowedTools RULES (also spelled --allowed-tools and --disallowed-tools) give the session a list of permission rules in one argument: each rule of --allowedTools is added as an allow rule, each of --disallowedTools as a deny rule, with the command line as their source, exactly as one --allow RULE or --deny RULE a rule would. The rules are separated by commas or blanks, those inside parentheses belonging to the rule: --allowedTools "Read Bash(git log:*)" is two rules, --disallowedTools Edit,MultiEdit two others. Both options may be repeated, 64 rules at most over the command line; a rule that is not Tool or Tool(pattern) stops the command (malformed rule). A tool --disallowedTools names without a pattern is left out of what the model is offered (agent-permissions.md); an allow rule does not restrict the session to the tools it names, it only lets their calls run without a question.

tools/check_headless.sh scripts it as a CI job would, without a model (a fixture replays the turns): the JSON result read with sed, the stream-json lines, piped input reaching the model, and the three exit statuses.

An agent other programs attach to

oklm agent -m MODEL --listen [PORT] runs the agent without a terminal and serves it over HTTP on loopback: its events as they come, and from the other side the user's messages and the answers to its permission questions. oklm-gui attaches to it (gui.md); any program that speaks HTTP can.

oklm agent -m qwen3-8b --listen        # the first free port from 7481
agent: listening on http://127.0.0.1:7481 (loopback only); the token is in /run/user/1000/oklm/agents/4242.json
  • Found by its file. At start the agent writes $XDG_RUNTIME_DIR/oklm/agents/PID.json (else ~/.cache/oklm/agents; %LOCALAPPDATA%\oklm\agents on Windows), readable by the user only: the port, a 256-bit token made for this run, the project (as an absolute path), the session and the model. The file is removed when the agent ends; one left by an agent that was killed names a process that is gone, which a client checks. The file also says when its process started (start_ticks): a pid that another process got since is not taken for the agent, and the file is removed.

  • Loopback and a token. Only 127.0.0.1 is listened on, and every request carries Authorization: Bearer TOKEN, compared in constant time. Whoever can read the file can drive the agent, with the agent's permissions: that is the user, and nobody else.

  • Routes. The listener answers five routes and no other (404), each with the token; they are not routes of oklm serve, whose own are in server.md:

    Method Path What it does
    GET /agent/state what the agent is and does now: its pid, project, session and model, whether it works, the last event's number, the question waiting, the messages queued, the context and what is used of it
    GET /agent/events the events after number after, waiting up to wait milliseconds for one (?after=N&wait=MS)
    POST /agent/message a user's message queued, {"text": "..."}
    POST /agent/permission the answer to a permission question, {"id": N, "answer": "yes"}
    POST /agent/interrupt the work in progress cancelled
  • Events. The agent's stream-json lines (the same as --output-format stream-json above), each numbered from 1, with three more types: the user message a client sent, permission_request (id, tool, subject, reason, always: the rule "always" would add, preview: the diff of an edit) and permission_result. GET /agent/events?after=N&wait=MS answers {"events": [{"seq": n, "event": {...}}], "last_seq": L, "busy": bool}, waiting up to MS (25 seconds at most) when nothing is new: a client asks again with the last number it has. The last 8192 events are kept, so a client attaching late reads what happened.

  • Messages. POST /agent/message {"text": "..."} queues a message (64 at most); the agent takes them in order, as if typed.

  • Permission questions. A tool call that would ask at a terminal raises a permission_request and waits two minutes for POST /agent/permission {"id": N, "answer": "yes" | "no" | "always", "feedback": "..."}; without an answer the call is refused. Any attached client may answer; the first answer counts.

  • The rest. GET /agent/state (pid, project, session, model, busy, the last event's number, the question waiting, the messages queued, and context with context_used: the model's window and what the agent's instructions, tools and conversation take of it, told once the model is loaded and again after each request, 0 before; the chat window reads them to search a document too long for what is left rather than send it whole: gui.md); POST /agent/interrupt cancels the work in progress.

It is the headless agent with another way in and out: the same loop, tools, permission rules, hooks and session file, which --resume can pick up later. On Windows it is the same listener: loopback only, the token from the system's random generator, and the file kept for the user by an access list of one entry (full control for that user, nothing inherited) on the directory and on the file, where POSIX has the modes 0700 and 0600; the default list of %LOCALAPPDATA% already keeps other accounts out, the system and the administrators excepted, as root is on POSIX. Run on 2026-11-13 with the MinGW build under Wine 11, not on Windows itself: the test below, and oklm.exe agent --listen asked for its state without the token and with a wrong one (401), with it (200), sent a message and read its events back; Wine keeps the file as mode 0770, the user's Unix group included. A listening agent ends when it is asked to, on every system: oklm agent --stop-listening PORT finds it by its port among the user's listeners and asks it (SIGTERM; on Windows the event of the registry of processes, server.md), as Ctrl-C in its console or a signal does: the turn in progress is cancelled, the listener is closed and its file removed, the session is kept (agent: asked to stop: the listener is closed, the session is kept, the status 0); one that has not ended after ten seconds is ended by the system. The listener is oklm_listen_* (include/oklm/harness/listen.h; oklm_listen_list() gives the agents listening now, from their files, which is what oklm-gui reads), tested by tests/harness/test_listen.c (the file, the list, the token, the events and their wait, the stream, the queue, a question answered, refused and timed out, the interrupt) and by hand with a client that sent a message, answered the Write tool's question and read the result.

Library use

The agent is a library (liboklm, the headers under oklm/harness/): the program creates a tool context and an agent, sends messages, and receives the events. Besides on_event and oklm_agent_cancel() above:

  • Its own tools: oklm_tool_ctx_add_tool(ctx, definition, fn, user) adds a tool from its JSON definition (name, description, parameters); agents of that context offer it after the built-in ones, check a call's arguments against its schema, decide it by the rules by its name (--allow Weather, or the mode: asked in the default mode), run the PreToolUse and PostToolUse hooks and audit it like any other; fn returns the text for the model. A name already taken is refused.
  • Hooks as functions: oklm_hooks_add_callback(hooks, event, matcher, fn, user); the function gets the event's JSON and answers as a command would (0, 2 to block with a reason, a JSON answer).
  • The permission question: the tool context's ask callback (oklm_ask_fn) is asked whenever the rules and the mode say ask, with the tool, its arguments, a preview of an edit; it answers yes, no, always (with the rules to add) or says what to do instead.
  • The output formats: oklm_headless_new() and oklm_headless_on_event() turn the events into the stream-json lines above for any FILE*.
oklm_tool_ctx tc = {0};
tc.root = ".";
oklm_perms_new(".", &tc.perms);
oklm_perms_add_rule(tc.perms, OKLM_SRC_CLI, OKLM_PERM_ALLOW, "Weather");
oklm_tool_ctx_add_tool(&tc, "{\"name\":\"Weather\",\"description\":\"The weather of a city\","
                            "\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},"
                            "\"required\":[\"city\"]}}", weather_fn, NULL);
oklm_agent_params p = oklm_agent_default_params();
p.backend = oklm_agent_http_backend(http);   /* or a model and context */
p.on_event = my_events;
oklm_agent_new(NULL, NULL, &tc, &p, &agent);
oklm_agent_send(agent, "What is the weather in Paris?", &reply);

On a published model

Qwen3-4B (Q4_K_M, tests/public_models.txt), thinking on, a scratch project holding a 6-line hello.c:

$ oklm agent -m models/qwen3-4b/Qwen3-4B-Q4_K_M.gguf --project build/agent_demo \
    --mode acceptEdits --allow "Bash(wc:*)" \
    -p "How many lines does hello.c have? Use wc. Then change the greeting printed by hello.c from hello to bonjour."
[Bash {"command":"wc -l hello.c"}] ok:
  | 6 hello.c
[Edit {"file_path":"hello.c","old_string":"hello","new_string":"bonjour","replace_all":false}] ok:
  | Replaced 1 occurrence
The file hello.c has 6 lines. The greeting in hello.c has been changed from "hello" to "bonjour".
$ oklm audit build/agent_demo/.oklm/audit/<session>.jsonl
2026-11-01T10:59:17Z  allow rule Bash  {"command":"wc -l hello.c"}  Bash(wc:*)
2026-11-01T10:59:17Z  allow mode Edit  {"file_path":"hello.c",...}  acceptEdits

The model writes both calls in one turn after thinking; both run, the first by the command-line rule, the second by the mode.

A task of several steps (add a function to a C file, use it, compile and run it), with --allow "Bash(cc:*)" --allow "Bash(./calc)":

[Read {"file_path":"calc.c","offset":1,"limit":20}] ok
[Edit {"file_path":"calc.c","old_string":"static int add(...) { return a + b; }\n\nint main(void) {",
       "new_string":"...\nint sub(int a, int b) { return a - b; }\n\nint main(void) {\n\tprintf(\"%d\\n\", sub(7, 4));"}] ok
[Bash {"command":"cc -o calc calc.c && ./calc"}] ok:
  | 3
  | 5
The function `sub(a, b)` has been added to `calc.c`, and `main` now prints `sub(7, 4)`. Compilation and
execution confirm the output is `3` (7 - 4), as expected.

A plan ticked: --mode acceptEdits --allow "Bash(make:*)" --allow "Bash(./app)" -p "Do these three steps and keep a task list with TodoWrite, ticking each step when done: 1) change the version printed by app.c from v1 to v2, 2) build it with make, 3) run ./app and check it prints v2." (commands sandboxed):

[TodoWrite ...] Task list (0 of 3 done)
[Edit {"file_path":"app.c","old_string":"v1","new_string":"v2"}] Replaced 1 occurrence
[TodoWrite ...] Task list (1 of 3 done)
[Bash {"command":"make"}] cc -o app app.c [exit code 0]
[TodoWrite ...] Task list (2 of 3 done)
[Bash {"command":"./app"}] v2 [exit code 0]
[TodoWrite ...] Task list (3 of 3 done)
The version in app.c was successfully updated from v1 to v2, the build completed without errors,
and running ./app confirmed it prints v2 as expected.

A program made to pass a project's test, in a project holding only a Makefile (test builds fizzbuzz and diffs ./fizzbuzz 15 against expected.txt), with --mode acceptEdits --allow "Bash(make:*)" and the project under /tmp (commands sandboxed), in 22 seconds:

$ oklm agent ... -p "Create fizzbuzz.c so that make test passes: the program takes N as its first argument and
    prints 1 to N, one per line, Fizz for multiples of 3, Buzz for multiples of 5, FizzBuzz for both. Check with make test."
[Write {"file_path":"fizzbuzz.c","content":"#include <stdio.h>\n#include <stdlib.h>\n..."}] ok:
  | Wrote 491 bytes to .../fizzbuzz.c
[Bash {"command":"make test"}] ok:
  | cc -Wall -o fizzbuzz fizzbuzz.c
  | ./fizzbuzz 15 > out.txt
  | diff -u expected.txt out.txt
  | [exit code 0]
The file `fizzbuzz.c` has been created and the test passed successfully.

Worded "write fizzbuzz.c ..., then run make test and fix the program until it passes", the model first wrote a program without <stdlib.h>, read the compiler's error from make test, added the include with Edit and ran the test again, which passed. (With the earlier prompt, which did not ask to check changes with the tests, it wrote the file and stopped, reasoning that it could not run commands.)

Skills end to end (tools/check_skills_e2e.sh [model] [options]): a project holds a skill the model cannot know, "the zorblax format", whose script writes it; asked to save a text in that format, Qwen3-4B (Q4_K_M, OpenCL) loads the skill by itself and runs the script, which writes the file, in about 25 s. Before the Skill result said that loading did nothing yet, the same model loaded the skill and then announced the file written without running anything. Qwen2.5-1.5B writes the file by hand without loading the skill.

A skill: oklm agent ... --allow "Bash(cc:*)" --allow "Bash(grep:*)" -p "Check calc.c with the right skill." loads c-check, runs its command (first piped into grep -n warning, which exits 1 when nothing matches, then alone) and answers that the file is clean. Without the grep rule the piped command asks, is refused (no terminal), and the model still claims the file is clean: a small model's answer is only as good as the results it actually got, which the audit log shows.

The same task in plan mode shows the 4B model's limits: it reads the file, writes a task list, tries to compile (denied: plan mode), then writes "ExitPlanMode" as text instead of calling it. Smaller models (Qwen2.5-1.5B, Qwen3-0.6B) make the first call and then end their turn announcing the second without writing it (the raw turns, with OKLM_LOG_LEVEL=DEBUG, show them ending on <|im_end|>).

Agent evals

make eval (or sh tools/eval_agent.sh [model.gguf] [--only NAME,...] [oklm agent options]) runs the tasks of tests/evals/ with a real model and judges each by what it left behind, not by what it says. A task is a directory: prompt (the request; @DIR@ and @PORT@ are replaced), setup.sh (makes its files in an empty directory, a repository for the git ones), check.sh (exit 0: passed), options (its permission mode and allow rules; @FAKE@ is the fake MCP server) and an optional teardown.sh. Each runs headless (-p, JSON output, nobody to approve), outside OKLM's own repository, under a 900 s limit. build/eval/report.md and report.json give each task's verdict, turns, tokens and seconds, and the pass rate; the session, audit log and checker output stay in build/eval/NAME/. Run it after a change to the prompt or the tools.

The seventeen tasks: fix a failing test, answer from a fetched page, call an MCP tool, follow a project's skill, /commit only the change, add a CLI flag, rename a function across files, find and explain a bug without fixing it, write a README from the code, start a dev server and wait for it, commit on a new branch in the history's style, find the commit that added a line; resolve a merge conflict, rebase onto main, say why a function was added (the git evals, 05 and 11 to 15); fix an error a program logs on startup, run the e2e tests and fix the failing one (the process evals, 10, 16 and 17). Tasks 18 to 25 need the project's documentation (a setting, a command, a constant, a reason, an exit status: 18 to 21, 24 and 25) or an earlier session (22 and 23); they measure the agent's retrieval (rag.md). Task 26 judges behaviour beside the outcome: two sub-agents started at once fix a bug each and add a line each to one CHANGELOG.md; it passes when both bugs are fixed, both lines are there and the rest of the file is intact, nothing else changed, no command threw work away (git checkout --, reset --hard, stash, clean, rm), the two Agent calls came in one reply, and the report names both fixed files and no file that does not exist. Qwen3-8B (2026-11-11), three runs: one pass; in the two others the second sub-agent explained dist.py's bug without fixing it and the final report still said both were fixed (once the first also rewrote the changelog without its heading). No command threw work away. With each report ending on the files the sub-agent wrote, four more runs: in one, the second sub-agent replaced the first one's changelog line with its own (an Edit, after a fresh Read) and the final report said both had added theirs; with lines a sibling wrote kept from the others, three more runs kept both lines every time (the hold not needed there), each failing on a fix not made instead (temp.py still subtracting 32, dist.py unchanged). Two of those failures were the task's own: a fix by the factor 0.621371 is off by 3e-7, and the test asked 1e-9 (now 1e-5). With the last command in each report line, a second ask for a sub-agent that wrote nothing, and Write taking Read's line numbers pasted back (a sub-agent had written its changelog as " 1 ## Unreleased", refused each time for dropping its sibling's line): 4 of 6 runs pass; the two others fail on the caller's own report (it said a sub-agent had added a changelog line that sub-agent said it could not add, and once answered with a plan instead of the report).

Qwen3-8B, the default model (Q4_K_M, OpenCL, 2026-11-07): 12 of 17, then 13 of 17 once the agent's own state was kept out of the repository's status, about 4 turns and 38 s a task:

Task Verdict What the model did Change
14 rebase onto main fail, then pass saw .oklm/ untracked in the git snapshot, called the tree dirty and stopped without rebasing .oklm/audit, jobs, sessions, worktrees each hold a .gitignore of * (oklm_git_ignore_dir())
08 explain a bug fail named the wrong line none: the model's
09 README fail guessed the options without reading wordcount.c none: the model's
15 why added fail gave the commit's subject (read the port from PORT) rather than the reason in its body (the platform assigns the port) none: the model's
16 startup error fail blamed the file's opening and wrapped it in try with broken indentation; the cause is the setting's name (listen_port against port) none: the model's

Every git and process task but 15 and 16 passes: 05, 10, 11, 12, 13, 14, 17.

GLM-4.7-Flash (Q4_K_M, oklm serve on OpenCL with a 16K context, the agent through --url, 2026-11-11): 21 of 25, 5 turns a task on average, 929 s and 724K prompt tokens in all; its 163 calls in the glm45 format all parsed (none left as text in an answer, none refused for its arguments, in the 25 transcripts). It passed every documentation task without retrieval (reading the docs with Glob and Read), and failed:

Task What the model did
09 README copied the working directory as 09-write-readereadme in every absolute path; the Write outside the project was refused and it never saw the typo
16 startup error replaced the lookups by .get() with a combined message instead of naming the missing setting
22 session branch asked for the branch name instead of looking at the earlier sessions
23 session changelog wrote an entry in its own style, the convention of the earlier session unread

All four are the model's.

GLM-4-9B-0414 (Q4_K_M, the same setup, 2026-11-11, its calls in the glm4 format): 3 of 25 (02, 03 and 04), 2 turns a task on average. Its 26 calls all parsed; it ends most tasks after its first turn on a plan written in prose ("First, I'll check the error log..."), once with the calls written inside a code fence; the format reads every call it opens a turn with. Reading the calls written inside a turn of text too (a line naming a tool, then its JSON) changes little: 3 of 25 again, 31 calls against 26, task 01 running its fenced build command and stopping after.

The review commands' seeded-bug measure (tools/eval_review.sh, six bugs, /code-review medium) finds 6 of 6 (2026-11-04) and 5 of 6 (2026-11-06) on Qwen3-8B, above the 80 % asked; Qwen3-4B found 2, then 0.

The earlier twelve tasks on Qwen3-4B (Q4_K_M, OpenCL, 2026-11-03): 7 of 12, about 3 turns and 53 s a task. The failures it showed, and what came of them:

Task What the model did Change
09 README read README.md (missing) and wrote a generic one a Read of a missing file now lists its directory (hidden names and those a Read deny rule covers left out); the model then names the program and make, but still guesses its options without reading wordcount.c
11 branch and commit three tasks in_progress refused the whole TodoWrite, and it stopped after the other calls of the turn TodoWrite keeps the first in progress and makes the others pending, with a note; passes
earlier runs piped or redirected an allowed command, refused with nobody to ask the refusal names the parts no rule covers and says to run the allowed command alone
08 explain a bug named line 17 for the bug on line 16, after reading numbered lines none: the model's
10 dev server put an invented detach on its curl and wrote $(curl ...) as the file's text in the same turn none: the model's (it passes on other runs)
12 git history took the latest commit from git log for the one that added the line none: the model's

Edit against a whole-file Write (Qwen3-4B Q4_K_M, OpenCL, 2026-11-08, tasks 01, 06 and 07, the ones that change an existing file; Edit and MultiEdit refused with --disallowedTools Edit,MultiEdit, so only Write can change a file). Three runs of each setting gave the same verdicts every time here (the sampling is greedy), so it is one sample of three tasks per setting, not nine:

Setting 01 fix a bug 06 add a flag 07 rename Written tokens (01, 06, 07)
Edit allowed pass, 38 s fail fail 1870, 1564, 4935
Write only fail, 172 s pass fail 9243, 1914, 4697

1 of 3 each, so the pass rate says nothing at this size. The cost differs on one task only: on 01 (a file of a few lines; the written tokens include the model's reasoning) Write only wrote 9243 tokens against 1870 and took 172 s against 38 s, and its rewrite failed the check; on 06 the two cost about the same (1914 against 1564) and 07 differs little. Inconclusive: Edit stays the default, and a file much longer than these would be the fair test of the whole-file rewrite.

Runs on OpenCL are not repeatable token for token, and the set that passes moves between runs: three full runs on the same day gave 7 of 12 each time, with 07 (a file left out of the rename) and 11 (the agent's own .oklm/ staged instead of the file) passing in some and failing in others. Two changes were tried and left out because they measured worse: a sentence in the system prompt asking to read code before writing about it (09 then read the Makefile, but 06 and 07 failed in every run), and a refused unread Edit showing the file and counting it as read (07 re-read the files and stopped without editing, 4 runs of 4).