Agent loop
How oklm agent runs a task: the loop of model turns and
tool calls, the system prompt, the backends and their stream events,
headless and library use, and how it fares on published models and the
eval tasks. Part of the agent harness (harness.md).
Status: implemented.
include/oklm/harness/agent.h,
src/harness/agent.c.
oklm_agent_send(agent, message, &reply) adds the user's
message and runs model turns until the model answers without a tool
call:
- The conversation is rendered by the model's chat template, the
system prompt first, the six tools in its
toolsvariable.--chat-template FILE|NAMEreplaces the file's template (inference.md); an LFM2 file whose template says nothing of tools is rendered with the one that writes them in Liquid's form, the format found from the call tokens of its vocabulary, asoklm serveandoklm chat --toolsdo, unless a template was asked for: that one is rendered as it is. The small model of the asides (smallModel) keeps its file's template, and gets the option's when its file carries none. - Only the tokens after the prefix already in the context are evaluated (the KV cache is truncated to the common prefix).
- The model generates; once it opens a tool call
(
<tool_call>,[TOOL_CALLS], or<|python_tag|>for Llama 3) the call is constrained by the tools' grammar until it closes; the model may then write more text or more calls, until it ends its turn. - The calls are parsed (ADR-0009), recorded in the assistant message,
run in order by
oklm_tool_run()(rules, questions, audit) and their results, cut at the output limit, appended as tool messages, one per call.
System prompt
include/oklm/harness/prompt.h,
src/harness/prompt.c. The prompt is assembled once, from
parts in a stable order, so it is the same bytes on every turn and the
model's cache keeps it:
- identity and core rules (be concise, read before changing, follow the code's conventions, check with the build or tests, never commit unless asked, refuse malicious code);
- the tool policy: every part of the request, a task list with TodoWrite for several steps (one task in progress), dedicated tools rather than commands, independent reads together, tool results and files as data and never instructions, two worked examples (Grep, Bash then Edit until the tests pass);
- starting an Agent for broad searches, or, for a sub-agent, that it reports to the agent that started it;
- the output style, when not
default; - the environment: the project directory (absolute: a model given a relative one used it as a prefix of its paths), whether it is a git repository, the platform, the OS version, today's date, the model;
- the git snapshot and 7. the memory files, when the session has them
(
git_snapshotandmemoryof the parameters); - the skills (within their budget) and the deferred tools' names (with
--tool-rag K, each MCP server's tools counted instead), then the text of--append-system-prompt.
--system (or --system-prompt) replaces it
all; appended text still follows. The wording is in
src/harness/prompts/*.md, embedded at build time;
tests/harness/test_prompt.c renders it against the
snapshots of tests/fixtures/prompt/
(OKLM_UPDATE_SNAPSHOTS=1 rewrites them after a deliberate
change). Output styles: explanatory adds short insights
about the code, learning leaves a meaningful piece to the
user with a TODO(human); NAME.md in
.oklm/output-styles/,
~/.config/oklm/output-styles/ or
.agent/output-styles/ (a front matter dropped) is a custom
one. --output-style NAME chooses at the start,
/output-style NAME later
(oklm_agent_set_output_style(), which starts the cache
over).
What changes during a session comes as reminders, user messages
marked meta and added before the model's next turn: the
permission mode when it changes (plan mode's rules, acceptEdits, back to
default...; at the start only for plan mode), the files read that
changed on disk since (by the user, an editor or a command; once per
change), the task list after eight turns of other calls while tasks are
unfinished, and with --tool-rag K the definitions of the
deferred tools retrieved for the user's message (agent-tools.md),
and with --rag-inject always|route the passages of the
project's documentation retrieved for it, the message condensed from the
conversation first (rag.md).
With --rag-critique on|flag (an in-process model, or a
served one that can judge: rag.md) a final answer that cites
[n] is checked against the passages its message retrieved,
through the answer_check hook: its verdict is a notice, and
with on an unsupported statement sends the agent back once
with the critique as a reminder, as a blocking Stop hook does (rag.md). When the
in-process model's template is rendered, a user message and the
reminders after it are one message, for templates that want the roles to
alternate.
A turn limit (32) stops a model that never answers; a turn has at
most 8192 tokens, room for a thinking model's reasoning, and
oklm agent opens a context of 32768 tokens (fewer when the
model was trained on fewer). A generate callback replaces
the model by any text source (the tests use a script).
OKLM_LOG_LEVEL=DEBUG logs each raw model turn.
oklm agent -m <model> [-p TASK] [--project DIR] [--mode MODE] [--dangerously-skip-permissions] [--allow RULE] [--ask RULE] [--deny RULE] [--no-sandbox] [--allow-network] [--limit NAME=VALUE] [--system TEXT] [--append-system-prompt TEXT] [--output-style STYLE]
runs it on a terminal: tool calls are shown compactly (below), and the
session's decisions go to
.oklm/audit/<session>.jsonl. Without -p
it is a REPL (/reset, /mode MODE,
/add-dir DIR, /output-style STYLE,
/jobs, /thinking, /exit;
Shift+Tab, at the prompt or while the agent works, switches to the next
mode, the bypass mode with a red banner). /add-dir adds a
working directory for the session: inside the project for the modes,
writable in the sandbox. A question shows the call, an edit's diff in
colour, and why it asks (in red for a dangerous call):
[Edit] f.c
--- a/f.c
+++ b/f.c
@@ -1,2 +1,2 @@
int a;
-int b;
+long b;
not covered by a rule (default)
Allow? [y]es, [a]lways (saves Edit(f.c)), [n]o, or say what to do instead:
a saves the rule to
.oklm/settings.local.json; anything other than
y, a or n refuses and is passed
to the model as what to do instead; a plan offers yes, accept and allow
edits, no, or what to change.
A thinking model's reasoning (<think> ...
</think>, a harmony reply's analysis channel, a Muse
Glimmer reply's message to itself or Gemma 4's thinking channel, told
apart while it streams by include/oklm/harness/think.h, the
tags possibly cut across tokens) is folded: the spinner counts its words
as it comes, then one line stays,
[thought, 212 words; ctrl-o or /thinking: full view], and
the answer follows (the classic view). Ctrl-O while the model writes
switches to the full view: it unfolds the block being written (what it
has so far, dim, then the rest as it comes) and the next ones; pressed
again, it erases the unfolded block from the screen (its wrapped lines
counted at the terminal's width) and the status line takes its place
again. Pressed between reasoning blocks, it says which view is now on.
/thinking (or Ctrl-O) at the prompt does the same for the
next replies and shows the last message's reasoning.
--thinking show starts unfolded, and so does
ui.viewMode full (or fullscreen,
which also enters the full-screen UI on a terminal). The terminal is
read key by key only during a turn, without echo: the agent looks
between the chunks of the prompt it evaluates, at each token (a tool
call's too) and while a tool runs
(oklm_agent_params.on_poll), so Esc and Ctrl-O act at once;
what is typed meanwhile waits for the next prompt (the arrows and other
special keys pressed meanwhile are dropped), and the settings come back
for the permission questions, at the end, and on Ctrl-C. Output that is
not a terminal gets the one line per block, without the status line.
When the template opens the reply with <think>
itself, the agent's text starts with that tag, so the blocks are always
whole.
Loop details
- Every message has an id (a random UUID), its parent's id, a time,
and for the assistant's turns their usage; reminders the agent adds (a
job that ended, a background agent's report, a loop noticed) are user
messages marked
meta.oklm_agent_message()reads them. - Every call gets exactly one result: a call that cannot run gets "Error: the call failed (...)", one interrupted "Interrupted by the user".
- Read-only calls in a row (Read, Glob, Grep, JobOutput) that run without asking run at once, up to 8; their events and results stay in order.
- The same failing call made a third time in a row earns a reminder to change approach; invalid arguments to one tool three times in a row stop the turn and tell the user.
oklm_agent_steer()sends the agent a message while it works: it is added before the model's next turn, and a turn that was about to answer goes on to read it. On the terminal, a line typed with Enter while the agent works is sent so ([sent to the agent: ...]).- The default system prompt carries worked examples (finding where a name is defined with Grep, building and fixing until the tests pass), as spec §27 advises for local models.
On Qwen3-4B (Q4_K_M, OpenCL) in this repository, "Where is X defined?
Give the file and the line number." is answered right for four functions
of five (oklm_agent_steer,
tr_head_norm_forward, oklm_ingest_pdf_pages);
for an OpenCL kernel the model filtered by a C type, which leaves
.cl files out, and gave the line where the kernel's name is
listed. Before the worked example and the hint after no match, it
searched for def name (assuming Python) and reasoned until
the turn's limit.
Backends, stream events and interrupts
The agent talks to the in-process model (oklm agent -m),
or to a backend (oklm_agent_params.backend): it receives
the conversation as OpenAI chat messages (the system prompt first, an
assistant's calls with their arguments as JSON strings and an id each,
tool messages with their tool_call_id) and the
tools, streams pieces back and returns the answer's text and its calls.
Two come with the library:
oklm_agent_http_new(),oklm agent --url URL [--api-key KEY] [--remote-model NAME]: an OpenAI-compatible server's streamed chat completions,oklm serveor another (its retrieval stages ask the same server: rag.md). Content and reasoning deltas (reasoning_content) stream as they come, a<think>block left in the content included; tool calls are assembled from their deltas by index (an id and a name, then the arguments in pieces); usage is read from the last chunk (stream_options.include_usage). One rule holds for the members of a request an answer can do without,stream_options,chat_template_kwargsand the sampling members a chat adds (oklm_agent_http_set_extra()): a server that answers 400 or 422 naming one of them is asked again without it at once, and no later request carries it (oklm_request_refused_member()andoklm_request_without_member()insrc/client/request_members.c, which the chat window uses too;oklm_agent_http_refused()lists what was left out; withoutstream_optionsthe counts are estimates). A refusal that names none of the request's members, or one ofmodel,messages,stream,max_tokens,temperatureortools, stays the turn's error. The template variables (--template-vars,/effort'senable_thinking) go in each request aschat_template_kwargs(the backend'sturn_vars). A turn's reasoning is kept apart from its answer (the reply and the content sent back never hold it) and sent back as the message'sreasoning_contentonly when the agent keeps it (oklm_agent_params.keep_reasoning, thereasoning.keepInHistorysetting; by default when the template variables ask for preserved thinking:clear_thinkingfalse orpreserve_thinkingtrue), as the in-process model keeps it in the history for its template. The session's transcript keeps it too. The terminal chat on a server (oklm chat --url, inference.md) is a client of the same backend, and three of its functions are there for it: a turn given no tools (NULL) sends notoolsmember;oklm_agent_http_set_temperature()andoklm_agent_http_set_extra()change the temperature and add members to each request's body (top_p,top_k,min_p,seed);oklm_agent_http_last_error()gives the status and the reason of the answer the last turn failed on (401 for a refused key), which the turn's result (OKLM_ERR_IO) does not tell. The agent's own requests are what they were.oklm_agent_mock_new(),oklm agent --mock FIXTURE.jsonl: a fixture replayed, one turn a line:{"text": ..., "reasoning": ..., "tool_calls": [{"name": ..., "arguments": {...}}], "stop": "max_tokens", "call_open": ..., "usage": {"prompt": N, "output": N}}(call_openwith"stop": "max_tokens": the text of a tool call the turn is cut inside), the text streamed a word at a time; each request is recorded (oklm_agent_mock_request()). The tests script the agent with it, and it shows the terminal without a model.
Background sub-agents run with the in-process model, a generate
callback, or a backend whose clone gives each one a copy of
its own (released with the sub-agent, which SendMessage may keep):
oklm agent --url gives each its own connection to the
server, with the same key, model and retries. A backend without
clone serves one turn at a time; with it, Agent runs in the
foreground and a background call is refused.
on_event receives what happens as it happens: TEXT and
REASONING pieces (the <think> blocks told apart,
whichever the model or backend), each TOOL_CALL before it runs and its
TOOL_RESULT, the USAGE of each model turn (prompt tokens, those already
in the cache, tokens written), and its STOP: end of turn, tool use, the
token limit, cancelled, or an error. on_text and
on_tool still receive the raw text (with its
<think> tags) and the results. Reasoning is not sent
back to a backend (a server may leave it in the content); the in-process
model's templates drop it themselves.
oklm_agent_cancel(), from any thread or a signal
handler, interrupts the work: the model's turn stops within a token (its
text so far is kept in the history), a running command is killed (its
process group, as at a timeout; its output so far ends with
[interrupted by the user]), the calls not run yet get
"Interrupted by the user" as their result, so every call has one, a
foreground sub-agent stops too, and oklm_agent_send()
returns with [interrupted]. The next message starts afresh.
On the terminal, Esc or Ctrl-C while the agent works interrupts it (a
second Ctrl-C before it has stopped ends the program); at the prompt
Ctrl-C clears the line, and twice on an empty line leaves.
Headless use
oklm agent -p TASK answers and exits. What a script
pipes in comes first
(cat log.txt | oklm agent -p "explain the error";
-p - takes standard input alone); a pipe that stays silent
for two seconds is taken as empty, so a caller that leaves it open does
not hang the agent.
--output-format text(the default) prints the answer as on a terminal.jsonprints one object at the end;stream-jsonprints an object per line as the work goes (oklm_headless_*inheadless.h):{"type":"system","subtype":"init","cwd":...,"model":...,"permissionMode":"default","tools":["Read",...],"session_id":...} {"type":"assistant","message":{"id":"msg_1","role":"assistant","content":[{"type":"thinking","thinking":...},{"type":"text","text":...}],"stop_reason":"tool_use"},"session_id":...} {"type":"assistant","message":{"id":"msg_1",...,"content":[{"type":"tool_use","id":"call_0","name":"Read","input":{"file_path":"notes.txt"}}]},...} {"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"call_0","content":...,"is_error":false}]},...} {"type":"result","subtype":"success","is_error":false,"result":"plum","num_turns":2,"duration_ms":5210,"usage":{"input_tokens":...,"cache_read_input_tokens":...,"output_tokens":...},"session_id":...}The result's
subtypeissuccess,error_max_turnsorerror_during_execution; itsresultis the answer without the reasoning. With a JSON output nothing else reaches standard output (the agent's own lines go to standard error) and nobody is asked: a call the rules do not allow is refused, so a script allows what it needs.--input-format stream-json(with--output-format stream-json) reads user messages from standard input, a JSON line each ({"type":"user","message":{"role":"user","content":"..."}}, the content a string or text parts), and writes a result after each: one conversation over several requests.Prompt commands work (
-p "/review src/x.c"); built-in commands answer with an error in the JSON formats (their output is for a terminal).--save-training FILEappends the run's whole conversation, rendered through the model's chat template with no trailing generation prompt, toFILEas one JSONL line{"model", "text"}: the exact in-process shape a turn is trained on (oklm_agent_training_text()). It needs an in-process model (a backend has no template). Running tool-use tasks this way captures fine-tuning data;tools/make_mcp_data.shdoes it against the tool zoo (tools/mcp_zoo.py) and keeps the successful runs (mcp-training-design.md).--no-tool-grammarleaves the model's tool calls unconstrained (they are constrained by the tools' grammar from the call's opening marker on otherwise;oklm_agent_params.free_tool_calls): how well a model writes its calls unaided, whichtools/mcp_loop.shmeasures.The exit status is 0 when the request (each request) succeeded, 1 when one ended in an error, 2 for a usage error. A message that was not answered because the context had no room for it is an error, in the text output as in the JSON ones (agent-sessions.md); an answer cut by
--max-tokensis not.The spec's flag names are read as OKLM's:
--model(-m),--permission-mode(--mode),--system-prompt(--system),--append-system-prompt,--add-dir DIR,--resume [ID],--verbose(accepted),--mcp-config FILEand--strict-mcp-config(agent-extensions.md).--allowedTools RULESand--disallowedTools RULES(also spelled--allowed-toolsand--disallowed-tools) give the session a list of permission rules in one argument: each rule of--allowedToolsis added as an allow rule, each of--disallowedToolsas a deny rule, with the command line as their source, exactly as one--allow RULEor--deny RULEa rule would. The rules are separated by commas or blanks, those inside parentheses belonging to the rule:--allowedTools "Read Bash(git log:*)"is two rules,--disallowedTools Edit,MultiEdittwo others. Both options may be repeated, 64 rules at most over the command line; a rule that is notToolorTool(pattern)stops the command (malformed rule). A tool--disallowedToolsnames without a pattern is left out of what the model is offered (agent-permissions.md); an allow rule does not restrict the session to the tools it names, it only lets their calls run without a question.
tools/check_headless.sh scripts it as a CI job would,
without a model (a fixture replays the turns): the JSON result read with
sed, the stream-json lines, piped input reaching the model,
and the three exit statuses.
An agent other programs attach to
oklm agent -m MODEL --listen [PORT] runs the agent
without a terminal and serves it over HTTP on loopback: its events as
they come, and from the other side the user's messages and the answers
to its permission questions. oklm-gui attaches to it (gui.md); any program that speaks HTTP
can.
oklm agent -m qwen3-8b --listen # the first free port from 7481
agent: listening on http://127.0.0.1:7481 (loopback only); the token is in /run/user/1000/oklm/agents/4242.json
Found by its file. At start the agent writes
$XDG_RUNTIME_DIR/oklm/agents/PID.json(else~/.cache/oklm/agents;%LOCALAPPDATA%\oklm\agentson Windows), readable by the user only: the port, a 256-bit token made for this run, the project (as an absolute path), the session and the model. The file is removed when the agent ends; one left by an agent that was killed names a process that is gone, which a client checks. The file also says when its process started (start_ticks): a pid that another process got since is not taken for the agent, and the file is removed.Loopback and a token. Only
127.0.0.1is listened on, and every request carriesAuthorization: Bearer TOKEN, compared in constant time. Whoever can read the file can drive the agent, with the agent's permissions: that is the user, and nobody else.Routes. The listener answers five routes and no other (404), each with the token; they are not routes of
oklm serve, whose own are in server.md:Method Path What it does GET/agent/statewhat the agent is and does now: its pid, project, session and model, whether it works, the last event's number, the question waiting, the messages queued, the context and what is used of it GET/agent/eventsthe events after number after, waiting up towaitmilliseconds for one (?after=N&wait=MS)POST/agent/messagea user's message queued, {"text": "..."}POST/agent/permissionthe answer to a permission question, {"id": N, "answer": "yes"}POST/agent/interruptthe work in progress cancelled Events. The agent's stream-json lines (the same as
--output-format stream-jsonabove), each numbered from 1, with three more types: theusermessage a client sent,permission_request(id,tool,subject,reason,always: the rule "always" would add,preview: the diff of an edit) andpermission_result.GET /agent/events?after=N&wait=MSanswers{"events": [{"seq": n, "event": {...}}], "last_seq": L, "busy": bool}, waiting up to MS (25 seconds at most) when nothing is new: a client asks again with the last number it has. The last 8192 events are kept, so a client attaching late reads what happened.Messages.
POST /agent/message {"text": "..."}queues a message (64 at most); the agent takes them in order, as if typed.Permission questions. A tool call that would ask at a terminal raises a
permission_requestand waits two minutes forPOST /agent/permission {"id": N, "answer": "yes" | "no" | "always", "feedback": "..."}; without an answer the call is refused. Any attached client may answer; the first answer counts.The rest.
GET /agent/state(pid, project, session, model, busy, the last event's number, the question waiting, the messages queued, andcontextwithcontext_used: the model's window and what the agent's instructions, tools and conversation take of it, told once the model is loaded and again after each request, 0 before; the chat window reads them to search a document too long for what is left rather than send it whole: gui.md);POST /agent/interruptcancels the work in progress.
It is the headless agent with another way in and out: the same loop,
tools, permission rules, hooks and session file, which
--resume can pick up later. On Windows it is the same
listener: loopback only, the token from the system's random generator,
and the file kept for the user by an access list of one entry (full
control for that user, nothing inherited) on the directory and on the
file, where POSIX has the modes 0700 and 0600; the default list of
%LOCALAPPDATA% already keeps other accounts out, the system
and the administrators excepted, as root is on POSIX. Run on 2026-11-13
with the MinGW build under Wine 11, not on Windows itself: the test
below, and oklm.exe agent --listen asked for its state
without the token and with a wrong one (401), with it (200), sent a
message and read its events back; Wine keeps the file as mode 0770, the
user's Unix group included. A listening agent ends when it is asked to,
on every system: oklm agent --stop-listening PORT finds it
by its port among the user's listeners and asks it (SIGTERM; on Windows
the event of the registry of processes, server.md), as Ctrl-C in its
console or a signal does: the turn in progress is cancelled, the
listener is closed and its file removed, the session is kept
(agent: asked to stop: the listener is closed, the session is kept,
the status 0); one that has not ended after ten seconds is ended by the
system. The listener is oklm_listen_*
(include/oklm/harness/listen.h;
oklm_listen_list() gives the agents listening now, from
their files, which is what oklm-gui reads), tested by
tests/harness/test_listen.c (the file, the list, the token,
the events and their wait, the stream, the queue, a question answered,
refused and timed out, the interrupt) and by hand with a client that
sent a message, answered the Write tool's question and read the
result.
Library use
The agent is a library (liboklm, the headers under
oklm/harness/): the program creates a tool context and an
agent, sends messages, and receives the events. Besides
on_event and oklm_agent_cancel() above:
- Its own tools:
oklm_tool_ctx_add_tool(ctx, definition, fn, user)adds a tool from its JSON definition (name,description,parameters); agents of that context offer it after the built-in ones, check a call's arguments against its schema, decide it by the rules by its name (--allow Weather, or the mode: asked in the default mode), run the PreToolUse and PostToolUse hooks and audit it like any other;fnreturns the text for the model. A name already taken is refused. - Hooks as functions:
oklm_hooks_add_callback(hooks, event, matcher, fn, user); the function gets the event's JSON and answers as a command would (0, 2 to block with a reason, a JSON answer). - The permission question: the tool context's
askcallback (oklm_ask_fn) is asked whenever the rules and the mode say ask, with the tool, its arguments, a preview of an edit; it answers yes, no, always (with the rules to add) or says what to do instead. - The output formats:
oklm_headless_new()andoklm_headless_on_event()turn the events into the stream-json lines above for anyFILE*.
oklm_tool_ctx tc = {0};
tc.root = ".";
oklm_perms_new(".", &tc.perms);
oklm_perms_add_rule(tc.perms, OKLM_SRC_CLI, OKLM_PERM_ALLOW, "Weather");
oklm_tool_ctx_add_tool(&tc, "{\"name\":\"Weather\",\"description\":\"The weather of a city\","
"\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},"
"\"required\":[\"city\"]}}", weather_fn, NULL);
oklm_agent_params p = oklm_agent_default_params();
p.backend = oklm_agent_http_backend(http); /* or a model and context */
p.on_event = my_events;
oklm_agent_new(NULL, NULL, &tc, &p, &agent);
oklm_agent_send(agent, "What is the weather in Paris?", &reply);On a published model
Qwen3-4B (Q4_K_M, tests/public_models.txt), thinking on,
a scratch project holding a 6-line hello.c:
$ oklm agent -m models/qwen3-4b/Qwen3-4B-Q4_K_M.gguf --project build/agent_demo \
--mode acceptEdits --allow "Bash(wc:*)" \
-p "How many lines does hello.c have? Use wc. Then change the greeting printed by hello.c from hello to bonjour."
[Bash {"command":"wc -l hello.c"}] ok:
| 6 hello.c
[Edit {"file_path":"hello.c","old_string":"hello","new_string":"bonjour","replace_all":false}] ok:
| Replaced 1 occurrence
The file hello.c has 6 lines. The greeting in hello.c has been changed from "hello" to "bonjour".
$ oklm audit build/agent_demo/.oklm/audit/<session>.jsonl
2026-11-01T10:59:17Z allow rule Bash {"command":"wc -l hello.c"} Bash(wc:*)
2026-11-01T10:59:17Z allow mode Edit {"file_path":"hello.c",...} acceptEdits
The model writes both calls in one turn after thinking; both run, the first by the command-line rule, the second by the mode.
A task of several steps (add a function to a C file, use it, compile
and run it), with
--allow "Bash(cc:*)" --allow "Bash(./calc)":
[Read {"file_path":"calc.c","offset":1,"limit":20}] ok
[Edit {"file_path":"calc.c","old_string":"static int add(...) { return a + b; }\n\nint main(void) {",
"new_string":"...\nint sub(int a, int b) { return a - b; }\n\nint main(void) {\n\tprintf(\"%d\\n\", sub(7, 4));"}] ok
[Bash {"command":"cc -o calc calc.c && ./calc"}] ok:
| 3
| 5
The function `sub(a, b)` has been added to `calc.c`, and `main` now prints `sub(7, 4)`. Compilation and
execution confirm the output is `3` (7 - 4), as expected.
A plan ticked:
--mode acceptEdits --allow "Bash(make:*)" --allow "Bash(./app)" -p "Do these three steps and keep a task list with TodoWrite, ticking each step when done: 1) change the version printed by app.c from v1 to v2, 2) build it with make, 3) run ./app and check it prints v2."
(commands sandboxed):
[TodoWrite ...] Task list (0 of 3 done)
[Edit {"file_path":"app.c","old_string":"v1","new_string":"v2"}] Replaced 1 occurrence
[TodoWrite ...] Task list (1 of 3 done)
[Bash {"command":"make"}] cc -o app app.c [exit code 0]
[TodoWrite ...] Task list (2 of 3 done)
[Bash {"command":"./app"}] v2 [exit code 0]
[TodoWrite ...] Task list (3 of 3 done)
The version in app.c was successfully updated from v1 to v2, the build completed without errors,
and running ./app confirmed it prints v2 as expected.
A program made to pass a project's test, in a project holding only a
Makefile (test builds fizzbuzz and diffs
./fizzbuzz 15 against expected.txt), with
--mode acceptEdits --allow "Bash(make:*)" and the project
under /tmp (commands sandboxed), in 22 seconds:
$ oklm agent ... -p "Create fizzbuzz.c so that make test passes: the program takes N as its first argument and
prints 1 to N, one per line, Fizz for multiples of 3, Buzz for multiples of 5, FizzBuzz for both. Check with make test."
[Write {"file_path":"fizzbuzz.c","content":"#include <stdio.h>\n#include <stdlib.h>\n..."}] ok:
| Wrote 491 bytes to .../fizzbuzz.c
[Bash {"command":"make test"}] ok:
| cc -Wall -o fizzbuzz fizzbuzz.c
| ./fizzbuzz 15 > out.txt
| diff -u expected.txt out.txt
| [exit code 0]
The file `fizzbuzz.c` has been created and the test passed successfully.
Worded "write fizzbuzz.c ..., then run make test and fix the program
until it passes", the model first wrote a program without
<stdlib.h>, read the compiler's error from
make test, added the include with Edit and ran the test
again, which passed. (With the earlier prompt, which did not ask to
check changes with the tests, it wrote the file and stopped, reasoning
that it could not run commands.)
Skills end to end
(tools/check_skills_e2e.sh [model] [options]): a project
holds a skill the model cannot know, "the zorblax format", whose script
writes it; asked to save a text in that format, Qwen3-4B (Q4_K_M,
OpenCL) loads the skill by itself and runs the script, which writes the
file, in about 25 s. Before the Skill result said that loading did
nothing yet, the same model loaded the skill and then announced the file
written without running anything. Qwen2.5-1.5B writes the file by hand
without loading the skill.
A skill:
oklm agent ... --allow "Bash(cc:*)" --allow "Bash(grep:*)" -p "Check calc.c with the right skill."
loads c-check, runs its command (first piped into
grep -n warning, which exits 1 when nothing matches, then
alone) and answers that the file is clean. Without the grep rule the
piped command asks, is refused (no terminal), and the model still claims
the file is clean: a small model's answer is only as good as the results
it actually got, which the audit log shows.
The same task in plan mode shows the 4B model's limits: it reads the
file, writes a task list, tries to compile (denied: plan mode), then
writes "ExitPlanMode" as text instead of calling it. Smaller models
(Qwen2.5-1.5B, Qwen3-0.6B) make the first call and then end their turn
announcing the second without writing it (the raw turns, with
OKLM_LOG_LEVEL=DEBUG, show them ending on
<|im_end|>).
Agent evals
make eval (or
sh tools/eval_agent.sh [model.gguf] [--only NAME,...] [oklm agent options])
runs the tasks of tests/evals/ with a real model and judges
each by what it left behind, not by what it says. A task is a directory:
prompt (the request; @DIR@ and
@PORT@ are replaced), setup.sh (makes its
files in an empty directory, a repository for the git ones),
check.sh (exit 0: passed), options (its
permission mode and allow rules; @FAKE@ is the fake MCP
server) and an optional teardown.sh. Each runs headless
(-p, JSON output, nobody to approve), outside OKLM's own
repository, under a 900 s limit. build/eval/report.md and
report.json give each task's verdict, turns, tokens and
seconds, and the pass rate; the session, audit log and checker output
stay in build/eval/NAME/. Run it after a change to the
prompt or the tools.
The seventeen tasks: fix a failing test, answer from a fetched page,
call an MCP tool, follow a project's skill, /commit only
the change, add a CLI flag, rename a function across files, find and
explain a bug without fixing it, write a README from the code, start a
dev server and wait for it, commit on a new branch in the history's
style, find the commit that added a line; resolve a merge conflict,
rebase onto main, say why a function was added (the git evals, 05 and 11
to 15); fix an error a program logs on startup, run the e2e tests and
fix the failing one (the process evals, 10, 16 and 17). Tasks 18 to 25
need the project's documentation (a setting, a command, a constant, a
reason, an exit status: 18 to 21, 24 and 25) or an earlier session (22
and 23); they measure the agent's retrieval (rag.md). Task 26 judges behaviour
beside the outcome: two sub-agents started at once fix a bug each and
add a line each to one CHANGELOG.md; it passes when both bugs are fixed,
both lines are there and the rest of the file is intact, nothing else
changed, no command threw work away (git checkout --,
reset --hard, stash, clean,
rm), the two Agent calls came in one reply, and the report
names both fixed files and no file that does not exist. Qwen3-8B
(2026-11-11), three runs: one pass; in the two others the second
sub-agent explained dist.py's bug without fixing it and the final report
still said both were fixed (once the first also rewrote the changelog
without its heading). No command threw work away. With each report
ending on the files the sub-agent wrote, four more runs: in one, the
second sub-agent replaced the first one's changelog line with its own
(an Edit, after a fresh Read) and the final report said both had added
theirs; with lines a sibling wrote kept from the others, three more runs
kept both lines every time (the hold not needed there), each failing on
a fix not made instead (temp.py still subtracting 32, dist.py
unchanged). Two of those failures were the task's own: a fix by the
factor 0.621371 is off by 3e-7, and the test asked 1e-9 (now 1e-5). With
the last command in each report line, a second ask for a sub-agent that
wrote nothing, and Write taking Read's line numbers pasted back (a
sub-agent had written its changelog as " 1 ## Unreleased", refused each
time for dropping its sibling's line): 4 of 6 runs pass; the two others
fail on the caller's own report (it said a sub-agent had added a
changelog line that sub-agent said it could not add, and once answered
with a plan instead of the report).
Qwen3-8B, the default model (Q4_K_M, OpenCL, 2026-11-07): 12 of 17, then 13 of 17 once the agent's own state was kept out of the repository's status, about 4 turns and 38 s a task:
| Task | Verdict | What the model did | Change |
|---|---|---|---|
| 14 rebase onto main | fail, then pass | saw .oklm/ untracked in the git snapshot, called the
tree dirty and stopped without rebasing |
.oklm/audit, jobs, sessions,
worktrees each hold a .gitignore of
* (oklm_git_ignore_dir()) |
| 08 explain a bug | fail | named the wrong line | none: the model's |
| 09 README | fail | guessed the options without reading wordcount.c |
none: the model's |
| 15 why added | fail | gave the commit's subject (read the port from PORT) rather than the reason in its body (the platform assigns the port) | none: the model's |
| 16 startup error | fail | blamed the file's opening and wrapped it in try with
broken indentation; the cause is the setting's name
(listen_port against port) |
none: the model's |
Every git and process task but 15 and 16 passes: 05, 10, 11, 12, 13, 14, 17.
GLM-4.7-Flash (Q4_K_M, oklm serve on OpenCL with a 16K
context, the agent through --url, 2026-11-11): 21 of 25, 5
turns a task on average, 929 s and 724K prompt tokens in all; its 163
calls in the glm45 format all parsed (none left as text in
an answer, none refused for its arguments, in the 25 transcripts). It
passed every documentation task without retrieval (reading the docs with
Glob and Read), and failed:
| Task | What the model did |
|---|---|
| 09 README | copied the working directory as 09-write-readereadme in
every absolute path; the Write outside the project was refused and it
never saw the typo |
| 16 startup error | replaced the lookups by .get() with a combined message
instead of naming the missing setting |
| 22 session branch | asked for the branch name instead of looking at the earlier sessions |
| 23 session changelog | wrote an entry in its own style, the convention of the earlier session unread |
All four are the model's.
GLM-4-9B-0414 (Q4_K_M, the same setup, 2026-11-11, its calls in the
glm4 format): 3 of 25 (02, 03 and 04), 2 turns a task on
average. Its 26 calls all parsed; it ends most tasks after its first
turn on a plan written in prose ("First, I'll check the error log..."),
once with the calls written inside a code fence; the format reads every
call it opens a turn with. Reading the calls written inside a turn of
text too (a line naming a tool, then its JSON) changes little: 3 of 25
again, 31 calls against 26, task 01 running its fenced build command and
stopping after.
The review commands' seeded-bug measure
(tools/eval_review.sh, six bugs,
/code-review medium) finds 6 of 6 (2026-11-04) and 5 of 6
(2026-11-06) on Qwen3-8B, above the 80 % asked; Qwen3-4B found 2, then
0.
The earlier twelve tasks on Qwen3-4B (Q4_K_M, OpenCL, 2026-11-03): 7 of 12, about 3 turns and 53 s a task. The failures it showed, and what came of them:
| Task | What the model did | Change |
|---|---|---|
| 09 README | read README.md (missing) and wrote a generic one |
a Read of a missing file now lists its directory (hidden names and
those a Read deny rule covers left out); the model then names the
program and make, but still guesses its options without
reading wordcount.c |
| 11 branch and commit | three tasks in_progress refused the whole TodoWrite,
and it stopped after the other calls of the turn |
TodoWrite keeps the first in progress and makes the others pending, with a note; passes |
| earlier runs | piped or redirected an allowed command, refused with nobody to ask | the refusal names the parts no rule covers and says to run the allowed command alone |
| 08 explain a bug | named line 17 for the bug on line 16, after reading numbered lines | none: the model's |
| 10 dev server | put an invented detach on its curl and
wrote $(curl ...) as the file's text in the same turn |
none: the model's (it passes on other runs) |
| 12 git history | took the latest commit from git log for the one that
added the line |
none: the model's |
Edit against a whole-file Write (Qwen3-4B Q4_K_M, OpenCL, 2026-11-08,
tasks 01, 06 and 07, the ones that change an existing file; Edit and
MultiEdit refused with --disallowedTools Edit,MultiEdit, so
only Write can change a file). Three runs of each setting gave the same
verdicts every time here (the sampling is greedy), so it is one sample
of three tasks per setting, not nine:
| Setting | 01 fix a bug | 06 add a flag | 07 rename | Written tokens (01, 06, 07) |
|---|---|---|---|---|
| Edit allowed | pass, 38 s | fail | fail | 1870, 1564, 4935 |
| Write only | fail, 172 s | pass | fail | 9243, 1914, 4697 |
1 of 3 each, so the pass rate says nothing at this size. The cost differs on one task only: on 01 (a file of a few lines; the written tokens include the model's reasoning) Write only wrote 9243 tokens against 1870 and took 172 s against 38 s, and its rewrite failed the check; on 06 the two cost about the same (1914 against 1564) and 07 differs little. Inconclusive: Edit stays the default, and a file much longer than these would be the fair test of the whole-file rewrite.
Runs on OpenCL are not repeatable token for token, and the set that
passes moves between runs: three full runs on the same day gave 7 of 12
each time, with 07 (a file left out of the rename) and 11 (the agent's
own .oklm/ staged instead of the file) passing in some and
failing in others. Two changes were tried and left out because they
measured worse: a sentence in the system prompt asking to read code
before writing about it (09 then read the Makefile, but 06
and 07 failed in every run), and a refused unread Edit showing the file
and counting it as read (07 re-read the files and stopped without
editing, 4 runs of 4).