OKLMdocumentation v1.0.3
On this page

ADR-0016: CPU worker team

Records how the CPU backend runs parallel loops: on a team of threads kept alive between operations, added to nilorea-library as N_PARALLEL_TEAM.

Status: accepted (2026-10-29).

Context

Decoding one token runs a few hundred matrix-vector products and many small row-wise ops, each taking tens to hundreds of microseconds. nilorea's n_parallel_for() starts and joins its threads on every call (tens of microseconds per thread), and its THREAD_POOL waits for completion by polling every millisecond; either would cost more than the work.

Decision

  • nilorea-library gains N_PARALLEL_TEAM (n_parallel_team_new(), n_parallel_team_run(), n_parallel_team_size(), n_parallel_team_free()): threads created once, woken per loop through a condition variable, with the caller doing the first slice.
  • The slicing depends only on the range, the grain and the team size, and slice k always runs on participant k, so per-slice reductions are reproducible.
  • A loop started from inside a loop of the same team runs inline (detected with a thread-local marker), so nested parallel code cannot deadlock. Concurrent callers are serialized by a run lock.
  • The CPU backend owns one team, sized by OKLM_THREADS, changeable with oklm_cpu_set_threads() between operations.
  • OKLM kernels never depend on the slicing for their results: each output element is computed by one thread in a fixed order, so results are the same for any thread count.

Amendment (2026-10-30)

  • Team threads poll the loop generation for a short while (N_PARALLEL_TEAM_SPIN polls with a CPU pause hint) before sleeping on the condition variable, and the caller polls the pending count before waiting: decoding issues loops a few microseconds apart, which then start without a kernel wake-up (SmolLM2-135M Q4_K_M generation 163 -> 198 t/s on 8 threads).
  • The default thread count is the number of physical cores (get_nb_cpu_physical_cores(), added to nilorea): on the i7-11700K (8 cores, 16 hardware threads) every corpus model decodes faster on 8 threads than on 16. OKLM_THREADS still overrides it.
  • Matrix products with at most four activation rows cut the weight rows into one even share per thread instead of 16-row tiles.

Amendment (2026-11-08)

  • The CPU backend has its own team (src/core/cpu/team_cpu.c) instead of N_PARALLEL_TEAM. On a machine shared with other processes, or with more threads than cores, decoding collapsed (Granite-1B-A400M: 72 t/s at 8 threads, 2 to 11 at 16; 2 t/s at 8 threads beside 8 spinning processes): every loop waited for each participant to run its fixed slice and check in, so whenever the kernel had one of them off its core, the loop waited a scheduler time slice, a few hundred times per token.
  • A loop is cut into contiguous chunks, at most four per participant, bounds fixed by the range, the grain and the team size. The caller and the workers claim them from one atomic word (loop generation, chunk count, next chunk) by compare-and-swap; the loop ends when the last chunk is done, and a worker that wakes late finds nothing left. Which thread runs a chunk depends on timing, so the guarantee that slice k runs on participant k is gone: no OKLM kernel used it (each output element is computed by the same code whatever chunk holds it, which the thread-count independence tests already required).
  • Matrix-vector products cut their weight rows into four tiles per thread instead of one share each, so a preempted thread's share is taken over.
  • Waiting threads poll 300 times, then alternate sched_yield() and 32 polls 512 times, then sleep on a condition variable; the 20000-poll spin took the core from the threads holding work on a loaded machine.
  • Numbers in benchmarks.md.

Consequences

Per-op overhead is one condition-variable broadcast and one wait. Ops issued concurrently from several application threads run one after the other on the team, which keeps the machine from being oversubscribed. The team is covered by OKLM's test_threads (a stalled participant, more threads than cores beside spinning threads, concurrent callers; ASan and TSan).