The Cost Was the Directory

A four-stage ring served at 0.55 tokens a second and we blamed the network. It was a readdir, and it got slower every time we ran it.

Four stages across two hosts decoded at 0.55 tok/s. Two stages on one host decoded at 20.19. The difference was thirty-six fold and the obvious culprit was the thing we had added: a hop across the machine boundary. We wrote that down as the finding and started looking for where the network time was going.

It was the wrong finding, and the way it was wrong is the interesting part. Comparing those two configurations moves two variables — the number of stages and the number of hosts — and we attributed the whole gap to one of them. Every measurement after that was an attempt to explain a number the experiment could not isolate.

The fix for a confounded comparison is to stop confounding it. We put three stages on one host, then four, then four split across two agents on that same host — the slow shape with the machine boundary removed and nothing else changed.

Stage count alone degrades gracefully. Splitting the same four stages across two agents on one machine costs another 2.6×, with no network involved. So the host boundary was not the mechanism — but the number still did not reach 0.55, and while we were measuring it something else surfaced: the same ring got slower the more we ran it.

Five consecutive runs of the identical request on the identical ring. Per-token cost rose linearly with the number of files in the agent's outbound journal. A ring that had been up all day was at 1.5 seconds a token because it had been up all day — which is exactly what a two-host ring is, and why the host boundary looked guilty.

fn occupied_bytes(&self) -> io::Result<u64> {
    let mut occupied = 0u64;
    for item in fs::read_dir(&self.root)? {
        let path = item?.path();
        let metadata = fs::symlink_metadata(&path)?;
        ...

fn require_capacity(&self, additional: u64) -> io::Result<()> {
    let occupied = self.occupied_bytes()?;   // before every single record

Every journal record asks whether it fits, and the answer is computed by listing the directory and stat-ing every entry in it. The journal has no deletion path at all — no remove_file, no prune, no retirement — so the directory only grows, and each write pays for every record the agent has ever written. Agent-to-agent hops write records a single-agent ring never writes, which is why splitting across two agents decayed so much faster than one.

A correlation is not a cause, so we changed exactly one thing. We padded the journal with 20,000 zero-byte entries. They charge nothing against the byte budget — the capacity check sums file sizes, and these have none — but each one is a directory entry that must be listed and stat-ed.

It is the walk. Not the bytes, and not the fsync either: moving the journal to tmpfs, where fsync is free but readdir still is not, recovered about ten percent and left the slope intact.

The agent keeps an occupancy figure per journal root and charges each write against it, walking the directory at startup and again only when the estimate approaches half the cap — where the exact number is what decides whether a write is refused. The estimate only ever over-counts, so it can refuse a write early but never admit one that does not fit.

Flat, and higher than the old best. The same 20,000-entry padding now costs nothing: 28.45 → 28.80 → 29.83. At 19,761 journal entries the production ring holds 28 tok/s, where before the patch it would have been under one.

The same investigation turned up a second cost. All the arithmetic runs on the GPU, but each stage server starts a 48-thread ggml CPU pool, and libgomp spins those threads while idle. Two stages therefore pegged all 96 logical cores of the machine, and the agent's per-event work — which is what actually paces the ring — ran on what was left. The stage servers were burning about 790 CPU-seconds each per 100-token run. Setting OMP_WAIT_POLICY=PASSIVE took agent CPU per token from 116 ms to 8 ms.

One thing this does not fix: the journal still grows without bound. Reading it is now cheap; retiring settled records is the follow-up.