Moving the Ring onto p4

Seven contract changes between an engine and its caller. Each one failed differently, and only one of them looked like an error.

We merged a new release of the p4 engine and the ring stopped serving. Not with a crash — the loader reported success, the agents reported ready, and nothing happened. Working back from that silence took a day and turned up seven places where our caller and the engine had drifted apart. What makes them worth writing down is not the count. It is that six of the seven did not produce an error.

Our loader addressed the LOAD command to the node it wanted to create. But a node does not exist until LOAD creates it, and the broker's rule for an event naming an unknown node is to forward it outbound rather than reject it. The command left the agent looking for somewhere else to go, found nowhere, and was dropped. No log line, because from the broker's point of view nothing had gone wrong.

The fix was to address LOAD to the *agent*, wrapped in the engine's backend-neutral lifecycle envelope, with the adapter's own command as an opaque body. Obvious in hindsight; invisible from the outside.

A model is loaded under a load generation, and each node is registered with a node generation. We had been treating these as independent — a timestamp for one, 1 for the other — and everything worked. The ring loaded. It answered a request correctly. Then the head node died.

let Endpoint::Node { generation, .. } = &event.envelope.source;
if *generation != receipt.load_generation {
    return Err("release owner census generation differs from source");
}

The receipt that closes out a finished request carries the load generation, and the node that sends it carries its own. When they differ the node is stopped. So the shape of the bug is: load succeeds, first request succeeds, head dies, every session after that hangs part-loaded. It reads exactly like a crash under load and not at all like a mismatch. Our loader now refuses a plan whose two numbers disagree, before anything is loaded.

The adapter compares the largest result a stage may return against the figure the stage server reports when it comes up, for exact equality. We could not know that figure without loading the model — so we loaded with a guess, and the failure told us all four stages' real numbers at once:

step37-s0: profile=33554432, READY=34419218444
step37-s1: profile=33554432, READY=34419218444
step37-s2: profile=33554432, READY=34419218444
step37-s3: profile=33554432, READY=59136012

34 GB. The agent's retained stores are 256 MiB, so that ring could never have been admitted. Reading the derivation out of the stage server showed why: the bound scales with n_batch × n_ubatch, and we had inherited a batch width of 2048 from a configuration that predated this check. At 128 rows — the width the production layout uses — the bound is 138 MB and fits. We now derive it in the plan from the same formula rather than carrying a remembered constant, and the tail stage's predicted figure came out to the exact number another deployment had recorded, which is the kind of agreement worth having before spending twenty minutes on a load.

• Loopback addresses in a two-host ring. A stage dials the next stage's agent at the address that agent advertises. Advertise 127.0.0.1 and host A dials itself. The ring's recorded configuration had been loopback all along — it could never have worked across hosts.
• The journal is mandatory. A model will not load without the agent's operational journal. We turned it off while chasing a different error and made the symptom worse in a way that looked like progress.
• The device name is backend-specific. The reference plan builder targets CUDA and emits --device CUDA0. The HIP build names its devices ROCm0. That plan loads the entire model and *then* fails to find the device.
• The native server is part of the release. An agent built from a newer tree wants capabilities the installed stage server does not report. Also discovered after a full model load.

Every one of these was cheap to fix and expensive to find, and the pattern is consistent: the costly failures were the ones where a correct-looking system did nothing, or did something once. The guards we added are all of the same shape — refuse early, at the place where the mistake is still legible. The loader writes the load generation to disk *before* the first command leaves, because it is otherwise unrecoverable. It refuses a generation mismatch rather than discovering it after the first request. It derives the result bound instead of remembering it.