In-flight ring

A pipeline that never drains: several requests occupy different stages at the same time, so no stage waits for the one in front to finish.

Kvasir serves a model as a pipeline of stages, each holding a contiguous slice of its layers. A stage runs its layers and passes the boundary — a hidden state, not weights — to the next. No stage holds the whole model, and nothing sits in the middle of the data path: the bridge submits to the head and reads from the tail, while the stages hand results to each other through their own agents.

If a pipeline finishes one request before admitting the next, every stage but one is idle at any moment — a four-stage ring runs at a quarter of its hardware. The in-flight design keeps several requests moving at once: while stage 3 decodes one request, stage 0 is already prefilling another. Stages report how long they held a batch and how long they had nothing to submit, so a ring that is starved looks different from a ring that is saturated.

           stage 0        stage 1        stage 2        stage 3
           layers 0-11    12-22          23-33          34-44

request A                                              decode
request B                 decode
request C  prefill

boundaries pass →  agent to agent, never through the caller

Rows from different requests are packed into one physical batch, and that exact membership is forwarded to every downstream stage rather than re-decided per hop. It is what lets a prefill and several decodes share a single pass, and it is why the size of a batch is a property of the load: the plan states the row and micro-batch widths up front, and those widths set the largest result a stage can ever return.

The ring is the latency path, and its granularity is a layer. Expert sharding removes that floor by cutting inside a layer, and plugs into the same serving fabric.