Phase 7 — Batched Dispatch Throughput (M4)
The swarm is a throughput fabric, not a latency play: batching dispatch calls amortizes per-request overhead 77× per token.
Measured · ROCm, expert FFN, n_used = 8
Bigger batches, more tok/s per worker
| batch | tok/s per worker |
|---|---|
| 1 | 688 |
| 16 | 4,255 |
| 64 | 8,130 |
| 256 | 30,666 |
| 512 | 53,067 |
From 1.45 ms/tok at batch 1 to 0.019 ms/tok at batch 512 — a 77× per-token improvement. The per-call time barely moves (1.45 → 9.6 ms) while the batch grows 512× — the GPU processes the batch nearly for free behind a fixed overhead. This is the throughput-fabric property that makes expert-parallel practical: batched dispatch amortizes per-request RTT and overhead.
Done
Batched dispatch throughput
- Worker
--bench: compute_dispatch timings for batch 1…512 → tok/s. - 53k tok/s per worker at batch 512 (ROCm) — batching amortizes the overhead.
- Hot-expert caching and multi-worker aggregate scaling (replica routing) stack on top of this.
With M4, the whole M0 → M4 pipeline is demonstrated on a real 122B: backbone offload · expert slices · verified workers · live decode dispatch (argmax MATCH) · distributed processes · coverage market · batched throughput.