Making WAN Dispatch Cheap: A Grounded Roadmap

Remote expert dispatch works and is byte-identical — but a WAN decode is round-trip-bound. Here's the plan to cut the cost, grounded in production numbers from DeepSeek, Petals and others (roadmap, not shipped).

Our own measurement, published plainly: dispatch costs about 110 KB per token per layer — 12.3 KB out (dispatch) plus 98.3 KB back (combine). The 8× asymmetry is because each selected expert returns its full output *before* the weighted sum. Directly linked, that's a 1.2% throughput overhead; through a CDN relay, ~28%. Those are the facts. The rest of this post is how we intend to close the gap — and why bytes are the easy part.

The single most important published result here isn't ours — it's Petals': as RTT goes from <5 ms to 100 ms, decode drops from 1.24 to 0.57 steps/s, while a 10× bandwidth cut changes it by ~0. Latency dominates; bandwidth is slack. That reframes the whole problem: shaving bytes is headroom, but cutting round trips is the substance. Every item below is ranked by how much round-trip cost it removes.

• Return weighted partial sums, not raw expert outputs. By linearity the backbone's combine is exact either way, but the worker returns one summed vector instead of 8 — that's the ~8× combine reduction, and it's precisely what DeepSeek-V3 / DeepEP do in production.
• Parallel star, not serial chain across multiple workers: ΣRTT collapses to max RTT.
• F16 on the wire — we already accept a cross-backend cosine of ~0.998, so F16 transport is inside existing tolerance; blockwise INT8/FP8 later, after our own argmax/cosine gate clears it against Q4_K_M weights (Petals showed INT8 over the real internet with no quality loss).
• Together these target ~110 KB → 9–12 KB per token (~12×) — real, but remember it's the *headroom*, not the bottleneck.

• Speculative decoding turns many tokens into one round trip. At a measured 80 ms WAN, break-even is only ~1.15–1.2 accepted tokens/step — so even a weak n-gram guess wins (vanilla Jacobi can backfire; technique choice matters). Our dispatch protocol already carries n_tokens > 1, so no wire change is needed.
• Continuous batching at the gateway folds concurrent requests into one trip; slot-affinity prefix caching keeps a session on the same replicas.
• Latency hiding: the shared expert is an independent additive term, so the backbone computes it *locally* during the remote round trip (ScMoE reports 1.82× over PCIe, no retraining). Keep hot experts local, send only cold ones remote (EPLB replicates the ~32 hottest for a 2.54× decode speedup in production).

Path policy: publicly-routable peers take the direct path (the 1.2% route); the relay is for NAT-bound devices only. And when wide 200 Gb/s links arrive, the 110 KB serializes in ~4.4 µs — the bandwidth term vanishes even before the reductions above, and a 794 MB slice ships in ~32 ms. But RTT is physics; it doesn't shrink — so speculative decoding and overlap remain the real levers even at 200 G. Where the fat pipe genuinely matters is multi-backbone federation (several backbones sharing one expert pool) and bandwidth-bound work: long-prompt prefill and large-batch throughput.

Our dispatch hook sits on build_moe_ffn — a single function 43 MoE architectures share in the inference engine. Three invariants are model-independent: the MoE math (routed = Σ wᵢ·Eᵢ(x), linear), the shared code path, and GGUF's standard stacked expert tensors (outermost ne[2] → block-aligned slicing). So onboarding a new model isn't a redesign — it's one pass through a per-model argmax/cosine verification gate.

The industry is converging on fine-grained MoE — smaller experts, more of them, higher sparsity (DeepSeek, Qwen, Kimi, GLM, gpt-oss all moved this way). Every step in that direction makes the swarm's unit of participation smaller and the scarcity market's grain finer. The models above aren't a wish list; each already flows through the same dispatch hook we run in production — onboarding is a verification gate, not an engineering project.