Expert sharding

Splitting a MoE at the expert grain, so a phone carries 42–340 MB of experts instead of a 1.4 GB layer.

Expert sharding drops the swarm's carrying unit from a layer (~1.4 GB on the 122B) to an expert (5.3 MB). A weak device downloads a slice of 8–64 experts (42–340 MB), loads it as a pure-function worker — no attention, no KV, no sampler — and computes its experts whenever the backbone's router selects them.

Two roles

Backbone × worker

The cut point inside one MoE layer (router runs once, on the backbone).

cur   = ffn_norm(x)                     # backbone
ids,p = top_k(softmax(cur @ router), 8) # backbone — authoritative
send  (cur rows, local_ids) → worker    # ~6 KB per decode step
recv  expert_out            ← worker    # worker: 3 mat-muls
x = x + combine(p, partials) + shared(cur)   # backbone — exact
  • The backbone keeps the dense path (attention, norms, router, shared expert, combine) and holds all experts as a RAM-offloaded fallback replica for churn tolerance.
  • Workers (linkcpp-expert-worker --serve) answer (n_used, n_tokens, cur, sel) → experts over one long-lived TCP stream — the same stream the 443 relay tunnels for phones.
  • Coverage self-heals through the expert coverage market: POST /api/expert-coverage heartbeats holdings, GET /api/expert-demand aggregates scarcity, POST /api/expert-volunteer assigns the scarcest range clipped to the node's budget.
Measured, not promised

Verified on real hardware

  • Sharded compute == monolithic to max|Δ| = 3.6e-12 (an exact regrouping, not an approximation).
  • Cross-process dispatch on a live 122B decode: argmax MATCH, logit cosine 0.99869 — byte-identical to in-process.
  • A Galaxy S25 autonomously downloaded its 1.58 GB slice and computed layer-0 experts every token: 8/8 tokens identical to the local run.
  • Batched dispatch reaches 53k tok/s per worker at batch 512 (ROCm) — the throughput-fabric property that makes the swarm practical.