Mixture of Experts (MoE)

A model whose FFNs are hundreds of independent experts, only a few of which fire per token.

A Mixture-of-Experts model replaces each layer's single FFN with a bank of independent expert FFNs plus a router that picks a few per token. Step-3.7-Flash, the 198B MoE serving today, has 288 experts per layer with top-8 routing. Qwen3.5-122B-A10B is the worked example below, because it is the one whose numbers were measured end to end:

49
layers
256
experts / layer
8
active / token
12,544
experts total
5.3 MB
one expert (Q4)
86%
of weight in experts
3072
n_embd
77.6 GB
full checkpoint

Each layer splits into a dense path — attention + KV, norms, the router (ffn_gate_inp), a shared expert — and an expert bank stored as three stacked tensors (ffn_up_exps, ffn_gate_exps, ffn_down_exps). The dense path is the minority of the bytes; the expert bank is 86% of the model.

  • The expert index is the outermost GGUF dimension, so each expert is a contiguous, quant-block-aligned slab — extraction is a byte-range copy, no dequantization.
  • Per token only 8 of 256 experts fire per layer, so a layer's decode-time expert traffic is a handful of small matrix multiplies over one hidden vector (~6 KB of dispatch).
  • Experts are mutually independent — ownership can be scattered across devices and re-balanced freely.

This is why MoE is the swarm's natural substrate: the weights come pre-packaged in device-sized, independently ownable units.