Mixture of Experts (MoE)

A model whose FFNs are hundreds of independent experts, only a few of which fire per token.

A Mixture-of-Experts model replaces each layer's single FFN with a bank of independent expert FFNs plus a router that picks a few per token. Qwen3.5-122B-A10B is the network's flagship example:

Each layer splits into a dense path — attention + KV, norms, the router (ffn_gate_inp), a shared expert — and an expert bank stored as three stacked tensors (ffn_up_exps, ffn_gate_exps, ffn_down_exps). The dense path is the minority of the bytes; the expert bank is 86% of the model.

• The expert index is the outermost GGUF dimension, so each expert is a contiguous, quant-block-aligned slab — extraction is a byte-range copy, no dequantization.
• Per token only 8 of 256 experts fire per layer, so a layer's decode-time expert traffic is a handful of small matrix multiplies over one hidden vector (~6 KB of dispatch).
• Experts are mutually independent — ownership can be scattered across devices and re-balanced freely.

This is why MoE is the swarm's natural substrate: the weights come pre-packaged in device-sized, independently ownable units.