Phase 2 — Inside a 122B MoE: Why the Weights Want to Be Sharded

A tensor-level analysis of Qwen3.5-122B: 86% of the bytes are 12,544 independent expert slabs, each one a clean byte-range copy away from standing alone.

Before designing anything, we took the 122B apart on disk. The question: if a swarm of weak devices is to carry this model, what is the natural unit of carrying? The answer fell out of the GGUF tensor layout itself.

Anatomy · Qwen3.5-122B-A10B (Q4_K_M)

What a MoE layer is actually made of

49
layers
256
experts / layer
8
active / token
12,544
experts total
5.3 MB
one expert (Q4)
86%
of weight in experts
3072
n_embd
77.6 GB
full checkpoint
Anatomical cutaway of a MoE model: slim dense spine beside a huge honeycomb of experts

Each layer splits into a dense path — attention + KV, the norms, the router (ffn_gate_inp), a shared expert — and an expert bank: 256 independent FFNs stored as three stacked tensors (ffn_up_exps, ffn_gate_exps, ffn_down_exps). The dense path is the minority of the bytes; the expert bank is 86% of the model.

The layout gift

Experts are contiguous, block-aligned slabs

  • The expert index is the outermost ggml dimension (ne[2]) of every expert tensor — expert *e* occupies one contiguous, quant-block-aligned slab of raw quantized bytes.
  • That makes per-expert extraction a byte-range copy: data[a:b], no dequantization, no re-packing — an expert-sliced mini-GGUF is cheap to produce and bit-faithful.
  • Per token only 8 of 256 experts fire per layer, chosen by the router — so at decode time a layer's expert traffic is a handful of small matrix multiplies over one hidden vector.
The implication

The carrying unit drops from 1.4 GB to 5.3 MB

At layer grain, the least a node can hold is ~1.4 GB — out of reach for most phones once the app, KV and OS take their share. At expert grain, the unit is 5.3 MB, and a realistic contribution is 8–64 experts (42–340 MB) — comfortably inside any modern device. The experts are mutually independent, so ownership can be scattered arbitrarily and re-balanced freely. This analysis is what made expert-level sharding the design bet: the weights were already packaged in swarm-sized units — the network just had to honor the packaging.