Phase 3 — Backbone Expert RAM Offload (M0)

Stream MoE expert FFNs from CPU RAM instead of VRAM, and a single 64 GB coordinator holds a 122B — with no graph surgery.

Expert FFNs don't have to live in VRAM. Streaming them from CPU RAM lets one coordinator hold a model whose experts exceed its VRAM — the foundation that lets weak nodes join a large MoE at all.

Planner verified · real 122B GGUF

A 122B fits a single 64 GB coordinator

Previously the ring placed weights VRAM-only, so the 122B (77.6 GB) was infeasible on a 64 GB GCD. With expert-offload rules the dry-run comes back feasible:

feasible
122B ring plan
62.6
VRAM GiB (≤ 64)
14.2
RAM GiB (experts)
10
offloaded layers
A coordinator siphoning expert tiles from VRAM into a RAM reservoir, stamped feasible

Planner output — inference engine -ot rule format.

node 0  layers [0,48]  vram=62.6  ram=14.2  ot_rules=10
sample: blk\.38\.ffn_(up|down|gate)_(ch|)exps=CPU   # inference engine -ot format
What was wired · pure Python, no C++ rebuild

Carrying the planner's offload rules into a real load

  • planner — already emits ot (comma-joined -ot rules) in each placement.
  • protocol.py — added the StageStartRequest.ot field.
  • runtime.py — forwards the placement's ot into the stage request.
  • stage_service.py — the coordinator launches with --override-tensor.
  • linkcpp-server forwards unknown args to stock llama-server, so -ot applies untouched.
# stage_service.py — coordinator branch
if request.ot:
    command += ["--override-tensor", request.ot]

Verified with an ot protocol round-trip test plus confirmation that the coordinator command emits --override-tensor; the hub redeployed cleanly with no regressions. Remaining at the time: the full 2-node 77 GB load (coordinator with offload + a phone holding a ~1.5 GB one-layer window), gated on server availability. The core of M0 — the backbone offload that lets weak nodes participate in a big MoE — was complete at the code and planner level.