Hub WAN interconnect (200G optics)

How hubs link at 200 Gb/s across a room, a campus, or a city: which optic at which distance, what plugs where, and what it takes to actually hit line rate.

When two hubs both have public routes, the expert-dispatch data plane should be a direct link — the 443 relay is for NAT'd edges. This entry is the concrete recipe for making that direct link 200 Gb/s-class with catalog parts. One rule organizes everything: the fiber is speed-neutral glass; the speed lives in the pluggable at each end.

Step 1 · pick by distance

The reach ladder

distancepartplugs into
same rack, 0.5–3 mQSFP56 DAC (passive copper)NIC ↔ NIC, no switch
same room, ≤30 mQSFP56 AOC (active optical)NIC ↔ NIC / switch
campus, 2–10 km200G FR4 (2 km) / LR4 (10 km) module + duplex LC, single-mode fiberNIC or switch QSFP56 cage
metro, ≤40 km200G ER4 module, single-mode fiberNIC or switch QSFP56 cage
region, ≤120 km400G ZR+ coherent module set to a 200G line rateswitch/router QSFP-DD cage (not the NIC)
long-haul, 100s of kmcarrier-leased 200G wavelength (or 2×100G) over DWDMyour switch hands off to the carrier
Step 2 · what plugs where

NIC side vs switch side

  • NIC side — ConnectX-6/7-class cards expose QSFP56 cages; DAC/AOC/FR4/LR4/ER4 all seat directly in the NIC. A GB10-class hub already has two 200 GbE QSFP ports on board, so a two-hub link needs exactly one cable and zero new hardware.
  • Switch side — coherent ZR+ optics are QSFP-DD form factor and belong in a switch or router; the hub's NIC then joins that switch at 200G over a short DAC. Use this tier when the far hub is tens of kilometers away.
  • The fiber itself — standard single-mode (G.652) duplex LC pairs, leased as dark fiber per strand. The same glass carries 100G today and 400G later; upgrades are a module swap, never civil works.
  • Beyond ~120 km — you stop buying parts and start leasing a wavelength from a carrier; the demarcation is an Ethernet handoff on your switch.

Three reference builds, cheapest first.

two-hub bench   : hub A qsfp0 ──QSFP56 DAC 1m── hub B qsfp0
campus pair     : hub A [LR4] ──dark fiber, ≤10km── [LR4] hub B
metro federation: hub ──DAC── switch [ZR+ @200G] ──SMF ≤120km── [ZR+] switch ──DAC── hub
Step 3 · actually hitting 200G

Line rate is a configuration, not a purchase

  • Use RDMA (RoCE) for the dispatch stream where available — GB10-class hosts feed the NIC through split PCIe links, and measured full speed (~185–190 Gb/s) shows up under RoCE with a correctly mapped topology; a mis-mapped path caps near half rate and untuned plain TCP lands far lower.
  • Enable jumbo frames (MTU 9000) end-to-end and keep TCP_NODELAY on the dispatch sockets (the hub already sets it).
  • Expect to *verify*, not assume: run a perftest between hubs after every physical change — the difference between 95 and 190 Gb/s is invisible until measured.
  • Keep the 443 relay as the fallback path — the dial policy is direct-first for public peers, relay for NAT. The relay's job is reach, the direct link's job is speed.

Why this matters to the architecture: decode latency is bounded by round-trip time (~5 µs/km in fiber — physics, unaffected by bandwidth), so a fat pipe buys prefill speed, batched-dispatch throughput, and near-instant expert-slice distribution, not lower per-token latency. That is exactly the hub-tier role in the two-tier design: capacity in the fat-pipe tier, reach in the relay tier.