NVIDIA Blackwell Joined the Swarm
A GB10 Grace Blackwell computed real 122B expert-FFN slices in CUDA and matched AMD ROCm bit-for-bit (cosine 1.0000000000) and Grace ARM CPU within tolerance. The cross-backend matrix is complete.
A swarm's premise is *whatever hardware shows up*. Numerical equivalence — the proof that CUDA, ROCm, Adreno and CPU workers all emit the same token — was already measured on ROCm, phone ARM and numpy. NVIDIA is the default and best-optimized path in stock ggml/inference engine, yet it was the one backend the matrix hadn't been closed on. Running real Blackwell hardware closes it.
GB10 Blackwell CUDA ↔ MI250 ROCm gfx90a: cosine 1.0000000000 — max abs diff 3.5×10⁻¹⁰. On the same real Qwen3.5-122B layer-0 expert slice, the two GPU backends are effectively bit-identical.
The cross-backend matrix
| Comparison | Hardware | cosine | max abs |
|---|---|---|---|
| CUDA ↔ ROCm | GB10 Blackwell ↔ MI250 gfx90a | 1.0000000000 | 3.5e-10 |
| CUDA ↔ CPU | GB10 Blackwell ↔ Grace ARM | 0.9997525825 | 2.6e-05 |
| CPU ↔ ROCm | Grace ARM ↔ MI250 gfx90a | 0.9997525823 | 2.6e-05 |
The two GPU backends (CUDA, ROCm) share kernel sources, so they land effectively bit-identical (10⁻¹⁰). GPU↔CPU carries ~10⁻³ per-op perturbation from a different accumulation order, but stays equivalent at cosine 0.99975 — the same pattern as the earlier ROCm↔phone-ARM 0.99992. The router-authority principle holds again on NVIDIA: the discrete decisions (argmax, expert selection) are invariant over this continuous perturbation.
What ran, on what
- Device — NVIDIA GB10 (Grace Blackwell), aarch64, compute 12.1 / sm_121a, 124.5 GB unified memory.
- Toolkit — CUDA 13.0.88 · gcc 13.3 · ggml 0.15.3; the pure-ggml/gguf expert-worker built with Blackwell kernels.
- Model — Qwen3.5-122B-A10B-Q4_K_M, layer-0 all experts (256 experts, n_embd 3072, n_ff 1024, Q4_K/Q6_K).
- Method — the 1.58 GB L0 slice streamed MI250 → GB10 (lossless compare); the same input (h/ids) run through CUDA, CPU and ROCm; float32 output vectors (36,864) compared by cosine, relative L2 and max-abs.
One real-hardware gotcha: the GB10's integrated GPU classifies as ggml device type ACCEL, not GPU — so init_by_type(GPU) found nothing. Fixed by selecting the first non-CPU device instead of hard-coding the GPU type.
The matrix is closed
For heterogeneous workers to serve one model, a CUDA box and a ROCm box must be interchangeable, and a GPU and a CPU must be numerically equivalent. With Blackwell measured, both hold across the full backend matrix: CUDA↔ROCm workers can stand in for each other, and GPU↔CPU workers agree within a bounded, well-conditioned tolerance. The most common accelerator on earth is now a verified swarm citizen.