Numerical Equivalence Across Heterogeneous Backends
CUDA, ROCm, Adreno and CPUs will never agree bit-for-bit. That the swarm still produces one coherent model is a designed property, not luck.
Exact vs equivalent — two different properties
- Within one backend — exact (3.6e-12): splitting experts across nodes and combining is the same weighted sum regrouped; the only difference is floating-point accumulation order. Oracle-verified.
- Across backends — equivalent (1e-3…1e-6): the same op on different hardware carries a per-op relative error around 1e-3–1e-6, and is never zero. The swarm lives in this regime.
"Exact" is what decomposition guarantees inside one device. "Equivalent" is what heterogeneous hardware gives you. The swarm's job is to keep equivalence from compounding into divergence.
Not a theory — measured on hardware
The same Qwen3.5-122B layer-0 expert FFN, computed by linkcpp-expert-worker on an MI250 (ROCm), a phone's ARM CPU (SM-S938N), and an x86 numpy reference — same inputs, same weights, different instruction sets and reduction orders:
| Backend pair | max|Δ| | cosine |
|---|---|---|
| ROCm (GPU) vs numpy (x86) | 7.9e-7 | 0.99996 |
| Phone ARM CPU vs numpy (x86) | 1.4e-6 | 0.99992 |
| ROCm GPU vs phone ARM CPU | 1.5e-6 | 0.99990 |
Three instruction sets, one computation — every pair equivalent (cosine ≈ 0.9999), no pair bit-identical (Δ ≈ 1e-6). The residuals are small because router authority pinned the inputs and the expert selection.
A later run on real NVIDIA GB10 Grace Blackwell hardware closed the matrix on the last backend: CUDA ↔ ROCm landed at cosine 1.0000000000 (max abs 3.5e-10, effectively bit-identical, since both GPU backends share kernel sources), and CUDA ↔ Grace ARM CPU at cosine 0.99975 — the same GPU↔CPU pattern seen above.
Floating-point addition is not associative
- Matmul reduction order — tensor cores, MFMA tiles, OpenCL workgroups and SIMD lanes each accumulate in different orders and tilings.
- FMA fusion —
a*b+crounded once (FMA) or twice, fused differently per backend. - Accumulation precision — F16/BF16 storage with F32 vs F16 accumulators (the biggest lever on divergence).
- Transcendental approximations — polynomial/table variants of exp (softmax), silu/sigmoid (swiglu), rsqrt (norms).
- Dequant + matmul path — dequantize-then-matmul vs fused quantized kernels round intermediates differently.
- Nondeterministic kernels — atomic/split-K reductions can differ run to run on the same device.
None of this is a bug. It is the price each accelerator's fast path pays.
One authority for decisions, enough precision for accumulation
ROUTER AUTHORITY — the core invariant. The only discrete decision inside the network is MoE routing (top-8 of 256). If every backend re-ran the router, borderline tokens would pick different experts and genuinely diverge. Kvasir runs the router once, on the backbone, and sends workers only the selected expert ids. A heterogeneous swarm may differ in the *magnitude* of each expert's output — it never differs in *which experts run*. This converts catastrophic discrete divergence into bounded continuous error, and is the coherence rule of heterogeneous expert sharding.
- Discrete argmax: decoding is an argmax over logits. A 1e-3 wobble flips a token only when two candidates are within 1e-3 — at most positions the margin is far larger, so tokens come out identical; the rare flips are positions as ambiguous as a different seed.
- Combine is addition: partial results merge as a probability-weighted sum. Independent ~1e-4 errors add incoherently — they grow like √k, not k — and there is no cancellation of large values, so the residual stays well-conditioned.
Divergence modes and defenses
| Divergence mode | Mechanism | Rule |
|---|---|---|
| Routing mismatch | Backends pick different top-8 for borderline tokens | Router authority — decided once on the backbone, ids dispatched |
| Trajectory fork | Per-token logit wobble eventually flips a token; the sequence forks like a new seed | Decode/sampling pinned to one node |
| Depth accumulation | 49 layers × ~1e-4 each → up to 1e-2 at the final logits | F32 accumulation at boundaries and combine |
| Self-nondeterminism | Atomic/split-K kernels vary run-to-run | Deterministic combine kernels; verification uses tolerances |
| Precision mismatch | One node accumulates F16, another F32 | Accumulation precision advertised as a capability; F32 nodes preferred for output ranks |
The measurement protocol
- Per-op delta — same inputs, A vs B relative error on matmul, swiglu, softmax, norm.
- Layer-boundary drift — residual delta after one layer, stacked to see whether depth accumulates as √L or L.
- End-to-end logit divergence — L∞, L2 and KL divergence over the full forward.
- Decision agreement — top-1 token agreement plus top-8 routing agreement (validating why router authority is necessary).
- Generation stability — greedy N tokens; the first index where A and B diverge.
- Task level — perplexity and eval-score deltas: the only metric a user actually feels.
A pass is a tolerance — "top-1 agreement ≥ 99.x%, KL ≤ ε". A node outside tolerance is marked unfit for sensitive ranks, not rejected outright.
Bit-agreement is impossible — and unnecessary
A homogeneous cluster can assume bit-exactness; a swarm cannot — its premise is *whatever hardware shows up*. So Kvasir treats numerical equivalence exactly like protocol compatibility: a first-class, measured contract. Backends and accumulation precision are advertised as node capabilities, router authority is enforced as an invariant, and every verification uses tolerances instead of bit-equality. Measured numerical equivalence + single-authority discrete decisions — that is what lets one model run on every GPU on earth at once. That is the swarm.