Numerical equivalence
Different backends never agree bit-for-bit; the swarm treats measured tolerance as a first-class contract.
CUDA, ROCm, Adreno and CPUs compute the same op with different reduction orders, FMA fusion, accumulators and transcendental approximations — so results differ by ~1e-6…1e-3 per op, by design, never bit-identically. A swarm made of whatever hardware shows up can't demand bit-exactness, so Kvasir measures equivalence instead.
| Backend pair (real 122B, layer-0 experts) | max|Δ| | cosine |
|---|---|---|
| CUDA (GB10 Blackwell) vs ROCm (MI250) | 3.5e-10 | 1.0000000000 |
| ROCm (MI250) vs numpy (x86) | 7.9e-7 | 0.99996 |
| Phone ARM CPU vs numpy (x86) | 1.4e-6 | 0.99992 |
| CUDA (GB10 Blackwell) vs Grace ARM CPU | 2.6e-5 | 0.99975 |
The full backend matrix is closed: the two GPU backends (CUDA, ROCm) share kernel sources and land effectively bit-identical (cosine 1.0000000000), while GPU↔CPU pairs stay equivalent at ~0.9997. A CUDA worker and a ROCm worker are interchangeable; a GPU worker and a CPU worker are numerically equivalent.
Why they differ
Floating-point addition is not associative
- Matmul reduction order — tensor cores, MFMA tiles, OpenCL workgroups and SIMD lanes accumulate in different orders.
- Accumulation precision — F16/BF16 storage with F32 vs F16 accumulators is the biggest lever on divergence.
- Transcendental approximations — exp (softmax), silu (swiglu) and rsqrt (norms) use different polynomial/table variants per backend.
The contract
Tolerances, capabilities, single authority
- Verification is a tolerance — "top-1 agreement ≥ 99.x%, KL ≤ ε" — never bit-equality.
- Backends and accumulation precision are advertised as node capabilities; F32-accumulating nodes are preferred for output-sensitive ranks.
- Out-of-tolerance nodes are marked unfit for sensitive ranks, not rejected outright.
- Discrete decisions (routing, sampling) are pinned to single authorities so continuous error can never become discrete divergence.