Thesis: Kvasir is a two-sided market settled in one token — consumers pay KVR to infer, nodes earn KVR to serve. The whole design succeeds or fails on one property: those two sides must form a virtuous cycle, where each turn makes the next turn easier. Get that wrong and any price policy eventually collapses; get it right and the network grows *cheaper* as it grows *larger*.

It's tempting to treat cost and reward as a tug-of-war — every dollar a consumer saves is a dollar a node doesn't earn. That framing is a trap. In a healthy network they are the same flywheel seen from two ends: payments become rewards, rewards become supply, supply becomes capacity and lower prices, lower prices become more usage, and more usage becomes more payments. The question isn't how to split a fixed pie; it's how to keep the wheel turning so the pie grows.

The engine of the cycle is a single rule already true in Kvasir: inference must be paid in KVR. That makes every unit of usage a unit of real demand for the token — utility, not speculation. Token demand supports the value of the KVR nodes earn; attractive rewards pull in supply; supply expands capacity and, through competition and finer expert-sharding, drives the marginal cost of serving down; cheaper, faster, more capable service pulls in more usage. Kvasir tightens the loop with a property no centralized API can copy: a participant can be consumer and supplier at once. The demand side and the supply side often grow inside the *same people*, which damps the imbalances that wreck one-sided markets.

A flywheel can spin down as easily as up. Naming the death spirals is how you design against them:

• Rewards are funded by real revenue. At steady state, what nodes earn comes from what consumers pay — not from open-ended token emission. Emission is a bootstrap subsidy that must *taper* as fee revenue grows. Kvasir already helps here by rewarding real work — KVR per tokens actually served × layer share, not mere presence — so subsidy can't leak to idle 'mercenary' nodes.
• KVR is the mandatory medium. Because you can't infer without paying KVR, usage is a permanent demand sink for the token. That anchors token value to real utility instead of speculation — the difference between a currency and a chip.
• Price floats inside a band. A floor kept above node marginal cost keeps serving worthwhile; a ceiling kept below centralized alternatives keeps Kvasir competitive. Between them, price moves — which is where the network's growth finally shows up as lower cost.

Today price is a governed constant — sensible for a devnet, but it means adding nodes raises *capacity*, not affordability. The design direction is a utilization-driven price: idle supply nudges the price down toward the floor, congestion nudges it up toward the ceiling. That single signal turns the intuition *"the more people share compute, the cheaper it gets"* into a rule the protocol enforces — while the floor keeps operators solvent so the supply that made it cheap doesn't evaporate. Because price is a sensitive economic parameter, it changes only under genesis-wallet authority with wallet-signature + 2FA, never a stray environment variable.

None of this requires exotic mechanism design. It requires discipline about three things: reward from revenue, value from usage, balance from a bounded floating price. Kvasir already ships the hard, honest parts — non-custodial settlement, work-proportional rewards, a token you must actually spend to use the network. The rest is the economic roadmap: the taper, the fee split that funds an insurance pool for failed inferences, and the thermostat. Built in that order, cost and reward stop fighting and start compounding.

The two GPU backends (CUDA, ROCm) share kernel sources, so they land effectively bit-identical (10⁻¹⁰). GPU↔CPU carries ~10⁻³ per-op perturbation from a different accumulation order, but stays equivalent at cosine 0.99975 — the same pattern as the earlier ROCm↔phone-ARM 0.99992. The router-authority principle holds again on NVIDIA: the discrete decisions (argmax, expert selection) are invariant over this continuous perturbation.

• Device — NVIDIA GB10 (Grace Blackwell), aarch64, compute 12.1 / sm_121a, 124.5 GB unified memory.
• Toolkit — CUDA 13.0.88 · gcc 13.3 · ggml 0.15.3; the pure-ggml/gguf expert-worker built with Blackwell kernels.
• Model — Qwen3.5-122B-A10B-Q4_K_M, layer-0 all experts (256 experts, n_embd 3072, n_ff 1024, Q4_K/Q6_K).
• Method — the 1.58 GB L0 slice streamed MI250 → GB10 (lossless compare); the same input (h/ids) run through CUDA, CPU and ROCm; float32 output vectors (36,864) compared by cosine, relative L2 and max-abs.

For heterogeneous workers to serve one model, a CUDA box and a ROCm box must be interchangeable, and a GPU and a CPU must be numerically equivalent. With Blackwell measured, both hold across the full backend matrix: CUDA↔ROCm workers can stand in for each other, and GPU↔CPU workers agree within a bounded, well-conditioned tolerance. The most common accelerator on earth is now a verified swarm citizen.

Kvasir's custody model keeps keys with users: wallets sign transactions, Solana records them, and the settlement service's only job is to verify what actually happened on chain before touching a balance. Payments follow *quote → payment → inference*, with every consumed transaction signature recorded in a one-shot usedSignatures registry so it can never be presented twice. That makes the settlement service the chokepoint — and the rule it must never break is: credit only what the chain proves, never what the client asserts.

Solana signatures are public. The stake-verification path checked that the vault *received* the expected KVR — but never *who sent it*. An attacker could watch devnet for a victim's KVR→vault transfer, then submit {owner: attacker, signature: victim's}: the vault-received check passed, the principal was credited to the attacker, and one unstake later the funds were theirs. Direct theft, using nothing but a block explorer.

verifyStakeTransfer(signature, owner, amount):
  delta(vault)  >= amount            # vault actually received it (old check)
  Σ debits from token accounts
    whose owner == credited owner    # NEW — sender binding
                >= amount            # summed across that owner's accounts
  # inference path (no owner): bound by private requestId
  # + one-shot usedSignatures instead

The node-reward endpoints minted claimable KVR from self-reported input: POST /api/node/contribution credited whatever units the client claimed — units: 1e9 and a claim call could drain the vault — and register/heartbeat honored self-declared hub/gateway roles (hourly infra rewards) and performance scores (reward multipliers). The fix gates every reward-affecting assertion behind a trusted reporter: only the M2M service token used by the hub's contribution poll, or an authenticated admin, may assert units, infra roles or perf tiers — enforced even in open LAN mode, because these mint KVR. The token comparison is constant-time, and wallet↔node linking still works freely; it just can't assert its own rewards anymore.

Settlement state was a lock-free read-modify-write, and every money operation *awaits* an on-chain payout or verification in the middle — yielding the event loop with a stale balance in hand. Two concurrent claims could both read the same 100 KVR pending balance and both pay out. This wasn't theoretical: the exploit test showed three concurrent claims paying 300 for a 100 balance.

withLock(key, fn)         # per-key promise chain, self-cleaning map
  stake / unstake / claim  → keyed by owner
  inference settlement     → keyed by requestId
inside the lock:
  usedSignatures check + credit   # no same-signature double-credit
  pay out FIRST, then debit       # failed payout leaves balance intact

• Replaying a victim's transfer signature under an attacker's wallet is now rejected ("not sent by owner"); legitimate stakes, over-claims and inference payments behave unchanged.
• Three concurrent claims against one balance pay exactly once; re-submitting an already-paid inference idempotently returns the same result.
• Self-asserted units, hub/gateway roles and perf tiers from unauthenticated clients no longer move a single lamport of rewards.

The through-line of all three fixes is one principle applied three ways: the chain is the source of truth, the service is a verifier, and every balance has exactly one writer. The settlement service still runs on Solana devnet — which is exactly where you want to find, exploit and fix these classes before mainnet raises the stakes.

prompt: "The capital of France is" → generated (with the phone in the loop): " Paris." — 8/8 tokens identical to the local run.

Even with the phone computing layer-0's experts for every token, the generated tokens are exactly the local ones — the correct "Paris.". TPS drops to 3.13 from 4.01 — the phone-dispatch round trip (MI250 → tunnel → phone, ~100 ms/token) costs 22%. Throughput comes back with batching and replicas (M4).

1. Phone knows the hub (hub.kvasir-ai.net) — already holds its wallet node-token
2. GET /api/proxy/models/…/expert-shard?layers=0:1&experts=0:256
   # partially downloads its own expert slice (1.58 GB, WiFi)
3. linkcpp-expert-worker --serve
   # loads the slice (Adreno device, CPU backend) + waits for dispatch
4. MI250 backbone decodes the 122B → every token, layer-0 experts
   dispatch to the phone → experts return → combine → "Paris."
   (8/8 identical to local)

• Partial download (the expert-shard endpoint), worker serving, backbone dispatch, live 122B generation and TPS — all verified on the real device.
• Correctness: with the phone participating, 8/8 tokens equal the local run, with the correct answer.
• Remaining: the in-app autonomous loop (poll expert-demand → volunteer → download → serve → register) is Kotlin wiring — this demo drove the mechanism directly.
• Transport: this demo used an SSH tunnel; production uses the 443 relay (already verified in the ring work).

Bigger batches, more tok/s per worker

From 1.45 ms/tok at batch 1 to 0.019 ms/tok at batch 512 — a 77× per-token improvement. The per-call time barely moves (1.45 → 9.6 ms) while the batch grows 512× — the GPU processes the batch nearly for free behind a fixed overhead. This is the throughput-fabric property that makes expert-parallel practical: batched dispatch amortizes per-request RTT and overhead.

• Worker --bench: compute_dispatch timings for batch 1…512 → tok/s.
• 53k tok/s per worker at batch 512 (ROCm) — batching amortizes the overhead.
• Hot-expert caching and multi-worker aggregate scaling (replica routing) stack on top of this.

Three workers register on layer 0: A = [0,128), B = [128,256), C = [0,128) as a second replica, with target_replicas = 2:

POST /api/expert-volunteer {"max_experts": 64}
  → {layer: 0, experts: [128, 192], scarcity: 0.5, replicas: 1, target: 2}

• POST /api/expert-coverage — workers heartbeat their (layer, expert-range) holdings.
• GET /api/expert-demand — per-expert replica aggregation → contiguous expert-range segments with scarcity scores.
• POST /api/expert-volunteer — assigns the scarcest range clipped to the node's budget.
• The existing layer market (self-enroll · partial download · rewards) re-parameterized to (layer, expert-range).

What follows in M4: batched dispatch for concurrent requests plus a hot-expert cache — tokens/s proportional to worker count — and replica routing (nearest/fastest worker) with churn fallback.

Backbone decode → TCP → worker → experts → same token

The expert worker serves the layer-0 slice as a separate process (ROCm), and the 122B backbone's build_moe_ffn dispatch callback ships (cur, sel) over TCP and receives the expert outputs. The logit cosine is exactly the in-process value (0.99868775) — the transport is lossless. Expert-parallel swarm compute works across a process boundary.

# worker: serving as a separate process
linkcpp-expert-worker --serve 52700 --model L0_all.gguf --layer 0 --n-embd 3072
# backbone: build_moe_ffn callback dispatches to the worker
linkcpp-moe-verify 122B.gguf ... --dispatch-port 52700
  → protocol: [n_used, n_tokens] + cur + sel  →  experts

• --serve mode: load the slice, listen on TCP, answer (n_used, n_tokens, cur, sel) → experts.
• --dispatch-port: the backbone callback sends/receives over TCP to a separate worker, replacing in-process compute.
• Measured on a live 122B decode with layer-0 dispatched out-of-process → argmax MATCH, cosine 0.99869 (= in-process, lossless).
• M2 core (earlier): the phone's ARM computed real 122B experts at cosine 0.99992 (Android cross-build).

Next from here: tunneling the same TCP stream through the 443 relay to workers on other machines and phones (the transport was already proven in the ring work), then the M3 coverage market and M4 batched throughput.

• Design (5 revisions since the blueprint) — EP architecture, partial-weight autonomous participation, the relay, worker kernel spec, and cross-backend numerical equivalence codified as core technology. Router authority written down as the coherence invariant.
• M0 — backbone expert RAM offload (planner verified): 122B is *feasible* on a single 64 GB coordinator — 10 expert layers offloaded to RAM, VRAM 62.6 GiB / RAM 14.2 GiB, wired via --override-tensor.
• M1 — expert-slice data path: per-expert mini-GGUF slicing (ne[2] slabs, byte copy without dequant) + the /expert-shard download endpoint.
• M1 — numerical oracle: dispatch + combine == monolithic with max|Δ| = 3.6e-12 on real layer-0 experts — sharding is an exact regrouping of the same weighted sum.
• M1 — C++ worker on hardware: linkcpp-expert-worker (pure ggml/gguf) built and run on ROCm, cosine 0.99995 vs the oracle; router → two C++ workers → combine matches monolithic at cosine 0.9997–0.9999.
• M2 core — a phone computes real 122B experts: Android cross-build, run on an SM-S938N, cosine 0.99992 vs the oracle.
• Numerics — 3-backend equivalence matrix: the same 122B computation on ROCm × phone ARM CPU × numpy — ROCm↔phone cosine 0.99990, ROCm↔numpy 0.99996, phone↔numpy 0.99992. All equivalent, none bit-identical.

The keystone has since landed: the follow-up posts on M2, M3, M4 and the live phone demo are the results of exactly this integration.

Exact vs equivalent — two different properties

• Within one backend — exact (3.6e-12): splitting experts across nodes and combining is the same weighted sum regrouped; the only difference is floating-point accumulation order. Oracle-verified.
• Across backends — equivalent (1e-3…1e-6): the same op on different hardware carries a per-op relative error around 1e-3–1e-6, and is never zero. The swarm lives in this regime.

"Exact" is what decomposition guarantees inside one device. "Equivalent" is what heterogeneous hardware gives you. The swarm's job is to keep equivalence from compounding into divergence.

The same Qwen3.5-122B layer-0 expert FFN, computed by linkcpp-expert-worker on an MI250 (ROCm), a phone's ARM CPU (SM-S938N), and an x86 numpy reference — same inputs, same weights, different instruction sets and reduction orders:

Three instruction sets, one computation — every pair equivalent (cosine ≈ 0.9999), no pair bit-identical (Δ ≈ 1e-6). The residuals are small because router authority pinned the inputs and the expert selection.

A later run on real NVIDIA GB10 Grace Blackwell hardware closed the matrix on the last backend: CUDA ↔ ROCm landed at cosine 1.0000000000 (max abs 3.5e-10, effectively bit-identical, since both GPU backends share kernel sources), and CUDA ↔ Grace ARM CPU at cosine 0.99975 — the same GPU↔CPU pattern seen above.

• Matmul reduction order — tensor cores, MFMA tiles, OpenCL workgroups and SIMD lanes each accumulate in different orders and tilings.
• FMA fusion — a*b+c rounded once (FMA) or twice, fused differently per backend.
• Accumulation precision — F16/BF16 storage with F32 vs F16 accumulators (the biggest lever on divergence).
• Transcendental approximations — polynomial/table variants of exp (softmax), silu/sigmoid (swiglu), rsqrt (norms).
• Dequant + matmul path — dequantize-then-matmul vs fused quantized kernels round intermediates differently.
• Nondeterministic kernels — atomic/split-K reductions can differ run to run on the same device.

None of this is a bug. It is the price each accelerator's fast path pays.

• Discrete argmax: decoding is an argmax over logits. A 1e-3 wobble flips a token only when two candidates are within 1e-3 — at most positions the margin is far larger, so tokens come out identical; the rare flips are positions as ambiguous as a different seed.
• Combine is addition: partial results merge as a probability-weighted sum. Independent ~1e-4 errors add incoherently — they grow like √k, not k — and there is no cancellation of large values, so the residual stays well-conditioned.

• Per-op delta — same inputs, A vs B relative error on matmul, swiglu, softmax, norm.
• Layer-boundary drift — residual delta after one layer, stacked to see whether depth accumulates as √L or L.
• End-to-end logit divergence — L∞, L2 and KL divergence over the full forward.
• Decision agreement — top-1 token agreement plus top-8 routing agreement (validating why router authority is necessary).
• Generation stability — greedy N tokens; the first index where A and B diverge.
• Task level — perplexity and eval-score deltas: the only metric a user actually feels.

A pass is a tolerance — "top-1 agreement ≥ 99.x%, KL ≤ ε". A node outside tolerance is marked unfit for sensitive ranks, not rejected outright.

A homogeneous cluster can assume bit-exactness; a swarm cannot — its premise is *whatever hardware shows up*. So Kvasir treats numerical equivalence exactly like protocol compatibility: a first-class, measured contract. Backends and accumulation precision are advertised as node capabilities, router authority is enforced as an invariant, and every verification uses tolerances instead of bit-equality. Measured numerical equivalence + single-authority discrete decisions — that is what lets one model run on every GPU on earth at once. That is the swarm.

An expert slice is a byte copy — no dequant

MoE expert tensors stack all experts along the outermost ggml dimension, so the reader exposes (n_expert, rows, row_bytes) of raw quantized bytes. Expert *e* is a quant-block-aligned contiguous slab — the slice is literally data[a:b], with no dequantization and no re-packing.

sliced = tensor.data[a:b]              # outermost axis = expert
writer.add_tensor(name, sliced, raw_dtype=tensor.tensor_type)
# router (ffn_gate_inp) & shared expert stay on the backbone → excluded
GET /api/proxy/models/{m}/expert-shard?layers=0:2&experts=0:16  # node-token authed

With real 122B layer-0 experts (dequantized reference), splitting the experts into 4 shards, computing each separately and combining matches the monolithic MoE FFN: sharding is an exact regrouping of the same weighted sum, not an approximation.

• Pure ggml/gguf (no libllama): loads the slice into a GPU backend and runs mul_mat_id(up/gate) → swiglu → mul_mat_id(down).
• ROCm build + run on an MI250: 122B layer-0, experts [0,8), 4 tokens.
• Cosine 0.99995 vs the oracle, allclose(1e-3) = True, max|Δ| = 7.9e-7 — this residual is itself the first measured instance of cross-backend equivalence (ROCm vs numpy).
• The same code path covers CUDA/Metal/Vulkan/CPU (mul_mat_id/swiglu are stock ggml; CUDA has a dedicated MoE kernel).

The hardest, riskiest piece — the on-device worker kernel — was verified here. What remained was backbone↔worker orchestration; the worker is a proven pure function consuming these slices.

Previously the ring placed weights VRAM-only, so the 122B (77.6 GB) was infeasible on a 64 GB GCD. With expert-offload rules the dry-run comes back feasible:

node 0  layers [0,48]  vram=62.6  ram=14.2  ot_rules=10
sample: blk\.38\.ffn_(up|down|gate)_(ch|)exps=CPU   # inference engine -ot format

• planner — already emits ot (comma-joined -ot rules) in each placement.
• protocol.py — added the StageStartRequest.ot field.
• runtime.py — forwards the placement's ot into the stage request.
• stage_service.py — the coordinator launches with --override-tensor.
• linkcpp-server forwards unknown args to stock llama-server, so -ot applies untouched.

# stage_service.py — coordinator branch
if request.ot:
    command += ["--override-tensor", request.ot]

Verified with an ot protocol round-trip test plus confirmation that the coordinator command emits --override-tensor; the hub redeployed cleanly with no regressions. Remaining at the time: the full 2-node 77 GB load (coordinator with offload + a phone holding a ~1.5 GB one-layer window), gated on server availability. The core of M0 — the backbone offload that lets weak nodes participate in a big MoE — was complete at the code and planner level.

The thesis: 86% of Qwen3.5-122B's weight is 12,544 mutually independent 5.3 MB experts. Shard at the expert grain and a weak device carries "8–64 experts (42–340 MB)" instead of "a 1.4 GB layer" — exactly the unit a phone can actually hold. MoE is the natural substrate of a swarm.

The expert index is the outermost dimension of every MoE tensor, so each expert is a contiguous, quant-block-aligned slab. An expert-sliced mini-GGUF is a clean byte-range copy — no dequantization, no re-packing.

• Backbone stage (strong node): attention + KV cache, all norms, the router, the shared expert, and the residual combine — the entire dense path. It also keeps all experts resident as a fallback replica (RAM offload), which gives the swarm churn tolerance.
• Expert worker (a phone): not a transformer. No attention, no KV, no sampler — a pure function (hidden, local_ids) → out made of three mat-muls, holding only its own expert slice. It fits any budget down to a 4 GB phone.

cur   = ffn_norm(x)                       # backbone
ids,p = top_k(softmax(cur @ router), 8)   # backbone — authoritative
── dispatch selected experts to owner nodes ──
send  (cur rows, local_ids)  →  worker    # ~6 KB per decode step
recv  expert_out             ←  worker
x = x + combine(p, partials) + shared(cur)  # backbone — numerically exact

Because the router runs exactly once on the backbone, each selected expert is computed exactly once by whichever node owns it. There is no approximation — the sharding only moves where the mat-muls happen.

Kvasir already runs an autonomous scarcity market for layer shards, verified on real devices: a NAT-bound phone polls the demand map, self-enrolls into the highest-reward segment, partially downloads only that window, loads it on its Adreno GPU, and completes ring inference — earning contribution rewards. Expert sharding reuses all of it — coverage map, max-reward self-enroll, partial download, per-node rewards — changing only the coverage unit from *layer ranges* to *(layer, expert-range)*.

• Reward-driven partial-weight download: conventional RPC/TP/PP setups ship the full checkpoint to every rank and a scheduler dictates placement. In Kvasir a node downloads only the slice it will compute, and picks that slice itself, by reward — a 254 MB stage mini-GGUF versus the 77.6 GB full model. This is how a 4 GB phone joins a model far bigger than itself.
• 443 relay data plane: Cloudflare's 80/443-only edge plus carrier NAT means no direct dial in either direction. A per-edge WebSocket bridge with a 1-byte role preamble lets both sides dial outbound (the phone opens zero inbound ports). Landing it meant fixing three real bugs — build-fingerprint agreement, node-token download auth, and a Kotlin Int.ushr frame-length bug that silently corrupted every frame ≥ 64 KiB (ushr uses only the low 5 bits of its shift; len ushr 56 became len ushr 24) — fixed by moving to Long shifts.

Decode is 49 serial layers, and a cross-internet round trip per layer costs 2.5–10 s per token. So the swarm's contest is serving models nobody can host alone, measured in aggregate throughput: batch dispatch amortizes RTT, the backbone keeps a hot-expert cache, and requests route to near replicas. The low-latency path stays with the pipeline ring.

• M0 — backbone expert RAM offload: run 122B on one coordinator, no graph surgery.
• M1 — single-host expert-parallel proof: expert-sliced mini-GGUF + worker runtime + dispatch, logits exactly matching monolithic.
• M2 — LAN + NAT phone workers computing real 122B experts through the 443 relay.
• M3 — expert-grain coverage market with replicas and churn fallback.
• M4 — throughput: batched dispatch + hot-expert cache, tokens/s scaling with worker count.

Each layer splits into a dense path — attention + KV, the norms, the router (ffn_gate_inp), a shared expert — and an expert bank: 256 independent FFNs stored as three stacked tensors (ffn_up_exps, ffn_gate_exps, ffn_down_exps). The dense path is the minority of the bytes; the expert bank is 86% of the model.

• The expert index is the outermost ggml dimension (ne[2]) of every expert tensor — expert *e* occupies one contiguous, quant-block-aligned slab of raw quantized bytes.
• That makes per-expert extraction a byte-range copy: data[a:b], no dequantization, no re-packing — an expert-sliced mini-GGUF is cheap to produce and bit-faithful.
• Per token only 8 of 256 experts fire per layer, chosen by the router — so at decode time a layer's expert traffic is a handful of small matrix multiplies over one hidden vector.

At layer grain, the least a node can hold is ~1.4 GB — out of reach for most phones once the app, KV and OS take their share. At expert grain, the unit is 5.3 MB, and a realistic contribution is 8–64 experts (42–340 MB) — comfortably inside any modern device. The experts are mutually independent, so ownership can be scattered arbitrarily and re-balanced freely. This analysis is what made expert-level sharding the design bet: the weights were already packaged in swarm-sized units — the network just had to honor the packaging.

The RPC master is a bottleneck and a gatekeeper

In the classic RPC topology one master opens the entire GGUF and dials out to every worker. That shape breaks in an open network three ways: the master must hold and serve the whole checkpoint; every worker must be dialable — phones behind carrier NAT are not; and the master is a single owner in a network that should have none.

• Every device stores the same model but loads only its contiguous layer window, then opens exactly two links: one to its predecessor, one to its successor.
• A request enters the ring; each node runs its layers and passes only the hidden-state boundary to its neighbor. The last rank samples the token and sends it back around — no central master, and no node holds the whole model.
• Placement comes from the planner's rank manifest — for Qwen3.5-122B, 49 layers split across whatever mix of GPU, CPU, NPU and phone shows up.

• Partial-shard download: a ring stage doesn't need the checkpoint — it needs its window. A stage mini-GGUF carries just those tensors (254 MB of 26 tensors versus the 77.6 GB full model), so a phone pulls ~1.5 GB for a one-layer window instead of everything.
• Mobile GPU path: the RPC route to a phone GPU proved infeasible (Adreno's OpenCL buffer layout doesn't survive RPC serialization), but a ring stage runs on the Adreno GPU directly — the stage owns its backend locally, so nothing crosses the wire but boundaries.
• NAT traversal: phones can't accept inbound connections, so the data plane runs through a 443 relay — a per-edge WebSocket bridge with a 1-byte role preamble that lets both ends dial outbound. The phone opens zero inbound ports.
• Self-enrollment market: stages are claimed, not assigned. A node polls the coverage/demand map, picks the highest-reward uncovered window, downloads that window, and joins — verified end-to-end with a NAT-bound phone completing ring inference and earning its contribution.

The ring is Kvasir's latency path: boundaries are small, hops are few, and decode flows around the loop without gathering anything centrally. Its limitation is granularity — the smallest unit a node can carry is a layer (~1.4 GB on the 122B). Removing that floor is what the expert-sharded swarm does; the ring remains the serving backbone it plugs into.

inference engine can already split a model across machines over RPC — but someone has to discover the GPUs, decide which layers go where, launch the right workers with the right budgets, check that every node speaks the same protocol, and expose an API developers can actually call. Doing that by hand for one cluster is a chore; doing it for an open network of strangers' devices is impossible. That coordination layer is linkcpp.

browser / SDK
  → hub :19000                      # FastAPI control plane (single Docker image)
  → GPU-less llama-server master    # per-controller, :8080+
  → ggml-rpc-server workers         # local slots, remote units, managed agents

• Three ways a machine joins: fixed local node slots with editable VRAM/RAM/CPU budgets; remote units — register another hub and import its nodes; and managed node agents — worker-only services that join over plain request/response HTTP, deliberately not a persistent stream, so they survive simple LAN/VPN routing.
• Compatibility gating is first-class: every unit, node and agent reports a protocol / runtime-pack identity plus backend details. Unit, runtime-pack, inference engine-revision and RPC-ABI mismatches are hard-blocked before bind, plan, load or infer — backend differences (CUDA/Metal/Vulkan/CPU) are tracked as capabilities, not rejections.
• The planner reads GGUF metadata and produces contiguous per-node layer placement, --tensor-split, KV-cache/layer/expert VRAM estimates, and optional expert-FFN offload to RAM.
• Gateways: every controller exposes OpenAI-compatible (/v1/chat/completions, /v1/responses, /v1/models) and Anthropic-compatible (/anthropic/v1/messages|models) endpoints, backed by the same loaded model — existing clients work unchanged.

This deliberate split — an unmodified data plane under an open control plane — is what everything later builds on: the ring runtime, the layer market, and eventually the expert-sharded swarm are all control-plane evolutions over the same stock compute.