Operating a hub

Operator notes for running a hub and gateway: hot-patch without rebuilds, survive restarts, keep the catalog registered, and lock the surface down to 443.

The hub (control plane) and gateway (public entry point) are the two long-lived services an operator keeps healthy. The hub and its RPC ports are unauthenticated by design — trusted host / LAN / VPN only — and all public traffic converges on the gateway's single 443 surface. These are the operating notes that keep that arrangement stable across code changes, restarts and reboots.

• Fast path: update hub/gateway code with docker cp <file> <container>:/app/... + docker restart — no image rebuild. But adding an environment variable can't be done this way (it needs a container recreate); prefer a runtime-config API that persists to hub-state instead.
• Compose drift: a long-running container can diverge from its compose file (network mode, entrypoint, env). Always docker inspect the real config before a docker compose up -d recreate — if it has drifted, recreation wipes the production settings. Use cp + restart.
• Diff before you patch: docker cp the in-container file out and diff it against repo HEAD before replacing it, so an earlier session's hot-patch isn't silently lost.

• A hub restart stops serving. Slots, controllers and bindings restore from hub-state.json, but a loaded model is runtime-only. After a restart, read each controller's last_load and re-fire POST /api/controllers/{cid}/serve — even a large model comes back in ~1 minute thanks to the page cache.
• Gateway watchdog: probe every served model with a 1-token request every 30 s and auto-reload from last_load on failure (with a cooldown). **Probe *all* models, not catalog[0]** — the moment a healthy model from another hub sorts to the front, a first-only probe misses a big model going down (a real bug, since fixed).
• Catalog TTL: POST /api/pay/hub/register has a 90 s TTL, so keep registration alive with a ~60 s heartbeat loop, made durable across reboots with an @reboot cron or a systemd unit.

• The hub (:19000) and RPC ports assume a trusted network; the only thing that should face the internet is the gateway on 443 (including its WebSocket relay passthrough).
• If a hub must sit on a public IP, firewall it to trusted IPs — but Docker's published ports are DNAT'd before the INPUT chain, so a rule on dport won't match. Filter in the DOCKER-USER chain using conntrack's original destination port (--ctorigdstport) instead, and persist the rules with a systemd oneshot ordered After=docker.service.

• Settlement is pull, not push: the hub accumulates contributions; the gateway polls GET /api/contributions and delta-credits KVR. If a hub restart resets its counters, the gateway re-baselines so nothing is double-paid. The expert-work rate is set by LINKCPP_EXPERT_UNITS_PER_MB.
• The `pkill` footgun: ssh host 'pkill -f X; ...' matches its *own* command line and kills itself. Use a character class in the pattern (X[x]), and never put the spawn and the pkill in the same remote command.