# v2: networking, anti-cheat and persistence

Who simulates what, netcode, the referee, anti-cheat and persistence tiers. Index:
`agents/world-format-v2.md` (the authority and P2P decisions). Schemas: `agents/v2-formats.md`
sections 7 (event log, T2 damage) and 16 (netcode rows). Brief: `agents/v2-k-referee.md`.
Built: referee v0 in `v2/net/` (goal 8).

## Who simulates (hybrid, decided 2026-09-26)

The server is authoritative for shared *systems*; peers own local *physics detail* (their car,
nearby props, debris) and hand ownership off as players move: the GTA Online model for physics.
The server can't run complex physics for everyone, and players are mostly in one city.
- **The server is the authority, referee and record:** traffic spine, police and wanted, fire,
  missions and leases, persistent damage, money, claims, canonical edits, and sanity checks on
  peer-owned state. (`server/CLAUDE.md` "Phase 2" had the server half; it points here.)
- **Interest management** decides what each player receives: nearby entities at full rate,
  further ones lower, far ones not at all. Per-player budgets (entities, bandwidth, physics
  bodies owned) are measured by the harness and set before multiplayer opens.
- **Peer-to-peer: yes (DECIDED 2026-09-27).** The mesh is on by default with a "relay only"
  privacy setting for anyone who doesn't want their IP exposed; under-18 accounts are forced
  relay-only (orchestrator, delegated 2026-09-27). Context: P2P exposes home IP addresses (a
  privacy and DDoS risk with kids and streamers; v1 never logs IPs), and Hetzner has no UK
  region, so **measure RTT** (Newcastle ISPs to Nuremberg/Helsinki, and between Newcastle ISPs)
  to tune K and the tier rates: if direct paths are only ~20 ms better, the mesh buys little and relay-only is simpler.

## Netcode v2 (designed 2026-09-27; v0 built)

- **Topology: server star plus a capped near mesh.** Every client has one link to its shard
  server over WebTransport (Chrome 97+, Firefox 114+, Safari/iOS 26.4+, ~91%), with WebSocket
  fallback for networks that block UDP. **Owners always uplink their state to the server**, so
  P2P is only a latency shortcut, never needed for correctness. Direct WebRTC links only
  between near-tier peers, at most K = 8, picked by the server with hysteresis (join 200 m,
  drop 280 m). Everything else is relayed by the server, SFU-style. A pair whose ICE fails uses
  the server path, **so no TURN server is needed** (this supersedes the earlier plan of a
  WebRTC mesh with a TURN fallback, and "TURN bandwidth is the real bill").
- **Ownership: leases with epochs.** Server table `eid -> {owner, epoch, lease_until}`.
  `claim{eid, epoch_seen, reason: drive|contact|spawn}` answered by `grant{epoch+1}` or `deny`;
  priority driver > contact > current owner > nearest. Authority follows interaction
  (Fiedler). Traffic-spine cars get a **physics loan** on collision and return to the spine at
  rest. Handoff sends a final state; the new owner starts from it at epoch+1. **Every packet
  carries `(eid, epoch)` and stale epochs are dropped**: that makes double ownership
  impossible. Leases lapse after 500 ms moving / 5 s resting; a disconnecting owner lapses
  everything it owned (debris dropped, cars parked, mission entities to the lease holder).
  Collisions with remote bodies send `impulse{eid, epoch, tick, J, point}` to the owner, clamped
  for plausibility (fixes v1's one-sided proxy collisions).
- **Replication:** per-recipient priority accumulators (Gaffer, "State Synchronization"),
  chunk-relative quantisation (18-bit horizontal ~1 mm over 256 m, smallest-three rotation,
  11-bit velocities, an `at_rest` bit), **quantised on the sender before stepping**, deltas
  against the last acked baseline. Car ~22 bytes, ped ~10, NPC intent ~8.

| Tier | Path | Rate |
|---|---|---|
| near, in mesh | P2P plus uplink | 30 Hz |
| near, not in mesh | server relay | 20 Hz |
| mid | server | 5 Hz |
| far | server | 1 Hz or intent |
| dormant | none | `where(seed, t)` pure field |

- **Channels:** `state` (unordered, no retransmits / datagrams: snapshots, inputs, acks),
  `events` (unordered reliable / a uni stream per event class, tick-tagged), `control` (ordered
  reliable: join, claim, grant, mesh, correction). Messages <= 1,150 bytes; pack many entities
  per packet (~60-70 bytes of per-packet overhead).
- **Prediction and clock:** the local car is owned by its driver, so no routine
  reconciliation; the referee's only override is `correction{eid, epoch, tick, state}` (snap
  physics, decay the visual error over ~150 ms). Clock sync NTP-style (best 4 of 16 RTTs, slew
  never step). Interpolation indexed by server tick, delay ~2.5 packet intervals (~80 ms at
  30 Hz). World clock, weather and radio hang off this timeline.
- **Lockstep bubbles** (input-only sync for private crews <= 8): delay-based lockstep (3
  ticks) first; rollback only if Rapier snapshot/restore stays under ~1 ms (unmeasured). The
  bubble is one owner (`owner = bubble:id`), reporting to the server at 5 Hz; checksum every 30
  ticks, resync on mismatch, fall back to hybrid after two. Requires no JS `Math.*` in the sim
  and the ARM determinism check (`simulation.md`).
- **Server:** Rust native, one process per shard (~100 players, one instance of Newcastle), on
  the same crate that compiles to WASM; the binary codec lives in the Rust core so client and
  server can't drift. Candidate crates `str0m` / `webrtc-rs`, `wtransport` / `quinn` (maturity
  unverified). Node running the same core is fine for the 50-player phase (v0 is Node).
  Connect tokens (netcode.io pattern): a signed ~30 s `{account, shard, expiry}` the shard
  verifies without a database.
- **Bandwidth (design):** ~192 kbps down typical (384 match day), ~40 kbps up, mesh up to
  8 x 24 kbps; server egress ~200 kbps per concurrent player.
- **Hetzner cost (live API, nbg1, 2026-09-27):** 50 concurrent ~EUR 50/month (1x CCX13 + object
  storage); 500 ~EUR 290 (2x CCX33); 5,000 ~EUR 600-700 on dedicated AX-class (unverified) or
  ~EUR 1,730 on cloud with overage.

### Referee v0 as built (2026-09-27, `v2/net/`)
Handoff at epochs 1 then 2 with never two owners per epoch; a resting prop converges to ~1 mm
on both clients; clock error <= 5 ms; ~12-34 kbps up and ~13-18 kbps down per client including
packet overhead (per-packet overhead dominates uplink); teleport corrected and revoked;
relay-only clients open no direct link. Car rows ~25 bytes (design ~22). Peers re-check speed
and teleport on direct rows (direct links bypass the referee); reporting disagreements back is
next. **Still to build:** baseline deltas and acks, real WebRTC/ICE, WebTransport, traffic
physics loans, runtime spawns, a 50-bot soak, and a per-row tick in formats section 16 so one
packet can mix owners.

## Anti-cheat (from day one)

The game runs on the player's machine, so they can edit memory, patch the JS/WASM or forge
messages. **Hashing the client can't stop that**: the cheater controls the code that computes
and sends the hash. What works:
- **Never trust a client about anything that matters.** Money, claims, mission rewards,
  canonical edits and persistent destruction are decided or checked by the server.
- **Plausibility checks on owned state:** speed, acceleration and teleport limits, owning
  things too far away, rate limits. Violations take the entity's ownership away, then flag,
  then kick.
- **Hashes where they work: state checksums.** In input-only sync every machine hashes its
  state each tick and they compare; a mismatch means a desync or a cheater, and the majority or
  server copy wins (lockstep RTS, rollback fighting games).
- **Cross-checking peers:** when a nearby peer's view of your car disagrees with what you
  report, that is a signal.
- **Server-authoritative modes** for anything competitive (ranked races, PvP).
- Accounts make bans stick.
- Warnings: GTA Online's peer-hosted sessions with no real referee are why it is rife with mod
  menus, so our referee is not optional. Age of Empires desynced despite exhaustive checksums
  from tiny float differences: checksums detect, they don't prove.

## Persistence

Every write names its tier; the tier belongs to the layer the edit goes to, not the system that
made it.

| Tier | Examples | Store | Survives restart | Lifetime |
|---|---|---|---|---|
| T0 ephemeral | debris, gibs, decals, ragdolls | memory | no | seconds to minutes |
| T1 session | mission layer, placed props, fires | server memory, snapshotted every 60 s | best effort | dies with session or mission |
| T2 timed world damage | wrecked bank front, carved voxels, burnt car | SQLite row + voxel diff blob | yes | `reset_policy{ttl, regrow}`, e.g. 1-24 h, eased back |
| T3 canonical | merged overlays | append-only event log + content blobs | forever | reverted only by a new event |
| T4 player-owned | claims, personal layers, inventory, money | SQLite, row per player, session-locked | forever | player export and delete |

- **Storage on a hobby budget:** one SQLite on the server, streamed continuously to Hetzner
  Object Storage by Litestream (point-in-time restore; about 6.49 EUR/month with ~1 TB). Voxel
  diffs and overlay assets are content-addressed blobs in the same bucket. Postgres only if
  several processes must write at once.
- **Canonical history is event-sourced:** `{seq, author, layer, registry_id, op, blob_hash,
  parent}`. State is a fold of the log, so we can revert an author, rewind an area, or re-merge
  after a pipeline rebuild (CoreProtect's rollback, keyed on registry ids). Nightly snapshots,
  30 daily copies in a second bucket.
- **Proposed layer:** a CRDT (Loro, Rust/WASM, movable tree) only where people co-edit one
  proposal. Canonical stays single-writer: the referee's log is linear and merges are events.
- **Player data:** session lock per player (Roblox ProfileService), atomic read-modify-write,
  staggered autosave. Money and rewards are only ever written by the server.
- **Restarts:** T2 and T4 flushed on change, everything flushed on clean shutdown. Say plainly
  that a crash loses T0 and T1 (Second Life region rollback).
- **Resets are design:** shared-world T2 damage decays on a timer; T3 and T4 never wipe.
  Private worlds choose: no reset, or a Rust-style wipe cycle.
- **Griefing:** T2 damage capped per player per area per hour; damage near claimed plots decays
  faster; every T2/T3 write carries its author so rollback-by-user is one query; claim
  interiors are instanced; proposed edits are agent-screened.
- **Saves** (single-player and private): `{world_version, mod_set_hashes, layer events since
  base, ECS snapshot of T1 and T4}`, the same serialiser the mission pressure valve needs.
  Browser: OPFS with export/import. Hosted private worlds: SQLite plus Litestream per world.