# v2: rendering, asset delivery, audio and browser support

Index: `agents/world-format-v2.md` (renderer decision). Briefs: `agents/v2-e-webgpu.md` (plan),
`agents/v2-g-renderer-spike.md`, `agents/v2-c-bench.md`, `agents/v2-j-audio.md`. Built: first
pixels in `v2/client/` (goal 6), the chunk converter in `v2/build/chunks/`, audio v0 in
`v2/audio/`, `v2/tools/recorder/`, `v2/tools/voice-batch/`; the benchmark harness on branch
`v2-bench`.

## Rendering

- **Renderer (decided 2026-09-26):** three.js WebGPURenderer (TSL). Target WebGPU (Baseline
  since Jan 2026; Firefox support is platform-limited, verify per platform); the WebGL2 backend
  covers the rest. Compute gives GPU culling and indirect draws (v1's 340-3,500 draw calls
  collapse), GPU particles/grass, clustered lighting (every street lamp lights the road), GPU
  timing queries. **Built (goal 6):** three r186 WebGPURenderer + TSL drawing Heaton from
  v2-chunks bundles, real WebGPU headless with no errors, 115 draw calls from the spawn, 857k
  triangles, 2.2 s load. Owner to open it on the M3. Converter follow-up: write the material
  strip as an array texture (512 x 9216 exceeds WebGPU's 8192 default).
- **Lighting:** no hardware ray tracing in the browser. Software tracing works because the city
  is a heightfield plus boxes (the LIDAR sun-shadow march already is one). Next: an SDF or voxel
  copy of the city for bounce light and AO (Lumen software-mode style), plus screen-space GI.
  Destruction voxels and lighting voxels may be the same data. Scope it like Teardown, which
  traces only AO, soft shadows and specular occlusion (no bounce GI) on a non-RT renderer.
- **Clean renderer first (decided 2026-09-26):** a physically based, Source-level-detail
  renderer is the baseline. Stylised looks are optional passes reading the same G-buffer, never
  baked into materials again. PS1 mode survives as one such pass.
  - **Surface-stable fractal dithering** (owner loved it): a dither pattern that sticks to
    surfaces and stays stable as the camera moves. Candidate stylised mode. It is runevision's
    Dither3D (MPL 2.0: only changes to the shader itself must be shared); Unity-only officially,
    but Godot and Shadertoy ports prove it's plain shader maths, so we port it to WGSL.
- **Camera as a sensor (from render.gabrielpenman.com):** render's `/home/gabriel/render/web/src/effects/` and
  `/home/gabriel/render/web/src/sensors/presets.js` model a physical camera: exposure and ISO grain (`SensorEffect`),
  blackbody white balance, aperture-driven depth of field with autofocus "personalities"
  (`autofocus.js`), metering (`metering.js`), volumetric fog with lamp inscattering,
  crepuscular shafts, camera motion blur, Chapman lens flare, depth AO, per-pass GPU timers
  (`/home/gabriel/render/web/src/perf/passTimers.js`). Take the model, not the React/postprocessing plumbing: port the passes
  to TSL. **Night vision** is a sensor preset (high gain, heavy grain, IR sensitivity, green
  phosphor, bloom); **thermal vision** is the same sensor reading the heat field (engines,
  people and burning cars glow; the cold river is black). Depth of field for cutscenes, photo
  mode and aiming, not always on.
- **From the WebGPU audit (2026-09-27, `agents/v2-e-webgpu.md`):**
  - v1 vendors three.js r186; v2 adds `three.webgpu.js` and `three.tsl.js`.
    `ShaderMaterial`/`onBeforeCompile` don't run on WebGPURenderer, so `ps1.js` is a rewrite.
  - **WebGPURenderer can be 2-10x slower** than WebGLRenderer with many unbatched draws
    (three.js #31055): batching, instancing and indirect draws are prerequisites. Its WebGL2
    backend is not a like-for-like fallback.
  - **The PS1 look is not only a post pass:** vertex snapping, per-vertex lighting and
    nearest-texel sampling live in the material. PS1 = low res + post quantise/dither + three
    material switches.
  - **WGSL uniformity rules** reject texture samples and derivatives in per-fragment varying
    branches: v2 materials are build-time variants, not v1's sentinel-driven uber-shader.
  - Stock nodes: TRAANode, GTAONode, SSRNode, BloomNode, DepthOfFieldNode, CSMShadowNode,
    PointsNodeMaterial (WebGPU has no point size above 1).
- **Upscaling for slow machines:** FSR 2/3-style temporal upscaling. WebGL2 path:
  `/home/gabriel/render/web/src/taau/` already built an FSR2-class TAAU (jitter, reprojected history, YCoCg
  variance clipping, RCAS), meant to become the MIT package `three-taau`. WebGPU path: start
  from pmndrs/upscaler (MIT FSR 2/3 port as TSL nodes; check Safari and Firefox first). Both
  need motion vectors, depth, a jitter hook and a reactive mask for particles/UI. FSR 3 frame
  generation has no known browser port: a research risk, not a plan.
- **Benchmark harness (decided 2026-09-26; built 2026-09-27 on branch `v2-bench`,** kept off
  the v2 line because it hooks v1's live runtime): `?bench=all&modes=8,0,6&upscale=nearest,fsr`
  flies fixed paths (Tyne Bridge, Grainger Town, Heaton Road) and records frame, CPU, GPU
  (timer queries), draw calls and triangles per pass, and heap; comparison page at /bench/;
  headless runner build/bench.py (both on that branch). "Pit everything against each other" as a system.
  **Measured v1 (headless, counts only):** PS1 path 340-790 draw calls and 1.7-2.4M triangles a
  frame; **hi-fi up to ~3,500 draw calls and 9.5M triangles**, with shadow cascades (960-1,420
  calls) and the depth prepass (a full repeat of the main pass) the biggest costs; JS heap
  0.85-1.3 GB. On the M3 (owner), add `&sync=1` for a second pass (Safari/macOS Chrome mostly
  lack GPU timer queries).

### Weather and sky rendering (WebGPU, TSL)
- **Sky:** pmndrs/sky (MIT, Hillaire model in TSL, aerial perspective); clouds, stars and
  moon ours.
- **Clouds:** Nubis-style raymarching with a weather map for high settings; a cheap slab above
  ~90% cover, which is most days.
- **Rain:** GPU compute particles occluded by a top-down depth map; splashes; ripple normals
  masked to puddles; surfaces darken and gloss with `wet`. Windscreen drops cleared by the
  wiper (a hinged part entity).
- **Snow:** coverage on up-facing surfaces; tracks rendered from below into a running height
  texture (Arkham Origins); the heat field melts it.
- **Fog:** volumetric fog with lamp inscattering, plus the haar volume.
- **Rainbows:** analytic, when the sun is behind the viewer, below 42°, with rain ahead.

## Asset delivery

**Measured 2026-09-27** (streaming analysis; M = measured, E = estimated):
- **v1 at spawn:** ~84 MB raw / **~56 MB on the wire in ~415 requests** (start-up files 25 MB
  wire incl. `world.json` loaded just for the minimap; Heaton's 254 facade PNGs 24 MB, the
  single biggest cost; near chunks 1.7 MB; mid/far LOD 4 MB). Esri imagery comes from Esri.
- **138 MB of the 308 MB is never fetched** (`streets.json`, `derived.json` are build
  intermediates; `ground/*.jpg` is a fallback only). v2 keeps build intermediates out of the
  served tree.
- nginx gzips JSON/bin at level 5 on the fly, no brotli module, `gzip_static` available, and
  **`Cache-Control: no-cache` on everything**, so returning players re-check ~415 URLs.

**Formats:**
- **Textures: KTX2 ETC1S everywhere** (UASTC is 4-5x bigger than JPEG on the wire). Against
  JPEG, ETC1S is about the same size on the wire; the win is 4-8x less GPU memory and no
  main-thread decode. Against PNG facades it is a big win (photo facade 0.14x, drawn 0.45-0.8x).
  Material id maps stay lossless (PNG or R8 + brotli). Phones drop the top mip instead of
  v1's separate sm/ copies.
- **Terrain stays a raster:** u16 heights per 250 m cell ~10 KB brotli (a float32 mesh is
  883 KB, 13 KB even after meshopt). Whole-map `terrain.bin` 12.5 MB -> 3.9 MB with delta + byte
  planes + brotli; DSM at 4 m ~1.7 MB (E).
- **Binary chunks don't save bytes** over brotli'd JSON (0.25 vs 0.26; already integer cm) but
  pay off in parse time and GC. meshopt on real mesh buffers (kerbs, pavements, roofs).

**Layout:**
- Keep the 250 m grid. Every payload content-addressed (`h/<sha>.bin|.ktx2`), served `public,
  max-age=31536000, immutable`, pre-compressed at build time; one small `world.<ver>.json`
  manifest (cell -> hashes, ~30 KB) served `no-cache`.
- Per cell: L0 near bundle (meshopt buffers, u16 heights, id map, props) + one ETC1S facade
  atlas; L1 mid 500 m tiles with KTX2 ground; L2 far as one ~1 MB pack in the core set.
- **Core (~5 MB, E):** manifest, binary road graph, 10 m global terrain, 4 m DSM, materials,
  tint, far pack. `world.json`/`registry.json` load lazily on first use.
- Nearest-first loading biased to direction of travel; ~60-90 requests instead of ~415. OPFS
  keyed by hash plus a service worker, so a rebuild downloads only changed cells.

| | v1 wire | v2 wire (E, design) |
|---|---|---|
| Whole world as served | ~110 MB fetchable | ~60 MB |
| **First playable (Heaton)** | **~56 MB** | **~20 MB** (core 5, near 1.3, facades ~10 +/-30%, mid 4) |

**Built (`v2/build/chunks/`):** Heaton first playable 42.8 MB / 246 requests (v1, the
converter's test area) to 11.0 MB / 70 (v2): better than the ~20 MB estimate. Prior art: Cesium
3D Tiles, Roblox StreamingEnabled.

## Audio

v1: `public/js/audio.js` synthesises effects with no 3D panning or reverb; `stations.js` has
seeded procedural stations (nothing to license); `radio.js` adds Frisk Radio (a live stream,
CORS-blocked so it can't be filtered) and MC Vypa's uploads (used with permission).
- **Audio is a consumer, never an input (principle 7).** It reads the snapshot and audio
  components (`emitter`, `engine{rpm, load, throttle}`, surfaceprop), reacts to events
  (`OnImpact{impulse, matA, matB}`, `OnBroken`, `OnIgnite`, `PlayVoice`) and queries fields
  (weather, wind, `acoustic_zone`, heat for fire crackle).
- **Mixer in Rust/WASM inside an AudioWorklet** (Firewheel has a web backend), fed from the sim
  worker by a SharedArrayBuffer ring. Native Web Audio only for decode and output: HRTF panner
  and convolver nodes are expensive, so spatialise ourselves. Built: a mixer skeleton (command
  ring + ducking), untested in a browser.
- **Spatial:** cheap pan plus distance filtering for most voices; HRTF only for the nearest 4-8
  (Steam Audio, Apache 2.0, WASM build experimental).
- **Acoustic zones are generated** (built): a build step tags volumes from the registry and
  heightfield (street canyon, park, under a bridge, Metro tunnel, open quay, interior), each
  with a reverb preset. Occlusion: a few round-robin Rapier raycasts per audible voice. Rooms
  and portals (Wwise's model) for interiors and tunnels.
- **Soundscapes are forge entities** (Source `env_soundscape`): a looping bed plus random
  one-shots (gulls, the Metro, pub noise).
- **Physics-driven sound** (choice built): impacts pick a sample by material pair with gain from
  impulse; scrapes loop from contact tangential speed; breaks play the material's break sound;
  engines use granular synthesis scrubbing a recorded RPM sweep (the current oscillator engine
  is the fallback). Vehicle sounds: `vehicles.md`.
- **Budget and LOD:** start at 32-48 mixed voices, 8 HRTF, 2 convolution reverbs, then measure.
  Rank by loudness at the listener; far voices go virtual; tiers full / filtered mono / folded
  into the zone bed. iOS still needs a gesture to unlock.
- **Mixing:** buses (music, dialogue, SFX, ambience, UI); dialogue and explosions duck music
  and ambience; snapshots swap bus settings on events (entering a car, a tunnel, a cutscene).
- **Radio:** a station is a catalogue entity; the car has `radio{station, on}`. Playback
  position is the world clock modulo track length, so everyone in a car hears the same thing
  with nothing networked (built: two synced clients agree within 22 ms). Live streams stay
  local-only. Mods can ship stations (community Geordie DJ packs).
- **Networking:** engines, tyres, wind, scrapes, ambience, radio and footsteps are derived
  locally from replicated state; gunshots, explosions, breaks, voice ids and horns are reliable
  events; small impacts on owned props are rebuilt by each peer.
- Voice line selection (Response Rules): `missions-and-forge.md`; performer rights:
  `platform-and-community.md`.

### Voice recording pipeline (research 2026-09-27; recorder and batch built)
Dynamic USB mics (Samson Q2U, Shure MV7+) 10-15 cm with a pop filter, a duvet behind mic and
actor; a browser recorder page (getUserMedia with processing off, AudioWorklet to 24-bit 48 kHz
WAV, never MediaRecorder's Opus; per-line cards, retakes, immediate resumable upload keyed by
line id, 10 s room tone). Offline batch: trim, DeepFilterNet (MIT/Apache) only if room tone is
above -60 dBFS, loudness ~-20 LUFS / -1 dBTP (per-performer gain; barks are too short for
integrated LUFS), Opus mono 32-40 kbps (built at 36 kb/s; ~6 MB per 500 lines; test Safari Ogg
Opus decode early), Rhubarb `-r phonetic` on the WAV master. Performer register kept apart from
assets (UK GDPR). Simplest minors rule: 16+ only, or unpaid under-16s with a guardian present.
**50 lines in an evening:** four emotion blocks, three reads each, then an ad-lib pass; takes
circled and batch-processed the same night. Uploader still stubbed.

## Browser compatibility (Sept 2026, MDN browser-compat-data and caniuse; not run on devices)

| Feature | Safari macOS / iOS 26 | Chrome desktop / Android | Firefox desktop / Android |
|---|---|---|---|
| WebGPU | 26 (macOS 26 Tahoe only) / 26 | 113 (Linux 144+, Intel Gen12+) / 121 | partial: Windows 141+, Apple silicon on Tahoe; no Intel Mac, no Linux / **no** |
| timestamp-query | contradictory data | yes (rounded) | no |
| WebTransport datagrams | 26.4 | 97 / yes | 114 / yes |
| SharedArrayBuffer (COOP+COEP) | 15.2 | yes | 79 / yes |
| COEP `credentialless` | **no** | 96 | 119 / **no** |
| OPFS sync handle in workers | 15.2 | 102 / 109 | 111 |
| Ogg Opus | macOS 15.4+ / iOS 18.4+ | yes | yes |
| `decodeAudioData` Opus | unverified | yes | yes |
| WASM SIMD, threads | yes | yes | yes |
| WASM memory64 | preview only / no | 133 | 134 |
| getUserMedia noiseSuppression / AGC off | **no** | yes | yes |

**Decisions this forces:**
1. Cross-origin isolation via `COOP: same-origin` + `COEP: require-corp`, loading every
   cross-origin image in CORS mode; don't rely on `credentialless` (Safari, Firefox Android
   lack it). Esri tiles send `Access-Control-Allow-Origin: *` (checked live); our CDN sends
   `Cross-Origin-Resource-Policy: cross-origin`.
2. Isolation breaks the Frisk Radio stream (proxy it or drop it), Google sign-in popups (use
   the redirect flow) and third-party iframes.
3. Voice files: Ogg Opus with an AAC fallback for older Macs; add a real-Safari
   `decodeAudioData` check before committing.
4. The voice recorder page is Chrome/Firefox desktop only (Safari keeps noise suppression and
   AGC on; the page warns).
5. The WASM core stays under 4 GB of 32-bit memory (no memory64 in Safari); relaxed SIMD only
   as a detected optional path.
6. **WebGL2 is a real fallback** with its own performance budget: Firefox Android, Firefox
   Linux, Intel Macs and Safari before Tahoe lack WebGPU.
7. GPU timing is Chrome-first: the bench uses timestamp queries where present, else CPU frame
   time; never a gate.
8. KTX2 transcode order: ASTC, ETC2, BC7, else RGBA8 (Windows/Linux desktops only expose BC).
9. WebTransport ~91% holds; iOS 26.0-26.3 needs the WebSocket fallback.