perf(gateway): pool backend conns; loadtest evaluate hot path

The loadtest harness never modelled game.evaluate — the debounced per-tile
play preview a real client fires several times per turn, the hottest gameplay
call. Model it (one evaluate per placed tile + reconsideration re-previews +
draft.save, human-paced; --eval / --eval-recon toggle it).

That realistic load surfaced the real bottleneck: the gateway's backend HTTP
client used the default transport (MaxIdleConnsPerHost=2), so every sync call
to the single backend host churned a fresh TCP connection — ~26500 TIME_WAIT
sockets at 500 players (near the ephemeral-port ceiling), burning ~1.75 gateway
cores while the backend sat near-idle. It was the unfixed root of the residual
transport_error the earlier passes chased on the client side.

Widen the keep-alive pool (backendMaxIdleConns=512, ~2x the observed 225-conn
peak). At 500 players the churn collapses to ~0 and peak gateway CPU drops ~7x
(~1.75 -> ~0.26 cores); postgres (~1.65 cores) becomes the busiest service.
This overturns the earlier "gateway is the binding constraint, scale it
horizontally" sizing — that was sizing around this bug, not a real floor.

Consolidate the loadtest trip reports into one loadtest/REPORT.md (drop the
R2/R7 split) and bake the finding into README / PRERELEASE / ARCHITECTURE /
TESTING.

loadtest/REPORT.md

@@ -0,0 +1,184 @@
+# loadtest — stress trip report
+
+The pre-release stress write-up for [`PRERELEASE.md`](../PRERELEASE.md). It drives the
+`scrabble/loadtest` harness against a freshly redeployed test contour to confirm the
+system holds at scale and to settle resource sizing before the prod cutover. The harness
+stays in the repo for repeats; see [`README.md`](README.md) for how to run it.
+
+This report supersedes the earlier per-phase notes. The harness has been through three
+passes: an early diagnostic, a tuning pass that sized container limits / `GOMAXPROCS`, and
+this final pass — which **added the per-tile `game.evaluate` preview to the model** (the
+hottest real gameplay call, previously unmodelled) and, with it, surfaced and fixed the
+**gateway→backend connection-pool bottleneck** described below. The numbers here are from
+that final pass.
+
+## What it models
+
+The harness seeds a large account population with pre-created sessions directly in
+Postgres, then drives virtual players through the **gateway edge protocol** (h2c) in real
+games assembled via the invitation flow. Each player owns its own `edge.Client` (its own
+h2c connection, like a real client), holds a live `Subscribe` stream, and per tick polls
+`game.state`, replays `game.history`, generates a legal **mid-ranked** move with the
+embedded `scrabble-solver`, and submits it (or passes/exchanges). A fraction of ticks
+exercise nudge / chat / check-word / draft / profile / stats. A separate **gateway-hammer**
+floods `games.list` to verify the rate limiter.
+
+### The evaluate hot path (this pass)
+
+A real client previews every tentative play as the user arranges tiles: the UI fires a
+debounced `game.evaluate` (legality + score) on each placement change while it is the
+player's turn. Over a single composed word that is **several evaluate calls per turn** —
+far more than the one `submit_play` — so `game.evaluate` is the single hottest gameplay
+request at scale. The earlier passes did not model it at all (they submitted directly),
+which understated the real load.
+
+This pass models it: when a player composes a play of *K* newly-placed tiles, it fires one
+`evaluate` per landed tile (a growing prefix of the tiles), plus a small number of
+full-composition re-previews for reconsideration, spaced by a human-paced gap (the client's
+250 ms debounce), then one `draft.save`, then `submit_play`. `--eval=false` reproduces the
+pre-evaluate harness for an A/B baseline; `--eval-recon` tunes the reconsideration count.
+
+`game.check_word` is a *different*, manual "look this word up" panel (throttled, on demand)
+— not the per-tile call — and is exercised separately as a secondary op.
+
+## Final run (eval-on, after the connection-pool fix)
+
+Contour: backend / postgres capped at 2 cores / 512 MiB (`GOMAXPROCS=2`), gateway at
+3 cores / 512 MiB (`GOMAXPROCS=3`), per the tuned `deploy/docker-compose.yml`. Gradual ramp
+**50 → 200 → 500** concurrent players, 4 min/step, `--tick 800ms`, gateway-hammer on. The
+harness ran as a one-shot container on `scrabble-internal`, capped at `--cpus 3`. The DB was
+wiped before the run (`DROP SCHEMA backend CASCADE`); the seeded population was removed by
+`--cleanup` afterwards.
+
+Per-operation results at the 500-player peak (740 s, gameplay rows; the hammer row is the
+limiter probe):
+
+| operation | count | req/s | p50 | p99 | max | notes |
+|-----------|------:|------:|----:|----:|----:|-------|
+| game.evaluate | 85 721 | 115.9 | 1 ms | 200 ms | 193 ms | **the hot path** — all ok |
+| game.state | 115 926 | 156.7 | 100 ms | 200 ms | 260 ms | transport_error 86 (0.07 %) |
+| game.history | 22 258 | 30.1 | 5 ms | 100 ms | 195 ms | all ok |
+| draft.save | 23 031 | 31.1 | 2 ms | 200 ms | 194 ms | all ok |
+| game.submit_play | 21 704 | 29.3 | 1 ms | 200 ms | 274 ms | ok 3 902; not_your_turn / illegal_play are concurrent-play races (see caveat) |
+| hammer:games.list | 522 756 | 706.7 | 1 ms | 2 ms | 53 ms | **99.97 % rate_limited** — limiter holds |
+
+- **Volume:** 802 200 total edge calls (1 084 req/s incl. the hammer; ~377 req/s of real
+  gameplay). `stream errors: 0`. Live events: 11 199 `opponent_moved`, 4 153 `your_turn`.
+- **`game.evaluate` is now the dominant gameplay write-path call** at ~116 req/s — second
+  only to the `game.state` poll — and it is cheap: p50 1 ms, p99 200 ms, effectively zero
+  errors. The backend serves it from the in-memory live-game cache plus a single
+  `GetGame` read.
+- **Latency stayed healthy** under the heavier evaluate load: every gameplay op p99 ≤ 200 ms.
+- **The limiter holds** unchanged: 99.97 % of the hammer rejected at p99 2 ms.
+
+### Peak CPU (500 players)
+
+| container | CPU peak | cap |
+|-----------|---------:|----:|
+| scrabble-postgres | **165 %** (~1.65 cores) | 200 % |
+| scrabble-backend | 77 % (~0.77 core) | 200 % |
+| scrabble-gateway | **26 %** (~0.26 core) | 300 % |
+| scrabble-loadtest (harness) | 42 % | 300 % |
+
+Memory stayed modest everywhere (Go services ≤ ~90 MiB). **Postgres is now the busiest
+service** — it has headroom (1.65 of 2 cores) but is the scaling axis. The gateway, after
+the fix below, is near-idle.
+
+## The headline finding: gateway→backend connection churn
+
+The gateway proxies every synchronous client call to the single backend host over REST.
+Its backend HTTP client used the default transport, whose **`MaxIdleConnsPerHost` is 2**
+(`http.DefaultMaxIdleConnsPerHost`). So the gateway kept only **2** keep-alive connections
+to the backend and opened — then closed — a fresh TCP connection for almost every other
+call. Measured at the gateway's network namespace:
+
+| | gateway→backend sockets |
+|---|---|
+| before (eval-on, 500 players) | **TIME_WAIT ≈ 26 500**, ESTABLISHED 2 |
+| after (eval-on, 500 players) | TIME_WAIT ≈ 0 (steady state), **ESTABLISHED ≈ 225 (reused)** |
+
+26 500 TIME_WAIT sockets is the connection **churn**: ~440 new connections per second,
+each a full TCP handshake + teardown, the socket then lingering 60 s. That count sits right
+under the ~28 000 ephemeral-port ceiling — the latent cliff that produced the residual
+`transport_error` the earlier passes chased on the *client* side (h2c streams) but never
+eliminated, because the real cause was here, on the *backend* side.
+
+The fix is one custom `http.Transport` with a wide idle pool
+(`gateway/internal/backendclient/client.go`, `backendMaxIdleConns`). Before / after, same
+eval-on workload at 500 players:
+
+| metric | before fix | after fix |
+|--------|-----------:|----------:|
+| gateway→backend TIME_WAIT | ~26 500 | **~0** |
+| gateway CPU peak | **175 %** (~1.75 cores) | **26 %** (~0.26 core) |
+| game.state p99 | 500 ms | 200 ms |
+
+**The churn was burning ~1.5 gateway cores of pure connection setup/teardown.** Removing it
+cut peak gateway CPU ~7× and erased the port-exhaustion cliff. The backend and postgres CPU
+are unchanged — they do the real work; only the gateway's wasted overhead disappeared. The
+pool settles at ~225 live connections at 500 players; the constant is set to 512 for ~2×
+headroom.
+
+## Sizing — why the old "≈150 concurrent / 2-core" figure was a bug, not a floor
+
+The earlier tuning pass concluded the gateway was the binding constraint — "size it for
+≥ 3 cores per 500 players, scale it horizontally" — and the single-host "minimum" tier
+topped out near ~150 concurrent. **That was sizing around the connection-churn bug.** The
+gateway drew ~1.75–3 cores not from proxying work but from churning backend connections;
+the backend behind it sat near-idle the whole time.
+
+With the churn fixed, at **500 concurrent players** the app draws roughly:
+
+- **gateway ≈ 0.26 core** (was ~3) — no longer the constraint,
+- **backend ≈ 0.77 core**,
+- **postgres ≈ 1.65 cores** — now the busiest, with headroom,
+
+≈ **2.7 app cores total** (down from the ~5.5-core contour peak the tuning pass recorded,
+*and* under a heavier, more realistic workload that now includes `game.evaluate`). Postgres,
+not the gateway, is the scaling axis.
+
+Revised single-host guidance (app + co-resident observability stack on one box):
+
+| tier | CPU | RAM | handles |
+|------|-----|-----|---------|
+| **Minimum** | 2 cores | 2 GiB | comfortably the low hundreds of concurrent — the gateway no longer eats cores; postgres + the observability stack set the limit |
+| **Average** | 4 cores | 4 GiB | 500 concurrent with headroom |
+| **Maximum** | 8 cores | 8 GiB | 500+ with full burst headroom and room to grow |
+
+The gateway's compose limit can drop well below its old 3 cores; it is now connection-pool
+bound, not connection-CPU bound. Memory was never the constraint. Disk is still dominated
+by observability retention (Tempo, Prometheus) + DB growth — unchanged from before.
+
+## Next optimisation (noted, not done)
+
+`game.evaluate` reads `GetGame` from Postgres on **every** call (to re-check seat
+membership and status) before validating against the cached live game. At ~116 evaluate
+req/s on top of the `game.state` / `game.history` reads, that is the bulk of the postgres
+load now. Caching the game metadata alongside the live engine game in
+`backend/internal/game` would cut it, but it touches persistence/cache coherency (a higher
+blast-radius change) and postgres still has headroom, so it is left as a deliberate
+follow-up rather than bundled here.
+
+## Caveat — harness fidelity
+
+The harness's `not_your_turn` and `illegal_play` on `submit_play` are concurrent-play
+artifacts, not system errors: it generates a move from a locally replayed board, and a
+fast opponent (or a transport hiccup) can move between the state fetch and the submit,
+leaving the move out of turn or illegal on the now-changed board. A real client previews
+with `evaluate` and only submits a legal, in-turn play. These rejections are cheap domain
+outcomes (HTTP-ok with a stable code) and do not change the request *load*, which is what
+the run measures. The harness also shares the host CPU with the contour (capped with
+`--cpus`); a fully isolated ceiling on separate hardware remains future work.
+
+## Re-running
+
+From the repo root:
+
+```sh
+docker build -f loadtest/Dockerfile -t scrabble-loadtest .
+docker run --rm --cpus=3 --name scrabble-loadtest --network scrabble-internal \
+  -e POSTGRES_PASSWORD="$TEST_POSTGRES_PASSWORD" scrabble-loadtest run --reset --cleanup
+```
+
+`--eval=false` reproduces the pre-evaluate baseline for comparison. The authoritative hard
+reset of the contour DB remains `DROP SCHEMA backend CASCADE` + a backend restart.