Merge pull request 'perf(gateway): pool backend conns; loadtest evaluate hot path' (#101) from feature/loadtest-evaluate-hotpath into development

EvaluatePlay (the hottest gameplay call, fired on every tile placement) now uses
the warm live-game cache directly: an active game stays cached (mutated in place
across moves, evicted only on finish), so the cached engine game and its immutable
seat list answer the membership check and the score with no DB read. The cold path
(eviction / first load) still loads and validates via the store. The seat list is
cached alongside the engine game for the membership fast path.

GetGame also folds its two round-trips (game, then seats) into one LEFT JOIN,
preserving the contract (same Game, a seatless game still returns empty seats, seat
order kept) — one round-trip for every remaining caller.

Measured at 500 players: evaluate p99 halves (200 -> 100 ms) and the per-op query
count drops. It does NOT cut postgres CPU — that is write-bound (per-move CommitMove
plus draft upserts and journal replays), the cheap indexed GetGame reads were never
its bottleneck, and postgres runs with headroom (~1.5 of 2 cores). So this is a
latency / query-volume optimization, not a DB-CPU one.

Regression cover: a non-player evaluate against a warm game asserts the cached-seat
membership path; the integration suite exercises GetGame's join across every game op.

PRERELEASE.md

@@ -311,7 +311,7 @@ Then Stage 18.
     hammer (99.97 % rejected, p99 2 ms). **Top finding:** ~14 % `transport_error` on `game.state` at 500
     players, under CPU saturation (backend/gateway/Postgres each ~1 core) and amplified by the harness's
     single shared `http2.Transport`; the harness itself peaked at 86 % of a core on the same host, so the
-    figures are pessimistic. Full trip report in [`../loadtest/REPORT-R2.md`](../loadtest/REPORT-R2.md);
+    figures are pessimistic. Full trip report in [`../loadtest/REPORT.md`](../loadtest/REPORT.md);
     it feeds R3 (h2c `MaxConcurrentStreams`/timeouts, body-size cap), R6 and R7 (per-player transports,
     separate hardware, pool/limit sizing).
   - **CI:** `./loadtest/...` added to the path filter + vet/build/test; `go.work.sum` carries the new deps.
@@ -454,7 +454,12 @@ Then Stage 18.
     one connection per player it bursts into its 2-core cap (the residual 2.49 % `transport_error`); backend
     ~0.85 core and postgres ~1.4 cores had headroom; **tempo reached its 1 GiB cap**; the backend pool sat at
     its `MaxOpenConns=25` cap (28 backends); docker logs were unbounded (~14 MiB / 30 min on the backend at
-    info). Full write-up in [`../loadtest/REPORT-R7.md`](../loadtest/REPORT-R7.md).
+    info). Full write-up in [`../loadtest/REPORT.md`](../loadtest/REPORT.md). *(Superseded in part: a
+    later pass modelling the `game.evaluate` hot path traced the gateway's CPU appetite to
+    **gateway→backend connection churn** — the default 2-idle-connection HTTP transport — not proxying
+    work. Pooling the connections cut peak gateway CPU ~7× (~1.75 → ~0.26 cores at 500 players) and
+    removed the ephemeral-port-exhaustion cliff behind the residual `transport_error`, so the gateway is
+    no longer the binding constraint — postgres is. The 3-core gateway cap below is now generous headroom.)*
   - **Round-2 tuning (owner-agreed, all in `deploy/docker-compose.yml`, no code change):** gateway **2 → 3
     cores + `GOMAXPROCS=3`**; tempo memory **1 → 2 GiB**; backend `MAX_OPEN_CONNS` **25 → 40**; a json-file
     **log-rotation** default (10m × 3) applied contour-wide via a YAML anchor (level stays info).
@@ -464,7 +469,7 @@ Then Stage 18.
     **burst** run (a single 100 → 500 jump) pegged the gateway at 3 cores (≈296 % sustained, 9.27 % error),
     confirming it is **connection-CPU-bound** — a true arrival spike is a **horizontal-scaling** lever, not
     more cores per node (recorded in the prod-sizing recommendation).
-  - **No schema change → no contour DB wipe.** Bake-back: `loadtest/REPORT-R7.md` (new), `loadtest/README.md`,
+  - **No schema change → no contour DB wipe.** Bake-back: `loadtest/REPORT.md`, `loadtest/README.md`,
     `docs/TESTING.md`, the telemetry/observability section of `docs/ARCHITECTURE.md`, the repo-layout line in `CLAUDE.md`.
 
 - **UI — Tab-bar navigation redesign** (owner ad-hoc, not on the raw TODO list): drop the hamburger

backend/internal/game/cache.go

@@ -63,6 +63,7 @@ type gameCache struct {
 
 type cachedGame struct {
 	game       *engine.Game
+	seats      []Seat
 	variant    string
 	lastAccess time.Time
 }
@@ -71,24 +72,27 @@ func newGameCache(ttl time.Duration, now func() time.Time) *gameCache {
 	return &gameCache{entries: make(map[uuid.UUID]*cachedGame), ttl: ttl, now: now}
 }
 
-// get returns the live game for id and refreshes its idle timer, or (nil, false).
-func (c *gameCache) get(id uuid.UUID) (*engine.Game, bool) {
+// get returns the live game and its immutable seat list for id and refreshes its idle
+// timer, or (nil, nil, false). The seats let a read check membership (and label seats)
+// without re-loading the game from the store, since seats never change after a game starts.
+func (c *gameCache) get(id uuid.UUID) (*engine.Game, []Seat, bool) {
 	c.mu.Lock()
 	defer c.mu.Unlock()
 	e, ok := c.entries[id]
 	if !ok {
-		return nil, false
+		return nil, nil, false
 	}
 	e.lastAccess = c.now()
-	return e.game, true
+	return e.game, e.seats, true
 }
 
-// put stores g as the live game for id. variant labels the entry so the active-
-// games gauge can report counts by variant without inspecting engine internals.
-func (c *gameCache) put(id uuid.UUID, g *engine.Game, variant string) {
+// put stores g as the live game for id together with its seat list. variant labels the
+// entry so the active-games gauge can report counts by variant without inspecting engine
+// internals; seats are the game's immutable seat standings for the membership fast path.
+func (c *gameCache) put(id uuid.UUID, g *engine.Game, variant string, seats []Seat) {
 	c.mu.Lock()
 	defer c.mu.Unlock()
-	c.entries[id] = &cachedGame{game: g, variant: variant, lastAccess: c.now()}
+	c.entries[id] = &cachedGame{game: g, seats: seats, variant: variant, lastAccess: c.now()}
 }
 
 // remove drops id from the cache (used on a finished game and after a failed

backend/internal/game/helpers_test.go

@@ -94,8 +94,8 @@ func TestGameCacheEviction(t *testing.T) {
 	cur := time.Unix(1_700_000_000, 0)
 	cache := newGameCache(time.Hour, func() time.Time { return cur })
 	id := uuid.New()
-	cache.put(id, nil, "scrabble_en")
-	if _, ok := cache.get(id); !ok {
+	cache.put(id, nil, "scrabble_en", nil)
+	if _, _, ok := cache.get(id); !ok {
 		t.Fatal("game must be resident after put")
 	}
 	cur = cur.Add(30 * time.Minute)
@@ -104,7 +104,7 @@ func TestGameCacheEviction(t *testing.T) {
 	if n := cache.sweep(); n != 1 {
 		t.Errorf("sweep evicted %d, want 1", n)
 	}
-	if _, ok := cache.get(id); ok {
+	if _, _, ok := cache.get(id); ok {
 		t.Error("game must be evicted after idle TTL")
 	}
 	if cache.size() != 0 {

backend/internal/game/service.go

@@ -287,12 +287,12 @@ func (svc *Service) Create(ctx context.Context, params CreateParams) (Game, erro
 	if err := svc.store.CreateGame(ctx, ins, seats, seeding.draws); err != nil {
 		return Game{}, err
 	}
-	svc.cache.put(id, g, params.Variant.String())
 	svc.metrics.recordStarted(ctx, params.Variant, params.VsAI)
 	created, err := svc.store.GetGame(ctx, id)
 	if err != nil {
 		return Game{}, err
 	}
+	svc.cache.put(id, g, params.Variant.String(), created.Seats)
 	// Honest-AI game seated with a robot: if the robot moves first, reply at once
 	// (the periodic driver is the fallback). No-op for every human-only game.
 	svc.triggerAI(created)
@@ -890,26 +890,35 @@ func (svc *Service) timeoutGame(ctx context.Context, gameID uuid.UUID, now time.
 // EvaluatePlay previews a tentative play for a seated player against the current
 // board without committing it: whether it is legal and what it would score.
 func (svc *Service) EvaluatePlay(ctx context.Context, gameID, accountID uuid.UUID, tiles []engine.TileRecord) (EvalResult, error) {
-	pre, err := svc.store.GetGame(ctx, gameID)
-	if err != nil {
-		return EvalResult{}, err
-	}
-	if _, ok := pre.seatOf(accountID); !ok {
-		return EvalResult{}, ErrNotAPlayer
-	}
-	if pre.Status == StatusFinished {
-		return EvalResult{}, ErrFinished
-	}
-
 	unlock := svc.locks.lock(gameID)
 	defer unlock()
-	g, err := svc.liveGame(ctx, pre)
-	if err != nil {
-		return EvalResult{}, err
+
+	// Hot path: an active game stays cached — the engine game is mutated in place across
+	// moves and evicted only when it finishes — so on a hit the cached live game and its
+	// immutable seat list answer the membership check and the score with no DB read. This
+	// preview is fired on every tile placement, the hottest gameplay call at scale.
+	g, seats, ok := svc.cache.get(gameID)
+	if !ok {
+		// Cold path: load and validate from the store, then replay into the cache.
+		pre, err := svc.store.GetGame(ctx, gameID)
+		if err != nil {
+			return EvalResult{}, err
+		}
+		if pre.Status == StatusFinished {
+			return EvalResult{}, ErrFinished
+		}
+		if g, err = svc.liveGame(ctx, pre); err != nil {
+			return EvalResult{}, err
+		}
+		seats = pre.Seats
 	}
+	if !seatedIn(seats, accountID) {
+		return EvalResult{}, ErrNotAPlayer
+	}
+
 	validateStart := time.Now()
 	rec, err := g.EvaluatePlay(tiles)
-	svc.metrics.recordValidate(ctx, pre.Variant, validateStart)
+	svc.metrics.recordValidate(ctx, g.Variant(), validateStart)
 	if err != nil {
 		if errors.Is(err, engine.ErrIllegalPlay) {
 			return EvalResult{Valid: false}, nil
@@ -1359,7 +1368,7 @@ func (svc *Service) ExportGCG(ctx context.Context, gameID uuid.UUID) (string, er
 // liveGame returns the live engine.Game for pre, rebuilding it from the journal
 // on a cache miss. Callers must hold the per-game lock.
 func (svc *Service) liveGame(ctx context.Context, pre Game) (*engine.Game, error) {
-	if g, ok := svc.cache.get(pre.ID); ok {
+	if g, _, ok := svc.cache.get(pre.ID); ok {
 		return g, nil
 	}
 	g, err := svc.replay(ctx, pre)
@@ -1374,7 +1383,7 @@ func (svc *Service) liveGame(ctx context.Context, pre Game) (*engine.Game, error
 		}
 	}
 	if !g.Over() {
-		svc.cache.put(pre.ID, g, pre.Variant.String())
+		svc.cache.put(pre.ID, g, pre.Variant.String(), pre.Seats)
 	}
 	return g, nil
 }

backend/internal/game/store.go

@@ -355,27 +355,33 @@ func (s *Store) ExpiredOpen(ctx context.Context, now time.Time) ([]OpenGame, err
 // GetGame loads the games row joined with its seats (ordered by seat), or
 // ErrNotFound.
 func (s *Store) GetGame(ctx context.Context, id uuid.UUID) (Game, error) {
-	gstmt := postgres.SELECT(table.Games.AllColumns).
-		FROM(table.Games).
+	// One round-trip: the game joined with its seats. A LEFT JOIN keeps a (would-be)
+	// seatless game returning the game with no seats, exactly as the prior two-query
+	// version did; ORDER BY seat preserves seat order. The games columns repeat per seat
+	// row — cheap at 2-4 seats, and one round-trip instead of two, which matters because
+	// GetGame is the universal "load the game" step on every game operation.
+	stmt := postgres.SELECT(table.Games.AllColumns, table.GamePlayers.AllColumns).
+		FROM(table.Games.LEFT_JOIN(table.GamePlayers, table.GamePlayers.GameID.EQ(table.Games.GameID))).
 		WHERE(table.Games.GameID.EQ(postgres.UUID(id))).
-		LIMIT(1)
-	var grow model.Games
-	if err := gstmt.QueryContext(ctx, s.db, &grow); err != nil {
-		if errors.Is(err, qrm.ErrNoRows) {
-			return Game{}, ErrNotFound
-		}
+		ORDER_BY(table.GamePlayers.Seat.ASC())
+	var rows []struct {
+		model.Games
+		model.GamePlayers
+	}
+	if err := stmt.QueryContext(ctx, s.db, &rows); err != nil {
 		return Game{}, fmt.Errorf("game: get %s: %w", id, err)
 	}
-
-	sstmt := postgres.SELECT(table.GamePlayers.AllColumns).
-		FROM(table.GamePlayers).
-		WHERE(table.GamePlayers.GameID.EQ(postgres.UUID(id))).
-		ORDER_BY(table.GamePlayers.Seat.ASC())
-	var srows []model.GamePlayers
-	if err := sstmt.QueryContext(ctx, s.db, &srows); err != nil {
-		return Game{}, fmt.Errorf("game: get seats %s: %w", id, err)
+	if len(rows) == 0 {
+		return Game{}, ErrNotFound
 	}
-	return projectGame(grow, srows)
+	seats := make([]model.GamePlayers, 0, len(rows))
+	for i := range rows {
+		// Skip the phantom all-NULL seat row a LEFT JOIN yields for a seatless game.
+		if rows[i].GamePlayers.GameID == id {
+			seats = append(seats, rows[i].GamePlayers)
+		}
+	}
+	return projectGame(rows[0].Games, seats)
 }
 
 // GetGameVariant reads just a game's variant — a cheap single-column lookup the edge uses

backend/internal/game/types.go

@@ -184,6 +184,18 @@ func (g Game) seatOf(accountID uuid.UUID) (int, bool) {
 	return 0, false
 }
 
+// seatedIn reports whether accountID holds a seat in seats. It backs the read-side
+// membership check against the cached, immutable seat list, so a hot read can skip
+// loading the game from the store.
+func seatedIn(seats []Seat, accountID uuid.UUID) bool {
+	for _, s := range seats {
+		if s.AccountID == accountID {
+			return true
+		}
+	}
+	return false
+}
+
 // MoveResult is the outcome of a committed transition: the decoded move and the
 // post-move game, plus the actor's own refilled rack and the bag size after the draw
 // (Rack/BagLen), so the mover renders the next state from the response without a

backend/internal/inttest/game_test.go

@@ -543,6 +543,12 @@ func TestEvaluatePlayPreview(t *testing.T) {
 	if bad.Valid {
 		t.Error("disconnected play must be invalid")
 	}
+
+	// A non-seated account cannot preview: with the game warm in the live cache, the
+	// membership check runs against the cached seat list (the hot path that skips GetGame).
+	if _, err := svc.EvaluatePlay(ctx, g.ID, provisionAccount(t), hint.Tiles); !errors.Is(err, game.ErrNotAPlayer) {
+		t.Errorf("evaluate by a non-player = %v, want ErrNotAPlayer", err)
+	}
 }
 
 // TestConcurrentSubmitSerialized confirms the per-game lock lets only one of two

docs/ARCHITECTURE.md

@@ -128,7 +128,11 @@ dropped). Horizontal scaling is explicit future work.
   and GCG are unaffected** (they stay decoded concrete characters, §9.1).
 - **gateway ↔ backend (sync)**: plain HTTP REST/JSON. The gateway injects
   `X-User-ID` for authenticated requests; `backend` never re-derives identity
-  from the body.
+  from the body. Because every sync call targets the one backend host, the
+  gateway's REST client widens its keep-alive pool well past the stdlib default
+  of 2 idle connections per host; otherwise the per-request connection churn
+  exhausts ephemeral ports and burns gateway CPU under load (see
+  [`../loadtest/REPORT.md`](../loadtest/REPORT.md)).
 - **backend → gateway (live)**: a single gRPC server-stream carries live events
   (your-turn, opponent-moved, chat, nudge). The gateway bridges them to the
   client's in-app stream while the app is open. Out-of-app delivery uses

docs/TESTING.md

@@ -133,9 +133,9 @@ tests or touching CI.
   engine tests do). It is **not** part of the per-PR suite's behavioural assertions: it
   runs ad hoc as a one-shot container against the contour, producing a trip report (bugs
   + a per-container resource profile) read off the **otelcol `docker_stats` +
-  postgres_exporter** Grafana dashboard on the contour. Two passes are recorded — the
-  early [`REPORT-R2.md`](../loadtest/REPORT-R2.md) and the final, tuned
-  [`REPORT-R7.md`](../loadtest/REPORT-R7.md). See [`../loadtest/README.md`](../loadtest/README.md).
+  postgres_exporter** Grafana dashboard on the contour. The findings — including the
+  `game.evaluate` hot-path model and the gateway→backend connection-pool fix — are written
+  up in [`REPORT.md`](../loadtest/REPORT.md). See [`../loadtest/README.md`](../loadtest/README.md).
 - **User feedback** — `internal/feedback` unit tests cover the attachment allow-list /
   content-type and the channel normaliser; the UI covers `detectChannel`, the attachment gate and
   the feedback wire round-trip (`channel` / `feedback` / `codec` tests) plus a Playwright e2e

gateway/internal/backendclient/client.go

@@ -22,6 +22,19 @@ import (
 	pushv1 "scrabble/pkg/proto/push/v1"
 )
 
+// backendMaxIdleConns sizes the REST keep-alive pool to the single backend host. The
+// default transport caps idle connections per host at 2 (http.DefaultMaxIdleConnsPerHost),
+// which — since every synchronous client call proxies to that one host — forces a fresh
+// TCP connection (and a lingering TIME_WAIT socket) for almost every request under load.
+// That connection churn burns gateway CPU and exhausts ephemeral ports at scale, all
+// while the backend itself sits near-idle. Pooling the connections lets them be reused.
+//
+// The stress harness measured the effect at 500 concurrent players: the churn collapsed
+// from ~26 500 TIME_WAIT sockets to ~0 and peak gateway CPU from ~1.75 to ~0.26 cores,
+// with the pool settling at ~225 live connections. 512 keeps ~2x headroom over that
+// observed peak so a burst never re-caps the pool. See loadtest/REPORT.md.
+const backendMaxIdleConns = 512
+
 // Client calls the backend's REST API and opens its push gRPC stream.
 type Client struct {
 	baseURL string
@@ -41,9 +54,14 @@ func New(httpURL, grpcAddr string, timeout time.Duration) (*Client, error) {
 	if err != nil {
 		return nil, fmt.Errorf("backendclient: dial push %s: %w", grpcAddr, err)
 	}
+	// Clone the default transport (keeping its proxy, dialer and timeouts) and widen the
+	// idle pool so REST calls to the backend reuse connections instead of churning them.
+	transport := http.DefaultTransport.(*http.Transport).Clone()
+	transport.MaxIdleConns = backendMaxIdleConns
+	transport.MaxIdleConnsPerHost = backendMaxIdleConns
 	return &Client{
 		baseURL: strings.TrimRight(httpURL, "/"),
-		http:    &http.Client{Timeout: timeout},
+		http:    &http.Client{Timeout: timeout, Transport: transport},
 		conn:    conn,
 		push:    pushv1.NewPushClient(conn),
 	}, nil

gateway/internal/backendclient/client_internal_test.go

@@ -0,0 +1,31 @@
+package backendclient
+
+import (
+	"net/http"
+	"testing"
+	"time"
+)
+
+// TestBackendTransportPoolsConnections guards the fix for the gateway->backend
+// connection churn. Every synchronous client call proxies to the single backend host,
+// so the REST client must widen the idle-connection pool past the default per-host cap
+// of 2 (http.DefaultMaxIdleConnsPerHost) — otherwise almost every request under load
+// opens a fresh TCP connection that then lingers in TIME_WAIT, burning gateway CPU and
+// exhausting ephemeral ports. Reverting to the default transport (`&http.Client{...}`
+// with no Transport) would silently reintroduce that, so assert the pool is widened.
+func TestBackendTransportPoolsConnections(t *testing.T) {
+	c, err := New("http://backend.invalid", "localhost:9090", time.Second)
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer func() { _ = c.Close() }()
+
+	tr, ok := c.http.Transport.(*http.Transport)
+	if !ok {
+		t.Fatalf("REST transport = %T, want a *http.Transport with a widened idle pool", c.http.Transport)
+	}
+	if tr.MaxIdleConnsPerHost <= http.DefaultMaxIdleConnsPerHost {
+		t.Errorf("MaxIdleConnsPerHost = %d, want > default %d (else per-call connection churn)",
+			tr.MaxIdleConnsPerHost, http.DefaultMaxIdleConnsPerHost)
+	}
+}

loadtest/README.md

@@ -15,10 +15,12 @@ and prints a trip-report summary. It stays in the repo for repeats.
 2. **Drive** (edge protocol over h2c): assembles real 2–4 player games via the
    invitation flow (`invitation.create` → `invitation.accept`, no robots), then runs
    each player's turn loop — poll `game.state`, replay `game.history`, generate a legal
-   **mid-ranked** move with the embedded `scrabble-solver`, and `game.submit_play`
-   (or pass/exchange). A fraction of turns exercise nudge / chat / check-word / draft /
-   profile-update / stats. Each player also holds a live `Subscribe` stream. The
-   moderate ramp is **50 → 200 → 500** concurrent players, ~12 min per step.
+   **mid-ranked** move with the embedded `scrabble-solver`, **compose it tile by tile with
+   the debounced `game.evaluate` preview a real client fires** (the hottest gameplay call),
+   persist a `draft.save`, and `game.submit_play` (or pass/exchange). A fraction of turns
+   exercise nudge / chat / check-word / draft / profile-update / stats. Each player also
+   holds a live `Subscribe` stream. The moderate ramp is **50 → 200 → 500** concurrent
+   players, ~12 min per step. `--eval=false` drops the evaluate model for an A/B baseline.
 3. **Hammer**: drives `games.list` from one account far above the per-user rate limit
    to verify the limiter holds (`rate_limited` results) and measure its cost.
 4. **Report**: per-operation latency percentiles, throughput, result-code breakdown,
@@ -72,6 +74,8 @@ Key `run` flags (env in parentheses):
 | `--games-per-player` | `0` (random 3–5) | target concurrent games per player |
 | `--tick` | `800ms` | per-player op cadence (keeps a player under the per-user limit) |
 | `--secondary-prob` | `0.08` | chance per tick of a non-move op |
+| `--eval` | `true` | model the per-tile `game.evaluate` preview (the gameplay hot path); `false` reproduces the pre-evaluate harness |
+| `--eval-recon` | `1` | extra full-composition evaluate re-previews per play (reconsideration), beyond one per placed tile |
 | `--hammer-workers` / `--hammer-dur` | `20` / `15s` | gateway-hammer (0 workers disables) |
 | `--reset` / `--cleanup` | `false` | delete harness rows before / after the run |
 
@@ -93,11 +97,11 @@ runs unconditionally. Use an **absolute** path (here via `$PWD`): `go test ./loa
 runs each package from its own directory, so a relative `BACKEND_DICT_DIR` would not
 resolve.
 
-## Trip reports
+## Trip report
 
-The two stress passes are written up in the repo: the early pass in
-[`REPORT-R2.md`](REPORT-R2.md) and the final, tuned pass in
-[`REPORT-R7.md`](REPORT-R7.md).
+The stress findings — the final run, the `game.evaluate` hot-path model, the
+gateway→backend connection-pool fix, and the revised sizing — are written up in
+[`REPORT.md`](REPORT.md).
 
 ## Caveat
 

loadtest/REPORT-R2.md

@@ -1,162 +0,0 @@
-# R2 — early stress-run trip report
-
-The early stress pass for `PRERELEASE.md` R2. It exercises the system through the
-**edge protocol** with the `scrabble/loadtest` harness, to surface logic/concurrency
-bugs and capture a resource baseline that feeds R3 (edge hardening), R6 (refactor) and
-R7 (final tuning). Pass bar: **diagnostic** — the run "passes" by completing without the
-harness crashing; findings are recorded below, not gated.
-
-## Method
-
-- **Driver:** the `scrabble/loadtest` module, run as a one-shot container on the
-  `scrabble-internal` docker network (reaching `postgres:5432` and `gateway:8081`
-  directly, bypassing the host→gateway hairpin).
-- **Seed:** 10 000 durable + 1 000 guest accounts with pre-created sessions written
-  directly to Postgres (token hash matches `backend/internal/session`), so the driver
-  authenticates without the per-IP-limited auth ops.
-- **Games:** assembled through the real **invitation** flow (`invitation.create` →
-  `invitation.accept`), 2–4 players each, no robots; variants spread over
-  scrabble_en / scrabble_ru / erudit_ru.
-- **Play:** each virtual player holds a live `Subscribe` stream and, per tick, polls
-  `game.state`, replays `game.history` and submits a **mid-ranked** legal move generated
-  locally by the embedded `scrabble-solver` (the edge carries no board), or
-  passes/exchanges; a fraction exercise nudge / chat / check-word / draft / profile /
-  stats. A separate **gateway-hammer** floods `games.list` from one account.
-- **Scale:** moderate ramp **50 → 200 → 500** concurrent players, 10 min/step (the
-  agreed moderate profile; harness and contour share this host's CPU).
-- **Resource capture:** `docker stats` (docker API) sampled every 28 s for per-container
-  CPU/memory; Prometheus for edge latency/throughput, `postgres_exporter` internals and
-  per-service Go runtime metrics.
-
-## Run configuration
-
-```
-loadtest run --durable 10000 --guest 1000 --steps 50,200,500 --step-dur 10m \
-             --tick 800ms --hammer-workers 20 --hammer-dur 15s --cleanup
-```
-
-Date: 2026-06-09. Contour: the R1-baseline schema, freshly deployed with the R2
-exporters. Seeded population removed by `--cleanup` afterwards.
-
-## Findings
-
-### Validated (fixed within R2)
-- **Harness draft payload.** `draft.save` first returned `bad_request`: the backend
-  draft DTO's `rack_order` is a string (the harness sent `[]`). Fixed → `ok`.
-- **Harness profile marker.** `profile.update` first returned `invalid_profile`: the
-  editable-display-name validator (`backend/internal/account/profile.go`) forbids digits
-  and colons, but the seed marker was `lt:…`. Switched the marker to a distinctive
-  letters-only string → `ok`. Cleanup still matches it.
-
-### By-design behaviour (correctly exercised, not bugs)
-- **`chat_not_your_turn`** — chat is gated to the sender's turn
-  (`backend/internal/social/chat.go`); off-turn posts are correctly rejected.
-- **`nudge_own_turn`** — you nudge the player whose turn it is, so a nudge on your own
-  turn is correctly rejected. The harness nudges/chats at random ticks, so a share of
-  these codes is expected.
-
-### Observability gap (key R7 input)
-- **cAdvisor yields only the root cgroup on the contour host.** Its docker factory
-  registers, but per-container init fails — `failed to identify the read-write layer ID
-  … /rootfs/var/lib/docker/image/overlayfs/…: no such file or directory` — because this
-  host's `/var/lib/docker` is a **separate XFS mount** not visible under cAdvisor's
-  `/rootfs` bind (the existing galaxy deployment on the same host has the same
-  limitation). So the **Scrabble — Resources** dashboard's per-container panels are empty
-  here, and per-container CPU/RSS for this run was captured via `docker stats` instead.
-  Postgres internals (`postgres_exporter`) and per-service Go runtime metrics
-  (`go_*` by `service_name`) work. **Recommendation for R7:** adopt the otelcol
-  **`docker_stats`** receiver (already the contrib image) — it reads per-container stats
-  via the docker API with no cgroup dependency — and/or run the final pass on hardware
-  where cAdvisor resolves containers. (Decision to confirm with the owner.)
-
-### Run results
-
-The ramp ran clean to 500 players with no harness crash, no deadlock and
-`stream errors: 0`; cleanup removed all 11 000 seeded accounts (and their ~941 games).
-
-- **Ramp:** step 1 = 50 players / 90 games, step 2 = 200 / 282, step 3 = 500 / 569.
-- **Volume (30 min):** 1.20 M total edge calls, 659 req/s average. Real gameplay at
-  scale: **48 870 committed plays**, 52 772 `your_turn` + 159 631 `opponent_moved`
-  events, **2 798 games finished**.
-- **Latency under load (peak, step 3):** `game.state` p50 ≈ 100 ms, p90/p99 in the
-  200–500 ms buckets, max 849 ms; `game.submit_play` similar (p99 ≤ 500 ms, max 490 ms).
-  Lobby ops stayed fast (invitation/games.list p99 ≤ 10 ms).
-- **Rate limiter holds.** The gateway-hammer sent 522 667 `games.list` from one account;
-  **522 486 (99.97 %) were `rate_limited`**, only 135 `ok` (the burst). Rejections are
-  cheap — p99 = 2 ms — and the gateway sustained ~16 k req/s of rejections during the
-  flood. The per-user limiter behaves as designed (R3 input: the cost is negligible).
-
-**Top finding — `transport_error` under saturation.** At 500 players ~14 % of
-`game.state` calls (72 429 / 519 067) and a few % of the other ops returned a Connect
-`transport_error` (not a domain code). It correlates with the CPU saturation below: the
-backend/gateway are pinned near one core each while the host also runs the 86 %-core
-harness, so the edge sheds load (resets/timeouts) at the knee. It is **amplified by a
-harness artifact** — all 500 virtual players multiplex over a *single* shared
-`http2.Transport`, so 500 persistent `Subscribe` streams plus Execute calls press on one
-HTTP/2 connection's concurrent-stream limit; real clients each use their own connection.
-**Actions:** R7 harness — give each player (or a pool) its own transport, and run on
-hardware not shared with the contour; R3 — confirm the gateway's h2c
-`MaxConcurrentStreams` and edge timeouts are sized for many persistent streams.
-
-**Minor findings:**
-- `unauthenticated` on a tiny share (188 / 519 067 `game.state`, ~0.04 %) — transient
-  session-resolve failures under load; worth a glance in R3 but not material.
-- one `internal` on `game.pass` (1 / 4 788).
-- `game_finished` dominates `chat.nudge`/`chat.post` (≈ 3 900 each): the harness keeps
-  secondary ops on games that already ended. Harness refinement — drop finished games
-  from the rotation (R7).
-- `nudge_own_turn` / `chat_not_your_turn` / `nudge_too_soon` are the expected turn/rate
-  gates, correctly exercised.
-
-## Resource baseline
-
-Per-container peak during step 3 (500 players), from `docker stats`:
-
-| container | peak CPU | memory |
-|-----------|---------:|-------:|
-| scrabble-backend | **99 %** (~1 core) | 91 MiB |
-| scrabble-gateway | **93 %** | 76 MiB |
-| scrabble-postgres | **90 %** | 69 MiB |
-| scrabble-loadtest (harness) | **86 %** | 42 MiB |
-| scrabble-otelcol | 10 % | 110 MiB |
-| scrabble-tempo | 9 % | 446 MiB |
-| prometheus / postgres-exporter | ~0 % | 46 / 16 MiB |
-
-- **The contour is CPU-bound at 500 concurrent players:** backend, gateway and Postgres
-  each saturate ~1 core (single-instance MVP config), so the system draws ~3 cores at
-  this scale; memory is modest (≤ 100 MiB per Go service). This is the sizing input for
-  R7 (pool sizes, GOMAXPROCS, container limits) and the prod cutover.
-- **Caveat:** the harness itself peaked at **86 % of a core** on the *same host*, so the
-  step-3 latency and `transport_error` figures are pessimistic — the contour competed
-  with the generator for CPU. A clean ceiling needs separate hardware (R7).
-- **Postgres:** peak 28 backend connections, ~5 581 commits/s at the peak, **100 % cache
-  hit ratio** (no disk reads) — the DB was comfortable; CPU, not I/O, is its limit here.
-- **Goroutines:** backend 638, gateway **1 698** (it holds the 500 `Subscribe` streams +
-  per-request goroutines), telegram 49 — all stable, no leak across the ramp.
-
-## Recommendations feeding later phases
-- **R3 (edge hardening):** the per-user limiter holds (99.97 % rejected, p99 2 ms) — add
-  the per-IP body-size cap on top. Investigate the **~14 % `transport_error` on
-  `game.state` at 500 players**: confirm the gateway h2c `MaxConcurrentStreams` and edge
-  read/write timeouts are sized for many persistent `Subscribe` streams, and glance at the
-  ~0.04 % transient `unauthenticated` resolves under load.
-- **R6 (refactor):** no logic bug forced a code change beyond the two harness-payload
-  fixes; the run surfaced no deadlock or goroutine leak across the ramp.
-- **R7 (final tuning + stress):** (1) fix the per-container observability gap — adopt the
-  otelcol `docker_stats` receiver so Grafana shows per-container CPU/RSS on the contour;
-  (2) refine the harness — per-player/pooled transports and dropping finished games from
-  the rotation — and run on hardware **not** shared with the contour; (3) size pools /
-  GOMAXPROCS / container limits from the CPU-bound peak (~1 core each for backend, gateway,
-  Postgres at 500 players).
-
-## Re-running
-
-See [`README.md`](README.md). Briefly, from the repo root:
-
-```sh
-docker build -f loadtest/Dockerfile -t scrabble-loadtest .
-docker run --rm --name scrabble-loadtest --network scrabble-internal \
-  -e POSTGRES_PASSWORD=… scrabble-loadtest run    # add --reset on a re-run
-```
-
-The harness stays in the repo for the R7 repeat.

loadtest/REPORT-R7.md

@@ -1,212 +0,0 @@
-# R7 — final stress-run trip report
-
-The final pre-release stress pass for [`PRERELEASE.md`](../PRERELEASE.md) R7. It re-runs
-the R2 harness (`scrabble/loadtest`) against the **final, refactored system** on a
-freshly redeployed contour, to confirm the system holds at scale and to settle the
-resource sizing (container limits, `GOMAXPROCS`, pools, rate limits, log levels) before
-the Stage 18 prod cutover. Pass bar: **diagnostic + a tuning decision** — the run
-"passes" by completing cleanly; the per-container resource profile drives the tuning
-recorded below. Companion to the early pass, [`REPORT-R2.md`](REPORT-R2.md).
-
-## What changed since the R2 pass
-
-- **Harness — per-player transports.** Each virtual player now owns its `edge.Client`
-  (its own `http2.Transport` / h2c connection carrying both its `Subscribe` stream and
-  its `Execute` calls), instead of all players multiplexing over one shared transport.
-  R2 traced the ~14 % `transport_error` on `game.state` at 500 players to that single
-  shared connection's stream limit; per-player connections mirror real clients and
-  remove the artifact, so this pass measures the system, not the harness.
-- **Harness — drop finished games.** `playTurn` reports a finished game and the player
-  drops it from its rotation, so secondary ops stop hitting `game_finished` on ended
-  games (the other R2 harness finding).
-- **Observability — otelcol `docker_stats`.** cAdvisor (which resolves only the root
-  cgroup on this host — separate-XFS `/var/lib/docker`) is replaced by the otelcol
-  `docker_stats` receiver, reading per-container CPU/memory/network from the Docker API.
-  Per-container panels now populate on the contour host. (`api_version` pinned to 1.44;
-  the daemon's minimum is 1.40.)
-- **Contour — container limits + `GOMAXPROCS`.** `deploy.resources.limits` now bound
-  every service; the Go services pin `GOMAXPROCS` to their CPU limit so the runtime
-  matches the cgroup quota. Starting values were generous over the R2 peak; this pass
-  validates them and settles the agreed sizing (below).
-
-## Method
-
-Unchanged from R2 except for the per-player transports and the dropped-finished-games
-refinement above:
-
-- **Driver:** the `scrabble/loadtest` module, run as a one-shot container on the
-  `scrabble-internal` docker network (reaching `postgres:5432` / `gateway:8081`
-  directly), capped at `--cpus 3` so the contour keeps the host's spare cores.
-- **Seed:** 10 000 durable + 1 000 guest accounts with pre-created sessions written
-  straight to Postgres (token hash matches `backend/internal/session`).
-- **Games:** assembled through the real **invitation** flow, 2–4 players each, no
-  robots; variants over scrabble_en / scrabble_ru / erudit_ru.
-- **Play:** each player holds a live `Subscribe` stream and, per tick, polls
-  `game.state`, replays `game.history` and submits a **mid-ranked** legal move generated
-  locally by the embedded `scrabble-solver`, or passes / exchanges; a fraction exercise
-  nudge / chat / check-word / draft / profile / stats. A separate **gateway-hammer**
-  floods `games.list` from one account.
-- **Scale:** the same moderate ramp **50 → 200 → 500** concurrent players, 10 min/step.
-- **Resource capture:** `docker stats` (docker API) sampled every ~20 s for per-container
-  CPU/memory; the otelcol **`docker_stats`** receiver → Prometheus → the Grafana
-  **Scrabble — Resources** dashboard for the same per-container series; `postgres_exporter`
-  internals and per-service Go runtime metrics.
-
-## Run configuration
-
-```
-docker run --rm --cpus=3 --name scrabble-loadtest --network scrabble-internal \
-  -e POSTGRES_PASSWORD=… scrabble-loadtest \
-  run --durable 10000 --guest 1000 --steps 50,200,500 --step-dur 10m \
-      --tick 800ms --hammer-workers 20 --hammer-dur 15s --reset --cleanup
-```
-
-Date: 2026-06-10. Contour: the R1-baseline schema, freshly redeployed with the R7
-container limits / `GOMAXPROCS` (backend/gateway/postgres capped at 2 cores + 512 MiB,
-`GOMAXPROCS=2`) and the `docker_stats` observability. Seeded population removed by
-`--cleanup` afterwards.
-
-## Findings
-
-The ramp ran clean to 500 players — no harness crash, no deadlock, `stream errors: 0` —
-and cleanup removed all 11 000 seeded accounts.
-
-- **Volume (1827 s):** 821 680 edge calls (449.7 req/s incl. the hammer). Real gameplay
-  at scale: **50 916 committed plays**, 4 817 passes, 2 931 games finished; 165 755
-  `opponent_moved` + 54 864 `your_turn` events.
-- **The per-player transport fix worked.** `game.state` returned `transport_error` on
-  **3 173 / 127 403 = 2.49 %** of calls — down from R2's ~14 % on the same step. Other
-  ops were lower still (`game.history` 0.43 %, `game.submit_play` 0.28 %). The residual
-  is the gateway bursting into its 2-core cap (see the profile below), not the harness.
-- **Dropping finished games worked.** `game_finished` on `chat.nudge` / `chat.post` fell
-  to **35 / 36** (R2: ≈ 3 900 each) — secondary ops no longer hammer ended games.
-- **The limiter holds.** The gateway-hammer sent 565 152 `games.list`; **564 979
-  (99.97 %) were `rate_limited`** (154 ok burst, 19 deadline), p99 = 2 ms, ~309 req/s of
-  rejections sustained — unchanged from R2.
-- **Latency (peak):** `game.state` p50 ≈ 100 ms, p99 in the 2000 ms bucket (max 2549 ms);
-  `game.submit_play` p50 100 / p99 1000 ms bucket. Lobby ops stayed fast
-  (invitation / games.list p99 ≤ 10 ms). The p99 tail correlates with the gateway
-  burst-throttling, not the backend (which stayed at ~0.85 core).
-
-## Resource profile
-
-Per-container peak during step 3 (500 players), with the R7 starting limits in force
-(backend/gateway/postgres capped at 2 cores / 512 MiB). Two CPU columns: `docker stats`
-samples a ~1 s window (catches bursts); the otelcol `docker_stats` receiver averages over
-its 30 s collection interval (smooths them) — they agree within sampling error, which
-validates the new observability path.
-
-| container | CPU burst (1 s) | CPU sustained (30 s) | CPU cap | mem peak | mem cap |
-|-----------|----------------:|---------------------:|--------:|---------:|--------:|
-| scrabble-gateway | **217 %** (at cap) | ~145 % | 200 % | 167 MiB | 512 MiB |
-| scrabble-postgres | 138 % | ~153 % | 200 % | 117 MiB | 512 MiB |
-| scrabble-backend | 85 % | ~89 % | 200 % | 116 MiB | 512 MiB |
-| scrabble-tempo | 33 % | — | (none) | **1024 MiB** (at cap) | 1024 MiB |
-| scrabble-otelcol | 11 % | — | (none) | 131 MiB | 512 MiB |
-| scrabble-loadtest (harness) | 157 % | — | 300 % | 369 MiB | — |
-
-- **The gateway is the binding constraint.** With one h2c connection per player it draws
-  ~1.45 cores sustained and **bursts to its 2-core cap** at 500 players, throttling
-  briefly — the source of the 2.49 % `transport_error`. R2 saw only ~0.93 core because
-  all 500 players shared one connection; the +~0.5 core is the realistic per-connection
-  overhead (500 separate HTTP/2 connections). This is a sizing fact, not a regression.
-- **backend is over-provisioned** (~0.85 core vs a 2-core cap); **postgres** (~1.4 cores)
-  has headroom; both stayed ≤ 120 MiB.
-- **tempo reached its 1 GiB memory cap** (R2: 446 MiB) — an OOM risk under sustained
-  tracing.
-- **Postgres backends peaked at 28**, with the backend pool at its `MaxOpenConns=25` cap.
-  Cache hit stayed ~100 % (no disk reads); CPU, not I/O, is the limit.
-- **docker log volume (30 min):** backend 14.2 MiB, gateway 4.6 MiB, postgres 0.04 MiB —
-  the backend's per-request latency line at info dominates, and json-file logs had no
-  rotation.
-
-## Tuning applied
-
-Agreed from the profile (all in `deploy/docker-compose.yml`; no code change — the pool
-is already env-driven):
-
-| knob | from | to | why |
-|------|------|----|-----|
-| gateway CPU + `GOMAXPROCS` | 2 cores / 2 | **3 cores / 3** | it bursts into the 2-core cap at 500 players (the 2.49 % `transport_error`); 3 absorbs the bursts |
-| tempo memory | 1 GiB | **2 GiB** | it reached the 1 GiB cap (OOM risk) |
-| backend `MAX_OPEN_CONNS` | 25 | **40** | the pool sat at its 25-conn cap at peak; headroom trims the p99 tail |
-| docker logs | unbounded | **json-file 10m × 3** | bound the ~14 MiB / 30 min backend log; level stays `info` |
-
-Left as-is: backend / postgres at 2 cores / 512 MiB (peak ~0.85 / ~1.4 cores — headroom
-is cheap on the shared host); the per-user rate limiter and `h2cMaxConcurrentStreams=250`
-(per-connection now, ~1 stream each — ample) and cache TTLs (no pressure observed).
-
-### Validation re-run
-
-Re-running the **same gradual ramp** (50 → 200 → 500) on the tuned contour confirms the
-fix:
-
-- **`game.state` `transport_error` fell to 0.72 %** (853 / 119 051), down from 2.49 % at
-  2 cores. The latency tail also improved — p99 in the 1000 ms bucket, max 1220 ms (was
-  the 2000 ms bucket, max 2549 ms).
-- The **gateway peaked at ~2 cores** (≈196 % on the 30 s gauge) — now comfortably **under
-  the 3-core cap**, so it no longer throttles. backend ~1 core, postgres ~1.3 cores.
-- **tempo peaked at ~1.27 GiB** — under the new 2 GiB cap (it would have OOM-ed at 1 GiB).
-- Drop-finished still holds (`game_finished` on chat 41/42); the limiter still rejects
-  99.97 % of the hammer at p99 2 ms; `stream errors: 0`.
-
-A separate **burst stress** (a single 100 → 500 jump — 400 players connecting at once)
-**pegged the gateway at 3 cores** (≈296 % sustained) and pushed `game.state`
-`transport_error` to 9.27 %. The gateway is **connection-CPU-bound and bursty**: average
-load is ~1 core, but a mass-simultaneous connection storm saturates whatever single-node
-cap it is given. Real arrivals are gradual (the canonical run), where 3 cores has
-headroom; the lever for a true arrival spike is **horizontal scaling**, not more cores per
-node — carried into the prod recommendation below.
-
-## Prod-sizing recommendation (Stage 18)
-
-The contour is **CPU-bound and gateway-led** at 500 concurrent players. Carry these to the
-prod contour env (the same compose, `PROD_*` values):
-
-- **gateway: ≥ 3 cores** per ~500 concurrent players, `GOMAXPROCS` pinned to the limit —
-  it scales with the **connection count**, not just the request rate; beyond one node's
-  worth, scale the gateway **horizontally** rather than vertically.
-- **backend: ~1–2 cores**, pool 40 — comfortable; the work is light per request.
-- **postgres: ~2 cores / ≥ 512 MiB** — ~1.4 cores at 500 players, 100 % cache hit.
-- **tempo: ≥ 2 GiB**; the Go services run under ~170 MiB (256 MiB would suffice, 512 is
-  safe); pin `GOMAXPROCS` to each CPU limit; keep json-file rotation.
-- Memory is not the constraint anywhere; CPU is.
-
-### VPS / VDS sizing (single-host contour)
-
-The whole contour (the app + the observability stack) runs on one host via
-`docker-compose`. The tiers below are grounded in the R7 profile (**≈5.5 cores / ≈2.5 GiB
-RAM peak at 500 concurrent players**; ≈0.5 GiB idle) and the **measured** on-disk
-footprint: prod images ≈2.4 GB; the Tempo volume **3.1 GB at 72 h** retention; Prometheus
-≈1–2 GB at 15 d; the game DB 23 MiB and growing with history. CPU and disk grow; RAM has
-the most slack.
-
-| tier | CPU | RAM | disk | handles |
-|------|-----|-----|------|---------|
-| **Minimum** | 2 cores | 2 GiB | 20 GiB | ~up to ~150 concurrent; lower the compose limits (gateway 1.5 / backend·postgres 1 / tempo 1 GiB) to fit the box |
-| **Average** (reasonable load) | 4 cores | 4 GiB | 40 GiB | ~300–400 concurrent comfortably; the tested 500 with occasional gateway burst-throttling |
-| **Maximum** (worry-free) | 8 cores | 8 GiB | 80 GiB | 500+ concurrent with full gateway burst headroom (its 3-core cap) + room to grow; the compose limits fit as-is |
-
-- The per-service limits in `docker-compose.yml` are tuned for the **Average/Maximum**
-  target (the gateway alone caps at 3 cores). On the **Minimum** tier, scale them down to
-  match the host or the caps over-subscribe it.
-- **Disk is dominated by observability retention + DB growth.** Tempo (72 h traces) and
-  Prometheus (15 d metrics) are the main levers — shorten the windows (or move Tempo to
-  object storage) to cut disk; Postgres grows with game history, so budget for months of
-  it; container logs are already capped (json-file 10m × 3 ≈ 30 MiB each).
-- **RAM** rarely binds: the contour peaks ≈2.5 GiB at 500 players and the sum of all
-  configured limits is ≈5.6 GiB, so 8 GiB never strains.
-- Beyond one host's worth of players, scale the **gateway horizontally** (it is
-  connection-CPU-bound) rather than ordering an ever-bigger box.
-
-## Re-running
-
-See [`README.md`](README.md). Briefly, from the repo root:
-
-```sh
-docker build -f loadtest/Dockerfile -t scrabble-loadtest .
-docker run --rm --cpus=3 --name scrabble-loadtest --network scrabble-internal \
-  -e POSTGRES_PASSWORD=… scrabble-loadtest run --reset --cleanup
-```
-
-The harness stays in the repo for future repeats.

loadtest/REPORT.md

@@ -0,0 +1,194 @@
+# loadtest — stress trip report
+
+The pre-release stress write-up for [`PRERELEASE.md`](../PRERELEASE.md). It drives the
+`scrabble/loadtest` harness against a freshly redeployed test contour to confirm the
+system holds at scale and to settle resource sizing before the prod cutover. The harness
+stays in the repo for repeats; see [`README.md`](README.md) for how to run it.
+
+This report supersedes the earlier per-phase notes. The harness has been through three
+passes: an early diagnostic, a tuning pass that sized container limits / `GOMAXPROCS`, and
+this final pass — which **added the per-tile `game.evaluate` preview to the model** (the
+hottest real gameplay call, previously unmodelled) and, with it, surfaced and fixed the
+**gateway→backend connection-pool bottleneck** described below. The numbers here are from
+that final pass.
+
+## What it models
+
+The harness seeds a large account population with pre-created sessions directly in
+Postgres, then drives virtual players through the **gateway edge protocol** (h2c) in real
+games assembled via the invitation flow. Each player owns its own `edge.Client` (its own
+h2c connection, like a real client), holds a live `Subscribe` stream, and per tick polls
+`game.state`, replays `game.history`, generates a legal **mid-ranked** move with the
+embedded `scrabble-solver`, and submits it (or passes/exchanges). A fraction of ticks
+exercise nudge / chat / check-word / draft / profile / stats. A separate **gateway-hammer**
+floods `games.list` to verify the rate limiter.
+
+### The evaluate hot path (this pass)
+
+A real client previews every tentative play as the user arranges tiles: the UI fires a
+debounced `game.evaluate` (legality + score) on each placement change while it is the
+player's turn. Over a single composed word that is **several evaluate calls per turn** —
+far more than the one `submit_play` — so `game.evaluate` is the single hottest gameplay
+request at scale. The earlier passes did not model it at all (they submitted directly),
+which understated the real load.
+
+This pass models it: when a player composes a play of *K* newly-placed tiles, it fires one
+`evaluate` per landed tile (a growing prefix of the tiles), plus a small number of
+full-composition re-previews for reconsideration, spaced by a human-paced gap (the client's
+250 ms debounce), then one `draft.save`, then `submit_play`. `--eval=false` reproduces the
+pre-evaluate harness for an A/B baseline; `--eval-recon` tunes the reconsideration count.
+
+`game.check_word` is a *different*, manual "look this word up" panel (throttled, on demand)
+— not the per-tile call — and is exercised separately as a secondary op.
+
+## Final run (eval-on, after the connection-pool fix)
+
+Contour: backend / postgres capped at 2 cores / 512 MiB (`GOMAXPROCS=2`), gateway at
+3 cores / 512 MiB (`GOMAXPROCS=3`), per the tuned `deploy/docker-compose.yml`. Gradual ramp
+**50 → 200 → 500** concurrent players, 4 min/step, `--tick 800ms`, gateway-hammer on. The
+harness ran as a one-shot container on `scrabble-internal`, capped at `--cpus 3`. The DB was
+wiped before the run (`DROP SCHEMA backend CASCADE`); the seeded population was removed by
+`--cleanup` afterwards.
+
+Per-operation results at the 500-player peak (740 s, gameplay rows; the hammer row is the
+limiter probe):
+
+| operation | count | req/s | p50 | p99 | max | notes |
+|-----------|------:|------:|----:|----:|----:|-------|
+| game.evaluate | 85 721 | 115.9 | 1 ms | 200 ms | 193 ms | **the hot path** — all ok |
+| game.state | 115 926 | 156.7 | 100 ms | 200 ms | 260 ms | transport_error 86 (0.07 %) |
+| game.history | 22 258 | 30.1 | 5 ms | 100 ms | 195 ms | all ok |
+| draft.save | 23 031 | 31.1 | 2 ms | 200 ms | 194 ms | all ok |
+| game.submit_play | 21 704 | 29.3 | 1 ms | 200 ms | 274 ms | ok 3 902; not_your_turn / illegal_play are concurrent-play races (see caveat) |
+| hammer:games.list | 522 756 | 706.7 | 1 ms | 2 ms | 53 ms | **99.97 % rate_limited** — limiter holds |
+
+- **Volume:** 802 200 total edge calls (1 084 req/s incl. the hammer; ~377 req/s of real
+  gameplay). `stream errors: 0`. Live events: 11 199 `opponent_moved`, 4 153 `your_turn`.
+- **`game.evaluate` is the dominant gameplay write-path call** at ~116 req/s — second only
+  to the `game.state` poll — and it is cheap: p50 1 ms, effectively zero errors. The backend
+  serves it straight from the in-memory live-game cache; on a warm hit it skips the database
+  entirely (see *Postgres read path* below, which halved its p99 to 100 ms).
+- **Latency stayed healthy** under the heavier evaluate load: every gameplay op p99 ≤ 200 ms.
+- **The limiter holds** unchanged: 99.97 % of the hammer rejected at p99 2 ms.
+
+### Peak CPU (500 players)
+
+| container | CPU peak | cap |
+|-----------|---------:|----:|
+| scrabble-postgres | **165 %** (~1.65 cores) | 200 % |
+| scrabble-backend | 77 % (~0.77 core) | 200 % |
+| scrabble-gateway | **26 %** (~0.26 core) | 300 % |
+| scrabble-loadtest (harness) | 42 % | 300 % |
+
+Memory stayed modest everywhere (Go services ≤ ~90 MiB). **Postgres is now the busiest
+service** — it has headroom (1.65 of 2 cores) but is the scaling axis. The gateway, after
+the fix below, is near-idle.
+
+## The headline finding: gateway→backend connection churn
+
+The gateway proxies every synchronous client call to the single backend host over REST.
+Its backend HTTP client used the default transport, whose **`MaxIdleConnsPerHost` is 2**
+(`http.DefaultMaxIdleConnsPerHost`). So the gateway kept only **2** keep-alive connections
+to the backend and opened — then closed — a fresh TCP connection for almost every other
+call. Measured at the gateway's network namespace:
+
+| | gateway→backend sockets |
+|---|---|
+| before (eval-on, 500 players) | **TIME_WAIT ≈ 26 500**, ESTABLISHED 2 |
+| after (eval-on, 500 players) | TIME_WAIT ≈ 0 (steady state), **ESTABLISHED ≈ 225 (reused)** |
+
+26 500 TIME_WAIT sockets is the connection **churn**: ~440 new connections per second,
+each a full TCP handshake + teardown, the socket then lingering 60 s. That count sits right
+under the ~28 000 ephemeral-port ceiling — the latent cliff that produced the residual
+`transport_error` the earlier passes chased on the *client* side (h2c streams) but never
+eliminated, because the real cause was here, on the *backend* side.
+
+The fix is one custom `http.Transport` with a wide idle pool
+(`gateway/internal/backendclient/client.go`, `backendMaxIdleConns`). Before / after, same
+eval-on workload at 500 players:
+
+| metric | before fix | after fix |
+|--------|-----------:|----------:|
+| gateway→backend TIME_WAIT | ~26 500 | **~0** |
+| gateway CPU peak | **175 %** (~1.75 cores) | **26 %** (~0.26 core) |
+| game.state p99 | 500 ms | 200 ms |
+
+**The churn was burning ~1.5 gateway cores of pure connection setup/teardown.** Removing it
+cut peak gateway CPU ~7× and erased the port-exhaustion cliff. The backend and postgres CPU
+are unchanged — they do the real work; only the gateway's wasted overhead disappeared. The
+pool settles at ~225 live connections at 500 players; the constant is set to 512 for ~2×
+headroom.
+
+## Sizing — why the old "≈150 concurrent / 2-core" figure was a bug, not a floor
+
+The earlier tuning pass concluded the gateway was the binding constraint — "size it for
+≥ 3 cores per 500 players, scale it horizontally" — and the single-host "minimum" tier
+topped out near ~150 concurrent. **That was sizing around the connection-churn bug.** The
+gateway drew ~1.75–3 cores not from proxying work but from churning backend connections;
+the backend behind it sat near-idle the whole time.
+
+With the churn fixed, at **500 concurrent players** the app draws roughly:
+
+- **gateway ≈ 0.26 core** (was ~3) — no longer the constraint,
+- **backend ≈ 0.77 core**,
+- **postgres ≈ 1.65 cores** — now the busiest, with headroom,
+
+≈ **2.7 app cores total** (down from the ~5.5-core contour peak the tuning pass recorded,
+*and* under a heavier, more realistic workload that now includes `game.evaluate`). Postgres,
+not the gateway, is the scaling axis.
+
+Revised single-host guidance (app + co-resident observability stack on one box):
+
+| tier | CPU | RAM | handles |
+|------|-----|-----|---------|
+| **Minimum** | 2 cores | 2 GiB | comfortably the low hundreds of concurrent — the gateway no longer eats cores; postgres + the observability stack set the limit |
+| **Average** | 4 cores | 4 GiB | 500 concurrent with headroom |
+| **Maximum** | 8 cores | 8 GiB | 500+ with full burst headroom and room to grow |
+
+The gateway's compose limit can drop well below its old 3 cores; it is now connection-pool
+bound, not connection-CPU bound. Memory was never the constraint. Disk is still dominated
+by observability retention (Tempo, Prometheus) + DB growth — unchanged from before.
+
+## Postgres read path (warm-cache optimization)
+
+Following this pass, `game.evaluate` no longer reads the database on the hot path. An
+active game is already resident in the in-memory live-game cache (mutated in place across
+moves, evicted only on finish), so the preview answers its seat-membership check from the
+cached immutable seat list and scores against the cached engine game — **no `GetGame` on a
+warm hit**. `GetGame` itself was also folded from two round-trips (game, then seats) into a
+single `LEFT JOIN`. Measured at 500 players, **`game.evaluate` p99 halved (200 → 100 ms)**
+and the per-operation query count dropped.
+
+It did **not** cut postgres CPU, and the measurement says why: postgres is **write-bound**,
+not read-bound. `pg_stat_user_tables` puts the cost in the per-move `CommitMove`
+transaction (a `game_moves` insert plus `games` / `game_players` updates), the debounced
+`game_drafts` upserts (~60 k in one run), and the journal replays — not the cheap, indexed,
+fully-cached `GetGame` lookups this change removed (one re-run even committed 28 % more
+plays, whose extra writes masked the saved reads). Postgres also runs with headroom
+(~1.5 of 2 cores), and the gateway fix freed ~3 cores on the box, so the lever if postgres
+ever caps is **more cores** (it is CPU-bound, not I/O), not riskier write-path surgery. So
+this change is a latency / query-volume win, deliberately not a DB-CPU one.
+
+## Caveat — harness fidelity
+
+The harness's `not_your_turn` and `illegal_play` on `submit_play` are concurrent-play
+artifacts, not system errors: it generates a move from a locally replayed board, and a
+fast opponent (or a transport hiccup) can move between the state fetch and the submit,
+leaving the move out of turn or illegal on the now-changed board. A real client previews
+with `evaluate` and only submits a legal, in-turn play. These rejections are cheap domain
+outcomes (HTTP-ok with a stable code) and do not change the request *load*, which is what
+the run measures. The harness also shares the host CPU with the contour (capped with
+`--cpus`); a fully isolated ceiling on separate hardware remains future work.
+
+## Re-running
+
+From the repo root:
+
+```sh
+docker build -f loadtest/Dockerfile -t scrabble-loadtest .
+docker run --rm --cpus=3 --name scrabble-loadtest --network scrabble-internal \
+  -e POSTGRES_PASSWORD="$TEST_POSTGRES_PASSWORD" scrabble-loadtest run --reset --cleanup
+```
+
+`--eval=false` reproduces the pre-evaluate baseline for comparison. The authoritative hard
+reset of the contour DB remains `DROP SCHEMA backend CASCADE` + a backend restart.

loadtest/cmd/loadtest/main.go

@@ -73,6 +73,8 @@ func cmdRun(ctx context.Context, log *slog.Logger, args []string) error {
 	gpp := fs.Int("games-per-player", 0, "target concurrent games per player (0 => random 3..5)")
 	tick := fs.Duration("tick", 800*time.Millisecond, "per-player operation cadence")
 	secProb := fs.Float64("secondary-prob", 0.08, "chance per tick of a non-move operation")
+	eval := fs.Bool("eval", true, "model the per-tile evaluate preview (the realistic gameplay hot path); --eval=false reproduces the pre-evaluate harness for an A/B baseline")
+	evalRecon := fs.Int("eval-recon", 1, "extra full-composition evaluate re-previews per play (reconsideration), beyond one per placed tile")
 	hammerWorkers := fs.Int("hammer-workers", 20, "gateway-hammer concurrent callers (0 disables)")
 	hammerDur := fs.Duration("hammer-dur", 15*time.Second, "gateway-hammer duration")
 	reset := fs.Bool("reset", false, "delete prior harness rows before seeding")
@@ -117,6 +119,7 @@ func cmdRun(ctx context.Context, log *slog.Logger, args []string) error {
 	cfg := scenario.RealisticConfig{
 		Steps: steps, StepDur: *stepDur, GamesPerPlayer: *gpp,
 		Tick: *tick, SecondaryProb: *secProb,
+		Eval: *eval, EvalRecon: *evalRecon,
 	}
 	if err := drv.RunRealistic(ctx, pool, cfg); err != nil && !errors.Is(err, context.Canceled) {
 		return err

loadtest/internal/edge/client.go

@@ -24,6 +24,7 @@ const (
 	msgSubmitPlay = "game.submit_play"
 	msgPass       = "game.pass"
 	msgExchange   = "game.exchange"
+	msgEvaluate   = "game.evaluate"
 	msgState      = "game.state"
 	msgHistory    = "game.history"
 	msgGamesList  = "games.list"

loadtest/internal/edge/encode.go

@@ -63,6 +63,33 @@ func submitPlay(gameID string, tiles []PlayTile) []byte {
 	return b.FinishedBytes()
 }
 
+// evalReq builds an EvalRequest payload (game id plus the tentative newly-placed tiles).
+// It mirrors submitPlay's shape — the backend infers the play's orientation the same way —
+// so a preview previews exactly what submitting those tiles would score.
+func evalReq(gameID string, tiles []PlayTile) []byte {
+	b := flatbuffers.NewBuilder(256)
+	gid := b.CreateString(gameID)
+	offs := make([]flatbuffers.UOffsetT, len(tiles))
+	for i, t := range tiles {
+		fb.PlayTileStart(b)
+		fb.PlayTileAddRow(b, int32(t.Row))
+		fb.PlayTileAddCol(b, int32(t.Col))
+		fb.PlayTileAddLetter(b, t.Letter)
+		fb.PlayTileAddBlank(b, t.Blank)
+		offs[i] = fb.PlayTileEnd(b)
+	}
+	fb.EvalRequestStartTilesVector(b, len(offs))
+	for i := len(offs) - 1; i >= 0; i-- {
+		b.PrependUOffsetT(offs[i])
+	}
+	tilesVec := b.EndVector(len(offs))
+	fb.EvalRequestStart(b)
+	fb.EvalRequestAddGameId(b, gid)
+	fb.EvalRequestAddTiles(b, tilesVec)
+	b.Finish(fb.EvalRequestEnd(b))
+	return b.FinishedBytes()
+}
+
 // exchange builds an ExchangeRequest payload swapping the listed rack tiles (alphabet
 // indices; 255 a blank).
 func exchange(gameID string, tiles []byte) []byte {

loadtest/internal/edge/ops.go

@@ -53,6 +53,15 @@ func (c *Client) Exchange(ctx context.Context, token, gameID string, tiles []byt
 	return decodeMoveResultGame(r.Payload), r.Code, nil
 }
 
+// Evaluate previews a tentative play's legality and score without committing it. It is
+// the per-tile composition call a real client fires (debounced) on every change while
+// arranging a word, so it is the hottest gameplay request at scale. The harness records
+// only the result code and latency; an illegal preview is a successful "ok" call.
+func (c *Client) Evaluate(ctx context.Context, token, gameID string, tiles []PlayTile) (string, error) {
+	r, err := c.execute(ctx, token, msgEvaluate, evalReq(gameID, tiles))
+	return r.Code, err
+}
+
 // Nudge prods the opponent whose turn it is.
 func (c *Client) Nudge(ctx context.Context, token, gameID string) (string, error) {
 	r, err := c.execute(ctx, token, msgNudge, gameAction(gameID))

loadtest/internal/scenario/scenario.go

@@ -42,19 +42,35 @@ type RealisticConfig struct {
 	GamesPerPlayer int           // target concurrent games per player; 0 => random 3..5
 	Tick           time.Duration // per-player operation cadence (keeps a player under the per-user limit)
 	SecondaryProb  float64       // chance per tick of a non-move operation
+	Eval           bool          // model the per-tile evaluate preview (the gameplay hot path); false reproduces the pre-evaluate harness
+	EvalRecon      int           // extra full-composition evaluate re-previews per play, beyond one per placed tile
 }
 
 // DefaultRealistic returns the moderate ramp: 50 -> 200
-// -> 500 concurrent players, ~12 minutes per step, ~1 op/s per player.
+// -> 500 concurrent players, ~12 minutes per step, ~1 op/s per player, with the
+// per-tile evaluate preview modelled (the realistic hot path).
 func DefaultRealistic() RealisticConfig {
 	return RealisticConfig{
 		Steps:         []int{50, 200, 500},
 		StepDur:       12 * time.Minute,
 		Tick:          800 * time.Millisecond,
 		SecondaryProb: 0.08,
+		Eval:          true,
+		EvalRecon:     1,
 	}
 }
 
+// evalGapBase and evalGapSpan bound the modelled pause between successive tile
+// placements: the client's 250 ms debounce coalesces faster drags into a single
+// evaluate, so a thoughtful player's previews are spaced by a gap drawn from
+// [base, base+span] — wide enough that a normal composition stays under the per-user
+// rate limit, the way a real one does (the limiter's cost is measured by the hammer,
+// not by self-inflicted rejections here).
+const (
+	evalGapBase = 250 * time.Millisecond
+	evalGapSpan = 500 * time.Millisecond
+)
+
 // RunRealistic runs the staged ramp. Each step activates more players (drawn from the
 // seeded pool), assembles a cohort of games for them and starts their turn loops; the
 // loops run until the whole ramp ends. Players from earlier steps keep playing, so
@@ -128,7 +144,7 @@ func (d *Driver) playerLoop(ctx context.Context, p seed.Account, games []*Game,
 				d.secondaryOp(ctx, c, p, g, rng)
 				continue
 			}
-			if d.playTurn(ctx, c, p, g, rng) {
+			if d.playTurn(ctx, c, p, g, cfg, rng) {
 				active = slices.DeleteFunc(active, func(x *Game) bool { return x == g })
 				gi = 0
 				if len(active) == 0 {
@@ -161,10 +177,10 @@ func (d *Driver) subscribeLoop(ctx context.Context, c *edge.Client, p seed.Accou
 }
 
 // playTurn plays one turn in g over the player's client when it is the player's
-// move: fetch state, replay history, pick a legal move and submit it (or exchange /
-// pass). It reports whether the game has finished, so the caller can drop it from the
-// rotation.
-func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g *Game, rng *rand.Rand) (finished bool) {
+// move: fetch state, replay history, pick a legal move, compose it (the per-tile
+// evaluate previews a real client fires) and submit it (or exchange / pass). It reports
+// whether the game has finished, so the caller can drop it from the rotation.
+func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g *Game, cfg RealisticConfig, rng *rand.Rand) (finished bool) {
 	seat := g.seatOf(p.ID.String())
 	if seat < 0 {
 		return false
@@ -196,6 +212,7 @@ func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g
 	}
 	switch action.Kind {
 	case "play":
+		d.composePlay(ctx, c, p, g, action.Tiles, cfg, rng)
 		t0 = time.Now()
 		_, code, _ := c.SubmitPlay(ctx, p.Token, g.ID, action.Tiles)
 		d.rec.Record("game.submit_play", code, time.Since(t0))
@@ -211,6 +228,59 @@ func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g
 	return false
 }
 
+// composePlay models a player arranging the chosen play tile by tile before committing:
+// the debounced evaluate preview the real client fires on each placement (a growing prefix
+// of the tiles), a few full-composition re-previews for reconsideration (recall a tile, try
+// another spot), and the single draft persistence the client debounces out. evaluate is the
+// hottest gameplay request at scale, so omitting it (the pre-evaluate harness) understated
+// the load; cfg.Eval false reproduces that baseline for an A/B comparison. Every step
+// honours ctx, so end-of-run cancellation never blocks on a sleep or an in-flight preview.
+func (d *Driver) composePlay(ctx context.Context, c *edge.Client, p seed.Account, g *Game, tiles []edge.PlayTile, cfg RealisticConfig, rng *rand.Rand) {
+	if !cfg.Eval || len(tiles) == 0 {
+		return
+	}
+	// One evaluate per landed tile: the growing prefix mirrors the client re-previewing
+	// after each placement (an early prefix is often illegal, which is still a successful
+	// "ok" round trip — exactly the backend work a real composition triggers).
+	for n := 1; n <= len(tiles); n++ {
+		if !jitterSleep(ctx, rng, evalGapBase, evalGapSpan) {
+			return
+		}
+		t0 := time.Now()
+		code, _ := c.Evaluate(ctx, p.Token, g.ID, tiles[:n])
+		d.rec.Record("game.evaluate", code, time.Since(t0))
+	}
+	for r := 0; r < cfg.EvalRecon; r++ {
+		if !jitterSleep(ctx, rng, evalGapBase, evalGapSpan) {
+			return
+		}
+		t0 := time.Now()
+		code, _ := c.Evaluate(ctx, p.Token, g.ID, tiles)
+		d.rec.Record("game.evaluate", code, time.Since(t0))
+	}
+	// The client persists the in-progress composition (debounced to one upsert). Its opaque
+	// JSON content does not affect the call's cost, so a minimal valid shape stands in.
+	t0 := time.Now()
+	code, _ := c.DraftSave(ctx, p.Token, g.ID, `{"rack_order":"","board_tiles":[]}`)
+	d.rec.Record("draft.save", code, time.Since(t0))
+}
+
+// jitterSleep pauses for a randomised gap in [base, base+span], modelling the human pause
+// between tile placements that the client's debounce coalesces into one evaluate. It
+// returns false if ctx is cancelled during the wait, so a composition unwinds promptly at
+// end of run.
+func jitterSleep(ctx context.Context, rng *rand.Rand, base, span time.Duration) bool {
+	d := base + time.Duration(rng.Int63n(int64(span)+1))
+	t := time.NewTimer(d)
+	defer t.Stop()
+	select {
+	case <-ctx.Done():
+		return false
+	case <-t.C:
+		return true
+	}
+}
+
 // secondaryOp exercises one of the non-move edge operations the plan calls out, so
 // the run touches nudge / chat / check-word / draft / profile / stats too, over the
 // player's own client.