perf(gateway): pool backend conns; loadtest evaluate hot path #101

Merged

developer merged 2 commits from feature/loadtest-evaluate-hotpath into development 2026-06-21 18:51:59 +00:00

Conversation 1 Commits 2 Files Changed 20

+471 -442

developer commented 2026-06-21 17:56:26 +00:00

Owner

Copy Link

Copy Source

Re-prepared and re-ran the pre-release load test against the current code, fixed the bottleneck it revealed, and trimmed the hot read path.

1. Harness now models the game.evaluate hot path

A real client fires a debounced game.evaluate (legality + score preview) on every tile placement while composing a play — several per turn, the single hottest gameplay call. The harness never modelled it; now it does (one evaluate per placed tile + reconsideration + draft.save, human-paced). --eval=false reproduces the pre-evaluate baseline. At 500 players game.evaluate is ~116 req/s.

2. Fix: gateway → backend connection churn (the real bottleneck)

The gateway's backend REST client used the default http.Transport (MaxIdleConnsPerHost = 2). Every sync call to the one backend host churned a TCP connection — ~26 500 TIME_WAIT sockets at 500 players, near the ephemeral-port ceiling, burning ~1.75 gateway cores while the backend sat near-idle. Widening the pool (backendMaxIdleConns = 512), measured before/after at 500 players:

metric	before	after
gateway→backend TIME_WAIT	~26 500	~0
peak gateway CPU	175 % (~1.75 cores)	~0.25 core
game.state p99	500 ms	200 ms

3. Postgres read path (latency win, honest about CPU)

game.evaluate no longer reads the DB on the hot path: an active game is warm in the live-game cache, so the preview checks membership against the cached seat list and scores the cached engine game — no GetGame on a warm hit. GetGame also folds its two queries into one LEFT JOIN. Result: game.evaluate p99 halved (200 → 100 ms) and per-op query count dropped.

It does not cut postgres CPU — pg_stat_user_tables shows postgres is write-bound (per-move CommitMove, draft upserts, journal replays), not read-bound, and it runs with headroom (~1.5 of 2 cores). So this is a latency / query-volume optimization, not a DB-CPU one; chasing the write path wasn't worth the risk for a service with headroom (and the gateway fix freed ~3 cores on the box if postgres ever needs more).

4. Sizing — the "≈150 concurrent / 2-core" floor was the churn bug

The old "gateway is the binding constraint, scale horizontally" conclusion was sizing around the churn. Fixed, the app draws ~2.7 cores at 500 players (gateway ~0.25, backend ~0.77, postgres ~1.65), down from the ~5.5-core peak the tuning pass recorded under a lighter workload. Full analysis in loadtest/REPORT.md.

5. Docs

Consolidated the two trip reports into one loadtest/REPORT.md (dropped the R2/R7 split); baked the findings into README / PRERELEASE / ARCHITECTURE / TESTING.

Tests

go build/vet, gofmt, gateway + loadtest + backend unit, and the integration suite (testcontainers Postgres, exercising GetGame's join + evaluate across every game op) all green locally. Added regression guards: the backend pool stays wider than the stdlib default; a non-player evaluate asserts the cached-seat membership path. Verified live on the test contour (before/after runs above).

Re-prepared and re-ran the pre-release load test against the current code, fixed the bottleneck it revealed, and trimmed the hot read path. ## 1. Harness now models the `game.evaluate` hot path A real client fires a debounced `game.evaluate` (legality + score preview) on **every tile placement** while composing a play — several per turn, the single hottest gameplay call. The harness never modelled it; now it does (one `evaluate` per placed tile + reconsideration + `draft.save`, human-paced). `--eval=false` reproduces the pre-evaluate baseline. At 500 players `game.evaluate` is ~116 req/s. ## 2. Fix: gateway → backend connection churn (the real bottleneck) The gateway's backend REST client used the default `http.Transport` (`MaxIdleConnsPerHost = 2`). Every sync call to the one backend host churned a TCP connection — **~26 500 TIME_WAIT sockets at 500 players**, near the ephemeral-port ceiling, burning **~1.75 gateway cores** while the backend sat near-idle. Widening the pool (`backendMaxIdleConns = 512`), measured before/after at 500 players: | metric | before | after | |---|---:|---:| | gateway→backend TIME_WAIT | ~26 500 | **~0** | | peak gateway CPU | 175 % (~1.75 cores) | **~0.25 core** | | `game.state` p99 | 500 ms | 200 ms | ## 3. Postgres read path (latency win, honest about CPU) `game.evaluate` no longer reads the DB on the hot path: an active game is warm in the live-game cache, so the preview checks membership against the cached seat list and scores the cached engine game — **no `GetGame` on a warm hit**. `GetGame` also folds its two queries into one `LEFT JOIN`. Result: **`game.evaluate` p99 halved (200 → 100 ms)** and per-op query count dropped. It does **not** cut postgres CPU — `pg_stat_user_tables` shows postgres is **write-bound** (per-move `CommitMove`, draft upserts, journal replays), not read-bound, and it runs with headroom (~1.5 of 2 cores). So this is a latency / query-volume optimization, not a DB-CPU one; chasing the write path wasn't worth the risk for a service with headroom (and the gateway fix freed ~3 cores on the box if postgres ever needs more). ## 4. Sizing — the "≈150 concurrent / 2-core" floor was the churn bug The old "gateway is the binding constraint, scale horizontally" conclusion was sizing around the churn. Fixed, the app draws ~2.7 cores at 500 players (gateway ~0.25, backend ~0.77, postgres ~1.65), down from the ~5.5-core peak the tuning pass recorded under a *lighter* workload. Full analysis in `loadtest/REPORT.md`. ## 5. Docs Consolidated the two trip reports into one `loadtest/REPORT.md` (dropped the R2/R7 split); baked the findings into `README` / `PRERELEASE` / `ARCHITECTURE` / `TESTING`. ## Tests `go build/vet`, `gofmt`, gateway + loadtest + backend unit, and the **integration suite** (testcontainers Postgres, exercising `GetGame`'s join + evaluate across every game op) all green locally. Added regression guards: the backend pool stays wider than the stdlib default; a non-player evaluate asserts the cached-seat membership path. Verified live on the test contour (before/after runs above).

developer added 1 commit 2026-06-21 17:56:26 +00:00

perf(gateway): pool backend conns; loadtest evaluate hot path ... e2771826fd

The loadtest harness never modelled game.evaluate — the debounced per-tile
play preview a real client fires several times per turn, the hottest gameplay
call. Model it (one evaluate per placed tile + reconsideration re-previews +
draft.save, human-paced; --eval / --eval-recon toggle it).

That realistic load surfaced the real bottleneck: the gateway's backend HTTP
client used the default transport (MaxIdleConnsPerHost=2), so every sync call
to the single backend host churned a fresh TCP connection — ~26500 TIME_WAIT
sockets at 500 players (near the ephemeral-port ceiling), burning ~1.75 gateway
cores while the backend sat near-idle. It was the unfixed root of the residual
transport_error the earlier passes chased on the client side.

Widen the keep-alive pool (backendMaxIdleConns=512, ~2x the observed 225-conn
peak). At 500 players the churn collapses to ~0 and peak gateway CPU drops ~7x
(~1.75 -> ~0.26 cores); postgres (~1.65 cores) becomes the busiest service.
This overturns the earlier "gateway is the binding constraint, scale it
horizontally" sizing — that was sizing around this bug, not a real floor.

Consolidate the loadtest trip reports into one loadtest/REPORT.md (drop the
R2/R7 split) and bake the finding into README / PRERELEASE / ARCHITECTURE /
TESTING.

developer added 1 commit 2026-06-21 18:47:15 +00:00

perf(backend): cut evaluate's DB round-trips; load the game in one query ... ecb21bd218

EvaluatePlay (the hottest gameplay call, fired on every tile placement) now uses
the warm live-game cache directly: an active game stays cached (mutated in place
across moves, evicted only on finish), so the cached engine game and its immutable
seat list answer the membership check and the score with no DB read. The cold path
(eviction / first load) still loads and validates via the store. The seat list is
cached alongside the engine game for the membership fast path.

GetGame also folds its two round-trips (game, then seats) into one LEFT JOIN,
preserving the contract (same Game, a seatless game still returns empty seats, seat
order kept) — one round-trip for every remaining caller.

Measured at 500 players: evaluate p99 halves (200 -> 100 ms) and the per-op query
count drops. It does NOT cut postgres CPU — that is write-bound (per-move CommitMove
plus draft upserts and journal replays), the cheap indexed GetGame reads were never
its bottleneck, and postgres runs with headroom (~1.5 of 2 cores). So this is a
latency / query-volume optimization, not a DB-CPU one.

Regression cover: a non-player evaluate against a warm game asserts the cached-seat
membership path; the integration suite exercises GetGame's join across every game op.

owner approved these changes 2026-06-21 18:51:14 +00:00

developer merged commit 62f42ed102 into development 2026-06-21 18:51:59 +00:00

developer deleted branch feature/loadtest-evaluate-hotpath 2026-06-21 18:51:59 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

owner

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

owner (Ilia) developer

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: developer/scrabble-game#101