perf(gateway): pool backend conns; loadtest evaluate hot path
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Has been skipped
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m5s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Has been skipped
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m5s
The loadtest harness never modelled game.evaluate — the debounced per-tile play preview a real client fires several times per turn, the hottest gameplay call. Model it (one evaluate per placed tile + reconsideration re-previews + draft.save, human-paced; --eval / --eval-recon toggle it). That realistic load surfaced the real bottleneck: the gateway's backend HTTP client used the default transport (MaxIdleConnsPerHost=2), so every sync call to the single backend host churned a fresh TCP connection — ~26500 TIME_WAIT sockets at 500 players (near the ephemeral-port ceiling), burning ~1.75 gateway cores while the backend sat near-idle. It was the unfixed root of the residual transport_error the earlier passes chased on the client side. Widen the keep-alive pool (backendMaxIdleConns=512, ~2x the observed 225-conn peak). At 500 players the churn collapses to ~0 and peak gateway CPU drops ~7x (~1.75 -> ~0.26 cores); postgres (~1.65 cores) becomes the busiest service. This overturns the earlier "gateway is the binding constraint, scale it horizontally" sizing — that was sizing around this bug, not a real floor. Consolidate the loadtest trip reports into one loadtest/REPORT.md (drop the R2/R7 split) and bake the finding into README / PRERELEASE / ARCHITECTURE / TESTING.
This commit is contained in:
+8
-3
@@ -311,7 +311,7 @@ Then Stage 18.
|
||||
hammer (99.97 % rejected, p99 2 ms). **Top finding:** ~14 % `transport_error` on `game.state` at 500
|
||||
players, under CPU saturation (backend/gateway/Postgres each ~1 core) and amplified by the harness's
|
||||
single shared `http2.Transport`; the harness itself peaked at 86 % of a core on the same host, so the
|
||||
figures are pessimistic. Full trip report in [`../loadtest/REPORT-R2.md`](../loadtest/REPORT-R2.md);
|
||||
figures are pessimistic. Full trip report in [`../loadtest/REPORT.md`](../loadtest/REPORT.md);
|
||||
it feeds R3 (h2c `MaxConcurrentStreams`/timeouts, body-size cap), R6 and R7 (per-player transports,
|
||||
separate hardware, pool/limit sizing).
|
||||
- **CI:** `./loadtest/...` added to the path filter + vet/build/test; `go.work.sum` carries the new deps.
|
||||
@@ -454,7 +454,12 @@ Then Stage 18.
|
||||
one connection per player it bursts into its 2-core cap (the residual 2.49 % `transport_error`); backend
|
||||
~0.85 core and postgres ~1.4 cores had headroom; **tempo reached its 1 GiB cap**; the backend pool sat at
|
||||
its `MaxOpenConns=25` cap (28 backends); docker logs were unbounded (~14 MiB / 30 min on the backend at
|
||||
info). Full write-up in [`../loadtest/REPORT-R7.md`](../loadtest/REPORT-R7.md).
|
||||
info). Full write-up in [`../loadtest/REPORT.md`](../loadtest/REPORT.md). *(Superseded in part: a
|
||||
later pass modelling the `game.evaluate` hot path traced the gateway's CPU appetite to
|
||||
**gateway→backend connection churn** — the default 2-idle-connection HTTP transport — not proxying
|
||||
work. Pooling the connections cut peak gateway CPU ~7× (~1.75 → ~0.26 cores at 500 players) and
|
||||
removed the ephemeral-port-exhaustion cliff behind the residual `transport_error`, so the gateway is
|
||||
no longer the binding constraint — postgres is. The 3-core gateway cap below is now generous headroom.)*
|
||||
- **Round-2 tuning (owner-agreed, all in `deploy/docker-compose.yml`, no code change):** gateway **2 → 3
|
||||
cores + `GOMAXPROCS=3`**; tempo memory **1 → 2 GiB**; backend `MAX_OPEN_CONNS` **25 → 40**; a json-file
|
||||
**log-rotation** default (10m × 3) applied contour-wide via a YAML anchor (level stays info).
|
||||
@@ -464,7 +469,7 @@ Then Stage 18.
|
||||
**burst** run (a single 100 → 500 jump) pegged the gateway at 3 cores (≈296 % sustained, 9.27 % error),
|
||||
confirming it is **connection-CPU-bound** — a true arrival spike is a **horizontal-scaling** lever, not
|
||||
more cores per node (recorded in the prod-sizing recommendation).
|
||||
- **No schema change → no contour DB wipe.** Bake-back: `loadtest/REPORT-R7.md` (new), `loadtest/README.md`,
|
||||
- **No schema change → no contour DB wipe.** Bake-back: `loadtest/REPORT.md`, `loadtest/README.md`,
|
||||
`docs/TESTING.md`, the telemetry/observability section of `docs/ARCHITECTURE.md`, the repo-layout line in `CLAUDE.md`.
|
||||
|
||||
- **UI — Tab-bar navigation redesign** (owner ad-hoc, not on the raw TODO list): drop the hamburger
|
||||
|
||||
Reference in New Issue
Block a user