perf(gateway): pool backend conns; loadtest evaluate hot path
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Has been skipped
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m5s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Has been skipped
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m5s
The loadtest harness never modelled game.evaluate — the debounced per-tile play preview a real client fires several times per turn, the hottest gameplay call. Model it (one evaluate per placed tile + reconsideration re-previews + draft.save, human-paced; --eval / --eval-recon toggle it). That realistic load surfaced the real bottleneck: the gateway's backend HTTP client used the default transport (MaxIdleConnsPerHost=2), so every sync call to the single backend host churned a fresh TCP connection — ~26500 TIME_WAIT sockets at 500 players (near the ephemeral-port ceiling), burning ~1.75 gateway cores while the backend sat near-idle. It was the unfixed root of the residual transport_error the earlier passes chased on the client side. Widen the keep-alive pool (backendMaxIdleConns=512, ~2x the observed 225-conn peak). At 500 players the churn collapses to ~0 and peak gateway CPU drops ~7x (~1.75 -> ~0.26 cores); postgres (~1.65 cores) becomes the busiest service. This overturns the earlier "gateway is the binding constraint, scale it horizontally" sizing — that was sizing around this bug, not a real floor. Consolidate the loadtest trip reports into one loadtest/REPORT.md (drop the R2/R7 split) and bake the finding into README / PRERELEASE / ARCHITECTURE / TESTING.
This commit is contained in:
+12
-8
@@ -15,10 +15,12 @@ and prints a trip-report summary. It stays in the repo for repeats.
|
||||
2. **Drive** (edge protocol over h2c): assembles real 2–4 player games via the
|
||||
invitation flow (`invitation.create` → `invitation.accept`, no robots), then runs
|
||||
each player's turn loop — poll `game.state`, replay `game.history`, generate a legal
|
||||
**mid-ranked** move with the embedded `scrabble-solver`, and `game.submit_play`
|
||||
(or pass/exchange). A fraction of turns exercise nudge / chat / check-word / draft /
|
||||
profile-update / stats. Each player also holds a live `Subscribe` stream. The
|
||||
moderate ramp is **50 → 200 → 500** concurrent players, ~12 min per step.
|
||||
**mid-ranked** move with the embedded `scrabble-solver`, **compose it tile by tile with
|
||||
the debounced `game.evaluate` preview a real client fires** (the hottest gameplay call),
|
||||
persist a `draft.save`, and `game.submit_play` (or pass/exchange). A fraction of turns
|
||||
exercise nudge / chat / check-word / draft / profile-update / stats. Each player also
|
||||
holds a live `Subscribe` stream. The moderate ramp is **50 → 200 → 500** concurrent
|
||||
players, ~12 min per step. `--eval=false` drops the evaluate model for an A/B baseline.
|
||||
3. **Hammer**: drives `games.list` from one account far above the per-user rate limit
|
||||
to verify the limiter holds (`rate_limited` results) and measure its cost.
|
||||
4. **Report**: per-operation latency percentiles, throughput, result-code breakdown,
|
||||
@@ -72,6 +74,8 @@ Key `run` flags (env in parentheses):
|
||||
| `--games-per-player` | `0` (random 3–5) | target concurrent games per player |
|
||||
| `--tick` | `800ms` | per-player op cadence (keeps a player under the per-user limit) |
|
||||
| `--secondary-prob` | `0.08` | chance per tick of a non-move op |
|
||||
| `--eval` | `true` | model the per-tile `game.evaluate` preview (the gameplay hot path); `false` reproduces the pre-evaluate harness |
|
||||
| `--eval-recon` | `1` | extra full-composition evaluate re-previews per play (reconsideration), beyond one per placed tile |
|
||||
| `--hammer-workers` / `--hammer-dur` | `20` / `15s` | gateway-hammer (0 workers disables) |
|
||||
| `--reset` / `--cleanup` | `false` | delete harness rows before / after the run |
|
||||
|
||||
@@ -93,11 +97,11 @@ runs unconditionally. Use an **absolute** path (here via `$PWD`): `go test ./loa
|
||||
runs each package from its own directory, so a relative `BACKEND_DICT_DIR` would not
|
||||
resolve.
|
||||
|
||||
## Trip reports
|
||||
## Trip report
|
||||
|
||||
The two stress passes are written up in the repo: the early pass in
|
||||
[`REPORT-R2.md`](REPORT-R2.md) and the final, tuned pass in
|
||||
[`REPORT-R7.md`](REPORT-R7.md).
|
||||
The stress findings — the final run, the `game.evaluate` hot-path model, the
|
||||
gateway→backend connection-pool fix, and the revised sizing — are written up in
|
||||
[`REPORT.md`](REPORT.md).
|
||||
|
||||
## Caveat
|
||||
|
||||
|
||||
Reference in New Issue
Block a user