perf(gateway): pool backend conns; loadtest evaluate hot path
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Has been skipped
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m5s

The loadtest harness never modelled game.evaluate — the debounced per-tile
play preview a real client fires several times per turn, the hottest gameplay
call. Model it (one evaluate per placed tile + reconsideration re-previews +
draft.save, human-paced; --eval / --eval-recon toggle it).

That realistic load surfaced the real bottleneck: the gateway's backend HTTP
client used the default transport (MaxIdleConnsPerHost=2), so every sync call
to the single backend host churned a fresh TCP connection — ~26500 TIME_WAIT
sockets at 500 players (near the ephemeral-port ceiling), burning ~1.75 gateway
cores while the backend sat near-idle. It was the unfixed root of the residual
transport_error the earlier passes chased on the client side.

Widen the keep-alive pool (backendMaxIdleConns=512, ~2x the observed 225-conn
peak). At 500 players the churn collapses to ~0 and peak gateway CPU drops ~7x
(~1.75 -> ~0.26 cores); postgres (~1.65 cores) becomes the busiest service.
This overturns the earlier "gateway is the binding constraint, scale it
horizontally" sizing — that was sizing around this bug, not a real floor.

Consolidate the loadtest trip reports into one loadtest/REPORT.md (drop the
R2/R7 split) and bake the finding into README / PRERELEASE / ARCHITECTURE /
TESTING.
This commit is contained in:
Ilia Denisov
2026-06-21 19:55:57 +02:00
parent dec6fac013
commit e2771826fd
14 changed files with 378 additions and 396 deletions
+3
View File
@@ -73,6 +73,8 @@ func cmdRun(ctx context.Context, log *slog.Logger, args []string) error {
gpp := fs.Int("games-per-player", 0, "target concurrent games per player (0 => random 3..5)")
tick := fs.Duration("tick", 800*time.Millisecond, "per-player operation cadence")
secProb := fs.Float64("secondary-prob", 0.08, "chance per tick of a non-move operation")
eval := fs.Bool("eval", true, "model the per-tile evaluate preview (the realistic gameplay hot path); --eval=false reproduces the pre-evaluate harness for an A/B baseline")
evalRecon := fs.Int("eval-recon", 1, "extra full-composition evaluate re-previews per play (reconsideration), beyond one per placed tile")
hammerWorkers := fs.Int("hammer-workers", 20, "gateway-hammer concurrent callers (0 disables)")
hammerDur := fs.Duration("hammer-dur", 15*time.Second, "gateway-hammer duration")
reset := fs.Bool("reset", false, "delete prior harness rows before seeding")
@@ -117,6 +119,7 @@ func cmdRun(ctx context.Context, log *slog.Logger, args []string) error {
cfg := scenario.RealisticConfig{
Steps: steps, StepDur: *stepDur, GamesPerPlayer: *gpp,
Tick: *tick, SecondaryProb: *secProb,
Eval: *eval, EvalRecon: *evalRecon,
}
if err := drv.RunRealistic(ctx, pool, cfg); err != nil && !errors.Is(err, context.Canceled) {
return err