perf(gateway): pool backend conns; loadtest evaluate hot path
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Has been skipped
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m5s

The loadtest harness never modelled game.evaluate — the debounced per-tile
play preview a real client fires several times per turn, the hottest gameplay
call. Model it (one evaluate per placed tile + reconsideration re-previews +
draft.save, human-paced; --eval / --eval-recon toggle it).

That realistic load surfaced the real bottleneck: the gateway's backend HTTP
client used the default transport (MaxIdleConnsPerHost=2), so every sync call
to the single backend host churned a fresh TCP connection — ~26500 TIME_WAIT
sockets at 500 players (near the ephemeral-port ceiling), burning ~1.75 gateway
cores while the backend sat near-idle. It was the unfixed root of the residual
transport_error the earlier passes chased on the client side.

Widen the keep-alive pool (backendMaxIdleConns=512, ~2x the observed 225-conn
peak). At 500 players the churn collapses to ~0 and peak gateway CPU drops ~7x
(~1.75 -> ~0.26 cores); postgres (~1.65 cores) becomes the busiest service.
This overturns the earlier "gateway is the binding constraint, scale it
horizontally" sizing — that was sizing around this bug, not a real floor.

Consolidate the loadtest trip reports into one loadtest/REPORT.md (drop the
R2/R7 split) and bake the finding into README / PRERELEASE / ARCHITECTURE /
TESTING.
This commit is contained in:
Ilia Denisov
2026-06-21 19:55:57 +02:00
parent dec6fac013
commit e2771826fd
14 changed files with 378 additions and 396 deletions
@@ -0,0 +1,31 @@
package backendclient
import (
"net/http"
"testing"
"time"
)
// TestBackendTransportPoolsConnections guards the fix for the gateway->backend
// connection churn. Every synchronous client call proxies to the single backend host,
// so the REST client must widen the idle-connection pool past the default per-host cap
// of 2 (http.DefaultMaxIdleConnsPerHost) — otherwise almost every request under load
// opens a fresh TCP connection that then lingers in TIME_WAIT, burning gateway CPU and
// exhausting ephemeral ports. Reverting to the default transport (`&http.Client{...}`
// with no Transport) would silently reintroduce that, so assert the pool is widened.
func TestBackendTransportPoolsConnections(t *testing.T) {
c, err := New("http://backend.invalid", "localhost:9090", time.Second)
if err != nil {
t.Fatalf("New: %v", err)
}
defer func() { _ = c.Close() }()
tr, ok := c.http.Transport.(*http.Transport)
if !ok {
t.Fatalf("REST transport = %T, want a *http.Transport with a widened idle pool", c.http.Transport)
}
if tr.MaxIdleConnsPerHost <= http.DefaultMaxIdleConnsPerHost {
t.Errorf("MaxIdleConnsPerHost = %d, want > default %d (else per-call connection churn)",
tr.MaxIdleConnsPerHost, http.DefaultMaxIdleConnsPerHost)
}
}