perf(gateway): pool backend conns; loadtest evaluate hot path
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Has been skipped
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m5s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Has been skipped
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m5s
The loadtest harness never modelled game.evaluate — the debounced per-tile play preview a real client fires several times per turn, the hottest gameplay call. Model it (one evaluate per placed tile + reconsideration re-previews + draft.save, human-paced; --eval / --eval-recon toggle it). That realistic load surfaced the real bottleneck: the gateway's backend HTTP client used the default transport (MaxIdleConnsPerHost=2), so every sync call to the single backend host churned a fresh TCP connection — ~26500 TIME_WAIT sockets at 500 players (near the ephemeral-port ceiling), burning ~1.75 gateway cores while the backend sat near-idle. It was the unfixed root of the residual transport_error the earlier passes chased on the client side. Widen the keep-alive pool (backendMaxIdleConns=512, ~2x the observed 225-conn peak). At 500 players the churn collapses to ~0 and peak gateway CPU drops ~7x (~1.75 -> ~0.26 cores); postgres (~1.65 cores) becomes the busiest service. This overturns the earlier "gateway is the binding constraint, scale it horizontally" sizing — that was sizing around this bug, not a real floor. Consolidate the loadtest trip reports into one loadtest/REPORT.md (drop the R2/R7 split) and bake the finding into README / PRERELEASE / ARCHITECTURE / TESTING.
This commit is contained in:
@@ -22,6 +22,19 @@ import (
|
||||
pushv1 "scrabble/pkg/proto/push/v1"
|
||||
)
|
||||
|
||||
// backendMaxIdleConns sizes the REST keep-alive pool to the single backend host. The
|
||||
// default transport caps idle connections per host at 2 (http.DefaultMaxIdleConnsPerHost),
|
||||
// which — since every synchronous client call proxies to that one host — forces a fresh
|
||||
// TCP connection (and a lingering TIME_WAIT socket) for almost every request under load.
|
||||
// That connection churn burns gateway CPU and exhausts ephemeral ports at scale, all
|
||||
// while the backend itself sits near-idle. Pooling the connections lets them be reused.
|
||||
//
|
||||
// The stress harness measured the effect at 500 concurrent players: the churn collapsed
|
||||
// from ~26 500 TIME_WAIT sockets to ~0 and peak gateway CPU from ~1.75 to ~0.26 cores,
|
||||
// with the pool settling at ~225 live connections. 512 keeps ~2x headroom over that
|
||||
// observed peak so a burst never re-caps the pool. See loadtest/REPORT.md.
|
||||
const backendMaxIdleConns = 512
|
||||
|
||||
// Client calls the backend's REST API and opens its push gRPC stream.
|
||||
type Client struct {
|
||||
baseURL string
|
||||
@@ -41,9 +54,14 @@ func New(httpURL, grpcAddr string, timeout time.Duration) (*Client, error) {
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("backendclient: dial push %s: %w", grpcAddr, err)
|
||||
}
|
||||
// Clone the default transport (keeping its proxy, dialer and timeouts) and widen the
|
||||
// idle pool so REST calls to the backend reuse connections instead of churning them.
|
||||
transport := http.DefaultTransport.(*http.Transport).Clone()
|
||||
transport.MaxIdleConns = backendMaxIdleConns
|
||||
transport.MaxIdleConnsPerHost = backendMaxIdleConns
|
||||
return &Client{
|
||||
baseURL: strings.TrimRight(httpURL, "/"),
|
||||
http: &http.Client{Timeout: timeout},
|
||||
http: &http.Client{Timeout: timeout, Transport: transport},
|
||||
conn: conn,
|
||||
push: pushv1.NewPushClient(conn),
|
||||
}, nil
|
||||
|
||||
Reference in New Issue
Block a user