From e2771826fd7c15464aaf58970cc46748716d46e6 Mon Sep 17 00:00:00 2001 From: Ilia Denisov Date: Sun, 21 Jun 2026 19:55:57 +0200 Subject: [PATCH] perf(gateway): pool backend conns; loadtest evaluate hot path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The loadtest harness never modelled game.evaluate — the debounced per-tile play preview a real client fires several times per turn, the hottest gameplay call. Model it (one evaluate per placed tile + reconsideration re-previews + draft.save, human-paced; --eval / --eval-recon toggle it). That realistic load surfaced the real bottleneck: the gateway's backend HTTP client used the default transport (MaxIdleConnsPerHost=2), so every sync call to the single backend host churned a fresh TCP connection — ~26500 TIME_WAIT sockets at 500 players (near the ephemeral-port ceiling), burning ~1.75 gateway cores while the backend sat near-idle. It was the unfixed root of the residual transport_error the earlier passes chased on the client side. Widen the keep-alive pool (backendMaxIdleConns=512, ~2x the observed 225-conn peak). At 500 players the churn collapses to ~0 and peak gateway CPU drops ~7x (~1.75 -> ~0.26 cores); postgres (~1.65 cores) becomes the busiest service. This overturns the earlier "gateway is the binding constraint, scale it horizontally" sizing — that was sizing around this bug, not a real floor. Consolidate the loadtest trip reports into one loadtest/REPORT.md (drop the R2/R7 split) and bake the finding into README / PRERELEASE / ARCHITECTURE / TESTING. --- PRERELEASE.md | 11 +- docs/ARCHITECTURE.md | 6 +- docs/TESTING.md | 6 +- gateway/internal/backendclient/client.go | 20 +- .../backendclient/client_internal_test.go | 31 +++ loadtest/README.md | 20 +- loadtest/REPORT-R2.md | 162 ------------- loadtest/REPORT-R7.md | 212 ------------------ loadtest/REPORT.md | 184 +++++++++++++++ loadtest/cmd/loadtest/main.go | 3 + loadtest/internal/edge/client.go | 1 + loadtest/internal/edge/encode.go | 27 +++ loadtest/internal/edge/ops.go | 9 + loadtest/internal/scenario/scenario.go | 82 ++++++- 14 files changed, 378 insertions(+), 396 deletions(-) create mode 100644 gateway/internal/backendclient/client_internal_test.go delete mode 100644 loadtest/REPORT-R2.md delete mode 100644 loadtest/REPORT-R7.md create mode 100644 loadtest/REPORT.md diff --git a/PRERELEASE.md b/PRERELEASE.md index b1239e8..6c1d91b 100644 --- a/PRERELEASE.md +++ b/PRERELEASE.md @@ -311,7 +311,7 @@ Then Stage 18. hammer (99.97 % rejected, p99 2 ms). **Top finding:** ~14 % `transport_error` on `game.state` at 500 players, under CPU saturation (backend/gateway/Postgres each ~1 core) and amplified by the harness's single shared `http2.Transport`; the harness itself peaked at 86 % of a core on the same host, so the - figures are pessimistic. Full trip report in [`../loadtest/REPORT-R2.md`](../loadtest/REPORT-R2.md); + figures are pessimistic. Full trip report in [`../loadtest/REPORT.md`](../loadtest/REPORT.md); it feeds R3 (h2c `MaxConcurrentStreams`/timeouts, body-size cap), R6 and R7 (per-player transports, separate hardware, pool/limit sizing). - **CI:** `./loadtest/...` added to the path filter + vet/build/test; `go.work.sum` carries the new deps. @@ -454,7 +454,12 @@ Then Stage 18. one connection per player it bursts into its 2-core cap (the residual 2.49 % `transport_error`); backend ~0.85 core and postgres ~1.4 cores had headroom; **tempo reached its 1 GiB cap**; the backend pool sat at its `MaxOpenConns=25` cap (28 backends); docker logs were unbounded (~14 MiB / 30 min on the backend at - info). Full write-up in [`../loadtest/REPORT-R7.md`](../loadtest/REPORT-R7.md). + info). Full write-up in [`../loadtest/REPORT.md`](../loadtest/REPORT.md). *(Superseded in part: a + later pass modelling the `game.evaluate` hot path traced the gateway's CPU appetite to + **gateway→backend connection churn** — the default 2-idle-connection HTTP transport — not proxying + work. Pooling the connections cut peak gateway CPU ~7× (~1.75 → ~0.26 cores at 500 players) and + removed the ephemeral-port-exhaustion cliff behind the residual `transport_error`, so the gateway is + no longer the binding constraint — postgres is. The 3-core gateway cap below is now generous headroom.)* - **Round-2 tuning (owner-agreed, all in `deploy/docker-compose.yml`, no code change):** gateway **2 → 3 cores + `GOMAXPROCS=3`**; tempo memory **1 → 2 GiB**; backend `MAX_OPEN_CONNS` **25 → 40**; a json-file **log-rotation** default (10m × 3) applied contour-wide via a YAML anchor (level stays info). @@ -464,7 +469,7 @@ Then Stage 18. **burst** run (a single 100 → 500 jump) pegged the gateway at 3 cores (≈296 % sustained, 9.27 % error), confirming it is **connection-CPU-bound** — a true arrival spike is a **horizontal-scaling** lever, not more cores per node (recorded in the prod-sizing recommendation). - - **No schema change → no contour DB wipe.** Bake-back: `loadtest/REPORT-R7.md` (new), `loadtest/README.md`, + - **No schema change → no contour DB wipe.** Bake-back: `loadtest/REPORT.md`, `loadtest/README.md`, `docs/TESTING.md`, the telemetry/observability section of `docs/ARCHITECTURE.md`, the repo-layout line in `CLAUDE.md`. - **UI — Tab-bar navigation redesign** (owner ad-hoc, not on the raw TODO list): drop the hamburger diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 991a19a..a10d3f9 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -128,7 +128,11 @@ dropped). Horizontal scaling is explicit future work. and GCG are unaffected** (they stay decoded concrete characters, §9.1). - **gateway ↔ backend (sync)**: plain HTTP REST/JSON. The gateway injects `X-User-ID` for authenticated requests; `backend` never re-derives identity - from the body. + from the body. Because every sync call targets the one backend host, the + gateway's REST client widens its keep-alive pool well past the stdlib default + of 2 idle connections per host; otherwise the per-request connection churn + exhausts ephemeral ports and burns gateway CPU under load (see + [`../loadtest/REPORT.md`](../loadtest/REPORT.md)). - **backend → gateway (live)**: a single gRPC server-stream carries live events (your-turn, opponent-moved, chat, nudge). The gateway bridges them to the client's in-app stream while the app is open. Out-of-app delivery uses diff --git a/docs/TESTING.md b/docs/TESTING.md index 4429c89..b0ffc7a 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -133,9 +133,9 @@ tests or touching CI. engine tests do). It is **not** part of the per-PR suite's behavioural assertions: it runs ad hoc as a one-shot container against the contour, producing a trip report (bugs + a per-container resource profile) read off the **otelcol `docker_stats` + - postgres_exporter** Grafana dashboard on the contour. Two passes are recorded — the - early [`REPORT-R2.md`](../loadtest/REPORT-R2.md) and the final, tuned - [`REPORT-R7.md`](../loadtest/REPORT-R7.md). See [`../loadtest/README.md`](../loadtest/README.md). + postgres_exporter** Grafana dashboard on the contour. The findings — including the + `game.evaluate` hot-path model and the gateway→backend connection-pool fix — are written + up in [`REPORT.md`](../loadtest/REPORT.md). See [`../loadtest/README.md`](../loadtest/README.md). - **User feedback** — `internal/feedback` unit tests cover the attachment allow-list / content-type and the channel normaliser; the UI covers `detectChannel`, the attachment gate and the feedback wire round-trip (`channel` / `feedback` / `codec` tests) plus a Playwright e2e diff --git a/gateway/internal/backendclient/client.go b/gateway/internal/backendclient/client.go index 527e71a..50ccbb7 100644 --- a/gateway/internal/backendclient/client.go +++ b/gateway/internal/backendclient/client.go @@ -22,6 +22,19 @@ import ( pushv1 "scrabble/pkg/proto/push/v1" ) +// backendMaxIdleConns sizes the REST keep-alive pool to the single backend host. The +// default transport caps idle connections per host at 2 (http.DefaultMaxIdleConnsPerHost), +// which — since every synchronous client call proxies to that one host — forces a fresh +// TCP connection (and a lingering TIME_WAIT socket) for almost every request under load. +// That connection churn burns gateway CPU and exhausts ephemeral ports at scale, all +// while the backend itself sits near-idle. Pooling the connections lets them be reused. +// +// The stress harness measured the effect at 500 concurrent players: the churn collapsed +// from ~26 500 TIME_WAIT sockets to ~0 and peak gateway CPU from ~1.75 to ~0.26 cores, +// with the pool settling at ~225 live connections. 512 keeps ~2x headroom over that +// observed peak so a burst never re-caps the pool. See loadtest/REPORT.md. +const backendMaxIdleConns = 512 + // Client calls the backend's REST API and opens its push gRPC stream. type Client struct { baseURL string @@ -41,9 +54,14 @@ func New(httpURL, grpcAddr string, timeout time.Duration) (*Client, error) { if err != nil { return nil, fmt.Errorf("backendclient: dial push %s: %w", grpcAddr, err) } + // Clone the default transport (keeping its proxy, dialer and timeouts) and widen the + // idle pool so REST calls to the backend reuse connections instead of churning them. + transport := http.DefaultTransport.(*http.Transport).Clone() + transport.MaxIdleConns = backendMaxIdleConns + transport.MaxIdleConnsPerHost = backendMaxIdleConns return &Client{ baseURL: strings.TrimRight(httpURL, "/"), - http: &http.Client{Timeout: timeout}, + http: &http.Client{Timeout: timeout, Transport: transport}, conn: conn, push: pushv1.NewPushClient(conn), }, nil diff --git a/gateway/internal/backendclient/client_internal_test.go b/gateway/internal/backendclient/client_internal_test.go new file mode 100644 index 0000000..244d2a0 --- /dev/null +++ b/gateway/internal/backendclient/client_internal_test.go @@ -0,0 +1,31 @@ +package backendclient + +import ( + "net/http" + "testing" + "time" +) + +// TestBackendTransportPoolsConnections guards the fix for the gateway->backend +// connection churn. Every synchronous client call proxies to the single backend host, +// so the REST client must widen the idle-connection pool past the default per-host cap +// of 2 (http.DefaultMaxIdleConnsPerHost) — otherwise almost every request under load +// opens a fresh TCP connection that then lingers in TIME_WAIT, burning gateway CPU and +// exhausting ephemeral ports. Reverting to the default transport (`&http.Client{...}` +// with no Transport) would silently reintroduce that, so assert the pool is widened. +func TestBackendTransportPoolsConnections(t *testing.T) { + c, err := New("http://backend.invalid", "localhost:9090", time.Second) + if err != nil { + t.Fatalf("New: %v", err) + } + defer func() { _ = c.Close() }() + + tr, ok := c.http.Transport.(*http.Transport) + if !ok { + t.Fatalf("REST transport = %T, want a *http.Transport with a widened idle pool", c.http.Transport) + } + if tr.MaxIdleConnsPerHost <= http.DefaultMaxIdleConnsPerHost { + t.Errorf("MaxIdleConnsPerHost = %d, want > default %d (else per-call connection churn)", + tr.MaxIdleConnsPerHost, http.DefaultMaxIdleConnsPerHost) + } +} diff --git a/loadtest/README.md b/loadtest/README.md index 5ffb762..f0b75eb 100644 --- a/loadtest/README.md +++ b/loadtest/README.md @@ -15,10 +15,12 @@ and prints a trip-report summary. It stays in the repo for repeats. 2. **Drive** (edge protocol over h2c): assembles real 2–4 player games via the invitation flow (`invitation.create` → `invitation.accept`, no robots), then runs each player's turn loop — poll `game.state`, replay `game.history`, generate a legal - **mid-ranked** move with the embedded `scrabble-solver`, and `game.submit_play` - (or pass/exchange). A fraction of turns exercise nudge / chat / check-word / draft / - profile-update / stats. Each player also holds a live `Subscribe` stream. The - moderate ramp is **50 → 200 → 500** concurrent players, ~12 min per step. + **mid-ranked** move with the embedded `scrabble-solver`, **compose it tile by tile with + the debounced `game.evaluate` preview a real client fires** (the hottest gameplay call), + persist a `draft.save`, and `game.submit_play` (or pass/exchange). A fraction of turns + exercise nudge / chat / check-word / draft / profile-update / stats. Each player also + holds a live `Subscribe` stream. The moderate ramp is **50 → 200 → 500** concurrent + players, ~12 min per step. `--eval=false` drops the evaluate model for an A/B baseline. 3. **Hammer**: drives `games.list` from one account far above the per-user rate limit to verify the limiter holds (`rate_limited` results) and measure its cost. 4. **Report**: per-operation latency percentiles, throughput, result-code breakdown, @@ -72,6 +74,8 @@ Key `run` flags (env in parentheses): | `--games-per-player` | `0` (random 3–5) | target concurrent games per player | | `--tick` | `800ms` | per-player op cadence (keeps a player under the per-user limit) | | `--secondary-prob` | `0.08` | chance per tick of a non-move op | +| `--eval` | `true` | model the per-tile `game.evaluate` preview (the gameplay hot path); `false` reproduces the pre-evaluate harness | +| `--eval-recon` | `1` | extra full-composition evaluate re-previews per play (reconsideration), beyond one per placed tile | | `--hammer-workers` / `--hammer-dur` | `20` / `15s` | gateway-hammer (0 workers disables) | | `--reset` / `--cleanup` | `false` | delete harness rows before / after the run | @@ -93,11 +97,11 @@ runs unconditionally. Use an **absolute** path (here via `$PWD`): `go test ./loa runs each package from its own directory, so a relative `BACKEND_DICT_DIR` would not resolve. -## Trip reports +## Trip report -The two stress passes are written up in the repo: the early pass in -[`REPORT-R2.md`](REPORT-R2.md) and the final, tuned pass in -[`REPORT-R7.md`](REPORT-R7.md). +The stress findings — the final run, the `game.evaluate` hot-path model, the +gateway→backend connection-pool fix, and the revised sizing — are written up in +[`REPORT.md`](REPORT.md). ## Caveat diff --git a/loadtest/REPORT-R2.md b/loadtest/REPORT-R2.md deleted file mode 100644 index a119b34..0000000 --- a/loadtest/REPORT-R2.md +++ /dev/null @@ -1,162 +0,0 @@ -# R2 — early stress-run trip report - -The early stress pass for `PRERELEASE.md` R2. It exercises the system through the -**edge protocol** with the `scrabble/loadtest` harness, to surface logic/concurrency -bugs and capture a resource baseline that feeds R3 (edge hardening), R6 (refactor) and -R7 (final tuning). Pass bar: **diagnostic** — the run "passes" by completing without the -harness crashing; findings are recorded below, not gated. - -## Method - -- **Driver:** the `scrabble/loadtest` module, run as a one-shot container on the - `scrabble-internal` docker network (reaching `postgres:5432` and `gateway:8081` - directly, bypassing the host→gateway hairpin). -- **Seed:** 10 000 durable + 1 000 guest accounts with pre-created sessions written - directly to Postgres (token hash matches `backend/internal/session`), so the driver - authenticates without the per-IP-limited auth ops. -- **Games:** assembled through the real **invitation** flow (`invitation.create` → - `invitation.accept`), 2–4 players each, no robots; variants spread over - scrabble_en / scrabble_ru / erudit_ru. -- **Play:** each virtual player holds a live `Subscribe` stream and, per tick, polls - `game.state`, replays `game.history` and submits a **mid-ranked** legal move generated - locally by the embedded `scrabble-solver` (the edge carries no board), or - passes/exchanges; a fraction exercise nudge / chat / check-word / draft / profile / - stats. A separate **gateway-hammer** floods `games.list` from one account. -- **Scale:** moderate ramp **50 → 200 → 500** concurrent players, 10 min/step (the - agreed moderate profile; harness and contour share this host's CPU). -- **Resource capture:** `docker stats` (docker API) sampled every 28 s for per-container - CPU/memory; Prometheus for edge latency/throughput, `postgres_exporter` internals and - per-service Go runtime metrics. - -## Run configuration - -``` -loadtest run --durable 10000 --guest 1000 --steps 50,200,500 --step-dur 10m \ - --tick 800ms --hammer-workers 20 --hammer-dur 15s --cleanup -``` - -Date: 2026-06-09. Contour: the R1-baseline schema, freshly deployed with the R2 -exporters. Seeded population removed by `--cleanup` afterwards. - -## Findings - -### Validated (fixed within R2) -- **Harness draft payload.** `draft.save` first returned `bad_request`: the backend - draft DTO's `rack_order` is a string (the harness sent `[]`). Fixed → `ok`. -- **Harness profile marker.** `profile.update` first returned `invalid_profile`: the - editable-display-name validator (`backend/internal/account/profile.go`) forbids digits - and colons, but the seed marker was `lt:…`. Switched the marker to a distinctive - letters-only string → `ok`. Cleanup still matches it. - -### By-design behaviour (correctly exercised, not bugs) -- **`chat_not_your_turn`** — chat is gated to the sender's turn - (`backend/internal/social/chat.go`); off-turn posts are correctly rejected. -- **`nudge_own_turn`** — you nudge the player whose turn it is, so a nudge on your own - turn is correctly rejected. The harness nudges/chats at random ticks, so a share of - these codes is expected. - -### Observability gap (key R7 input) -- **cAdvisor yields only the root cgroup on the contour host.** Its docker factory - registers, but per-container init fails — `failed to identify the read-write layer ID - … /rootfs/var/lib/docker/image/overlayfs/…: no such file or directory` — because this - host's `/var/lib/docker` is a **separate XFS mount** not visible under cAdvisor's - `/rootfs` bind (the existing galaxy deployment on the same host has the same - limitation). So the **Scrabble — Resources** dashboard's per-container panels are empty - here, and per-container CPU/RSS for this run was captured via `docker stats` instead. - Postgres internals (`postgres_exporter`) and per-service Go runtime metrics - (`go_*` by `service_name`) work. **Recommendation for R7:** adopt the otelcol - **`docker_stats`** receiver (already the contrib image) — it reads per-container stats - via the docker API with no cgroup dependency — and/or run the final pass on hardware - where cAdvisor resolves containers. (Decision to confirm with the owner.) - -### Run results - -The ramp ran clean to 500 players with no harness crash, no deadlock and -`stream errors: 0`; cleanup removed all 11 000 seeded accounts (and their ~941 games). - -- **Ramp:** step 1 = 50 players / 90 games, step 2 = 200 / 282, step 3 = 500 / 569. -- **Volume (30 min):** 1.20 M total edge calls, 659 req/s average. Real gameplay at - scale: **48 870 committed plays**, 52 772 `your_turn` + 159 631 `opponent_moved` - events, **2 798 games finished**. -- **Latency under load (peak, step 3):** `game.state` p50 ≈ 100 ms, p90/p99 in the - 200–500 ms buckets, max 849 ms; `game.submit_play` similar (p99 ≤ 500 ms, max 490 ms). - Lobby ops stayed fast (invitation/games.list p99 ≤ 10 ms). -- **Rate limiter holds.** The gateway-hammer sent 522 667 `games.list` from one account; - **522 486 (99.97 %) were `rate_limited`**, only 135 `ok` (the burst). Rejections are - cheap — p99 = 2 ms — and the gateway sustained ~16 k req/s of rejections during the - flood. The per-user limiter behaves as designed (R3 input: the cost is negligible). - -**Top finding — `transport_error` under saturation.** At 500 players ~14 % of -`game.state` calls (72 429 / 519 067) and a few % of the other ops returned a Connect -`transport_error` (not a domain code). It correlates with the CPU saturation below: the -backend/gateway are pinned near one core each while the host also runs the 86 %-core -harness, so the edge sheds load (resets/timeouts) at the knee. It is **amplified by a -harness artifact** — all 500 virtual players multiplex over a *single* shared -`http2.Transport`, so 500 persistent `Subscribe` streams plus Execute calls press on one -HTTP/2 connection's concurrent-stream limit; real clients each use their own connection. -**Actions:** R7 harness — give each player (or a pool) its own transport, and run on -hardware not shared with the contour; R3 — confirm the gateway's h2c -`MaxConcurrentStreams` and edge timeouts are sized for many persistent streams. - -**Minor findings:** -- `unauthenticated` on a tiny share (188 / 519 067 `game.state`, ~0.04 %) — transient - session-resolve failures under load; worth a glance in R3 but not material. -- one `internal` on `game.pass` (1 / 4 788). -- `game_finished` dominates `chat.nudge`/`chat.post` (≈ 3 900 each): the harness keeps - secondary ops on games that already ended. Harness refinement — drop finished games - from the rotation (R7). -- `nudge_own_turn` / `chat_not_your_turn` / `nudge_too_soon` are the expected turn/rate - gates, correctly exercised. - -## Resource baseline - -Per-container peak during step 3 (500 players), from `docker stats`: - -| container | peak CPU | memory | -|-----------|---------:|-------:| -| scrabble-backend | **99 %** (~1 core) | 91 MiB | -| scrabble-gateway | **93 %** | 76 MiB | -| scrabble-postgres | **90 %** | 69 MiB | -| scrabble-loadtest (harness) | **86 %** | 42 MiB | -| scrabble-otelcol | 10 % | 110 MiB | -| scrabble-tempo | 9 % | 446 MiB | -| prometheus / postgres-exporter | ~0 % | 46 / 16 MiB | - -- **The contour is CPU-bound at 500 concurrent players:** backend, gateway and Postgres - each saturate ~1 core (single-instance MVP config), so the system draws ~3 cores at - this scale; memory is modest (≤ 100 MiB per Go service). This is the sizing input for - R7 (pool sizes, GOMAXPROCS, container limits) and the prod cutover. -- **Caveat:** the harness itself peaked at **86 % of a core** on the *same host*, so the - step-3 latency and `transport_error` figures are pessimistic — the contour competed - with the generator for CPU. A clean ceiling needs separate hardware (R7). -- **Postgres:** peak 28 backend connections, ~5 581 commits/s at the peak, **100 % cache - hit ratio** (no disk reads) — the DB was comfortable; CPU, not I/O, is its limit here. -- **Goroutines:** backend 638, gateway **1 698** (it holds the 500 `Subscribe` streams + - per-request goroutines), telegram 49 — all stable, no leak across the ramp. - -## Recommendations feeding later phases -- **R3 (edge hardening):** the per-user limiter holds (99.97 % rejected, p99 2 ms) — add - the per-IP body-size cap on top. Investigate the **~14 % `transport_error` on - `game.state` at 500 players**: confirm the gateway h2c `MaxConcurrentStreams` and edge - read/write timeouts are sized for many persistent `Subscribe` streams, and glance at the - ~0.04 % transient `unauthenticated` resolves under load. -- **R6 (refactor):** no logic bug forced a code change beyond the two harness-payload - fixes; the run surfaced no deadlock or goroutine leak across the ramp. -- **R7 (final tuning + stress):** (1) fix the per-container observability gap — adopt the - otelcol `docker_stats` receiver so Grafana shows per-container CPU/RSS on the contour; - (2) refine the harness — per-player/pooled transports and dropping finished games from - the rotation — and run on hardware **not** shared with the contour; (3) size pools / - GOMAXPROCS / container limits from the CPU-bound peak (~1 core each for backend, gateway, - Postgres at 500 players). - -## Re-running - -See [`README.md`](README.md). Briefly, from the repo root: - -```sh -docker build -f loadtest/Dockerfile -t scrabble-loadtest . -docker run --rm --name scrabble-loadtest --network scrabble-internal \ - -e POSTGRES_PASSWORD=… scrabble-loadtest run # add --reset on a re-run -``` - -The harness stays in the repo for the R7 repeat. diff --git a/loadtest/REPORT-R7.md b/loadtest/REPORT-R7.md deleted file mode 100644 index 2f03999..0000000 --- a/loadtest/REPORT-R7.md +++ /dev/null @@ -1,212 +0,0 @@ -# R7 — final stress-run trip report - -The final pre-release stress pass for [`PRERELEASE.md`](../PRERELEASE.md) R7. It re-runs -the R2 harness (`scrabble/loadtest`) against the **final, refactored system** on a -freshly redeployed contour, to confirm the system holds at scale and to settle the -resource sizing (container limits, `GOMAXPROCS`, pools, rate limits, log levels) before -the Stage 18 prod cutover. Pass bar: **diagnostic + a tuning decision** — the run -"passes" by completing cleanly; the per-container resource profile drives the tuning -recorded below. Companion to the early pass, [`REPORT-R2.md`](REPORT-R2.md). - -## What changed since the R2 pass - -- **Harness — per-player transports.** Each virtual player now owns its `edge.Client` - (its own `http2.Transport` / h2c connection carrying both its `Subscribe` stream and - its `Execute` calls), instead of all players multiplexing over one shared transport. - R2 traced the ~14 % `transport_error` on `game.state` at 500 players to that single - shared connection's stream limit; per-player connections mirror real clients and - remove the artifact, so this pass measures the system, not the harness. -- **Harness — drop finished games.** `playTurn` reports a finished game and the player - drops it from its rotation, so secondary ops stop hitting `game_finished` on ended - games (the other R2 harness finding). -- **Observability — otelcol `docker_stats`.** cAdvisor (which resolves only the root - cgroup on this host — separate-XFS `/var/lib/docker`) is replaced by the otelcol - `docker_stats` receiver, reading per-container CPU/memory/network from the Docker API. - Per-container panels now populate on the contour host. (`api_version` pinned to 1.44; - the daemon's minimum is 1.40.) -- **Contour — container limits + `GOMAXPROCS`.** `deploy.resources.limits` now bound - every service; the Go services pin `GOMAXPROCS` to their CPU limit so the runtime - matches the cgroup quota. Starting values were generous over the R2 peak; this pass - validates them and settles the agreed sizing (below). - -## Method - -Unchanged from R2 except for the per-player transports and the dropped-finished-games -refinement above: - -- **Driver:** the `scrabble/loadtest` module, run as a one-shot container on the - `scrabble-internal` docker network (reaching `postgres:5432` / `gateway:8081` - directly), capped at `--cpus 3` so the contour keeps the host's spare cores. -- **Seed:** 10 000 durable + 1 000 guest accounts with pre-created sessions written - straight to Postgres (token hash matches `backend/internal/session`). -- **Games:** assembled through the real **invitation** flow, 2–4 players each, no - robots; variants over scrabble_en / scrabble_ru / erudit_ru. -- **Play:** each player holds a live `Subscribe` stream and, per tick, polls - `game.state`, replays `game.history` and submits a **mid-ranked** legal move generated - locally by the embedded `scrabble-solver`, or passes / exchanges; a fraction exercise - nudge / chat / check-word / draft / profile / stats. A separate **gateway-hammer** - floods `games.list` from one account. -- **Scale:** the same moderate ramp **50 → 200 → 500** concurrent players, 10 min/step. -- **Resource capture:** `docker stats` (docker API) sampled every ~20 s for per-container - CPU/memory; the otelcol **`docker_stats`** receiver → Prometheus → the Grafana - **Scrabble — Resources** dashboard for the same per-container series; `postgres_exporter` - internals and per-service Go runtime metrics. - -## Run configuration - -``` -docker run --rm --cpus=3 --name scrabble-loadtest --network scrabble-internal \ - -e POSTGRES_PASSWORD=… scrabble-loadtest \ - run --durable 10000 --guest 1000 --steps 50,200,500 --step-dur 10m \ - --tick 800ms --hammer-workers 20 --hammer-dur 15s --reset --cleanup -``` - -Date: 2026-06-10. Contour: the R1-baseline schema, freshly redeployed with the R7 -container limits / `GOMAXPROCS` (backend/gateway/postgres capped at 2 cores + 512 MiB, -`GOMAXPROCS=2`) and the `docker_stats` observability. Seeded population removed by -`--cleanup` afterwards. - -## Findings - -The ramp ran clean to 500 players — no harness crash, no deadlock, `stream errors: 0` — -and cleanup removed all 11 000 seeded accounts. - -- **Volume (1827 s):** 821 680 edge calls (449.7 req/s incl. the hammer). Real gameplay - at scale: **50 916 committed plays**, 4 817 passes, 2 931 games finished; 165 755 - `opponent_moved` + 54 864 `your_turn` events. -- **The per-player transport fix worked.** `game.state` returned `transport_error` on - **3 173 / 127 403 = 2.49 %** of calls — down from R2's ~14 % on the same step. Other - ops were lower still (`game.history` 0.43 %, `game.submit_play` 0.28 %). The residual - is the gateway bursting into its 2-core cap (see the profile below), not the harness. -- **Dropping finished games worked.** `game_finished` on `chat.nudge` / `chat.post` fell - to **35 / 36** (R2: ≈ 3 900 each) — secondary ops no longer hammer ended games. -- **The limiter holds.** The gateway-hammer sent 565 152 `games.list`; **564 979 - (99.97 %) were `rate_limited`** (154 ok burst, 19 deadline), p99 = 2 ms, ~309 req/s of - rejections sustained — unchanged from R2. -- **Latency (peak):** `game.state` p50 ≈ 100 ms, p99 in the 2000 ms bucket (max 2549 ms); - `game.submit_play` p50 100 / p99 1000 ms bucket. Lobby ops stayed fast - (invitation / games.list p99 ≤ 10 ms). The p99 tail correlates with the gateway - burst-throttling, not the backend (which stayed at ~0.85 core). - -## Resource profile - -Per-container peak during step 3 (500 players), with the R7 starting limits in force -(backend/gateway/postgres capped at 2 cores / 512 MiB). Two CPU columns: `docker stats` -samples a ~1 s window (catches bursts); the otelcol `docker_stats` receiver averages over -its 30 s collection interval (smooths them) — they agree within sampling error, which -validates the new observability path. - -| container | CPU burst (1 s) | CPU sustained (30 s) | CPU cap | mem peak | mem cap | -|-----------|----------------:|---------------------:|--------:|---------:|--------:| -| scrabble-gateway | **217 %** (at cap) | ~145 % | 200 % | 167 MiB | 512 MiB | -| scrabble-postgres | 138 % | ~153 % | 200 % | 117 MiB | 512 MiB | -| scrabble-backend | 85 % | ~89 % | 200 % | 116 MiB | 512 MiB | -| scrabble-tempo | 33 % | — | (none) | **1024 MiB** (at cap) | 1024 MiB | -| scrabble-otelcol | 11 % | — | (none) | 131 MiB | 512 MiB | -| scrabble-loadtest (harness) | 157 % | — | 300 % | 369 MiB | — | - -- **The gateway is the binding constraint.** With one h2c connection per player it draws - ~1.45 cores sustained and **bursts to its 2-core cap** at 500 players, throttling - briefly — the source of the 2.49 % `transport_error`. R2 saw only ~0.93 core because - all 500 players shared one connection; the +~0.5 core is the realistic per-connection - overhead (500 separate HTTP/2 connections). This is a sizing fact, not a regression. -- **backend is over-provisioned** (~0.85 core vs a 2-core cap); **postgres** (~1.4 cores) - has headroom; both stayed ≤ 120 MiB. -- **tempo reached its 1 GiB memory cap** (R2: 446 MiB) — an OOM risk under sustained - tracing. -- **Postgres backends peaked at 28**, with the backend pool at its `MaxOpenConns=25` cap. - Cache hit stayed ~100 % (no disk reads); CPU, not I/O, is the limit. -- **docker log volume (30 min):** backend 14.2 MiB, gateway 4.6 MiB, postgres 0.04 MiB — - the backend's per-request latency line at info dominates, and json-file logs had no - rotation. - -## Tuning applied - -Agreed from the profile (all in `deploy/docker-compose.yml`; no code change — the pool -is already env-driven): - -| knob | from | to | why | -|------|------|----|-----| -| gateway CPU + `GOMAXPROCS` | 2 cores / 2 | **3 cores / 3** | it bursts into the 2-core cap at 500 players (the 2.49 % `transport_error`); 3 absorbs the bursts | -| tempo memory | 1 GiB | **2 GiB** | it reached the 1 GiB cap (OOM risk) | -| backend `MAX_OPEN_CONNS` | 25 | **40** | the pool sat at its 25-conn cap at peak; headroom trims the p99 tail | -| docker logs | unbounded | **json-file 10m × 3** | bound the ~14 MiB / 30 min backend log; level stays `info` | - -Left as-is: backend / postgres at 2 cores / 512 MiB (peak ~0.85 / ~1.4 cores — headroom -is cheap on the shared host); the per-user rate limiter and `h2cMaxConcurrentStreams=250` -(per-connection now, ~1 stream each — ample) and cache TTLs (no pressure observed). - -### Validation re-run - -Re-running the **same gradual ramp** (50 → 200 → 500) on the tuned contour confirms the -fix: - -- **`game.state` `transport_error` fell to 0.72 %** (853 / 119 051), down from 2.49 % at - 2 cores. The latency tail also improved — p99 in the 1000 ms bucket, max 1220 ms (was - the 2000 ms bucket, max 2549 ms). -- The **gateway peaked at ~2 cores** (≈196 % on the 30 s gauge) — now comfortably **under - the 3-core cap**, so it no longer throttles. backend ~1 core, postgres ~1.3 cores. -- **tempo peaked at ~1.27 GiB** — under the new 2 GiB cap (it would have OOM-ed at 1 GiB). -- Drop-finished still holds (`game_finished` on chat 41/42); the limiter still rejects - 99.97 % of the hammer at p99 2 ms; `stream errors: 0`. - -A separate **burst stress** (a single 100 → 500 jump — 400 players connecting at once) -**pegged the gateway at 3 cores** (≈296 % sustained) and pushed `game.state` -`transport_error` to 9.27 %. The gateway is **connection-CPU-bound and bursty**: average -load is ~1 core, but a mass-simultaneous connection storm saturates whatever single-node -cap it is given. Real arrivals are gradual (the canonical run), where 3 cores has -headroom; the lever for a true arrival spike is **horizontal scaling**, not more cores per -node — carried into the prod recommendation below. - -## Prod-sizing recommendation (Stage 18) - -The contour is **CPU-bound and gateway-led** at 500 concurrent players. Carry these to the -prod contour env (the same compose, `PROD_*` values): - -- **gateway: ≥ 3 cores** per ~500 concurrent players, `GOMAXPROCS` pinned to the limit — - it scales with the **connection count**, not just the request rate; beyond one node's - worth, scale the gateway **horizontally** rather than vertically. -- **backend: ~1–2 cores**, pool 40 — comfortable; the work is light per request. -- **postgres: ~2 cores / ≥ 512 MiB** — ~1.4 cores at 500 players, 100 % cache hit. -- **tempo: ≥ 2 GiB**; the Go services run under ~170 MiB (256 MiB would suffice, 512 is - safe); pin `GOMAXPROCS` to each CPU limit; keep json-file rotation. -- Memory is not the constraint anywhere; CPU is. - -### VPS / VDS sizing (single-host contour) - -The whole contour (the app + the observability stack) runs on one host via -`docker-compose`. The tiers below are grounded in the R7 profile (**≈5.5 cores / ≈2.5 GiB -RAM peak at 500 concurrent players**; ≈0.5 GiB idle) and the **measured** on-disk -footprint: prod images ≈2.4 GB; the Tempo volume **3.1 GB at 72 h** retention; Prometheus -≈1–2 GB at 15 d; the game DB 23 MiB and growing with history. CPU and disk grow; RAM has -the most slack. - -| tier | CPU | RAM | disk | handles | -|------|-----|-----|------|---------| -| **Minimum** | 2 cores | 2 GiB | 20 GiB | ~up to ~150 concurrent; lower the compose limits (gateway 1.5 / backend·postgres 1 / tempo 1 GiB) to fit the box | -| **Average** (reasonable load) | 4 cores | 4 GiB | 40 GiB | ~300–400 concurrent comfortably; the tested 500 with occasional gateway burst-throttling | -| **Maximum** (worry-free) | 8 cores | 8 GiB | 80 GiB | 500+ concurrent with full gateway burst headroom (its 3-core cap) + room to grow; the compose limits fit as-is | - -- The per-service limits in `docker-compose.yml` are tuned for the **Average/Maximum** - target (the gateway alone caps at 3 cores). On the **Minimum** tier, scale them down to - match the host or the caps over-subscribe it. -- **Disk is dominated by observability retention + DB growth.** Tempo (72 h traces) and - Prometheus (15 d metrics) are the main levers — shorten the windows (or move Tempo to - object storage) to cut disk; Postgres grows with game history, so budget for months of - it; container logs are already capped (json-file 10m × 3 ≈ 30 MiB each). -- **RAM** rarely binds: the contour peaks ≈2.5 GiB at 500 players and the sum of all - configured limits is ≈5.6 GiB, so 8 GiB never strains. -- Beyond one host's worth of players, scale the **gateway horizontally** (it is - connection-CPU-bound) rather than ordering an ever-bigger box. - -## Re-running - -See [`README.md`](README.md). Briefly, from the repo root: - -```sh -docker build -f loadtest/Dockerfile -t scrabble-loadtest . -docker run --rm --cpus=3 --name scrabble-loadtest --network scrabble-internal \ - -e POSTGRES_PASSWORD=… scrabble-loadtest run --reset --cleanup -``` - -The harness stays in the repo for future repeats. diff --git a/loadtest/REPORT.md b/loadtest/REPORT.md new file mode 100644 index 0000000..ef6361d --- /dev/null +++ b/loadtest/REPORT.md @@ -0,0 +1,184 @@ +# loadtest — stress trip report + +The pre-release stress write-up for [`PRERELEASE.md`](../PRERELEASE.md). It drives the +`scrabble/loadtest` harness against a freshly redeployed test contour to confirm the +system holds at scale and to settle resource sizing before the prod cutover. The harness +stays in the repo for repeats; see [`README.md`](README.md) for how to run it. + +This report supersedes the earlier per-phase notes. The harness has been through three +passes: an early diagnostic, a tuning pass that sized container limits / `GOMAXPROCS`, and +this final pass — which **added the per-tile `game.evaluate` preview to the model** (the +hottest real gameplay call, previously unmodelled) and, with it, surfaced and fixed the +**gateway→backend connection-pool bottleneck** described below. The numbers here are from +that final pass. + +## What it models + +The harness seeds a large account population with pre-created sessions directly in +Postgres, then drives virtual players through the **gateway edge protocol** (h2c) in real +games assembled via the invitation flow. Each player owns its own `edge.Client` (its own +h2c connection, like a real client), holds a live `Subscribe` stream, and per tick polls +`game.state`, replays `game.history`, generates a legal **mid-ranked** move with the +embedded `scrabble-solver`, and submits it (or passes/exchanges). A fraction of ticks +exercise nudge / chat / check-word / draft / profile / stats. A separate **gateway-hammer** +floods `games.list` to verify the rate limiter. + +### The evaluate hot path (this pass) + +A real client previews every tentative play as the user arranges tiles: the UI fires a +debounced `game.evaluate` (legality + score) on each placement change while it is the +player's turn. Over a single composed word that is **several evaluate calls per turn** — +far more than the one `submit_play` — so `game.evaluate` is the single hottest gameplay +request at scale. The earlier passes did not model it at all (they submitted directly), +which understated the real load. + +This pass models it: when a player composes a play of *K* newly-placed tiles, it fires one +`evaluate` per landed tile (a growing prefix of the tiles), plus a small number of +full-composition re-previews for reconsideration, spaced by a human-paced gap (the client's +250 ms debounce), then one `draft.save`, then `submit_play`. `--eval=false` reproduces the +pre-evaluate harness for an A/B baseline; `--eval-recon` tunes the reconsideration count. + +`game.check_word` is a *different*, manual "look this word up" panel (throttled, on demand) +— not the per-tile call — and is exercised separately as a secondary op. + +## Final run (eval-on, after the connection-pool fix) + +Contour: backend / postgres capped at 2 cores / 512 MiB (`GOMAXPROCS=2`), gateway at +3 cores / 512 MiB (`GOMAXPROCS=3`), per the tuned `deploy/docker-compose.yml`. Gradual ramp +**50 → 200 → 500** concurrent players, 4 min/step, `--tick 800ms`, gateway-hammer on. The +harness ran as a one-shot container on `scrabble-internal`, capped at `--cpus 3`. The DB was +wiped before the run (`DROP SCHEMA backend CASCADE`); the seeded population was removed by +`--cleanup` afterwards. + +Per-operation results at the 500-player peak (740 s, gameplay rows; the hammer row is the +limiter probe): + +| operation | count | req/s | p50 | p99 | max | notes | +|-----------|------:|------:|----:|----:|----:|-------| +| game.evaluate | 85 721 | 115.9 | 1 ms | 200 ms | 193 ms | **the hot path** — all ok | +| game.state | 115 926 | 156.7 | 100 ms | 200 ms | 260 ms | transport_error 86 (0.07 %) | +| game.history | 22 258 | 30.1 | 5 ms | 100 ms | 195 ms | all ok | +| draft.save | 23 031 | 31.1 | 2 ms | 200 ms | 194 ms | all ok | +| game.submit_play | 21 704 | 29.3 | 1 ms | 200 ms | 274 ms | ok 3 902; not_your_turn / illegal_play are concurrent-play races (see caveat) | +| hammer:games.list | 522 756 | 706.7 | 1 ms | 2 ms | 53 ms | **99.97 % rate_limited** — limiter holds | + +- **Volume:** 802 200 total edge calls (1 084 req/s incl. the hammer; ~377 req/s of real + gameplay). `stream errors: 0`. Live events: 11 199 `opponent_moved`, 4 153 `your_turn`. +- **`game.evaluate` is now the dominant gameplay write-path call** at ~116 req/s — second + only to the `game.state` poll — and it is cheap: p50 1 ms, p99 200 ms, effectively zero + errors. The backend serves it from the in-memory live-game cache plus a single + `GetGame` read. +- **Latency stayed healthy** under the heavier evaluate load: every gameplay op p99 ≤ 200 ms. +- **The limiter holds** unchanged: 99.97 % of the hammer rejected at p99 2 ms. + +### Peak CPU (500 players) + +| container | CPU peak | cap | +|-----------|---------:|----:| +| scrabble-postgres | **165 %** (~1.65 cores) | 200 % | +| scrabble-backend | 77 % (~0.77 core) | 200 % | +| scrabble-gateway | **26 %** (~0.26 core) | 300 % | +| scrabble-loadtest (harness) | 42 % | 300 % | + +Memory stayed modest everywhere (Go services ≤ ~90 MiB). **Postgres is now the busiest +service** — it has headroom (1.65 of 2 cores) but is the scaling axis. The gateway, after +the fix below, is near-idle. + +## The headline finding: gateway→backend connection churn + +The gateway proxies every synchronous client call to the single backend host over REST. +Its backend HTTP client used the default transport, whose **`MaxIdleConnsPerHost` is 2** +(`http.DefaultMaxIdleConnsPerHost`). So the gateway kept only **2** keep-alive connections +to the backend and opened — then closed — a fresh TCP connection for almost every other +call. Measured at the gateway's network namespace: + +| | gateway→backend sockets | +|---|---| +| before (eval-on, 500 players) | **TIME_WAIT ≈ 26 500**, ESTABLISHED 2 | +| after (eval-on, 500 players) | TIME_WAIT ≈ 0 (steady state), **ESTABLISHED ≈ 225 (reused)** | + +26 500 TIME_WAIT sockets is the connection **churn**: ~440 new connections per second, +each a full TCP handshake + teardown, the socket then lingering 60 s. That count sits right +under the ~28 000 ephemeral-port ceiling — the latent cliff that produced the residual +`transport_error` the earlier passes chased on the *client* side (h2c streams) but never +eliminated, because the real cause was here, on the *backend* side. + +The fix is one custom `http.Transport` with a wide idle pool +(`gateway/internal/backendclient/client.go`, `backendMaxIdleConns`). Before / after, same +eval-on workload at 500 players: + +| metric | before fix | after fix | +|--------|-----------:|----------:| +| gateway→backend TIME_WAIT | ~26 500 | **~0** | +| gateway CPU peak | **175 %** (~1.75 cores) | **26 %** (~0.26 core) | +| game.state p99 | 500 ms | 200 ms | + +**The churn was burning ~1.5 gateway cores of pure connection setup/teardown.** Removing it +cut peak gateway CPU ~7× and erased the port-exhaustion cliff. The backend and postgres CPU +are unchanged — they do the real work; only the gateway's wasted overhead disappeared. The +pool settles at ~225 live connections at 500 players; the constant is set to 512 for ~2× +headroom. + +## Sizing — why the old "≈150 concurrent / 2-core" figure was a bug, not a floor + +The earlier tuning pass concluded the gateway was the binding constraint — "size it for +≥ 3 cores per 500 players, scale it horizontally" — and the single-host "minimum" tier +topped out near ~150 concurrent. **That was sizing around the connection-churn bug.** The +gateway drew ~1.75–3 cores not from proxying work but from churning backend connections; +the backend behind it sat near-idle the whole time. + +With the churn fixed, at **500 concurrent players** the app draws roughly: + +- **gateway ≈ 0.26 core** (was ~3) — no longer the constraint, +- **backend ≈ 0.77 core**, +- **postgres ≈ 1.65 cores** — now the busiest, with headroom, + +≈ **2.7 app cores total** (down from the ~5.5-core contour peak the tuning pass recorded, +*and* under a heavier, more realistic workload that now includes `game.evaluate`). Postgres, +not the gateway, is the scaling axis. + +Revised single-host guidance (app + co-resident observability stack on one box): + +| tier | CPU | RAM | handles | +|------|-----|-----|---------| +| **Minimum** | 2 cores | 2 GiB | comfortably the low hundreds of concurrent — the gateway no longer eats cores; postgres + the observability stack set the limit | +| **Average** | 4 cores | 4 GiB | 500 concurrent with headroom | +| **Maximum** | 8 cores | 8 GiB | 500+ with full burst headroom and room to grow | + +The gateway's compose limit can drop well below its old 3 cores; it is now connection-pool +bound, not connection-CPU bound. Memory was never the constraint. Disk is still dominated +by observability retention (Tempo, Prometheus) + DB growth — unchanged from before. + +## Next optimisation (noted, not done) + +`game.evaluate` reads `GetGame` from Postgres on **every** call (to re-check seat +membership and status) before validating against the cached live game. At ~116 evaluate +req/s on top of the `game.state` / `game.history` reads, that is the bulk of the postgres +load now. Caching the game metadata alongside the live engine game in +`backend/internal/game` would cut it, but it touches persistence/cache coherency (a higher +blast-radius change) and postgres still has headroom, so it is left as a deliberate +follow-up rather than bundled here. + +## Caveat — harness fidelity + +The harness's `not_your_turn` and `illegal_play` on `submit_play` are concurrent-play +artifacts, not system errors: it generates a move from a locally replayed board, and a +fast opponent (or a transport hiccup) can move between the state fetch and the submit, +leaving the move out of turn or illegal on the now-changed board. A real client previews +with `evaluate` and only submits a legal, in-turn play. These rejections are cheap domain +outcomes (HTTP-ok with a stable code) and do not change the request *load*, which is what +the run measures. The harness also shares the host CPU with the contour (capped with +`--cpus`); a fully isolated ceiling on separate hardware remains future work. + +## Re-running + +From the repo root: + +```sh +docker build -f loadtest/Dockerfile -t scrabble-loadtest . +docker run --rm --cpus=3 --name scrabble-loadtest --network scrabble-internal \ + -e POSTGRES_PASSWORD="$TEST_POSTGRES_PASSWORD" scrabble-loadtest run --reset --cleanup +``` + +`--eval=false` reproduces the pre-evaluate baseline for comparison. The authoritative hard +reset of the contour DB remains `DROP SCHEMA backend CASCADE` + a backend restart. diff --git a/loadtest/cmd/loadtest/main.go b/loadtest/cmd/loadtest/main.go index aaec899..29fb3fc 100644 --- a/loadtest/cmd/loadtest/main.go +++ b/loadtest/cmd/loadtest/main.go @@ -73,6 +73,8 @@ func cmdRun(ctx context.Context, log *slog.Logger, args []string) error { gpp := fs.Int("games-per-player", 0, "target concurrent games per player (0 => random 3..5)") tick := fs.Duration("tick", 800*time.Millisecond, "per-player operation cadence") secProb := fs.Float64("secondary-prob", 0.08, "chance per tick of a non-move operation") + eval := fs.Bool("eval", true, "model the per-tile evaluate preview (the realistic gameplay hot path); --eval=false reproduces the pre-evaluate harness for an A/B baseline") + evalRecon := fs.Int("eval-recon", 1, "extra full-composition evaluate re-previews per play (reconsideration), beyond one per placed tile") hammerWorkers := fs.Int("hammer-workers", 20, "gateway-hammer concurrent callers (0 disables)") hammerDur := fs.Duration("hammer-dur", 15*time.Second, "gateway-hammer duration") reset := fs.Bool("reset", false, "delete prior harness rows before seeding") @@ -117,6 +119,7 @@ func cmdRun(ctx context.Context, log *slog.Logger, args []string) error { cfg := scenario.RealisticConfig{ Steps: steps, StepDur: *stepDur, GamesPerPlayer: *gpp, Tick: *tick, SecondaryProb: *secProb, + Eval: *eval, EvalRecon: *evalRecon, } if err := drv.RunRealistic(ctx, pool, cfg); err != nil && !errors.Is(err, context.Canceled) { return err diff --git a/loadtest/internal/edge/client.go b/loadtest/internal/edge/client.go index 9b7d41b..7cb702d 100644 --- a/loadtest/internal/edge/client.go +++ b/loadtest/internal/edge/client.go @@ -24,6 +24,7 @@ const ( msgSubmitPlay = "game.submit_play" msgPass = "game.pass" msgExchange = "game.exchange" + msgEvaluate = "game.evaluate" msgState = "game.state" msgHistory = "game.history" msgGamesList = "games.list" diff --git a/loadtest/internal/edge/encode.go b/loadtest/internal/edge/encode.go index eeb6c3a..8075f76 100644 --- a/loadtest/internal/edge/encode.go +++ b/loadtest/internal/edge/encode.go @@ -63,6 +63,33 @@ func submitPlay(gameID string, tiles []PlayTile) []byte { return b.FinishedBytes() } +// evalReq builds an EvalRequest payload (game id plus the tentative newly-placed tiles). +// It mirrors submitPlay's shape — the backend infers the play's orientation the same way — +// so a preview previews exactly what submitting those tiles would score. +func evalReq(gameID string, tiles []PlayTile) []byte { + b := flatbuffers.NewBuilder(256) + gid := b.CreateString(gameID) + offs := make([]flatbuffers.UOffsetT, len(tiles)) + for i, t := range tiles { + fb.PlayTileStart(b) + fb.PlayTileAddRow(b, int32(t.Row)) + fb.PlayTileAddCol(b, int32(t.Col)) + fb.PlayTileAddLetter(b, t.Letter) + fb.PlayTileAddBlank(b, t.Blank) + offs[i] = fb.PlayTileEnd(b) + } + fb.EvalRequestStartTilesVector(b, len(offs)) + for i := len(offs) - 1; i >= 0; i-- { + b.PrependUOffsetT(offs[i]) + } + tilesVec := b.EndVector(len(offs)) + fb.EvalRequestStart(b) + fb.EvalRequestAddGameId(b, gid) + fb.EvalRequestAddTiles(b, tilesVec) + b.Finish(fb.EvalRequestEnd(b)) + return b.FinishedBytes() +} + // exchange builds an ExchangeRequest payload swapping the listed rack tiles (alphabet // indices; 255 a blank). func exchange(gameID string, tiles []byte) []byte { diff --git a/loadtest/internal/edge/ops.go b/loadtest/internal/edge/ops.go index ab86fa1..7c782ee 100644 --- a/loadtest/internal/edge/ops.go +++ b/loadtest/internal/edge/ops.go @@ -53,6 +53,15 @@ func (c *Client) Exchange(ctx context.Context, token, gameID string, tiles []byt return decodeMoveResultGame(r.Payload), r.Code, nil } +// Evaluate previews a tentative play's legality and score without committing it. It is +// the per-tile composition call a real client fires (debounced) on every change while +// arranging a word, so it is the hottest gameplay request at scale. The harness records +// only the result code and latency; an illegal preview is a successful "ok" call. +func (c *Client) Evaluate(ctx context.Context, token, gameID string, tiles []PlayTile) (string, error) { + r, err := c.execute(ctx, token, msgEvaluate, evalReq(gameID, tiles)) + return r.Code, err +} + // Nudge prods the opponent whose turn it is. func (c *Client) Nudge(ctx context.Context, token, gameID string) (string, error) { r, err := c.execute(ctx, token, msgNudge, gameAction(gameID)) diff --git a/loadtest/internal/scenario/scenario.go b/loadtest/internal/scenario/scenario.go index 29bf611..fb4213a 100644 --- a/loadtest/internal/scenario/scenario.go +++ b/loadtest/internal/scenario/scenario.go @@ -42,19 +42,35 @@ type RealisticConfig struct { GamesPerPlayer int // target concurrent games per player; 0 => random 3..5 Tick time.Duration // per-player operation cadence (keeps a player under the per-user limit) SecondaryProb float64 // chance per tick of a non-move operation + Eval bool // model the per-tile evaluate preview (the gameplay hot path); false reproduces the pre-evaluate harness + EvalRecon int // extra full-composition evaluate re-previews per play, beyond one per placed tile } // DefaultRealistic returns the moderate ramp: 50 -> 200 -// -> 500 concurrent players, ~12 minutes per step, ~1 op/s per player. +// -> 500 concurrent players, ~12 minutes per step, ~1 op/s per player, with the +// per-tile evaluate preview modelled (the realistic hot path). func DefaultRealistic() RealisticConfig { return RealisticConfig{ Steps: []int{50, 200, 500}, StepDur: 12 * time.Minute, Tick: 800 * time.Millisecond, SecondaryProb: 0.08, + Eval: true, + EvalRecon: 1, } } +// evalGapBase and evalGapSpan bound the modelled pause between successive tile +// placements: the client's 250 ms debounce coalesces faster drags into a single +// evaluate, so a thoughtful player's previews are spaced by a gap drawn from +// [base, base+span] — wide enough that a normal composition stays under the per-user +// rate limit, the way a real one does (the limiter's cost is measured by the hammer, +// not by self-inflicted rejections here). +const ( + evalGapBase = 250 * time.Millisecond + evalGapSpan = 500 * time.Millisecond +) + // RunRealistic runs the staged ramp. Each step activates more players (drawn from the // seeded pool), assembles a cohort of games for them and starts their turn loops; the // loops run until the whole ramp ends. Players from earlier steps keep playing, so @@ -128,7 +144,7 @@ func (d *Driver) playerLoop(ctx context.Context, p seed.Account, games []*Game, d.secondaryOp(ctx, c, p, g, rng) continue } - if d.playTurn(ctx, c, p, g, rng) { + if d.playTurn(ctx, c, p, g, cfg, rng) { active = slices.DeleteFunc(active, func(x *Game) bool { return x == g }) gi = 0 if len(active) == 0 { @@ -161,10 +177,10 @@ func (d *Driver) subscribeLoop(ctx context.Context, c *edge.Client, p seed.Accou } // playTurn plays one turn in g over the player's client when it is the player's -// move: fetch state, replay history, pick a legal move and submit it (or exchange / -// pass). It reports whether the game has finished, so the caller can drop it from the -// rotation. -func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g *Game, rng *rand.Rand) (finished bool) { +// move: fetch state, replay history, pick a legal move, compose it (the per-tile +// evaluate previews a real client fires) and submit it (or exchange / pass). It reports +// whether the game has finished, so the caller can drop it from the rotation. +func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g *Game, cfg RealisticConfig, rng *rand.Rand) (finished bool) { seat := g.seatOf(p.ID.String()) if seat < 0 { return false @@ -196,6 +212,7 @@ func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g } switch action.Kind { case "play": + d.composePlay(ctx, c, p, g, action.Tiles, cfg, rng) t0 = time.Now() _, code, _ := c.SubmitPlay(ctx, p.Token, g.ID, action.Tiles) d.rec.Record("game.submit_play", code, time.Since(t0)) @@ -211,6 +228,59 @@ func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g return false } +// composePlay models a player arranging the chosen play tile by tile before committing: +// the debounced evaluate preview the real client fires on each placement (a growing prefix +// of the tiles), a few full-composition re-previews for reconsideration (recall a tile, try +// another spot), and the single draft persistence the client debounces out. evaluate is the +// hottest gameplay request at scale, so omitting it (the pre-evaluate harness) understated +// the load; cfg.Eval false reproduces that baseline for an A/B comparison. Every step +// honours ctx, so end-of-run cancellation never blocks on a sleep or an in-flight preview. +func (d *Driver) composePlay(ctx context.Context, c *edge.Client, p seed.Account, g *Game, tiles []edge.PlayTile, cfg RealisticConfig, rng *rand.Rand) { + if !cfg.Eval || len(tiles) == 0 { + return + } + // One evaluate per landed tile: the growing prefix mirrors the client re-previewing + // after each placement (an early prefix is often illegal, which is still a successful + // "ok" round trip — exactly the backend work a real composition triggers). + for n := 1; n <= len(tiles); n++ { + if !jitterSleep(ctx, rng, evalGapBase, evalGapSpan) { + return + } + t0 := time.Now() + code, _ := c.Evaluate(ctx, p.Token, g.ID, tiles[:n]) + d.rec.Record("game.evaluate", code, time.Since(t0)) + } + for r := 0; r < cfg.EvalRecon; r++ { + if !jitterSleep(ctx, rng, evalGapBase, evalGapSpan) { + return + } + t0 := time.Now() + code, _ := c.Evaluate(ctx, p.Token, g.ID, tiles) + d.rec.Record("game.evaluate", code, time.Since(t0)) + } + // The client persists the in-progress composition (debounced to one upsert). Its opaque + // JSON content does not affect the call's cost, so a minimal valid shape stands in. + t0 := time.Now() + code, _ := c.DraftSave(ctx, p.Token, g.ID, `{"rack_order":"","board_tiles":[]}`) + d.rec.Record("draft.save", code, time.Since(t0)) +} + +// jitterSleep pauses for a randomised gap in [base, base+span], modelling the human pause +// between tile placements that the client's debounce coalesces into one evaluate. It +// returns false if ctx is cancelled during the wait, so a composition unwinds promptly at +// end of run. +func jitterSleep(ctx context.Context, rng *rand.Rand, base, span time.Duration) bool { + d := base + time.Duration(rng.Int63n(int64(span)+1)) + t := time.NewTimer(d) + defer t.Stop() + select { + case <-ctx.Done(): + return false + case <-t.C: + return true + } +} + // secondaryOp exercises one of the non-move edge operations the plan calls out, so // the run touches nudge / chat / check-word / draft / profile / stats too, over the // player's own client.