scrabble-game/loadtest/README.md

loadtest — stress harness

Reusable load/stress harness. It seeds a large account population with pre-created sessions, drives virtual players through the gateway edge protocol in realistic games, hammers the rate limiter, and prints a trip-report summary. It stays in the repo for repeats.

What it does

1. Seed (direct Postgres, schema backend): inserts --durable durable accounts (each with a confirmed email identity) + --guest guest accounts and an active sessions row per account, then hands the plaintext bearer tokens to the driver. Token hashes match backend/internal/session (hex(sha256(token))), so the seeded sessions resolve. Every row carries a distinctive display-name marker for cleanup.
2. Drive (edge protocol over h2c): assembles real 2–4 player games via the invitation flow (invitation.create → invitation.accept, no robots), then runs each player's turn loop — poll game.state, replay game.history, generate a legal mid-ranked move with the embedded scrabble-solver, compose it tile by tile with the debounced game.evaluate preview a real client fires (the hottest gameplay call), persist a draft.save, and game.submit_play (or pass/exchange). A fraction of turns exercise nudge / chat / check-word / draft / profile-update / stats. Each player also holds a live Subscribe stream. The moderate ramp is 50 → 200 → 500 concurrent players, ~12 min per step. --eval=false drops the evaluate model for an A/B baseline.
3. Hammer: drives games.list from one account far above the per-user rate limit to verify the limiter holds (rate_limited results) and measure its cost.
4. Report: per-operation latency percentiles, throughput, result-code breakdown, live-event tally and the aggregate error rate.

The driver runs the solver locally because the edge protocol carries no board: the client reconstructs it from decoded history (the same invariant as the UI).

Connection model

The harness reaches Postgres and the gateway directly, so run it as a one-shot container on the contour's docker network (this bypasses the host→gateway hairpin):

# from the repo root (DICT_VERSION has no default — pass the scrabble-dictionary release tag)
docker build --build-arg DICT_VERSION=v1.3.0 -f loadtest/Dockerfile -t scrabble-loadtest .

docker run --rm --cpus=3 --name scrabble-loadtest --network scrabble-internal \
  -e POSTGRES_PASSWORD="$TEST_POSTGRES_PASSWORD" \
  scrabble-loadtest run

Each virtual player gets its own edge.Client (its own h2c connection), mirroring real clients rather than multiplexing every player over one transport. Defaults assume the contour service names: postgres:5432 and gateway:8081. The DAWGs are baked into the image (/opt/dawg, pinned to the dictionary release). On a host shared with the contour, cap the harness (--cpus=3) so the contour keeps the spare cores. Run with --name scrabble-loadtest so the harness's own CPU/memory show up as a scrabble-* series in the metrics (keeping it separable from the system under test). Capture the resource baseline from the Grafana Scrabble — Resources dashboard (the otelcol docker_stats receiver + postgres_exporter), or from docker stats directly, while the run is in progress.

Commands & flags

loadtest run [flags]      seed, drive the ramp + hammer, print the report
loadtest cleanup [flags]  delete everything the harness seeded (matched by the display-name marker)

Key run flags (env in parentheses):

flag	default	meaning
--gateway (LOADTEST_GATEWAY_URL)	http://gateway:8081	gateway base URL
--dsn (LOADTEST_DSN)	from POSTGRES_*	backend Postgres DSN (schema backend)
--dawg (LOADTEST_DAWG_DIR)	/dawg (image: /opt/dawg)	committed *.dawg directory
--durable / --guest	10000 / 1000	accounts to seed
--steps	50,200,500	concurrent-player ramp steps
--step-dur	12m	hold time per step
--games-per-player	0 (random 3–5)	target concurrent games per player
--tick	800ms	per-player op cadence (keeps a player under the per-user limit)
--secondary-prob	0.08	chance per tick of a non-move op
--eval	true	model the per-tile game.evaluate preview (the gameplay hot path); false reproduces the pre-evaluate harness
--eval-recon	1	extra full-composition evaluate re-previews per play (reconsideration), beyond one per placed tile
--hammer-workers / --hammer-dur	20 / 15s	gateway-hammer (0 workers disables)
--reset / --cleanup	false	delete harness rows before / after the run

run re-seeds every time (plaintext tokens are never stored), so pass --reset to clear a prior run's rows first. The authoritative hard reset of the contour remains the DB wipe (DROP SCHEMA backend CASCADE + backend restart).

Build & test

go build ./loadtest/...
go vet ./loadtest/...
BACKEND_DICT_DIR="$PWD/../scrabble-solver/dawg" go test -count=1 ./loadtest/...

The DAWG-backed moves test runs only when BACKEND_DICT_DIR is set (as the engine tests use); the pure logic (hashing, board replay, rack build, move selection, report) runs unconditionally. Use an absolute path (here via $PWD): go test ./loadtest/... runs each package from its own directory, so a relative BACKEND_DICT_DIR would not resolve.

Trip report

The stress findings — the final run, the game.evaluate hot-path model, the gateway→backend connection-pool fix, and the revised sizing — are written up in REPORT.md.

Caveat

The harness shares the host CPU with the contour, so its own scrabble-loadtest container series is read alongside the system under test; capping it with --cpus keeps the contour's quota. Per-player transports removed the shared-transport artifact that previously inflated transport_error, so the figures reflect the system. A fully isolated ceiling on separate hardware remains future work.