Bump DICT_VERSION v1.2.1 -> v1.3.0 across the seed surface: .env.example, the
compose build-arg default, both backend Dockerfile stages, the loadtest
Dockerfile, the CI dawg-download version, and the deploy/backend docs.
v1.3.0 drops the abbreviation class from the Russian word list (scrabble-dictionary
#6). This only seeds a FRESH volume; a live contour/prod volume is unaffected (the
.seed_version marker wins — seed-drift guard) and moves to v1.3.0 through the admin
console (ARCHITECTURE §5). Per-contour deploy still overrides via TEST_/PROD_DICT_VERSION.
The project is live in production, so the staged-development scaffolding is removed.
- Delete the staged trackers PLAN.md and PRERELEASE.md.
- Rewrite CLAUDE.md: drop the per-stage workflow; codify the ongoing development
principles (How we work) and the production model (Branching, CI & production):
manual prod-deploy / prod-rollback, semver release tags, Ansible provisioning,
expand-contract migrations.
- De-stage the living docs (README, ARCHITECTURE, TESTING, deploy/ansible, loadtest,
platform/telegram READMEs) and the docker-compose tuning comments: drop the
Stage N / R1-R7 / pre-release labels, keep every number and rationale, and fix the
now-dangling PLAN.md / PRERELEASE.md references to describe the current state.
- Reword stale 'later stage' Go doc comments for subsystems that have shipped.
EvaluatePlay (the hottest gameplay call, fired on every tile placement) now uses
the warm live-game cache directly: an active game stays cached (mutated in place
across moves, evicted only on finish), so the cached engine game and its immutable
seat list answer the membership check and the score with no DB read. The cold path
(eviction / first load) still loads and validates via the store. The seat list is
cached alongside the engine game for the membership fast path.
GetGame also folds its two round-trips (game, then seats) into one LEFT JOIN,
preserving the contract (same Game, a seatless game still returns empty seats, seat
order kept) — one round-trip for every remaining caller.
Measured at 500 players: evaluate p99 halves (200 -> 100 ms) and the per-op query
count drops. It does NOT cut postgres CPU — that is write-bound (per-move CommitMove
plus draft upserts and journal replays), the cheap indexed GetGame reads were never
its bottleneck, and postgres runs with headroom (~1.5 of 2 cores). So this is a
latency / query-volume optimization, not a DB-CPU one.
Regression cover: a non-player evaluate against a warm game asserts the cached-seat
membership path; the integration suite exercises GetGame's join across every game op.
The loadtest harness never modelled game.evaluate — the debounced per-tile
play preview a real client fires several times per turn, the hottest gameplay
call. Model it (one evaluate per placed tile + reconsideration re-previews +
draft.save, human-paced; --eval / --eval-recon toggle it).
That realistic load surfaced the real bottleneck: the gateway's backend HTTP
client used the default transport (MaxIdleConnsPerHost=2), so every sync call
to the single backend host churned a fresh TCP connection — ~26500 TIME_WAIT
sockets at 500 players (near the ephemeral-port ceiling), burning ~1.75 gateway
cores while the backend sat near-idle. It was the unfixed root of the residual
transport_error the earlier passes chased on the client side.
Widen the keep-alive pool (backendMaxIdleConns=512, ~2x the observed 225-conn
peak). At 500 players the churn collapses to ~0 and peak gateway CPU drops ~7x
(~1.75 -> ~0.26 cores); postgres (~1.65 cores) becomes the busiest service.
This overturns the earlier "gateway is the binding constraint, scale it
horizontally" sizing — that was sizing around this bug, not a real floor.
Consolidate the loadtest trip reports into one loadtest/REPORT.md (drop the
R2/R7 split) and bake the finding into README / PRERELEASE / ARCHITECTURE /
TESTING.
The dictionary release moved to v1.2.1 while DICT_VERSION stayed pinned at
v1.0.0 in CI and the image/compose seed defaults. Two problems:
1. CI validated against a stale dictionary.
2. The contour seed could be bumped on a live volume, which silently relabels
the already-seeded bytes — voiding games pinned to the prior label and
serving the wrong dictionary for new ones. The flat DAWGs carry no embedded
version, so this drift was undetectable.
Changes:
- Seed-drift guard: OpenWithVersions records the flat dir's version in a
.seed_version marker on first boot and refuses to start when a later
BACKEND_DICT_VERSION disagrees. DICT_VERSION is now the seed for a *fresh*
volume only; a live contour migrates through the admin console (old versions
stay resident, in-progress games keep replaying).
- Track the current release: CI's DICT_VERSION centralised to one workflow-level
env (v1.2.1); image/compose/.env seed defaults bumped to v1.2.1. The deploy
job keeps reading the per-contour vars.TEST_DICT_VERSION.
- Docs: ARCHITECTURE §5 (decision record), backend/deploy READMEs, PRERELEASE
tracker (DV row).
Verified locally against the v1.2.1 artifact: gofmt, build, vet, unit and
integration (-tags=integration) all green.
Bumps the engine to scrabble-solver v1.1.1, where a single-word-per-turn play
must form its word along its own line through an existing tile: a multi-tile play
that touches the board only perpendicular to itself (the contour РЮМ/КЕД/ОР cases)
no longer connects. For a single tile that abuts the board on both axes the engine
now plays the higher-scoring legal orientation instead of the geometrically longer
one (playDirection), so a real word is never rejected in favour of a non-word.
Reworks the single-word solver/engine tests for the corrected rule (no longer a
superset of standard play) and updates ARCHITECTURE/FUNCTIONAL/PRERELEASE.
Add a per-game rule chosen on New Game for Russian variants (default off = the
single-word rule; on = standard Scrabble). Off, only the main word along the play
direction is validated and scored; perpendicular cross-words are ignored,
including in robot move generation. The rule rides every create and enqueue
request and joins the matchmaking key, so games and auto-match stay one uniform
path; "Russian-only" is a UI affordance (English always sends standard and shows
no toggle).
- Engine: consume scrabble-solver v1.1.0's PlayOptions{IgnoreCrossWords}, threaded
through engine.Options.MultipleWordsPerTurn -> playOpts() into validate, score
and generate.
- Backend: thread the flag through game CreateParams/Game + store (games column),
lobby InvitationSettings + invitation row, and the matchmaker queue key (variant
+ rule); persisted, so a rebuilt-from-journal game keeps it. Baseline migration
gains multiple_words_per_turn (DB not versioned); jet regenerated.
- Edge: multiple_words_per_turn added to the EnqueueRequest / CreateInvitationRequest
FlatBuffers tables (Go + TS regenerated) and threaded through the gateway.
- UI: a "Multiple words per turn" toggle on New Game, shown for Russian variants
only (auto-match and friend invite), default off; English silently sends standard.
- Tests: backend engine/matchmaker; UI unit (gating) + Playwright e2e (solver
corner-case + GCG fixtures ship in v1.1.0). Docs + PRERELEASE tracker updated.
A single tile that only extended a word perpendicular to the client-declared
direction was rejected: the UI always sent dir=H for one-tile plays (the
dirOverride/Controls toggle was orphaned in the Stage 7 game rework), so placing
"А" above "БАК" to form "АБАК" failed the solver's main-word-length check even
though the word is in the dictionary.
Make the backend infer a play's orientation from the placed tiles and the board
(internal/engine.resolveDirection): two or more tiles by the line they share, a
lone tile by the axis it abuts (longer word wins, horizontal on a tie). Direction
becomes an output, not an input: drop dir from the SubmitPlay/Eval wire requests
and add it to EvalResult. Journal replay keeps trusting the stored "H"/"V"
(SubmitPlayDir) so a rebuilt game matches the one committed.
UI: stop computing/sending direction; the preview now shows the words a move
forms with its total score (game.previewWords); the make-move control is disabled
until the play is confirmed legal; the "your turn" label hides while tiles are
pending. Delete the orphaned Controls.svelte.
Regenerate the FlatBuffers bindings (Go + TS) and update the gateway transcode
and the loadtest edge client to the new contract. Bake the decision into
ARCHITECTURE.md (§5/§9.1), FUNCTIONAL.md (+ _ru) and the backend README.
A practical single-host ordering guide — CPU cores, RAM, disk at three tiers —
grounded in the R7 profile (~5.5 cores / ~2.5 GiB peak at 500 players) and the
measured on-disk footprint (images ~2.4 GB; Tempo 3.1 GB at 72 h; the game DB
23 MiB and growing). Notes which knobs move disk (Tempo/Prometheus retention,
Postgres growth) and that the gateway scales horizontally past one host.
- loadtest/REPORT-R7.md: the final stress-run report — method, the 500-player resource
profile, the agreed tuning, the validation (transport_error 2.49% -> 0.72% at 3 gateway
cores; the burst run showing connection-bound behavior), and the prod-sizing
recommendation for Stage 18.
- loadtest/README.md: per-player transports, --cpus capping, docker_stats (was cAdvisor),
the absolute BACKEND_DICT_DIR for ./loadtest/... , and report links.
- docs/TESTING.md + docs/ARCHITECTURE.md: observability now uses the otelcol docker_stats
receiver (cAdvisor removed); links to both trip reports.
- CLAUDE.md: repo-layout line reflects docker_stats + per-service limits.
- PRERELEASE.md: R7 marked done in the tracker + heading; a Refinements entry recording
the decisions, findings, applied tuning and validation.
This is the final pre-release hardening phase; Stage 18 (prod cutover) is next.
Each virtual player now builds its own edge.Client (its own h2c connection
carrying both the Subscribe stream and the Execute calls), instead of every
player multiplexing over a single shared http2.Transport. The R2 trip report
traced the ~14% transport_error on game.state at 500 players to that single
shared transport; per-player connections mirror real clients and isolate the
artifact. The assembly burst and the gateway-hammer each get their own client.
playTurn now reports when a game has finished so playerLoop drops it from the
rotation (slices.DeleteFunc); once no active game remains the player idles while
still holding its stream. This stops secondary ops from hammering game_finished
on already-ended games (the other R2 harness finding).
Ran the moderate early pass (50/200/500, 10 min/step) against the contour: ramped
clean to 500 players, 1.2 M edge calls, 48 870 plays, 2 798 games finished, no
crash/deadlock; cleanup removed all 11 000 seeded accounts. The per-user limiter held
under the gateway-hammer (99.97 % rejected, p99 2 ms).
Top finding: ~14 % transport_error on game.state at 500 players under CPU saturation
(backend/gateway/Postgres each ~1 core), amplified by the harness's single shared
http2.Transport (the harness itself peaked at 86 % of a core on the same host).
Observability finding: cAdvisor yields only the root cgroup on the contour host
(separate XFS /var/lib/docker); per-container metrics captured via docker stats; R7
should adopt the otelcol docker_stats receiver. Full report in loadtest/REPORT-R2.md;
PRERELEASE refinements logged; R2 marked done.
- display-name marker: letters-only 'Zzloadtest' (the editable-name validator
forbids digits/colons), so profile.update resends the seeded name successfully.
- draft.save: rack_order is a string in the backend draft DTO (was sent as []),
fixing the bad_request.
Both confirmed ok against the contour. chat_not_your_turn / nudge_own_turn are
by-design turn gates (backend/internal/social/chat.go), correctly exercised.
New scrabble/loadtest module (the pre-release stress harness): seeds 1000 guest +
10000 durable accounts with pre-created sessions directly in Postgres (token hash
matches backend/internal/session), drives virtual players through the edge protocol
(real 2-4p games assembled via invitations, mid-ranked legal moves generated locally
by the embedded scrabble-solver — the edge carries no board, so the client replays
history), plus nudge/chat/check-word/draft/profile/stats and a gateway-hammer that
verifies the rate limiter. Prints a trip-report summary (per-op latency percentiles,
result codes, live-event tally). Go unit tests cover the pure pieces; the DAWG-backed
move test runs under BACKEND_DICT_DIR.
Contour: add cAdvisor + postgres_exporter + a 'Scrabble - Resources' Grafana
dashboard and the two Prometheus scrape jobs, for the R2/R7 stress-run resource
baseline.
CI: gate ./loadtest/... (path filter + vet/build/test). Docs: TESTING, ARCHITECTURE,
project CLAUDE repo layout.
Ilia Denisov