When the Telegram first name yields no usable letters, fall back to the @username
taken whole (trimmed + length-capped, never character-stripped like the real name)
rather than a sanitized form; the generated placeholder is reached only when no
username is set. Precedence: real name -> @username (verbatim) -> placeholder.
- Lobby friend-invitation card: icon-only checkmark/cross actions stacked in a
min-width right column; the middle column (From <name> + flag + variant rules,
like New Game) grows and wraps. The cross now opens a decline-confirmation modal
(mirroring the in-game resign confirm) instead of declining on first tap.
- New Game with a friend: a lone offered variant is pre-selected and its picker
disabled (nothing else to choose); relabel 'Тип игры' -> 'Вариант' and
'Подсказок на игрока' -> 'Подсказки'.
- Quick game: pin the Start button to the bottom of the screen, mirroring the
friend-game Send-invitation button.
The project is live in production, so the staged-development scaffolding is removed.
- Delete the staged trackers PLAN.md and PRERELEASE.md.
- Rewrite CLAUDE.md: drop the per-stage workflow; codify the ongoing development
principles (How we work) and the production model (Branching, CI & production):
manual prod-deploy / prod-rollback, semver release tags, Ansible provisioning,
expand-contract migrations.
- De-stage the living docs (README, ARCHITECTURE, TESTING, deploy/ansible, loadtest,
platform/telegram READMEs) and the docker-compose tuning comments: drop the
Stage N / R1-R7 / pre-release labels, keep every number and rationale, and fix the
now-dangling PLAN.md / PRERELEASE.md references to describe the current state.
- Reword stale 'later stage' Go doc comments for subsystems that have shipped.
- prod-deploy.yaml is now four visible sequential jobs (build -> deploy-main ->
deploy-bot -> verify) so the rollout stages show in the Actions UI; the
per-service rolling stays in the deploy-main log.
- prod-rollback.yaml: a separate manual workflow_dispatch. Leave target_version
blank to roll back to the previous deployed version (the host now tracks
DEPLOYED_TAG + PREVIOUS_TAG), or pick a release tag. Re-deploys an already
published image rolling + health-gated, image-only (no rebuild, no DB migration).
- prod-deploy.sh tracks the previous tag (commit_tag) for the blank-input rollback.
- Docs: ARCHITECTURE §13 + deploy/README runbook cover versioning + rollback.
pkg/version.Version (default "dev") is set at link time via -ldflags from each
service Dockerfile's VERSION build-arg, which the deploy passes as the git tag
(git describe --tags). It surfaces as the OpenTelemetry service.version resource
attribute (so Grafana/Tempo are version-aware), alongside the SPA's existing
About version. Adds the VERSION build-arg to the backend/gateway/validator/bot
compose builds and a serviceResource test covering service.name + service.version.
- ARCHITECTURE §13 prod bullet -> the realized mechanism: registry transport,
two-host, rolling + auto-rollback, migration maintenance window, node_exporter,
the undersized launch; the contour paragraph notes node_exporter + the
telegram-local profile.
- deploy/README gains a prod rollout runbook (how to run, migrations/restore, cert
rotation, sizing/monitoring, the full PROD_ set) + node_exporter row, the
telegram-local profile note, and the soft AWG_CONF note.
- PLAN Stage 18 records the resolved open details and the remaining live cutover
(pending erudit-game.ru DNS); the tracker reads 'machinery built; cutover pending DNS'.
- PRERELEASE TX/AG note the prod wiring is built.
A workflow_dispatch-only rollout from master (confirm=deploy):
- .gitea/workflows/prod-deploy.yaml builds + pushes the images to the registry,
ships the compose/config/certs/env over SSH, deploys the main host via
prod-deploy.sh, then the bot host, then verifies the public site.
- deploy/prod-deploy.sh rolls the main stack one service at a time in dependency
order (postgres->backend->gateway->landing->validator->caddy), health-checking
after each; any failure rolls the whole stack back to the previous tag. A schema
migration adds a maintenance window: the backend (sole writer) is stopped for a
consistent pg_dump before migrating; image rollback stays DB-safe (expand-contract),
the dump is kept for a manual restore.
- prod overlay: pull the four main images from the registry by tag.
- Runtime secrets reach the host via a sourced env.sh (single-quoted values keep the
bcrypt hash's literal $ intact, unlike a --env-file).
Split the contour across the two prod hosts and retune for the small main host:
- Gate vpn+bot to the telegram-local profile. The CI test deploy now passes
--profile telegram-local so the test contour still brings them; the prod main
host omits both, and the prod bot runs standalone from docker-compose.bot.yml.
- docker-compose.prod.yml (main-host overlay): publish caddy 80/443 (no host
caddy in prod; caddy owns ACME) and gateway 9443 (the remote bot dials in over
mTLS); GOMAXPROCS=2, smaller memory caps and 7d Prometheus retention for the
2 vCPU / 1.9 GiB host. It launches deliberately undersized; resize reactively.
- docker-compose.bot.yml: standalone bot for the tg host (no VPN, OTLP off since
otelcol is unreachable from there, dials the main host's bot-link).
- Add node_exporter + a Prometheus scrape so host memory pressure (the OOM
signal on the tight main host), not just per-container docker_stats, is visible.
- Soften AWG_CONF to a default: only the profiled vpn sidecar consumes it, and
compose interpolates profiled-out services too, so prod must not require it.
Idempotent playbooks under deploy/ansible/ prepare both production hosts:
docker-ce + compose plugin, a non-sudo deploy service account holding the CI
deploy key, key-only sshd, default-deny ufw, fail2ban, unattended upgrades and
chrony. The main host also opens 80/443/9443 and creates the external edge
network; the tg host verifies direct Bot API egress (the no-VPN assumption).
The application is deployed separately by the prod-deploy workflow (later
phase), running as the deploy account this playbook provisions.
Inside Telegram, a failed initData authentication (e.g. the backend down
during a deploy) dropped the user onto the web login screen — the /app/
experience, which has no place inside the Mini App. bootstrap now retries the
launch a few times in silence and then renders a dedicated boot-error screen
with a Retry button (new BootError.svelte, app.bootError), never falling back
to the web sign-in. A blocked account is still terminal and goes straight to
the blocked screen.
The profile "Link an account" section (email + Telegram link) is hidden while
sign-in is provider-only; the anonymous /app/ guest whose upgrade path this is
comes later. The flow is kept wired (`hidden` on .emailbox) and its two e2e
specs are skipped, both to be re-enabled together.
Adds i18n boot.* copy (en/ru), a mock authTelegram failure hook plus an e2e
covering the retry screen, and bakes both behaviours into FUNCTIONAL(.md/_ru).
EvaluatePlay (the hottest gameplay call, fired on every tile placement) now uses
the warm live-game cache directly: an active game stays cached (mutated in place
across moves, evicted only on finish), so the cached engine game and its immutable
seat list answer the membership check and the score with no DB read. The cold path
(eviction / first load) still loads and validates via the store. The seat list is
cached alongside the engine game for the membership fast path.
GetGame also folds its two round-trips (game, then seats) into one LEFT JOIN,
preserving the contract (same Game, a seatless game still returns empty seats, seat
order kept) — one round-trip for every remaining caller.
Measured at 500 players: evaluate p99 halves (200 -> 100 ms) and the per-op query
count drops. It does NOT cut postgres CPU — that is write-bound (per-move CommitMove
plus draft upserts and journal replays), the cheap indexed GetGame reads were never
its bottleneck, and postgres runs with headroom (~1.5 of 2 cores). So this is a
latency / query-volume optimization, not a DB-CPU one.
Regression cover: a non-player evaluate against a warm game asserts the cached-seat
membership path; the integration suite exercises GetGame's join across every game op.
The loadtest harness never modelled game.evaluate — the debounced per-tile
play preview a real client fires several times per turn, the hottest gameplay
call. Model it (one evaluate per placed tile + reconsideration re-previews +
draft.save, human-paced; --eval / --eval-recon toggle it).
That realistic load surfaced the real bottleneck: the gateway's backend HTTP
client used the default transport (MaxIdleConnsPerHost=2), so every sync call
to the single backend host churned a fresh TCP connection — ~26500 TIME_WAIT
sockets at 500 players (near the ephemeral-port ceiling), burning ~1.75 gateway
cores while the backend sat near-idle. It was the unfixed root of the residual
transport_error the earlier passes chased on the client side.
Widen the keep-alive pool (backendMaxIdleConns=512, ~2x the observed 225-conn
peak). At 500 players the churn collapses to ~0 and peak gateway CPU drops ~7x
(~1.75 -> ~0.26 cores); postgres (~1.65 cores) becomes the busiest service.
This overturns the earlier "gateway is the binding constraint, scale it
horizontally" sizing — that was sizing around this bug, not a real floor.
Consolidate the loadtest trip reports into one loadtest/REPORT.md (drop the
R2/R7 split) and bake the finding into README / PRERELEASE / ARCHITECTURE /
TESTING.
The main bot is now an admin in the moderated discussion group and receives its
messages (allowed_updates includes message). Its default handler replied to
every message with a Mini App launch button — an inline web_app button, which
Telegram permits only in private chats — so replying in the group failed with
BUTTON_TYPE_INVALID (silently: the send fails, no user-facing error). Reply only
in a private chat; in the group the bot only manages permissions. The promo bot
gets the same guard.
- Bake the final default-allow + mute-the-ineligible strategy into
docs/ARCHITECTURE.md, docs/FUNCTIONAL.md (+_ru), platform/telegram/README.md,
the deploy compose comment and the PRERELEASE tracker. The live test proved a
per-user grant cannot exceed a deny-by-default group (Telegram intersects the
chat default with the per-user permission), so the chat allows sending by
default and the bot restricts the ineligible instead of granting the eligible.
- Lower the per-event chat_member trace and eligibility evaluation to Debug;
keep the actual mute/unmute actions, the startup self-check and warnings at
Info, so prod logs only what the bot did.
- Update game.searchingForOpponent (Searching -> Waiting for opponent / Поиск ->
Ждём соперника) and the quickmatch e2e assertions to match.
Telegram intersects the chat default with each user's permissions, so a per-user
grant can never exceed a deny-by-default group: the original default-deny +
grant design could not let any user write (can_send=true was AND-ed with the
denying default). Invert it — the chat allows sending by default and the bot
MUTES an ineligible member (unregistered, admin-suspended, or chat_muted) and
restores an eligible one it had muted, acting only when the current state
differs (idempotent, no self-loop). The block/unblock/chat_muted/registration
path already sets can_send to the eligibility, so it is unchanged.
The CanSendMessages loop-guard skipped exactly the stuck case — a restricted
member whose chat_member event reports can_send=true yet who cannot actually
write. Replace it with a precise loop guard (skip only the bot's own restrict
action, i.e. the update whose performer is the bot) and grant any eligible
in-chat member (member or restricted) otherwise. Also log the new member's
can_send, is_member and the actor id for full visibility.
A default-deny discussion group reports a present or freshly joined member as
`restricted` (no send right), not `member`. The join filter required `member`,
so the real case never matched and a registered user stayed muted. Grant any
eligible in-chat member (member or restricted) that still lacks the send right,
with a loop guard (skip when send is already allowed) so the bot's own grant
does not re-fire. Revoking a now-ineligible user stays the chat-gate path's job,
so this never fights a chat_muted/block.
Two follow-ups from a contour test where a user joined the chat, then
registered, and got no write access — with silent logs.
Observability: log every chat_member update (chat id, configured id, user,
old->new status), the eligibility result and the grant outcome; plus a startup
self-check that warns loudly when the bot is not an administrator in the chat
with the restrict-members ("Ban users") right — the common misconfiguration,
previously invisible in the logs.
Grant on first registration: a user who joins the moderated chat BEFORE
registering is covered by no chat_member event, so the join-time grant never
fires for them. ProvisionTelegram now reports first contact, and the Telegram
auth handler emits chat_access_changed on it, so the gateway re-evaluates and
grants write access if the user is already in the chat.
tgbot.New validates the token with getMe, so a bad or unreachable promo token
would otherwise return an error from run() and crash-loop the whole bot process
— taking the main game bot down with it, since they share the container. Log it
and skip the promo bot instead; the main bot and bot-link are unaffected.
Add a second standalone promo bot to the bot container (answers /start with a
localized message + a URL button into the main bot's Mini App) and gate write
access in a channel's linked discussion chat: grant on join when the Telegram
user is registered and neither admin-suspended nor holding a new chat_muted
role, and revoke/grant on the matching moderation change for a member currently
in the chat.
Eligibility (registered AND NOT suspended AND NOT chat_muted; the game
suspension dominates) is resolved once in the backend and reached two ways: the
bot's join-time unary ResolveChatEligibility over the existing mTLS bot-link,
and a backend chat_access_changed event -> gateway -> ChatGate command
(idempotent; a temporary-block-expiry sweeper may over-emit). The bot guards the
block/unblock path with getChatMember, since bots cannot list members.
A web_app button cannot open another bot's Mini App (it signs initData with the
sending bot's token), so the promo button is a t.me ?startapp URL reusing the
UI's VITE_TELEGRAM_LINK. The bot must be a chat admin with the restrict-members
right and chat_member in its allowed updates.
No schema change: chat_muted reuses the data-driven account_roles table.
The dismiss check fired at the instant a word finished laying, so a word
was never held: ЭРУДИТ fell straight into the lobby (too fast) and
ЗАГРУЗКА got no readable pause. The pause was a *leading* gap before the
next word, not a hold after the current one.
Move the hold to after each word and run the check after it: every word
(ЭРУДИТ included) now stays up for PAUSE_MS before the splash either
dismisses or lays the next word. prefixMs = WORD_MS + PAUSE_MS,
cycleMs = 2*(WORD_MS + PAUSE_MS).
On a cold app open the lobby's game list arrives over the network; on a
slow link the empty "no games yet" line flashed before the games loaded.
Add a full-screen tile splash that lays a Scrabble crossword of ЭРУДИТ /
ЗАГРУЗКА / ОЖИДАНИЕ (Эрудит point values, hardcoded since the alphabet
table is not cached at boot) until the lobby's first load settles, then
removes itself to reveal the populated list.
- lib/splash.ts: pure layout + reveal schedule (unit-tested).
- components/Splash.svelte: App-level overlay; per-tile drop-in; loops
ЗАГРУЗКА → ОЖИДАНИЕ until ready, dismisses on a word boundary. Static
ЭРУДИТ under reduced motion / the mock build.
- app state: lobbyReady (set by Lobby on first settle) + splashDone,
reset on logout.
- App.svelte: overlay while routeIsLobby && !splashDone; the plain text
splash now only covers non-lobby deep-links during bootstrap.
- docs: UI_DESIGN + FUNCTIONAL (+ _ru).
The @honeypot block both deleted and set X-Scrabble-Honeypot in one reverse_proxy.
Caddy applies header_up deletions *after* sets, so the tag we set was immediately
stripped: the gateway never saw it, and a decoy hit (e.g. GET /.env) fell through
to the gateway's /app redirect (308) instead of tripping the honeypot. Drop the
delete — the bare set already replaces any client-supplied value. The real
endpoints keep stripping the header in the @gateway block (delete-only, no
conflicting set). Caught on the live test contour (no caddy locally).
Add a prod-only, in-memory IP ban enforced at the edge, fed by three signals:
sustained rate-limiter rejections (the IP-keyed public/email/admin classes — the
user class stays the backend soft-flag's concern), a honeypot decoy-path hit (the
contour caddy tags decoys with X-Scrabble-Honeypot and routes them to the gateway),
and a honeytoken (a planted bearer, GATEWAY_HONEYTOKEN). A banned IP is refused with
429 by the abuseGuard middleware before any work — covering the Connect edge, the
live stream and the static SPA/landing the per-op limiter never gated.
The ban is off by default: it keys by the real client IP the shared-NAT test contour
does not expose, so a ban there would be self-inflicted; detection still logs in the
contour, only the ban action is gated (GATEWAY_ABUSE_BAN_ENABLED). Rejection bans last
GATEWAY_ABUSE_BAN_DURATION; tripwire/honeytoken hits are near-zero-false-positive and
earn longer fixed bans. Each ban increments gateway_abuse_banned_total{reason}.
Operators see and lift active bans on the admin console's Throttled page; the gateway
syncs its active set to the backend every 30s (POST /api/v1/internal/bans/sync,
backend/internal/banview) and applies the operator unbans the response returns.
PRERELEASE phase AG. Docs baked into ARCHITECTURE / FUNCTIONAL (+ru) / both READMEs.
The gateway and bot run on distroless nonroot (UID 65532) and bind-mount the
cert dir read-only, but gen-certs.sh wrote the keys 0600 (owner-only, the deploy
user), so both crash-looped at boot with "open /certs/*.key: permission denied"
and the deploy probe correctly failed (the contour's gateway was down).
The .crt files were already 0644 (openssl default); make the leaf keys 0644 too
so UID 65532 can read them. These are ephemeral TEST certificates regenerated
every deploy on the trusted runner; prod keys come from PROD_ secrets. The CA
private key stays 0600 (containers never read it).
Move all Telegram egress off the main host. The single connector held the
bot token, long-polled Telegram and answered the gateway/backend over the
trusted internal network, so the whole component (including login validation)
shared fate with its VPN sidecar. Split it into two binaries that share the
token:
- cmd/validator (home, no VPN): Mini App initData + Login Widget HMAC only,
never calls the Bot API. The gateway dials it for Telegram auth, so game
login is now independent of Telegram reachability.
- cmd/bot (remote): Bot API long-poll + sendMessage, the only component
reaching Telegram. It holds no inbound port — it dials the gateway over a
new reverse mTLS bot-link (pkg/proto/botlink/v1) and executes the send
commands the gateway pushes.
The gateway funnels sends to the bot-link: out-of-app push is fire-and-forget
(at-most-once, dropped if no bot is connected); the backend admin broadcasts
reach a gateway-served relay that forwards them and awaits the bot's ack
(SendToUser/SendToGameChannel contract preserved). mTLS (pkg/mtls) is the one
inter-service link that leaves the trusted segment; validator<->gateway and
the relay stay plaintext internal. The bot is Telegram-rate-limited.
One bot now; the gateway bot registry, an owns_updates flag and per-command
ids leave seams for N later. Webhook rejected (one URL per token, adds inbound
+ a static address).
The unified test contour runs the split (the bot keeps its VPN sidecar and
dials the gateway by its internal name; bot-link certs from deploy/gen-certs.sh,
generated in CI). The prod wiring — the bot on a separate host (no VPN), the
gateway bot-link port published, PROD_ certs with scheduled rotation, an SSH
deploy of both hosts together — is the deferred final stage (PRERELEASE.md TX,
Stage 18).
Docs: ARCHITECTURE, PRERELEASE (phase TX), platform/telegram + gateway +
backend + deploy READMEs, FUNCTIONAL(+ru), CLAUDE.md, .env.example.
The lobby tinted only the viewer's own number, and a tie counted as
"leading" — so an even score showed only the viewer's number green,
reading as if the viewer were ahead. A fresh 0:0 board did the same,
accenting the start of a game where nobody has scored.
scoreStanding is now per-seat: the viewer's seat stays green when
leading or tied and red when trailing; an opponent's seat greens only
when it ties the viewer for the lead, so an equal non-zero score paints
both numbers green. When the top score is 0 (nobody has moved) every
number is left muted, like a finished game.
The post-deploy probe checked only the static landing and the gateway-served SPA shell, so a crash-looping backend passed the deploy green. Add an http://backend:8080/readyz probe on the internal network (and dump backend logs on failure) so an unready backend fails the deploy loudly.
The seed-drift guard shipped as refuse-boot: the backend exited when
BACKEND_DICT_VERSION disagreed with the flat dir's recorded .seed_version. On
the test contour that turned a harmless-in-intent action — bumping the
TEST_DICT_VERSION variable to the active release (v1.2.1) on a volume seeded as
v1.0.0 — into a crash loop, because DICT_VERSION is the *seed* of a fresh
volume, not the active version (which the admin console drives).
Make the marker authoritative instead: OpenWithVersions resolves the flat dir's
version from .seed_version when present and ignores bootVersion on an
already-seeded volume; bootVersion only seeds a fresh volume's marker. So a
bumped build seed on a live volume is a no-op (it can't relabel live bytes and
can't void games pinned to the prior label), and it correctly seeds the next
fresh volume. The subdirectory scan now skips the resolved seed, so a version
also present as a subdir (e.g. v1.2.1 uploaded via the console while the build
seed is bumped to v1.2.1) is still loaded rather than shadowed by the flat bytes.
Tests: marker-wins over a bumped boot version; a bumped boot keeps the matching
subdir resident (the live-contour case). Docs updated (ARCHITECTURE §5, READMEs,
compose/.env, PRERELEASE DV) from "refuses to boot" to "marker wins / ignored".
Verified locally against v1.2.1: gofmt, build, vet, unit, integration green.
The dictionary release moved to v1.2.1 while DICT_VERSION stayed pinned at
v1.0.0 in CI and the image/compose seed defaults. Two problems:
1. CI validated against a stale dictionary.
2. The contour seed could be bumped on a live volume, which silently relabels
the already-seeded bytes — voiding games pinned to the prior label and
serving the wrong dictionary for new ones. The flat DAWGs carry no embedded
version, so this drift was undetectable.
Changes:
- Seed-drift guard: OpenWithVersions records the flat dir's version in a
.seed_version marker on first boot and refuses to start when a later
BACKEND_DICT_VERSION disagrees. DICT_VERSION is now the seed for a *fresh*
volume only; a live contour migrates through the admin console (old versions
stay resident, in-progress games keep replaying).
- Track the current release: CI's DICT_VERSION centralised to one workflow-level
env (v1.2.1); image/compose/.env seed defaults bumped to v1.2.1. The deploy
job keeps reading the per-contour vars.TEST_DICT_VERSION.
- Docs: ARCHITECTURE §5 (decision record), backend/deploy READMEs, PRERELEASE
tracker (DV row).
Verified locally against the v1.2.1 artifact: gofmt, build, vet, unit and
integration (-tags=integration) all green.
Consolidate the incremental goose migrations (00001-00014) into one
baseline. There is no production data, so the squash carries no data
migration. The baseline was generated from the end-state schema and
verified schema-identical to the squashed set (pg_dump diff) plus a full
integration run; the default house ad-campaign seed is carried over (a
schema-only dump omits it). The per-feature narrative that lived in the
squashed migrations is preserved in git history and docs/ARCHITECTURE.md.
Deploy note: this breaks goose's version continuity, so the test contour
DB must be wiped once — DROP SCHEMA backend CASCADE + restart backend —
for goose to re-apply the single baseline fresh. No prod data exists.
Collapse the two per-language Telegram bots into one unified bot and
replace language-based variant gating with explicit per-user variant
preferences.
- Telegram: one bot; drop service_language and the supported_languages
set everywhere (DB, account, auth, FlatBuffers Session wire, gateway,
connector proto). The single bot renders chat and out-of-app push in
the recipient's preferred_language; remove the game-language push
routing override (notify Intent.Language / push Event.language).
- Preferences: new accounts.variant_preferences (text[], DB default
{erudit_ru}, CHECK non-empty + subset of the three variants). Gates
the New Game picker, vs-AI and the friend invitation the player
creates, enforced server-side (HTTP 400 otherwise); an invited friend
may still accept any variant. Edited on the Settings screen; variants
are Erudit-first everywhere.
- Admin: drop the per-bot language selectors (broadcast / send-to-user)
and the feedback channel_lang column/field.
- Env/CI: collapse TELEGRAM_BOT_TOKEN_{EN,RU}, TELEGRAM_GAME_CHANNEL_ID_{EN,RU},
VITE_TELEGRAM_LINK{,_EN,_RU} and VITE_TELEGRAM_GAME_CHANNEL_NAME_{EN,RU}
to single unsuffixed names; drop GATEWAY_DEFAULT_SUPPORTED_LANGUAGES.
- Docs updated (ARCHITECTURE, FUNCTIONAL + _ru, platform/telegram, gateway,
backend, ui, UI_DESIGN, PRERELEASE).
The migration squash is deferred to a follow-up PR.
Ilia Denisov
developer