Compare commits

..

33 Commits

Author SHA1 Message Date
developer 24017bcb7f Merge pull request 'Promote development → master (deploy v2: versioning + visible jobs + rollback)' (#106) from development into master 2026-06-22 06:01:03 +00:00
developer 40d8f06588 Merge pull request 'Deploy v2 — release versioning + visible deploy jobs + manual rollback' (#105) from feature/release-versioning into development
CI / changes (push) Successful in 2s
CI / unit (push) Successful in 11s
CI / integration (push) Successful in 15s
CI / ui (push) Successful in 57s
CI / changes (pull_request) Successful in 2s
CI / gate (push) Successful in 0s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 16s
CI / ui (pull_request) Successful in 56s
CI / deploy (push) Successful in 1m19s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Has been skipped
2026-06-22 05:40:55 +00:00
Ilia Denisov c59e522732 feat(deploy): visible prod-deploy jobs + manual prod-rollback
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 16s
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m23s
- prod-deploy.yaml is now four visible sequential jobs (build -> deploy-main ->
  deploy-bot -> verify) so the rollout stages show in the Actions UI; the
  per-service rolling stays in the deploy-main log.
- prod-rollback.yaml: a separate manual workflow_dispatch. Leave target_version
  blank to roll back to the previous deployed version (the host now tracks
  DEPLOYED_TAG + PREVIOUS_TAG), or pick a release tag. Re-deploys an already
  published image rolling + health-gated, image-only (no rebuild, no DB migration).
- prod-deploy.sh tracks the previous tag (commit_tag) for the blank-input rollback.
- Docs: ARCHITECTURE §13 + deploy/README runbook cover versioning + rollback.
2026-06-22 07:37:08 +02:00
Ilia Denisov 8d45ae6e3b feat: stamp the build version into every service
pkg/version.Version (default "dev") is set at link time via -ldflags from each
service Dockerfile's VERSION build-arg, which the deploy passes as the git tag
(git describe --tags). It surfaces as the OpenTelemetry service.version resource
attribute (so Grafana/Tempo are version-aware), alongside the SPA's existing
About version. Adds the VERSION build-arg to the backend/gateway/validator/bot
compose builds and a serviceResource test covering service.name + service.version.
2026-06-22 07:28:27 +02:00
developer 2c4f4b10dc Merge pull request 'Promote development → master (initial production release: pre-release line + Stage 18)' (#104) from development into master 2026-06-22 05:05:48 +00:00
developer 520a9092fe Merge pull request 'Stage 18 — prod contour deploy (two-host registry rollout, rolling + auto-rollback)' (#103) from feature/prod-contour-deploy into development
CI / changes (push) Successful in 2s
CI / unit (push) Successful in 11s
CI / integration (push) Successful in 17s
CI / ui (push) Successful in 57s
CI / changes (pull_request) Successful in 2s
CI / gate (push) Successful in 0s
CI / deploy (push) Successful in 1m4s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 56s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Has been skipped
2026-06-22 04:59:46 +00:00
Ilia Denisov 9f970495ee fix(deploy): guard cd and split DOCKER_GID assignment (shellcheck)
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 16s
CI / ui (pull_request) Successful in 56s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m7s
cd $COMPOSE_DIR now aborts on failure instead of deploying from the wrong dir;
DOCKER_GID is declared then exported so the subshell exit isn't masked.
2026-06-22 00:35:20 +02:00
Ilia Denisov 3d9ba3ac3d docs(deploy): bake Stage 18 prod-deploy decisions into the live docs
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 16s
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m8s
- ARCHITECTURE §13 prod bullet -> the realized mechanism: registry transport,
  two-host, rolling + auto-rollback, migration maintenance window, node_exporter,
  the undersized launch; the contour paragraph notes node_exporter + the
  telegram-local profile.
- deploy/README gains a prod rollout runbook (how to run, migrations/restore, cert
  rotation, sizing/monitoring, the full PROD_ set) + node_exporter row, the
  telegram-local profile note, and the soft AWG_CONF note.
- PLAN Stage 18 records the resolved open details and the remaining live cutover
  (pending erudit-game.ru DNS); the tracker reads 'machinery built; cutover pending DNS'.
- PRERELEASE TX/AG note the prod wiring is built.
2026-06-22 00:30:30 +02:00
Ilia Denisov 171b71b7e0 feat(deploy): manual prod-deploy pipeline with rolling rollback (Stage 18)
A workflow_dispatch-only rollout from master (confirm=deploy):

- .gitea/workflows/prod-deploy.yaml builds + pushes the images to the registry,
  ships the compose/config/certs/env over SSH, deploys the main host via
  prod-deploy.sh, then the bot host, then verifies the public site.
- deploy/prod-deploy.sh rolls the main stack one service at a time in dependency
  order (postgres->backend->gateway->landing->validator->caddy), health-checking
  after each; any failure rolls the whole stack back to the previous tag. A schema
  migration adds a maintenance window: the backend (sole writer) is stopped for a
  consistent pg_dump before migrating; image rollback stays DB-safe (expand-contract),
  the dump is kept for a manual restore.
- prod overlay: pull the four main images from the registry by tag.
- Runtime secrets reach the host via a sourced env.sh (single-quoted values keep the
  bcrypt hash's literal $ intact, unlike a --env-file).
2026-06-22 00:25:09 +02:00
Ilia Denisov 2b399d0838 feat(deploy): prod compose split + host-memory monitoring (Stage 18)
Split the contour across the two prod hosts and retune for the small main host:

- Gate vpn+bot to the telegram-local profile. The CI test deploy now passes
  --profile telegram-local so the test contour still brings them; the prod main
  host omits both, and the prod bot runs standalone from docker-compose.bot.yml.
- docker-compose.prod.yml (main-host overlay): publish caddy 80/443 (no host
  caddy in prod; caddy owns ACME) and gateway 9443 (the remote bot dials in over
  mTLS); GOMAXPROCS=2, smaller memory caps and 7d Prometheus retention for the
  2 vCPU / 1.9 GiB host. It launches deliberately undersized; resize reactively.
- docker-compose.bot.yml: standalone bot for the tg host (no VPN, OTLP off since
  otelcol is unreachable from there, dials the main host's bot-link).
- Add node_exporter + a Prometheus scrape so host memory pressure (the OOM
  signal on the tight main host), not just per-container docker_stats, is visible.
- Soften AWG_CONF to a default: only the profiled vpn sidecar consumes it, and
  compose interpolates profiled-out services too, so prod must not require it.
2026-06-22 00:12:43 +02:00
Ilia Denisov f5f45e7afb feat(deploy): Ansible provisioning for prod hosts (Stage 18)
Idempotent playbooks under deploy/ansible/ prepare both production hosts:
docker-ce + compose plugin, a non-sudo deploy service account holding the CI
deploy key, key-only sshd, default-deny ufw, fail2ban, unattended upgrades and
chrony. The main host also opens 80/443/9443 and creates the external edge
network; the tg host verifies direct Bot API egress (the no-VPN assumption).

The application is deployed separately by the prod-deploy workflow (later
phase), running as the deploy account this playbook provisions.
2026-06-21 23:54:57 +02:00
developer b54cb8878d Merge pull request 'fix(ui): retry Mini App launch on backend failure; hide account linking' (#102) from feature/tg-boot-retry-hide-linking into development
CI / changes (push) Successful in 2s
CI / unit (push) Has been skipped
CI / integration (push) Has been skipped
CI / ui (push) Successful in 57s
CI / gate (push) Successful in 0s
CI / deploy (push) Successful in 1m6s
2026-06-21 19:38:37 +00:00
Ilia Denisov e336638ca8 fix(ui): retry Mini App launch on backend failure; hide account linking
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Has been skipped
CI / integration (pull_request) Has been skipped
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m8s
Inside Telegram, a failed initData authentication (e.g. the backend down
during a deploy) dropped the user onto the web login screen — the /app/
experience, which has no place inside the Mini App. bootstrap now retries the
launch a few times in silence and then renders a dedicated boot-error screen
with a Retry button (new BootError.svelte, app.bootError), never falling back
to the web sign-in. A blocked account is still terminal and goes straight to
the blocked screen.

The profile "Link an account" section (email + Telegram link) is hidden while
sign-in is provider-only; the anonymous /app/ guest whose upgrade path this is
comes later. The flow is kept wired (`hidden` on .emailbox) and its two e2e
specs are skipped, both to be re-enabled together.

Adds i18n boot.* copy (en/ru), a mock authTelegram failure hook plus an e2e
covering the retry screen, and bakes both behaviours into FUNCTIONAL(.md/_ru).
2026-06-21 21:23:27 +02:00
developer 62f42ed102 Merge pull request 'perf(gateway): pool backend conns; loadtest evaluate hot path' (#101) from feature/loadtest-evaluate-hotpath into development
CI / changes (push) Successful in 2s
CI / unit (push) Successful in 10s
CI / integration (push) Successful in 15s
CI / ui (push) Has been skipped
CI / gate (push) Successful in 1s
CI / deploy (push) Successful in 1m26s
2026-06-21 18:51:58 +00:00
Ilia Denisov ecb21bd218 perf(backend): cut evaluate's DB round-trips; load the game in one query
CI / changes (pull_request) Successful in 3s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 17s
CI / ui (pull_request) Has been skipped
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 2m18s
EvaluatePlay (the hottest gameplay call, fired on every tile placement) now uses
the warm live-game cache directly: an active game stays cached (mutated in place
across moves, evicted only on finish), so the cached engine game and its immutable
seat list answer the membership check and the score with no DB read. The cold path
(eviction / first load) still loads and validates via the store. The seat list is
cached alongside the engine game for the membership fast path.

GetGame also folds its two round-trips (game, then seats) into one LEFT JOIN,
preserving the contract (same Game, a seatless game still returns empty seats, seat
order kept) — one round-trip for every remaining caller.

Measured at 500 players: evaluate p99 halves (200 -> 100 ms) and the per-op query
count drops. It does NOT cut postgres CPU — that is write-bound (per-move CommitMove
plus draft upserts and journal replays), the cheap indexed GetGame reads were never
its bottleneck, and postgres runs with headroom (~1.5 of 2 cores). So this is a
latency / query-volume optimization, not a DB-CPU one.

Regression cover: a non-player evaluate against a warm game asserts the cached-seat
membership path; the integration suite exercises GetGame's join across every game op.
2026-06-21 20:47:13 +02:00
Ilia Denisov e2771826fd perf(gateway): pool backend conns; loadtest evaluate hot path
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Has been skipped
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m5s
The loadtest harness never modelled game.evaluate — the debounced per-tile
play preview a real client fires several times per turn, the hottest gameplay
call. Model it (one evaluate per placed tile + reconsideration re-previews +
draft.save, human-paced; --eval / --eval-recon toggle it).

That realistic load surfaced the real bottleneck: the gateway's backend HTTP
client used the default transport (MaxIdleConnsPerHost=2), so every sync call
to the single backend host churned a fresh TCP connection — ~26500 TIME_WAIT
sockets at 500 players (near the ephemeral-port ceiling), burning ~1.75 gateway
cores while the backend sat near-idle. It was the unfixed root of the residual
transport_error the earlier passes chased on the client side.

Widen the keep-alive pool (backendMaxIdleConns=512, ~2x the observed 225-conn
peak). At 500 players the churn collapses to ~0 and peak gateway CPU drops ~7x
(~1.75 -> ~0.26 cores); postgres (~1.65 cores) becomes the busiest service.
This overturns the earlier "gateway is the binding constraint, scale it
horizontally" sizing — that was sizing around this bug, not a real floor.

Consolidate the loadtest trip reports into one loadtest/REPORT.md (drop the
R2/R7 split) and bake the finding into README / PRERELEASE / ARCHITECTURE /
TESTING.
2026-06-21 19:55:57 +02:00
developer dec6fac013 Merge pull request 'fix(telegram): reply to /start only in private chats' (#100) from feature/telegram-private-reply-guard into development
CI / changes (push) Successful in 2s
CI / unit (push) Successful in 11s
CI / integration (push) Successful in 16s
CI / ui (push) Has been skipped
CI / gate (push) Successful in 0s
CI / deploy (push) Successful in 1m8s
2026-06-21 15:32:11 +00:00
Ilia Denisov c494da553a fix(telegram): reply to /start only in private chats
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Has been skipped
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m7s
The main bot is now an admin in the moderated discussion group and receives its
messages (allowed_updates includes message). Its default handler replied to
every message with a Mini App launch button — an inline web_app button, which
Telegram permits only in private chats — so replying in the group failed with
BUTTON_TYPE_INVALID (silently: the send fails, no user-facing error). Reply only
in a private chat; in the group the bot only manages permissions. The promo bot
gets the same guard.
2026-06-21 17:28:01 +02:00
developer fa8abf22db Merge pull request 'feat(telegram): promo bot + channel-chat moderation gate' (#99) from feature/telegram-promo-bot-chat-moderation into development
CI / changes (push) Successful in 2s
CI / unit (push) Successful in 10s
CI / integration (push) Successful in 16s
CI / ui (push) Successful in 56s
CI / gate (push) Successful in 0s
CI / deploy (push) Successful in 1m6s
2026-06-21 15:21:23 +00:00
Ilia Denisov 1ba789a1f1 docs(telegram): invert chat-gate strategy in docs; tune logs; i18n text
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m15s
- Bake the final default-allow + mute-the-ineligible strategy into
  docs/ARCHITECTURE.md, docs/FUNCTIONAL.md (+_ru), platform/telegram/README.md,
  the deploy compose comment and the PRERELEASE tracker. The live test proved a
  per-user grant cannot exceed a deny-by-default group (Telegram intersects the
  chat default with the per-user permission), so the chat allows sending by
  default and the bot restricts the ineligible instead of granting the eligible.
- Lower the per-event chat_member trace and eligibility evaluation to Debug;
  keep the actual mute/unmute actions, the startup self-check and warnings at
  Info, so prod logs only what the bot did.
- Update game.searchingForOpponent (Searching -> Waiting for opponent / Поиск ->
  Ждём соперника) and the quickmatch e2e assertions to match.
2026-06-21 17:15:10 +02:00
Ilia Denisov bdd1cc7d85 fix(telegram): invert the chat gate — mute the ineligible (default-allow)
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 19s
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m17s
Telegram intersects the chat default with each user's permissions, so a per-user
grant can never exceed a deny-by-default group: the original default-deny +
grant design could not let any user write (can_send=true was AND-ed with the
denying default). Invert it — the chat allows sending by default and the bot
MUTES an ineligible member (unregistered, admin-suspended, or chat_muted) and
restores an eligible one it had muted, acting only when the current state
differs (idempotent, no self-loop). The block/unblock/chat_muted/registration
path already sets can_send to the eligibility, so it is unchanged.
2026-06-21 16:50:44 +02:00
Ilia Denisov 0ab1719ee9 fix(telegram): grant in-chat members regardless of reported can_send
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 16s
CI / ui (pull_request) Successful in 56s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m18s
The CanSendMessages loop-guard skipped exactly the stuck case — a restricted
member whose chat_member event reports can_send=true yet who cannot actually
write. Replace it with a precise loop guard (skip only the bot's own restrict
action, i.e. the update whose performer is the bot) and grant any eligible
in-chat member (member or restricted) otherwise. Also log the new member's
can_send, is_member and the actor id for full visibility.
2026-06-21 16:25:57 +02:00
Ilia Denisov 380f82438c fix(telegram): grant write to restricted members in default-deny chats
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m22s
A default-deny discussion group reports a present or freshly joined member as
`restricted` (no send right), not `member`. The join filter required `member`,
so the real case never matched and a registered user stayed muted. Grant any
eligible in-chat member (member or restricted) that still lacks the send right,
with a loop guard (skip when send is already allowed) so the bot's own grant
does not re-fire. Revoking a now-ineligible user stays the chat-gate path's job,
so this never fights a chat_muted/block.
2026-06-21 16:12:58 +02:00
Ilia Denisov a404513037 feat(telegram): chat-gate observability + grant on first registration
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 56s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m0s
Two follow-ups from a contour test where a user joined the chat, then
registered, and got no write access — with silent logs.

Observability: log every chat_member update (chat id, configured id, user,
old->new status), the eligibility result and the grant outcome; plus a startup
self-check that warns loudly when the bot is not an administrator in the chat
with the restrict-members ("Ban users") right — the common misconfiguration,
previously invisible in the logs.

Grant on first registration: a user who joins the moderated chat BEFORE
registering is covered by no chat_member event, so the join-time grant never
fires for them. ProvisionTelegram now reports first contact, and the Telegram
auth handler emits chat_access_changed on it, so the gateway re-evaluates and
grants write access if the user is already in the chat.
2026-06-21 15:19:21 +02:00
Ilia Denisov b22b624d28 fix(telegram): keep a failed promo-bot construction non-fatal
CI / changes (pull_request) Successful in 3s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 17s
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m17s
tgbot.New validates the token with getMe, so a bad or unreachable promo token
would otherwise return an error from run() and crash-loop the whole bot process
— taking the main game bot down with it, since they share the container. Log it
and skip the promo bot instead; the main bot and bot-link are unaffected.
2026-06-21 14:50:01 +02:00
Ilia Denisov e71e40eef5 feat(telegram): promo bot + channel-chat moderation gate
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 19s
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s
Add a second standalone promo bot to the bot container (answers /start with a
localized message + a URL button into the main bot's Mini App) and gate write
access in a channel's linked discussion chat: grant on join when the Telegram
user is registered and neither admin-suspended nor holding a new chat_muted
role, and revoke/grant on the matching moderation change for a member currently
in the chat.

Eligibility (registered AND NOT suspended AND NOT chat_muted; the game
suspension dominates) is resolved once in the backend and reached two ways: the
bot's join-time unary ResolveChatEligibility over the existing mTLS bot-link,
and a backend chat_access_changed event -> gateway -> ChatGate command
(idempotent; a temporary-block-expiry sweeper may over-emit). The bot guards the
block/unblock path with getChatMember, since bots cannot list members.

A web_app button cannot open another bot's Mini App (it signs initData with the
sending bot's token), so the promo button is a t.me ?startapp URL reusing the
UI's VITE_TELEGRAM_LINK. The bot must be a chat admin with the restrict-members
right and chat_member in its allowed updates.

No schema change: chat_muted reuses the data-driven account_roles table.
2026-06-21 14:46:51 +02:00
developer 41d21f3f6f Merge pull request 'feat(ui): tile-crossword loading splash for cold lobby open' (#98) from feature/lobby-splash-tiles into development
CI / changes (push) Successful in 2s
CI / unit (push) Has been skipped
CI / integration (push) Has been skipped
CI / ui (push) Successful in 57s
CI / gate (push) Successful in 1s
CI / deploy (push) Successful in 1m15s
2026-06-21 10:19:15 +00:00
Ilia Denisov 9642cafc1f fix(ui): hold each splash word for the pause before the readiness check
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Has been skipped
CI / integration (pull_request) Has been skipped
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m18s
The dismiss check fired at the instant a word finished laying, so a word
was never held: ЭРУДИТ fell straight into the lobby (too fast) and
ЗАГРУЗКА got no readable pause. The pause was a *leading* gap before the
next word, not a hold after the current one.

Move the hold to after each word and run the check after it: every word
(ЭРУДИТ included) now stays up for PAUSE_MS before the splash either
dismisses or lays the next word. prefixMs = WORD_MS + PAUSE_MS,
cycleMs = 2*(WORD_MS + PAUSE_MS).
2026-06-21 10:42:31 +02:00
Ilia Denisov ba6ee90278 feat(ui): tile-crossword loading splash for cold lobby open
CI / changes (pull_request) Successful in 3s
CI / unit (pull_request) Has been skipped
CI / integration (pull_request) Has been skipped
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m17s
On a cold app open the lobby's game list arrives over the network; on a
slow link the empty "no games yet" line flashed before the games loaded.
Add a full-screen tile splash that lays a Scrabble crossword of ЭРУДИТ /
ЗАГРУЗКА / ОЖИДАНИЕ (Эрудит point values, hardcoded since the alphabet
table is not cached at boot) until the lobby's first load settles, then
removes itself to reveal the populated list.

- lib/splash.ts: pure layout + reveal schedule (unit-tested).
- components/Splash.svelte: App-level overlay; per-tile drop-in; loops
  ЗАГРУЗКА → ОЖИДАНИЕ until ready, dismisses on a word boundary. Static
  ЭРУДИТ under reduced motion / the mock build.
- app state: lobbyReady (set by Lobby on first settle) + splashDone,
  reset on logout.
- App.svelte: overlay while routeIsLobby && !splashDone; the plain text
  splash now only covers non-lobby deep-links during bootstrap.
- docs: UI_DESIGN + FUNCTIONAL (+ _ru).
2026-06-21 10:29:39 +02:00
developer e79c1ea891 Merge pull request 'feat(gateway): temporary IP ban (fail2ban) + honeypot/honeytoken (PRERELEASE AG)' (#97) from feature/abuse-ip-ban-honeypot into development
CI / changes (push) Successful in 1s
CI / unit (push) Successful in 11s
CI / integration (push) Successful in 15s
CI / ui (push) Successful in 56s
CI / gate (push) Successful in 0s
CI / deploy (push) Successful in 1m15s
2026-06-21 07:28:30 +00:00
Ilia Denisov cf9fa75d62 fix(deploy): honeypot tag dropped — Caddy applies header_up delete after set
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 14s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m3s
The @honeypot block both deleted and set X-Scrabble-Honeypot in one reverse_proxy.
Caddy applies header_up deletions *after* sets, so the tag we set was immediately
stripped: the gateway never saw it, and a decoy hit (e.g. GET /.env) fell through
to the gateway's /app redirect (308) instead of tripping the honeypot. Drop the
delete — the bare set already replaces any client-supplied value. The real
endpoints keep stripping the header in the @gateway block (delete-only, no
conflicting set). Caught on the live test contour (no caddy locally).
2026-06-21 09:02:23 +02:00
Ilia Denisov 81b44c2b02 docs(prerelease): mark phase AG done (code + test contour)
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 14s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m6s
2026-06-21 08:58:10 +02:00
Ilia Denisov 041106d623 feat(gateway): temporary IP ban (fail2ban) fed by rejections + honeypot/honeytoken
CI / changes (pull_request) Successful in 3s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 17s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m11s
Add a prod-only, in-memory IP ban enforced at the edge, fed by three signals:
sustained rate-limiter rejections (the IP-keyed public/email/admin classes — the
user class stays the backend soft-flag's concern), a honeypot decoy-path hit (the
contour caddy tags decoys with X-Scrabble-Honeypot and routes them to the gateway),
and a honeytoken (a planted bearer, GATEWAY_HONEYTOKEN). A banned IP is refused with
429 by the abuseGuard middleware before any work — covering the Connect edge, the
live stream and the static SPA/landing the per-op limiter never gated.

The ban is off by default: it keys by the real client IP the shared-NAT test contour
does not expose, so a ban there would be self-inflicted; detection still logs in the
contour, only the ban action is gated (GATEWAY_ABUSE_BAN_ENABLED). Rejection bans last
GATEWAY_ABUSE_BAN_DURATION; tripwire/honeytoken hits are near-zero-false-positive and
earn longer fixed bans. Each ban increments gateway_abuse_banned_total{reason}.

Operators see and lift active bans on the admin console's Throttled page; the gateway
syncs its active set to the backend every 30s (POST /api/v1/internal/bans/sync,
backend/internal/banview) and applies the operator unbans the response returns.

PRERELEASE phase AG. Docs baked into ARCHITECTURE / FUNCTIONAL (+ru) / both READMEs.
2026-06-21 08:54:20 +02:00
124 changed files with 6229 additions and 613 deletions
+10 -2
View File
@@ -260,11 +260,16 @@ jobs:
GM_BASICAUTH_HASH: ${{ secrets.TEST_GM_BASICAUTH_HASH }}
GRAFANA_ADMIN_PASSWORD: ${{ secrets.TEST_GRAFANA_ADMIN_PASSWORD }}
TELEGRAM_BOT_TOKEN: ${{ secrets.TEST_TELEGRAM_BOT_TOKEN }}
TELEGRAM_PROMO_BOT_TOKEN: ${{ secrets.TEST_TELEGRAM_PROMO_BOT_TOKEN }}
GM_BASICAUTH_USER: ${{ vars.TEST_GM_BASICAUTH_USER }}
GRAFANA_ROOT_URL: ${{ vars.TEST_GRAFANA_ROOT_URL }}
CADDY_SITE_ADDRESS: ${{ vars.TEST_CADDY_SITE_ADDRESS }}
TELEGRAM_MINIAPP_URL: ${{ vars.TEST_TELEGRAM_MINIAPP_URL }}
TELEGRAM_GAME_CHANNEL_ID: ${{ vars.TEST_TELEGRAM_GAME_CHANNEL_ID }}
TELEGRAM_CHAT_ID: ${{ vars.TEST_TELEGRAM_CHAT_ID }}
TELEGRAM_BOT_USERNAME: ${{ vars.TEST_TELEGRAM_BOT_USERNAME }}
# The promo button reuses the UI's Mini App link variable.
TELEGRAM_BOT_LINK: ${{ vars.TEST_VITE_TELEGRAM_LINK }}
# The test contour always uses Telegram's test environment — pinned here,
# not an operator variable. The prod workflow leaves it false.
TELEGRAM_TEST_ENV: "true"
@@ -296,8 +301,11 @@ jobs:
# App version for the About screen: the git tag if present, else the short SHA
# (the test checkout is shallow/untagged, so this is the SHA here — fine).
export APP_VERSION="$(git -C "$GITHUB_WORKSPACE" describe --tags --always 2>/dev/null || echo dev)"
docker compose --ansi never build --progress plain
docker compose --ansi never up -d --remove-orphans
# The telegram-local profile brings the bot + its VPN sidecar; prod runs the
# bot on its own host instead (deploy/docker-compose.bot.yml), and the prod
# main host omits both. Without the profile they would not start here.
docker compose --ansi never --profile telegram-local build --progress plain
docker compose --ansi never --profile telegram-local up -d --remove-orphans
# The config-only services bind-mount the reseeded config dir. A plain `up -d`
# leaves them on the previous bind mount (the dir was rm'd + recreated), so a
# changed Caddyfile or Grafana dashboard is ignored — force-recreate them to
+266
View File
@@ -0,0 +1,266 @@
# Manual production rollout. Runs ONLY from master, ONLY on workflow_dispatch with
# confirm=deploy (development->master is merged + green first; this is the separate,
# deliberate prod step). Visible sequential jobs from most to least significant:
# build -> deploy-main -> deploy-bot -> verify
# The per-service rolling (postgres->backend->gateway->landing->validator->caddy),
# health-gating and auto-rollback live in deploy/prod-deploy.sh on the main host and
# show in the deploy-main log. Manual post-deploy rollback is prod-rollback.yaml.
# See deploy/README.md (prod runbook).
name: prod-deploy
run-name: "prod deploy ${{ github.sha }}"
on:
workflow_dispatch:
inputs:
confirm:
description: 'Type "deploy" to confirm a production rollout from master.'
required: true
default: ""
permissions:
contents: read
env:
NO_COLOR: "1"
DOCKER_CLI_HINTS: "false"
REGISTRY: docker.iliadenisov.ru/developer
jobs:
build:
if: ${{ github.ref == 'refs/heads/master' && inputs.confirm == 'deploy' }}
runs-on: ubuntu-latest
defaults:
run:
shell: bash
outputs:
tag: ${{ steps.ver.outputs.tag }}
env:
PROD_REGISTRY_USER: ${{ vars.PROD_REGISTRY_USER }}
PROD_REGISTRY_PASSWORD: ${{ secrets.PROD_REGISTRY_PASSWORD }}
VITE_TELEGRAM_BOT_ID: ${{ vars.PROD_VITE_TELEGRAM_BOT_ID }}
VITE_TELEGRAM_LINK: ${{ vars.PROD_VITE_TELEGRAM_LINK }}
VITE_TELEGRAM_GAME_CHANNEL_NAME: ${{ vars.PROD_VITE_TELEGRAM_GAME_CHANNEL_NAME }}
VITE_GATEWAY_URL: ${{ vars.PROD_VITE_GATEWAY_URL }}
POSTGRES_PASSWORD: ${{ secrets.PROD_POSTGRES_PASSWORD }}
GM_BASICAUTH_HASH: ${{ secrets.PROD_GM_BASICAUTH_HASH }}
TELEGRAM_MINIAPP_URL: ${{ vars.PROD_TELEGRAM_MINIAPP_URL }}
DICT_VERSION: ${{ vars.PROD_DICT_VERSION }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Compute version tag
id: ver
run: echo "tag=$(git describe --tags --always)" >> "$GITHUB_OUTPUT"
- name: Registry login
run: echo "$PROD_REGISTRY_PASSWORD" | docker login "${REGISTRY%%/*}" -u "$PROD_REGISTRY_USER" --password-stdin
- name: Build and push images
working-directory: deploy
run: |
export TAG="${{ steps.ver.outputs.tag }}" APP_VERSION="${{ steps.ver.outputs.tag }}" SCRABBLE_CONFIG_DIR=.
# The four main-stack images via compose (reuses the build args, incl. VERSION);
# the bot separately, since it is profiled out of the prod compose.
docker compose -f docker-compose.yml -f docker-compose.prod.yml build
docker compose -f docker-compose.yml -f docker-compose.prod.yml push backend gateway landing validator
docker build -f ../platform/telegram/Dockerfile --target bot --build-arg VERSION="$TAG" -t "$REGISTRY/scrabble-telegram-bot:$TAG" ..
docker push "$REGISTRY/scrabble-telegram-bot:$TAG"
deploy-main:
needs: build
runs-on: ubuntu-latest
defaults:
run:
shell: bash
env:
TAG: ${{ needs.build.outputs.tag }}
PROD_REGISTRY_USER: ${{ vars.PROD_REGISTRY_USER }}
PROD_REGISTRY_PASSWORD: ${{ secrets.PROD_REGISTRY_PASSWORD }}
PROD_SSH_KEY: ${{ secrets.PROD_SSH_KEY }}
PROD_SSH_KNOWN_HOSTS: ${{ secrets.PROD_SSH_KNOWN_HOSTS }}
MAIN_HOST: ${{ vars.PROD_MAIN_HOST }}
POSTGRES_PASSWORD: ${{ secrets.PROD_POSTGRES_PASSWORD }}
GM_BASICAUTH_HASH: ${{ secrets.PROD_GM_BASICAUTH_HASH }}
GRAFANA_ADMIN_PASSWORD: ${{ secrets.PROD_GRAFANA_ADMIN_PASSWORD }}
TELEGRAM_BOT_TOKEN: ${{ secrets.PROD_TELEGRAM_BOT_TOKEN }}
PROD_BOTLINK_CA: ${{ secrets.PROD_BOTLINK_CA }}
PROD_BOTLINK_GATEWAY_CERT: ${{ secrets.PROD_BOTLINK_GATEWAY_CERT }}
PROD_BOTLINK_GATEWAY_KEY: ${{ secrets.PROD_BOTLINK_GATEWAY_KEY }}
GM_BASICAUTH_USER: ${{ vars.PROD_GM_BASICAUTH_USER }}
GRAFANA_ROOT_URL: ${{ vars.PROD_GRAFANA_ROOT_URL }}
CADDY_SITE_ADDRESS: ${{ vars.PROD_CADDY_SITE_ADDRESS }}
LOG_LEVEL: ${{ vars.PROD_LOG_LEVEL }}
DICT_VERSION: ${{ vars.PROD_DICT_VERSION }}
POSTGRES_DB: ${{ vars.PROD_POSTGRES_DB }}
POSTGRES_USER: ${{ vars.PROD_POSTGRES_USER }}
TELEGRAM_MINIAPP_URL: ${{ vars.PROD_TELEGRAM_MINIAPP_URL }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up SSH
run: |
mkdir -p ~/.ssh && chmod 700 ~/.ssh
printf '%s\n' "$PROD_SSH_KEY" > ~/.ssh/id_deploy && chmod 600 ~/.ssh/id_deploy
printf '%s\n' "$PROD_SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- name: Determine previous tag and migration
run: |
ssh_main() { ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$MAIN_HOST" "$@"; }
PREV_TAG="$(ssh_main 'cat /opt/scrabble/DEPLOYED_TAG 2>/dev/null || echo none')"
MIGRATION=0
if [ "$PREV_TAG" != none ]; then
if ! git cat-file -e "$PREV_TAG^{commit}" 2>/dev/null; then
MIGRATION=1
elif git diff --name-only "$PREV_TAG..$TAG" -- backend/internal/postgres/migrations/ | grep -q .; then
MIGRATION=1
fi
fi
{ echo "PREV_TAG=$PREV_TAG"; echo "MIGRATION=$MIGRATION"; } >> "$GITHUB_ENV"
echo "prev=$PREV_TAG migration=$MIGRATION"
- name: Render main env + certs
run: |
umask 077
mkdir -p stage/certs-main
cat > stage/env.sh <<EOF
export REGISTRY='$REGISTRY'
export SCRABBLE_CONFIG_DIR='/opt/scrabble'
export POSTGRES_DB='${POSTGRES_DB:-scrabble}'
export POSTGRES_USER='${POSTGRES_USER:-scrabble}'
export POSTGRES_PASSWORD='$POSTGRES_PASSWORD'
export GM_BASICAUTH_USER='${GM_BASICAUTH_USER:-gm}'
export GM_BASICAUTH_HASH='$GM_BASICAUTH_HASH'
export GRAFANA_ADMIN_PASSWORD='$GRAFANA_ADMIN_PASSWORD'
export GRAFANA_ROOT_URL='$GRAFANA_ROOT_URL'
export CADDY_SITE_ADDRESS='$CADDY_SITE_ADDRESS'
export LOG_LEVEL='${LOG_LEVEL:-info}'
export DICT_VERSION='$DICT_VERSION'
export APP_VERSION='$TAG'
export TELEGRAM_BOT_TOKEN='$TELEGRAM_BOT_TOKEN'
export TELEGRAM_MINIAPP_URL='$TELEGRAM_MINIAPP_URL'
export GATEWAY_ABUSE_BAN_ENABLED='true'
EOF
printf '%s\n' "$PROD_BOTLINK_CA" > stage/certs-main/ca.crt
printf '%s\n' "$PROD_BOTLINK_GATEWAY_CERT" > stage/certs-main/gateway.crt
printf '%s\n' "$PROD_BOTLINK_GATEWAY_KEY" > stage/certs-main/gateway.key
chmod 644 stage/certs-main/*
- name: Deploy the main host
run: |
ssh_main() { ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$MAIN_HOST" "$@"; }
ssh_main 'mkdir -p /opt/scrabble/compose'
tar -C deploy -czf - docker-compose.yml docker-compose.prod.yml prod-deploy.sh \
| ssh_main 'tar -C /opt/scrabble/compose -xzf -'
tar -C deploy -czf - caddy otelcol prometheus tempo grafana \
| ssh_main 'tar -C /opt/scrabble -xzf -'
tar -C stage -czf - certs-main \
| ssh_main 'rm -rf /opt/scrabble/certs && mkdir -p /opt/scrabble/certs && tar -C /opt/scrabble/certs --strip-components=1 -xzf -'
scp -i ~/.ssh/id_deploy -o BatchMode=yes stage/env.sh "deploy@$MAIN_HOST:/opt/scrabble/env.sh"
echo "$PROD_REGISTRY_PASSWORD" | ssh_main "docker login ${REGISTRY%%/*} -u $PROD_REGISTRY_USER --password-stdin"
ssh_main "TAG='$TAG' PREV_TAG='$PREV_TAG' MIGRATION='$MIGRATION' bash /opt/scrabble/compose/prod-deploy.sh"
deploy-bot:
needs: [build, deploy-main]
runs-on: ubuntu-latest
defaults:
run:
shell: bash
env:
TAG: ${{ needs.build.outputs.tag }}
PROD_REGISTRY_USER: ${{ vars.PROD_REGISTRY_USER }}
PROD_REGISTRY_PASSWORD: ${{ secrets.PROD_REGISTRY_PASSWORD }}
PROD_SSH_KEY: ${{ secrets.PROD_SSH_KEY }}
PROD_SSH_KNOWN_HOSTS: ${{ secrets.PROD_SSH_KNOWN_HOSTS }}
TG_HOST: ${{ vars.PROD_TG_HOST }}
MAIN_HOST: ${{ vars.PROD_MAIN_HOST }}
TELEGRAM_BOT_TOKEN: ${{ secrets.PROD_TELEGRAM_BOT_TOKEN }}
TELEGRAM_PROMO_BOT_TOKEN: ${{ secrets.PROD_TELEGRAM_PROMO_BOT_TOKEN }}
PROD_BOTLINK_CA: ${{ secrets.PROD_BOTLINK_CA }}
PROD_BOTLINK_BOT_CERT: ${{ secrets.PROD_BOTLINK_BOT_CERT }}
PROD_BOTLINK_BOT_KEY: ${{ secrets.PROD_BOTLINK_BOT_KEY }}
LOG_LEVEL: ${{ vars.PROD_LOG_LEVEL }}
TELEGRAM_MINIAPP_URL: ${{ vars.PROD_TELEGRAM_MINIAPP_URL }}
TELEGRAM_GAME_CHANNEL_ID: ${{ vars.PROD_TELEGRAM_GAME_CHANNEL_ID }}
TELEGRAM_CHAT_ID: ${{ vars.PROD_TELEGRAM_CHAT_ID }}
TELEGRAM_BOT_USERNAME: ${{ vars.PROD_TELEGRAM_BOT_USERNAME }}
TELEGRAM_BOT_LINK: ${{ vars.PROD_VITE_TELEGRAM_LINK }}
steps:
- uses: actions/checkout@v4
- name: Set up SSH
run: |
mkdir -p ~/.ssh && chmod 700 ~/.ssh
printf '%s\n' "$PROD_SSH_KEY" > ~/.ssh/id_deploy && chmod 600 ~/.ssh/id_deploy
printf '%s\n' "$PROD_SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- name: Render bot env + certs
run: |
umask 077
mkdir -p stage/certs-bot
cat > stage/env.bot.sh <<EOF
export SCRABBLE_CONFIG_DIR='/opt/scrabble'
export BOT_IMAGE='$REGISTRY/scrabble-telegram-bot:$TAG'
export BOTLINK_GATEWAY_ADDR='$MAIN_HOST:9443'
export TELEGRAM_BOT_TOKEN='$TELEGRAM_BOT_TOKEN'
export TELEGRAM_MINIAPP_URL='$TELEGRAM_MINIAPP_URL'
export TELEGRAM_GAME_CHANNEL_ID='$TELEGRAM_GAME_CHANNEL_ID'
export TELEGRAM_CHAT_ID='$TELEGRAM_CHAT_ID'
export TELEGRAM_PROMO_BOT_TOKEN='$TELEGRAM_PROMO_BOT_TOKEN'
export TELEGRAM_BOT_USERNAME='$TELEGRAM_BOT_USERNAME'
export TELEGRAM_BOT_LINK='$TELEGRAM_BOT_LINK'
export LOG_LEVEL='${LOG_LEVEL:-info}'
EOF
printf '%s\n' "$PROD_BOTLINK_CA" > stage/certs-bot/ca.crt
printf '%s\n' "$PROD_BOTLINK_BOT_CERT" > stage/certs-bot/bot.crt
printf '%s\n' "$PROD_BOTLINK_BOT_KEY" > stage/certs-bot/bot.key
chmod 644 stage/certs-bot/*
- name: Deploy the bot host
run: |
ssh_tg() { ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$TG_HOST" "$@"; }
ssh_tg 'mkdir -p /opt/scrabble/compose'
tar -C deploy -czf - docker-compose.bot.yml | ssh_tg 'tar -C /opt/scrabble/compose -xzf -'
tar -C stage -czf - certs-bot \
| ssh_tg 'rm -rf /opt/scrabble/certs && mkdir -p /opt/scrabble/certs && tar -C /opt/scrabble/certs --strip-components=1 -xzf -'
scp -i ~/.ssh/id_deploy -o BatchMode=yes stage/env.bot.sh "deploy@$TG_HOST:/opt/scrabble/env.bot.sh"
echo "$PROD_REGISTRY_PASSWORD" | ssh_tg "docker login ${REGISTRY%%/*} -u $PROD_REGISTRY_USER --password-stdin"
ssh_tg 'set -a; . /opt/scrabble/env.bot.sh; set +a; cd /opt/scrabble/compose;
docker compose -f docker-compose.bot.yml pull;
docker compose -f docker-compose.bot.yml up -d'
ssh_tg 'for i in $(seq 1 20); do
s=$(docker inspect -f "{{.State.Status}}" scrabble-telegram-bot 2>/dev/null || echo missing)
r=$(docker inspect -f "{{.State.Restarting}}" scrabble-telegram-bot 2>/dev/null || echo true)
if [ "$s" = running ] && [ "$r" = false ]; then
c1=$(docker inspect -f "{{.RestartCount}}" scrabble-telegram-bot); sleep 5
c2=$(docker inspect -f "{{.RestartCount}}" scrabble-telegram-bot)
[ "$c1" = "$c2" ] && { echo "bot healthy"; exit 0; }
fi
sleep 3
done
echo "bot not healthy:"; docker logs --tail 80 scrabble-telegram-bot; exit 1'
verify:
needs: [deploy-main, deploy-bot]
runs-on: ubuntu-latest
defaults:
run:
shell: bash
env:
PROD_SSH_KEY: ${{ secrets.PROD_SSH_KEY }}
PROD_SSH_KNOWN_HOSTS: ${{ secrets.PROD_SSH_KNOWN_HOSTS }}
MAIN_HOST: ${{ vars.PROD_MAIN_HOST }}
CADDY_SITE_ADDRESS: ${{ vars.PROD_CADDY_SITE_ADDRESS }}
steps:
- name: Set up SSH
run: |
mkdir -p ~/.ssh && chmod 700 ~/.ssh
printf '%s\n' "$PROD_SSH_KEY" > ~/.ssh/id_deploy && chmod 600 ~/.ssh/id_deploy
printf '%s\n' "$PROD_SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- name: Verify the public site
run: |
domain="${CADDY_SITE_ADDRESS%% *}"
ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$MAIN_HOST" "for i in \$(seq 1 20); do
if curl -fsS -k --resolve $domain:443:127.0.0.1 https://$domain/ -o /dev/null &&
curl -fsS -k --resolve $domain:443:127.0.0.1 https://$domain/app/ -o /dev/null &&
docker run --rm --network scrabble-internal alpine:3.20 wget -q -T 5 -O /dev/null http://backend:8080/readyz; then
echo 'public site + /app/ + backend healthy'; exit 0
fi
sleep 5
done
echo 'public verify failed; recent caddy + gateway + backend logs:'
docker logs --tail 40 scrabble-caddy; docker logs --tail 40 scrabble-gateway; docker logs --tail 40 scrabble-backend
exit 1"
+223
View File
@@ -0,0 +1,223 @@
# Manual production rollback. Runs ONLY from master, ONLY on workflow_dispatch with
# confirm=rollback. Re-deploys an already-published image tag (no build): leave
# target_version blank to roll back to the previously deployed version (read from the
# main host), or set it to a specific release tag from the Releases page. The
# re-deploy is the same rolling, health-gated path as prod-deploy (TAG=target,
# MIGRATION=0 — rollback is image-only and never migrates the DB; image rollback is
# DB-safe under the expand-contract rule). See deploy/README.md (prod runbook).
name: prod-rollback
run-name: "prod rollback ${{ inputs.target_version || 'previous' }}"
on:
workflow_dispatch:
inputs:
confirm:
description: 'Type "rollback" to confirm a production rollback.'
required: true
default: ""
target_version:
description: "Release tag to roll back to (blank = the previous deployed version)."
required: false
default: ""
permissions:
contents: read
env:
NO_COLOR: "1"
DOCKER_CLI_HINTS: "false"
REGISTRY: docker.iliadenisov.ru/developer
jobs:
rollback-main:
if: ${{ github.ref == 'refs/heads/master' && inputs.confirm == 'rollback' }}
runs-on: ubuntu-latest
defaults:
run:
shell: bash
outputs:
target: ${{ steps.resolve.outputs.target }}
env:
PROD_REGISTRY_USER: ${{ vars.PROD_REGISTRY_USER }}
PROD_REGISTRY_PASSWORD: ${{ secrets.PROD_REGISTRY_PASSWORD }}
PROD_SSH_KEY: ${{ secrets.PROD_SSH_KEY }}
PROD_SSH_KNOWN_HOSTS: ${{ secrets.PROD_SSH_KNOWN_HOSTS }}
MAIN_HOST: ${{ vars.PROD_MAIN_HOST }}
POSTGRES_PASSWORD: ${{ secrets.PROD_POSTGRES_PASSWORD }}
GM_BASICAUTH_HASH: ${{ secrets.PROD_GM_BASICAUTH_HASH }}
GRAFANA_ADMIN_PASSWORD: ${{ secrets.PROD_GRAFANA_ADMIN_PASSWORD }}
TELEGRAM_BOT_TOKEN: ${{ secrets.PROD_TELEGRAM_BOT_TOKEN }}
PROD_BOTLINK_CA: ${{ secrets.PROD_BOTLINK_CA }}
PROD_BOTLINK_GATEWAY_CERT: ${{ secrets.PROD_BOTLINK_GATEWAY_CERT }}
PROD_BOTLINK_GATEWAY_KEY: ${{ secrets.PROD_BOTLINK_GATEWAY_KEY }}
GM_BASICAUTH_USER: ${{ vars.PROD_GM_BASICAUTH_USER }}
GRAFANA_ROOT_URL: ${{ vars.PROD_GRAFANA_ROOT_URL }}
CADDY_SITE_ADDRESS: ${{ vars.PROD_CADDY_SITE_ADDRESS }}
LOG_LEVEL: ${{ vars.PROD_LOG_LEVEL }}
DICT_VERSION: ${{ vars.PROD_DICT_VERSION }}
POSTGRES_DB: ${{ vars.PROD_POSTGRES_DB }}
POSTGRES_USER: ${{ vars.PROD_POSTGRES_USER }}
TELEGRAM_MINIAPP_URL: ${{ vars.PROD_TELEGRAM_MINIAPP_URL }}
INPUT_TARGET: ${{ inputs.target_version }}
steps:
- uses: actions/checkout@v4
- name: Set up SSH
run: |
mkdir -p ~/.ssh && chmod 700 ~/.ssh
printf '%s\n' "$PROD_SSH_KEY" > ~/.ssh/id_deploy && chmod 600 ~/.ssh/id_deploy
printf '%s\n' "$PROD_SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- name: Resolve rollback target
id: resolve
run: |
ssh_main() { ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$MAIN_HOST" "$@"; }
CURRENT="$(ssh_main 'cat /opt/scrabble/DEPLOYED_TAG 2>/dev/null || echo none')"
if [ -n "$INPUT_TARGET" ]; then
TARGET="$INPUT_TARGET"
else
TARGET="$(ssh_main 'cat /opt/scrabble/PREVIOUS_TAG 2>/dev/null || echo none')"
fi
if [ -z "$TARGET" ] || [ "$TARGET" = none ]; then
echo "no rollback target (no PREVIOUS_TAG on the host and no target_version input)"; exit 1
fi
if [ "$TARGET" = "$CURRENT" ]; then
echo "target $TARGET is already the deployed version; nothing to do"; exit 1
fi
echo "rolling back: current=$CURRENT -> target=$TARGET"
echo "target=$TARGET" >> "$GITHUB_OUTPUT"
{ echo "TARGET=$TARGET"; echo "CURRENT=$CURRENT"; } >> "$GITHUB_ENV"
- name: Render main env + certs
run: |
umask 077
mkdir -p stage/certs-main
cat > stage/env.sh <<EOF
export REGISTRY='$REGISTRY'
export SCRABBLE_CONFIG_DIR='/opt/scrabble'
export POSTGRES_DB='${POSTGRES_DB:-scrabble}'
export POSTGRES_USER='${POSTGRES_USER:-scrabble}'
export POSTGRES_PASSWORD='$POSTGRES_PASSWORD'
export GM_BASICAUTH_USER='${GM_BASICAUTH_USER:-gm}'
export GM_BASICAUTH_HASH='$GM_BASICAUTH_HASH'
export GRAFANA_ADMIN_PASSWORD='$GRAFANA_ADMIN_PASSWORD'
export GRAFANA_ROOT_URL='$GRAFANA_ROOT_URL'
export CADDY_SITE_ADDRESS='$CADDY_SITE_ADDRESS'
export LOG_LEVEL='${LOG_LEVEL:-info}'
export DICT_VERSION='$DICT_VERSION'
export APP_VERSION='$TARGET'
export TELEGRAM_BOT_TOKEN='$TELEGRAM_BOT_TOKEN'
export TELEGRAM_MINIAPP_URL='$TELEGRAM_MINIAPP_URL'
export GATEWAY_ABUSE_BAN_ENABLED='true'
EOF
printf '%s\n' "$PROD_BOTLINK_CA" > stage/certs-main/ca.crt
printf '%s\n' "$PROD_BOTLINK_GATEWAY_CERT" > stage/certs-main/gateway.crt
printf '%s\n' "$PROD_BOTLINK_GATEWAY_KEY" > stage/certs-main/gateway.key
chmod 644 stage/certs-main/*
- name: Roll the main host back
run: |
ssh_main() { ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$MAIN_HOST" "$@"; }
ssh_main 'mkdir -p /opt/scrabble/compose'
tar -C deploy -czf - docker-compose.yml docker-compose.prod.yml prod-deploy.sh \
| ssh_main 'tar -C /opt/scrabble/compose -xzf -'
tar -C deploy -czf - caddy otelcol prometheus tempo grafana \
| ssh_main 'tar -C /opt/scrabble -xzf -'
tar -C stage -czf - certs-main \
| ssh_main 'rm -rf /opt/scrabble/certs && mkdir -p /opt/scrabble/certs && tar -C /opt/scrabble/certs --strip-components=1 -xzf -'
scp -i ~/.ssh/id_deploy -o BatchMode=yes stage/env.sh "deploy@$MAIN_HOST:/opt/scrabble/env.sh"
echo "$PROD_REGISTRY_PASSWORD" | ssh_main "docker login ${REGISTRY%%/*} -u $PROD_REGISTRY_USER --password-stdin"
# Image-only rollback: no migration window (TAG=target, MIGRATION=0). A failed
# rollback's auto-revert returns to the current version (PREV_TAG=$CURRENT).
ssh_main "TAG='$TARGET' PREV_TAG='$CURRENT' MIGRATION=0 bash /opt/scrabble/compose/prod-deploy.sh"
rollback-bot:
needs: rollback-main
runs-on: ubuntu-latest
defaults:
run:
shell: bash
env:
TARGET: ${{ needs.rollback-main.outputs.target }}
PROD_REGISTRY_USER: ${{ vars.PROD_REGISTRY_USER }}
PROD_REGISTRY_PASSWORD: ${{ secrets.PROD_REGISTRY_PASSWORD }}
PROD_SSH_KEY: ${{ secrets.PROD_SSH_KEY }}
PROD_SSH_KNOWN_HOSTS: ${{ secrets.PROD_SSH_KNOWN_HOSTS }}
TG_HOST: ${{ vars.PROD_TG_HOST }}
MAIN_HOST: ${{ vars.PROD_MAIN_HOST }}
TELEGRAM_BOT_TOKEN: ${{ secrets.PROD_TELEGRAM_BOT_TOKEN }}
TELEGRAM_PROMO_BOT_TOKEN: ${{ secrets.PROD_TELEGRAM_PROMO_BOT_TOKEN }}
PROD_BOTLINK_CA: ${{ secrets.PROD_BOTLINK_CA }}
PROD_BOTLINK_BOT_CERT: ${{ secrets.PROD_BOTLINK_BOT_CERT }}
PROD_BOTLINK_BOT_KEY: ${{ secrets.PROD_BOTLINK_BOT_KEY }}
LOG_LEVEL: ${{ vars.PROD_LOG_LEVEL }}
TELEGRAM_MINIAPP_URL: ${{ vars.PROD_TELEGRAM_MINIAPP_URL }}
TELEGRAM_GAME_CHANNEL_ID: ${{ vars.PROD_TELEGRAM_GAME_CHANNEL_ID }}
TELEGRAM_CHAT_ID: ${{ vars.PROD_TELEGRAM_CHAT_ID }}
TELEGRAM_BOT_USERNAME: ${{ vars.PROD_TELEGRAM_BOT_USERNAME }}
TELEGRAM_BOT_LINK: ${{ vars.PROD_VITE_TELEGRAM_LINK }}
steps:
- uses: actions/checkout@v4
- name: Set up SSH
run: |
mkdir -p ~/.ssh && chmod 700 ~/.ssh
printf '%s\n' "$PROD_SSH_KEY" > ~/.ssh/id_deploy && chmod 600 ~/.ssh/id_deploy
printf '%s\n' "$PROD_SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- name: Render bot env + certs
run: |
umask 077
mkdir -p stage/certs-bot
cat > stage/env.bot.sh <<EOF
export SCRABBLE_CONFIG_DIR='/opt/scrabble'
export BOT_IMAGE='$REGISTRY/scrabble-telegram-bot:$TARGET'
export BOTLINK_GATEWAY_ADDR='$MAIN_HOST:9443'
export TELEGRAM_BOT_TOKEN='$TELEGRAM_BOT_TOKEN'
export TELEGRAM_MINIAPP_URL='$TELEGRAM_MINIAPP_URL'
export TELEGRAM_GAME_CHANNEL_ID='$TELEGRAM_GAME_CHANNEL_ID'
export TELEGRAM_CHAT_ID='$TELEGRAM_CHAT_ID'
export TELEGRAM_PROMO_BOT_TOKEN='$TELEGRAM_PROMO_BOT_TOKEN'
export TELEGRAM_BOT_USERNAME='$TELEGRAM_BOT_USERNAME'
export TELEGRAM_BOT_LINK='$TELEGRAM_BOT_LINK'
export LOG_LEVEL='${LOG_LEVEL:-info}'
EOF
printf '%s\n' "$PROD_BOTLINK_CA" > stage/certs-bot/ca.crt
printf '%s\n' "$PROD_BOTLINK_BOT_CERT" > stage/certs-bot/bot.crt
printf '%s\n' "$PROD_BOTLINK_BOT_KEY" > stage/certs-bot/bot.key
chmod 644 stage/certs-bot/*
- name: Roll the bot host back
run: |
ssh_tg() { ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$TG_HOST" "$@"; }
ssh_tg 'mkdir -p /opt/scrabble/compose'
tar -C deploy -czf - docker-compose.bot.yml | ssh_tg 'tar -C /opt/scrabble/compose -xzf -'
tar -C stage -czf - certs-bot \
| ssh_tg 'rm -rf /opt/scrabble/certs && mkdir -p /opt/scrabble/certs && tar -C /opt/scrabble/certs --strip-components=1 -xzf -'
scp -i ~/.ssh/id_deploy -o BatchMode=yes stage/env.bot.sh "deploy@$TG_HOST:/opt/scrabble/env.bot.sh"
echo "$PROD_REGISTRY_PASSWORD" | ssh_tg "docker login ${REGISTRY%%/*} -u $PROD_REGISTRY_USER --password-stdin"
ssh_tg 'set -a; . /opt/scrabble/env.bot.sh; set +a; cd /opt/scrabble/compose;
docker compose -f docker-compose.bot.yml pull;
docker compose -f docker-compose.bot.yml up -d'
verify:
needs: [rollback-main, rollback-bot]
runs-on: ubuntu-latest
defaults:
run:
shell: bash
env:
PROD_SSH_KEY: ${{ secrets.PROD_SSH_KEY }}
PROD_SSH_KNOWN_HOSTS: ${{ secrets.PROD_SSH_KNOWN_HOSTS }}
MAIN_HOST: ${{ vars.PROD_MAIN_HOST }}
CADDY_SITE_ADDRESS: ${{ vars.PROD_CADDY_SITE_ADDRESS }}
steps:
- name: Set up SSH
run: |
mkdir -p ~/.ssh && chmod 700 ~/.ssh
printf '%s\n' "$PROD_SSH_KEY" > ~/.ssh/id_deploy && chmod 600 ~/.ssh/id_deploy
printf '%s\n' "$PROD_SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- name: Verify the public site
run: |
domain="${CADDY_SITE_ADDRESS%% *}"
ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$MAIN_HOST" "for i in \$(seq 1 20); do
if curl -fsS -k --resolve $domain:443:127.0.0.1 https://$domain/ -o /dev/null &&
docker run --rm --network scrabble-internal alpine:3.20 wget -q -T 5 -O /dev/null http://backend:8080/readyz; then
echo 'rolled-back site healthy'; exit 0
fi
sleep 5
done
echo 'verify failed'; docker logs --tail 40 scrabble-caddy; docker logs --tail 40 scrabble-backend; exit 1"
+23 -13
View File
@@ -51,7 +51,7 @@ independent (see ARCHITECTURE §9.1).
| 15 | Dual Telegram bots & language-gated variants | **done** |
| 16 | Deploy infra & test contour (Dockerfiles, gateway static UI, compose, observability) | **done** |
| 17 | Test-contour verification & defect fixes | **done** |
| 18 | Prod contour deploy (SSH export/import, manual after merge) | todo |
| 18 | Prod contour deploy (registry, two-host, rolling + auto-rollback; manual after merge) | machinery built; first cutover pending DNS |
| 19 | User feedback (in-app submit + attachment, admin review/reply, account roles) | **done** |
Scaffolding is incremental: `go.work` lists only existing modules; each stage
@@ -413,18 +413,28 @@ raw list is kept here as the record of what the first contour run surfaced.
"что-то пошло не так". при этом "new -> эрудит" работает. Попробуй посмотреть в логах сейчас, может что-то есть. Или как-то иначе проанализируй, или давай вместе будем смотреть, если не получится.
### Stage 18 — Prod contour deploy
Scope: the **production contour** on a remote host over SSH. Deploy by **container export/import**
(`docker save``scp`/ssh → `docker load``docker compose up` on the remote), the SSH key + host IP
in Gitea secrets; **strictly manual** (`workflow_dispatch`) after `development` is merged to `master`
(the Stage 16 branch model: `feature/* → development → master`, merge gated green). Two-contour config
uses **`TEST_`/`PROD_` secret/variable prefixes** — Gitea 1.26 has no deployment environments (verified:
the `environments` API 404s), so a flat prefixed namespace is the convention.
Reuses the Stage 16 `deploy/docker-compose.yml` as-is, mapping the **`PROD_`** set onto the same
unprefixed compose vars. **No host caddy on prod**, so the contour's own caddy terminates TLS — set
`CADDY_SITE_ADDRESS` to the prod domain so caddy does its own ACME (the Caddyfile is already
parameterised for this; the test contour leaves it `:80` behind the host caddy).
Open details (re-interview): export/import vs a registry trade-off; prod domain/cert source (ACME vs a
provided cert) at the contour caddy; prod VPN; rollback.
Scope: the **production contour** on **two remote hosts** over SSH — main (full stack, `erudit-game.ru`)
and tg (the bot only). Resolved open details (re-interviewed):
- **Transport: a registry** (not export/import) — build + push to `docker.iliadenisov.ru`, the hosts pull by tag.
- **Cert: ACME** at the contour caddy (`CADDY_SITE_ADDRESS=erudit-game.ru www.erudit-game.ru`, no host caddy).
- **No prod VPN** — the bot host has native Bot API egress (verified `api.telegram.org` → 200).
- **Rollback** — rolling per-service deploy (least → most dependent), health-gated, auto-rollback to the
previous image tag; a maintenance window + consistent `pg_dump` only on a schema migration
(expand-contract keeps the auto-rollback image-only; the dump is a manual safety net).
**Strictly manual** (`workflow_dispatch` from `master`, `confirm=deploy`) after `development → master`
is merged green. `TEST_`/`PROD_` prefixed Gitea secrets/variables (Gitea 1.26 has no deployment
environments — the `environments` API 404s). Hosts are provisioned by **`deploy/ansible/`** (docker, a
non-sudo `deploy` user with the CI key, key-only sshd, ufw, fail2ban). The main host is **launch-sized**
(2 vCPU / 1.9 GiB): `docker-compose.prod.yml` trims the R7 limits (`GOMAXPROCS=2`, smaller caps, 7d
Prometheus retention) and adds `node_exporter` for host-memory monitoring (launch undersized, resize at
Selectel reactively). `vpn`+`bot` are gated to a `telegram-local` compose profile (test only); the prod
bot runs standalone from `docker-compose.bot.yml`. `GATEWAY_ABUSE_BAN_ENABLED=true`.
**Built:** `deploy/ansible/` (both hosts provisioned + verified), the compose split + `node_exporter`,
`.gitea/workflows/prod-deploy.yaml` + `deploy/prod-deploy.sh`, the full `PROD_` secret/variable set.
**Remaining (acceptance):** the **first live cutover** — waits on the `erudit-game.ru` DNS delegation
(`A`/`www` → the main host) that ACME requires; then run the workflow and verify the public site end-to-end.
### Stage 19 — User feedback *(done)*
A user→operator feedback channel, sequenced after the numbered stages but shipped **before** the Stage 18
+69 -4
View File
@@ -39,7 +39,9 @@ the edge before prod. Each phase maps back to the owner's raw pre-release TODO l
| FM | First-move tile draw (official rules): each seated player draws a tile, the one closest to "A" leads (a blank beats every letter), ties re-drawing until a single leader; **honest per-draw `crypto/rand` entropy**, not the bag seed, so the **record** (`game_setup_draws`, migration `00013`) — not a seed — is the only account of the outcome, kept for future **tournaments** (designed as a discrete per-tile "player N draws" step). Friend/AI draws at create; **auto-match draws at *open*** against a synthetic `uuid.Nil` opponent whose draw rows are back-filled on join, so the opener's seat is fixed up front and the existing open-game pre-move is preserved (no reseating, no play-gating). Admin `/_gm/games/:id` gains the recorded draw list + a simple **step-by-step board replay** (`ReplayTimeline`). | owner ad-hoc | **done** |
| SB | Single Telegram bot + per-user variant preferences: the two per-language bots collapse into **one** (drop `accounts.service_language`, `supported_languages`, the `*_EN`/`*_RU` env vars and game-language push routing — the single bot renders in the recipient's `preferred_language`); New Game variant gating moves to a profile **`variant_preferences`** set (default Erudit only, Erudit-first, server-enforced on the caller's auto-match/vs-AI/invitation-create paths, an invited friend may accept any variant); env vars collapse to unsuffixed `TELEGRAM_BOT_TOKEN`/`TELEGRAM_GAME_CHANNEL_ID`/`VITE_TELEGRAM_LINK`/`VITE_TELEGRAM_GAME_CHANNEL_NAME` and `GATEWAY_DEFAULT_SUPPORTED_LANGUAGES` is removed; wire drops `service_language`/`supported_languages` (Session, ValidateInitDataResponse) + the push `language` routing field and adds `variant_preferences` to Profile/UpdateProfile. | owner ad-hoc | **done** |
| DV | Dictionary version hygiene: CI + image/compose seed track the current release (`v1.2.1`); a **seed-drift guard** records the flat dir's seed in an authoritative `.seed_version` marker so a bumped build seed on a live volume is ignored (it can't relabel live bytes — which would mis-serve the dictionary + void games pinned to the prior label); `DICT_VERSION` is the fresh-volume seed only, a live contour migrates through the admin console | owner ad-hoc | **done** |
| TX | Telegram egress off the main host: split the connector into a home **validator** (Mini App / Login-Widget HMAC, no VPN, no Bot API — so game login no longer depends on Telegram being reachable) and a remote **bot** (Bot API long-poll + `sendMessage`) that holds **no inbound port** and dials the gateway over a reverse **mTLS bot-link** (`pkg/proto/botlink/v1`); the gateway funnels out-of-app push (fire-and-forget, at-most-once) and the backend admin broadcasts (a relay that awaits the bot's ack) down the link. The bot is Telegram-rate-limited; **one bot now**, with seams (a bot registry + `owns_updates` + command ids) for N later; **no webhook** (rejected: one URL per token, adds inbound + a static address). The **unified test contour** runs the split (the bot keeps its VPN sidecar and dials the gateway by its internal name; certs from `deploy/gen-certs.sh`). The **prod** wiring — the bot on a separate host (no VPN), the gateway bot-link port published, `PROD_` certs with scheduled rotation, an SSH deploy of both hosts together — is the **deferred final stage** (Stage 18). | owner ad-hoc | **done** (code + test contour; prod wiring Stage 18) |
| TX | Telegram egress off the main host: split the connector into a home **validator** (Mini App / Login-Widget HMAC, no VPN, no Bot API — so game login no longer depends on Telegram being reachable) and a remote **bot** (Bot API long-poll + `sendMessage`) that holds **no inbound port** and dials the gateway over a reverse **mTLS bot-link** (`pkg/proto/botlink/v1`); the gateway funnels out-of-app push (fire-and-forget, at-most-once) and the backend admin broadcasts (a relay that awaits the bot's ack) down the link. The bot is Telegram-rate-limited; **one bot now**, with seams (a bot registry + `owns_updates` + command ids) for N later; **no webhook** (rejected: one URL per token, adds inbound + a static address). The **unified test contour** runs the split (the bot keeps its VPN sidecar and dials the gateway by its internal name; certs from `deploy/gen-certs.sh`). The **prod** wiring — the bot on a separate host (no VPN), the gateway bot-link port published, `PROD_` certs, an SSH deploy of both hosts together — is **built in Stage 18** (the two-host registry rollout; first cutover pending the `erudit-game.ru` DNS). | owner ad-hoc | **done** (code + test contour; prod wiring built — Stage 18) |
| AG | Anti-abuse IP ban + honeypot/honeytoken (prod-only): a fail2ban-style in-memory `ratelimit.Banlist` keyed by client IP, fed by sustained rate-limiter rejections (the IP-keyed public/email/admin classes — the user class stays the soft-flag's concern), a **honeypot** decoy path (the contour caddy tags `/.env`, `/.git`, `/wp-*`, … with `X-Scrabble-Honeypot` and routes them to the gateway), and a **honeytoken** (`GATEWAY_HONEYTOKEN`, a planted bearer). The `abuseGuard` edge middleware refuses a banned IP with **429** before any work — closing the R3 gap that the static SPA/landing was outside the token bucket. Off by default — it keys by the real client IP the shared-NAT test contour does not expose (detection still logs there); enabled in prod via `GATEWAY_ABUSE_BAN_ENABLED`. Operators see + lift bans on the console **Throttled** page; the gateway syncs its active set to the backend (`/api/v1/internal/bans/sync`, `internal/banview`) every 30 s and applies operator unbans. | owner ad-hoc | **done** (code + test contour; ban on in prod via Stage 18 — machinery built, cutover pending DNS) |
| CM | Channel-chat moderation + promo bot: a second standalone bot in the bot container answers `/start` with a localized message + a **URL** button into the **main** bot's Mini App (`?startapp`; a `web_app` button would sign initData with the promo token, which the main validator rejects). The **main** bot gates write access in a channel's linked discussion chat. The chat **allows sending by default** and the bot only restricts (Telegram intersects the chat default with the per-user permission, so a per-user grant cannot exceed a deny-by-default group): it **mutes** a member who is not registered or is admin-suspended or holding a new **`chat_muted`** role, and **un-mutes** an eligible one it had muted, for a member currently in the chat (a `getChatMember` guard, since bots cannot list members). Eligibility = `registered AND NOT suspended AND NOT chat_muted` (the game suspension dominates), resolved once in the backend and reached two ways: the bot's `ResolveChatEligibility` on a `chat_member` event over the existing mTLS bot-link, and a backend `chat_access_changed` event → gateway → `ChatGate` command (emitted on block/unblock, a `chat_muted` change, a first registration, or a temporary-block expiry via a sweeper; idempotent). No schema change — `chat_muted` reuses `account_roles`. | owner ad-hoc | **done** |
| → | Stage 18 — prod contour deploy | — | see [`PLAN.md`](PLAN.md) |
## Key findings (these reshaped the raw list — read before starting a phase)
@@ -83,6 +85,19 @@ the edge before prod. Each phase maps back to the owner's raw pre-release TODO l
- **Rate-abuse (TODO 8):** metric + Grafana + admin view **plus a conservative auto-flag**
a *soft, reversible* "suspected high-rate" marker for operator review, tunable threshold,
**no auto-ban**.
- **Anti-abuse IP ban (AG, owner ad-hoc):** a honeypot was considered and rejected as a *DDoS*
defence — it detects/deceives but does not shed volumetric load, cannot cover the real
endpoints, and a tarpit backfires under flood; volumetric L3/L4 is an upstream/CDN concern,
out of scope. The effective layer is a **temporary IP ban** (fail2ban-style) that the honeypot
and honeytoken merely *feed*. This does **not** reverse the TODO-8 "no auto-ban": that decision
governs the **account** soft-flag (still never a gate); the IP ban is a separate, IP-keyed,
**prod-only** layer with an **operator unban** in the console. Decisions: banlist lives in the
existing `ratelimit` package (smallest surface); the decoy path list is a **single source of
truth in the caddy** (it tags requests with a header — the gateway keeps no second list);
bans are in-memory + single-instance (like `ratewatch`), auto-expiring, **plus** an admin
console view + manual unban over a bidirectional 30 s sync (operator control = owner's choice).
An active-bans Grafana **gauge** was trimmed (the console view + the `gateway_abuse_banned_total`
counter cover it) to keep the diff focused.
- **Open auto-match (owner ad-hoc):** a quick game **enters a real game at once and waits inside
it** (status `open`, the opponent seat empty); a second human searching the same variant+rule
joins it, or a robot fills it after a **90 s + random 090 s** wait, pushing the in-app
@@ -296,7 +311,7 @@ Then Stage 18.
hammer (99.97 % rejected, p99 2 ms). **Top finding:** ~14 % `transport_error` on `game.state` at 500
players, under CPU saturation (backend/gateway/Postgres each ~1 core) and amplified by the harness's
single shared `http2.Transport`; the harness itself peaked at 86 % of a core on the same host, so the
figures are pessimistic. Full trip report in [`../loadtest/REPORT-R2.md`](../loadtest/REPORT-R2.md);
figures are pessimistic. Full trip report in [`../loadtest/REPORT.md`](../loadtest/REPORT.md);
it feeds R3 (h2c `MaxConcurrentStreams`/timeouts, body-size cap), R6 and R7 (per-player transports,
separate hardware, pool/limit sizing).
- **CI:** `./loadtest/...` added to the path filter + vet/build/test; `go.work.sum` carries the new deps.
@@ -439,7 +454,12 @@ Then Stage 18.
one connection per player it bursts into its 2-core cap (the residual 2.49 % `transport_error`); backend
~0.85 core and postgres ~1.4 cores had headroom; **tempo reached its 1 GiB cap**; the backend pool sat at
its `MaxOpenConns=25` cap (28 backends); docker logs were unbounded (~14 MiB / 30 min on the backend at
info). Full write-up in [`../loadtest/REPORT-R7.md`](../loadtest/REPORT-R7.md).
info). Full write-up in [`../loadtest/REPORT.md`](../loadtest/REPORT.md). *(Superseded in part: a
later pass modelling the `game.evaluate` hot path traced the gateway's CPU appetite to
**gateway→backend connection churn** — the default 2-idle-connection HTTP transport — not proxying
work. Pooling the connections cut peak gateway CPU ~7× (~1.75 → ~0.26 cores at 500 players) and
removed the ephemeral-port-exhaustion cliff behind the residual `transport_error`, so the gateway is
no longer the binding constraint — postgres is. The 3-core gateway cap below is now generous headroom.)*
- **Round-2 tuning (owner-agreed, all in `deploy/docker-compose.yml`, no code change):** gateway **2 → 3
cores + `GOMAXPROCS=3`**; tempo memory **1 → 2 GiB**; backend `MAX_OPEN_CONNS` **25 → 40**; a json-file
**log-rotation** default (10m × 3) applied contour-wide via a YAML anchor (level stays info).
@@ -449,7 +469,7 @@ Then Stage 18.
**burst** run (a single 100 → 500 jump) pegged the gateway at 3 cores (≈296 % sustained, 9.27 % error),
confirming it is **connection-CPU-bound** — a true arrival spike is a **horizontal-scaling** lever, not
more cores per node (recorded in the prod-sizing recommendation).
- **No schema change → no contour DB wipe.** Bake-back: `loadtest/REPORT-R7.md` (new), `loadtest/README.md`,
- **No schema change → no contour DB wipe.** Bake-back: `loadtest/REPORT.md`, `loadtest/README.md`,
`docs/TESTING.md`, the telemetry/observability section of `docs/ARCHITECTURE.md`, the repo-layout line in `CLAUDE.md`.
- **UI — Tab-bar navigation redesign** (owner ad-hoc, not on the raw TODO list): drop the hamburger
@@ -565,3 +585,48 @@ Then Stage 18.
(`game_limit_test.go`: count rule + HTTP gate 409 + accept bypass), server unit (error mapping), gateway
transcode round-trip, UI codec + lobbycache unit, e2e (`gamelimit.spec.ts`). Bake-back: `docs/FUNCTIONAL.md`
(+`_ru`), `docs/ARCHITECTURE.md` §8, `docs/UI_DESIGN.md`, `backend/README.md`.
- **CM — Channel-chat moderation + promo bot** (owner ad-hoc, not on the raw TODO list):
- **Locked decisions (interview):** the promo bot is a **goroutine in `cmd/bot`** (its own token, no
bot-link); the moderated chat's default-no-send is configured by a **human** in the group settings (the
bot only grants, never `setChatPermissions`); a non-eligible joiner is **left muted silently**; a
temporary-suspension expiry is handled by a **backend sweeper** that emits the re-evaluate event; and a new
**`chat_muted` role** is a chat-only mute with the **game suspension dominating**
(`eligible = registered AND NOT suspended AND NOT chat_muted`).
- **Bot API reality (verified against the docs):** a cross-bot Mini App launch must be a **URL button** to the
main bot's `t.me/<bot>?startapp` link — a `web_app` button signs initData with the *sending* bot's token,
which the main validator rejects — so the promo button reuses the UI's `VITE_TELEGRAM_LINK`. `chat_member`
updates arrive **only** when the bot is a chat **admin** with the "Ban users" right (the client label for the
Bot API `can_restrict_members`) and `chat_member` is in `allowed_updates`; bots cannot list members but can
`getChatMember` a single user, which is the membership guard on the block/unblock path.
- **Wire:** `pkg/proto/botlink/v1` gains a `ChatGateCommand` in the `Command` oneof and a unary
`ResolveChatEligibility`; the backend gains `notify.KindChatAccessChanged` (no payload, infra-only — never an
out-of-app message) and an internal `POST /api/v1/internal/chat-access` resolver; the gateway resolves the
join (by external_id) and the event (by user_id) through it and pushes the chat-gate command fire-and-forget
(at-most-once, recovered by the next moderation action or a re-join).
- **No schema change → no contour DB wipe:** `chat_muted` is a new `account.KnownRoles` entry (the
`account_roles` table is data-driven). The suspension-expiry sweeper is a new `account.SuspensionSweeper`
(a 1-minute window, idempotent) started in `cmd/backend`, alongside the guest reaper.
- **Deploy:** new `TEST_`/`PROD_` `TELEGRAM_PROMO_BOT_TOKEN` (secret), `TELEGRAM_BOT_USERNAME` and
`TELEGRAM_CHAT_ID` (variables); the promo link reuses the existing `*_VITE_TELEGRAM_LINK` variable as
`TELEGRAM_BOT_LINK`. The bot must be promoted to admin in the real discussion group, and the group default
set to no-send, as part of the Stage 18 prod cutover (the test contour exercises the code path).
- **Bake-back:** `docs/ARCHITECTURE.md`, `docs/FUNCTIONAL.md` (+`_ru`), `platform/telegram/README.md`,
`backend/README.md`, Go Doc comments. Tests: backend resolver truth table + publish on block/unblock/role +
the sweeper window (unit + integration); gateway hub `ResolveChatEligibility` + the chat-gate command; bot
`chat_member` grant + `ApplyChatGate` getChatMember-guard; promo `/start` localization + URL button; config
parsing.
- **Post-contour-test fixes (same PR):** a live test drove three corrections. (1) **Strategy
inversion (the key one)** — the original "group default no-send, bot grants the eligible" cannot
work: Telegram intersects the chat default with each user's permission, so a per-user grant never
exceeds a deny-by-default group (the bot set `can_send=true` yet the user still could not write).
The group now **allows sending by default** and the bot only **restricts** — it mutes an ineligible
member (unregistered / admin-suspended / `chat_muted`) and un-mutes an eligible one it had muted,
acting only when the current state differs (idempotent; the bot's own change is skipped by matching
the actor id to the bot). A present member in a default-allow group can appear as `restricted` with
`is_member`, so the gate reads both. (2) **Join-before-register** — a user who joins before
registering is covered by no `chat_member` event, so `ProvisionTelegram` now reports first contact
and the Telegram auth handler emits `chat_access_changed` on it. (3) **Observability** — a startup
self-check logs whether the bot is an admin-with-restrict in the chat (it caught a misconfigured
`TELEGRAM_CHAT_ID` set to a channel id, not the discussion-group id); the per-event trace is at
Debug, the actual mute/unmute and warnings at Info.
+3 -1
View File
@@ -33,7 +33,9 @@ COPY backend ./backend
# Reduce the workspace to what the backend needs: backend + pkg. loadtest and the
# gateway replace it requires are not in this context, so drop both.
RUN go work edit -dropuse=./gateway -dropuse=./platform/telegram -dropuse=./loadtest -dropreplace=scrabble/gateway@v0.0.0
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/backend ./backend/cmd/backend
# VERSION (the deploy passes the git tag) is stamped into the binary via the linker.
ARG VERSION=dev
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags "-X scrabble/pkg/version.Version=${VERSION}" -o /out/backend ./backend/cmd/backend
# --- runtime -----------------------------------------------------------------
FROM gcr.io/distroless/static-debian12:nonroot
+15
View File
@@ -106,6 +106,13 @@ Telegram `external_id`, language and `notifications_in_app_only` flag) lets the
route out-of-app push to the Telegram bot over the gateway bot-link; the Telegram login
seeds a new account's language and display name from the launch fields, and the
`accounts.notifications_in_app_only` flag (default true).
The gateway-only `POST /api/v1/internal/chat-access` resolves a Telegram identity (the
bot's join-time query) or an account id (a `chat_access_changed` event) to its
**moderated-chat write eligibility**`registered AND NOT suspended AND NOT chat_muted`.
That event is emitted on an admin block/unblock, a `chat_muted` role grant/revoke, or — via
the `account.SuspensionSweeper` started in `cmd/backend` — a temporary block lapsing;
`chat_muted` is an `account.KnownRoles` entry, a chat-only mute distinct from the game
suspension (which dominates it).
`accounts.is_guest` marks an ephemeral guest — a durable row
with no identity, excluded from statistics. The server-rendered
**admin console** at `/_gm` (`internal/adminconsole` + `internal/server/handlers_admin_console.go`;
@@ -148,6 +155,13 @@ rejected calls within `BACKEND_HIGHRATE_FLAG_WINDOW` gets the soft, reversible
`accounts.flagged_high_rate_at` marker (set-once; a badge in the user list and a
**Clear** action on the user card; never an automatic ban).
The gateway also syncs its active IP bans (prod-only — see ARCHITECTURE §11) to
`POST /api/v1/internal/bans/sync`; `internal/banview` mirrors them for the console's
**Throttled** page (an **Active IP bans** panel with an **Unban** action) and returns
the operator's pending unbans in the response, which the gateway applies on its next
sync. Like `ratewatch` it is in-memory and resets on restart — the enforced ban lives
in the gateway, not here.
## Package layout
```
@@ -173,6 +187,7 @@ internal/adminconsole/ # server-rendered admin console (Go templates + embedded
internal/ads/ # advertising banner: campaigns + bilingual messages + display timings, weighted-rotation feed (ActiveSet)
internal/connector/ # backend gRPC client to the gateway bot-link relay (operator broadcasts)
internal/ratewatch/ # gateway rate-limit reports: episode window for the console + the high-rate auto-flag
internal/banview/ # gateway active-ban mirror: the console's Active IP bans panel + the operator unban backchannel
```
## Configuration (environment)
+16
View File
@@ -15,11 +15,13 @@ import (
"syscall"
"time"
"github.com/google/uuid"
"go.uber.org/zap"
"scrabble/backend/internal/account"
"scrabble/backend/internal/accountmerge"
"scrabble/backend/internal/ads"
"scrabble/backend/internal/banview"
"scrabble/backend/internal/config"
"scrabble/backend/internal/connector"
"scrabble/backend/internal/engine"
@@ -159,6 +161,15 @@ func run(ctx context.Context, cfg config.Config, logger *zap.Logger) error {
zap.Duration("interval", cfg.GuestReapInterval),
zap.Duration("retention", cfg.GuestRetention))
// Re-evaluate moderated-chat write access when a temporary block self-expires:
// no operator action fires then, so the sweeper emits the chat-access-changed
// event for lapsed blocks and the gateway re-pushes the chat-gate command.
chatSweeper := account.NewSuspensionSweeper(accounts, func(id uuid.UUID) {
hub.Publish(notify.ChatAccessChanged(id))
}, logger)
go chatSweeper.Run(ctx)
logger.Info("suspension expiry sweeper started", zap.Duration("interval", chatSweeper.Interval()))
// Lobby & social domains. Their REST and stream surface lives in the gateway,
// so they are handed to the server (like the route groups) for the handlers.
mailer := newMailer(cfg.SMTP, logger)
@@ -211,6 +222,10 @@ func run(ctx context.Context, cfg config.Config, logger *zap.Logger) error {
zap.Int("flag_threshold", cfg.RateWatch.FlagThreshold),
zap.Duration("flag_window", cfg.RateWatch.FlagWindow))
// Ban observability: mirror the gateway's active IP bans for the admin console's
// active-bans panel and collect operator unban requests.
banView := banview.New()
// Advertising-banner domain: campaign rotation feeding the profile.get banner
// block and the banner admin console section.
adsSvc := ads.NewService(ads.NewStore(db))
@@ -233,6 +248,7 @@ func run(ctx context.Context, cfg config.Config, logger *zap.Logger) error {
DictDir: cfg.Game.DictDir,
Connector: conn,
RateWatch: rateWatch,
BanView: banView,
Ads: adsSvc,
Notifier: hub,
})
+28 -8
View File
@@ -151,14 +151,26 @@ func (s *Store) ProvisionRobot(ctx context.Context, externalID, displayName stri
return modelToAccount(row), nil
}
// ProvisionTelegram provisions (or finds) the account bound to a Telegram
// identity. On first contact only, it seeds the new account's preferred language
// from the Telegram client languageCode (when it maps to a supported language) and
// its display name sanitized from firstName (falling back to username, then to a
// generated placeholder when neither yields any letters); an already-existing
// account is returned unchanged, so a later profile edit is never overwritten.
func (s *Store) ProvisionTelegram(ctx context.Context, externalID, languageCode, username, firstName string) (Account, error) {
return s.provision(ctx, KindTelegram, externalID, telegramSeed(languageCode, username, firstName))
// ProvisionTelegram provisions (or finds) the account bound to a Telegram identity,
// reporting whether this call created it (first contact). On first contact only, it
// seeds the new account's preferred language from the Telegram client languageCode
// (when it maps to a supported language) and its display name sanitized from firstName
// (falling back to username, then to a generated placeholder when neither yields any
// letters); an already-existing account is returned unchanged, so a later profile edit
// is never overwritten. The created flag lets the auth handler re-evaluate moderated-
// chat write access on first registration — the path of a user who joined the chat
// before registering, whom no chat_member event covers.
func (s *Store) ProvisionTelegram(ctx context.Context, externalID, languageCode, username, firstName string) (Account, bool, error) {
// Pre-check whether the identity already exists so the caller can act on first
// contact. A race with a concurrent create only over- or under-reports created for
// that one call, which the idempotent chat-access re-evaluation tolerates.
_, err := s.findByIdentity(ctx, KindTelegram, externalID)
created := errors.Is(err, ErrNotFound)
if err != nil && !created {
return Account{}, false, err
}
acc, err := s.provision(ctx, KindTelegram, externalID, telegramSeed(languageCode, username, firstName))
return acc, created, err
}
// provision finds the account for (kind, externalID) or creates it with seed,
@@ -303,6 +315,14 @@ func (s *Store) CountAccounts(ctx context.Context) (int, error) {
return int(dest.Count), nil
}
// AccountByIdentity returns the account bound to (kind, externalID), or ErrNotFound
// when none exists. Unlike ProvisionByIdentity it never creates one: the chat-access
// resolver uses it to tell a registered Telegram user (eligible to be granted chat
// write access) from an unregistered one (left muted).
func (s *Store) AccountByIdentity(ctx context.Context, kind, externalID string) (Account, error) {
return s.findByIdentity(ctx, kind, externalID)
}
// findByIdentity joins identities to accounts and returns the matching account,
// or ErrNotFound.
func (s *Store) findByIdentity(ctx context.Context, kind, externalID string) (Account, error) {
+9 -1
View File
@@ -24,11 +24,19 @@ const (
// unconditionally, overriding the usual eligibility (a free account with an
// empty hint wallet otherwise sees it). See internal/ads.
RoleNoBanner = "no_banner"
// RoleChatMuted forbids the account from writing in the moderated Telegram
// discussion chat, without otherwise restricting the game (the chat-only
// counterpart to a full account suspension). It is one input to the chat-access
// gate; an active admin suspension mutes the player regardless, so this role only
// matters for an account that is not suspended. Granting or revoking it re-pushes
// the chat-gate command for a member currently in the chat.
RoleChatMuted = "chat_muted"
)
// KnownRoles is the set of roles the console may grant or revoke; an operator
// cannot assign an unrecognised role.
var KnownRoles = []string{RoleFeedbackBanned, RoleNoBanner}
var KnownRoles = []string{RoleFeedbackBanned, RoleNoBanner, RoleChatMuted}
// IsKnownRole reports whether role is a recognised account role.
func IsKnownRole(role string) bool {
+25
View File
@@ -161,6 +161,31 @@ func (s *Store) queryCurrentSuspension(ctx context.Context, accountID uuid.UUID,
return modelToSuspension(row), true, nil
}
// SuspensionsExpiredBetween returns the distinct account ids whose temporary block lapsed in the
// half-open window (since, until]: a non-lifted suspension with a blocked_until in that range. The
// chat-access sweeper uses it to re-evaluate chat write access when a temporary block self-expires,
// since no operator action fires then. An account that still has another active block may be
// included; the eligibility resolver returns the true state, so emitting for it is harmless.
func (s *Store) SuspensionsExpiredBetween(ctx context.Context, since, until time.Time) ([]uuid.UUID, error) {
rows, err := s.db.QueryContext(ctx,
`SELECT DISTINCT account_id FROM backend.account_suspensions
WHERE lifted_at IS NULL AND blocked_until > $1 AND blocked_until <= $2`,
since.UTC(), until.UTC())
if err != nil {
return nil, fmt.Errorf("account: suspensions expired between: %w", err)
}
defer rows.Close()
var out []uuid.UUID
for rows.Next() {
var id uuid.UUID
if err := rows.Scan(&id); err != nil {
return nil, fmt.Errorf("account: scan expired suspension: %w", err)
}
out = append(out, id)
}
return out, rows.Err()
}
// invalidateSuspension drops the account's cached block so the next CurrentSuspension re-reads it.
// Called after Suspend and LiftSuspension.
func (s *Store) invalidateSuspension(accountID uuid.UUID) {
@@ -0,0 +1,84 @@
package account
import (
"context"
"time"
"github.com/google/uuid"
"go.uber.org/zap"
)
// suspensionSweepInterval is how often the sweeper re-checks for temporary blocks
// that lapsed. A minute is well under the coarsest block grain (operators pick day
// presets) while keeping the query trivial.
const suspensionSweepInterval = time.Minute
// suspensionExpiryQuerier is the slice of the account store the sweeper depends on:
// the accounts whose temporary block lapsed in a window. *Store satisfies it; a fake
// drives the sweeper's unit tests.
type suspensionExpiryQuerier interface {
SuspensionsExpiredBetween(ctx context.Context, since, until time.Time) ([]uuid.UUID, error)
}
// SuspensionSweeper re-evaluates chat write access when a temporary block self-
// expires. No operator action fires on expiry — the suspension gate just re-reads
// the wall clock — so without this a temporarily blocked player would stay muted in
// the moderated discussion chat after their block lapsed. Each tick it finds blocks
// that expired since the previous tick and calls onExpire for the affected accounts;
// onExpire is wired to publish the chat-access-changed event, after which the gateway
// re-resolves the true eligibility. A liberal call (an account that still has another
// active block) is therefore harmless. The window is in-memory, so a block that
// expires while the process is down is not re-granted until the next operator action
// or the player rejoins — an accepted best-effort gap.
type SuspensionSweeper struct {
store suspensionExpiryQuerier
onExpire func(accountID uuid.UUID)
log *zap.Logger
// since is the upper bound of the previous swept window; the next sweep covers
// (since, now]. It advances only on a successful query, so a failed tick retries
// the same window rather than dropping expiries.
since time.Time
}
// NewSuspensionSweeper builds the sweeper over the account store, the per-account
// expiry callback (publishing the chat-access-changed event) and a logger. The first
// window opens at construction time, so blocks that lapsed earlier are not re-emitted.
func NewSuspensionSweeper(store *Store, onExpire func(accountID uuid.UUID), log *zap.Logger) *SuspensionSweeper {
if log == nil {
log = zap.NewNop()
}
return &SuspensionSweeper{store: store, onExpire: onExpire, log: log, since: time.Now().UTC()}
}
// Interval reports the sweep cadence, for the startup log line.
func (w *SuspensionSweeper) Interval() time.Duration { return suspensionSweepInterval }
// Run sweeps every Interval until ctx is cancelled.
func (w *SuspensionSweeper) Run(ctx context.Context) {
ticker := time.NewTicker(suspensionSweepInterval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
w.sweep(ctx)
}
}
}
// sweep emits a chat-access-changed signal for every account whose temporary block
// lapsed in (since, now], then advances the window. On a query error it keeps the
// window so the next tick retries it.
func (w *SuspensionSweeper) sweep(ctx context.Context) {
now := time.Now().UTC()
ids, err := w.store.SuspensionsExpiredBetween(ctx, w.since, now)
if err != nil {
w.log.Warn("suspension expiry sweep failed", zap.Error(err))
return
}
w.since = now
for _, id := range ids {
w.onExpire(id)
}
}
@@ -0,0 +1,80 @@
package account
import (
"context"
"errors"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/zap"
)
// fakeExpiryQuerier records the `since` bound of each call and replays a scripted
// result/error per call, so the sweeper's window and dispatch logic is testable
// without a database.
type fakeExpiryQuerier struct {
results [][]uuid.UUID
errs []error
sinces []time.Time
idx int
}
func (f *fakeExpiryQuerier) SuspensionsExpiredBetween(_ context.Context, since, _ time.Time) ([]uuid.UUID, error) {
f.sinces = append(f.sinces, since)
i := f.idx
f.idx++
if i < len(f.errs) && f.errs[i] != nil {
return nil, f.errs[i]
}
if i < len(f.results) {
return f.results[i], nil
}
return nil, nil
}
func newSweeper(store suspensionExpiryQuerier, onExpire func(uuid.UUID)) *SuspensionSweeper {
return &SuspensionSweeper{
store: store,
onExpire: onExpire,
log: zap.NewNop(),
since: time.Now().Add(-time.Minute).UTC(),
}
}
func TestSuspensionSweeperDispatchesAndAdvances(t *testing.T) {
id1, id2 := uuid.New(), uuid.New()
fake := &fakeExpiryQuerier{results: [][]uuid.UUID{{id1, id2}, nil}}
var got []uuid.UUID
w := newSweeper(fake, func(id uuid.UUID) { got = append(got, id) })
first := w.since
w.sweep(context.Background())
assert.Equal(t, []uuid.UUID{id1, id2}, got, "every expired account is dispatched")
assert.True(t, w.since.After(first), "the window advances on success")
// A second sweep opens the next window at the previous upper bound.
prev := w.since
w.sweep(context.Background())
require.Len(t, fake.sinces, 2)
assert.True(t, fake.sinces[1].After(fake.sinces[0]), "consecutive windows are contiguous and forward")
assert.True(t, fake.sinces[1].Equal(prev), "the next window starts at the previous upper bound")
}
func TestSuspensionSweeperKeepsWindowOnError(t *testing.T) {
fake := &fakeExpiryQuerier{errs: []error{errors.New("db down")}}
w := newSweeper(fake, func(uuid.UUID) { t.Fatal("onExpire must not run when the query fails") })
before := w.since
w.sweep(context.Background())
assert.True(t, w.since.Equal(before), "the window is retained on error so the next tick retries it")
}
func TestNewSuspensionSweeperDefaults(t *testing.T) {
w := NewSuspensionSweeper(nil, func(uuid.UUID) {}, nil)
assert.Equal(t, time.Minute, w.Interval())
assert.NotNil(t, w.log, "a nil logger is tolerated")
assert.WithinDuration(t, time.Now().UTC(), w.since, time.Second, "the first window opens at construction time")
}
@@ -5,6 +5,26 @@
list is in-memory and resets on a backend restart. An account sustaining
{{.FlagThreshold}}+ rejected calls within {{.FlagWindow}} is soft-flagged for review
below — never banned automatically; clear the flag on the user card.</p>
<section class="panel"><h2>Active IP bans</h2>
<p class="note">Temporary IP bans the gateway is currently enforcing (in-memory, prod-only;
reset on a gateway restart). Unban applies on the gateway's next sync.</p>
<table class="list">
<thead><tr><th>IP</th><th>Reason</th><th>Since</th><th>Expires</th><th></th></tr></thead>
<tbody>
{{range .Bans}}
<tr>
<td><code>{{.IP}}</code></td>
<td>{{.Reason}}</td>
<td>{{.Since}}</td>
<td>{{.Expires}}</td>
<td><form class="form" method="post" action="/_gm/bans/unban"><input type="hidden" name="ip" value="{{.IP}}"><button type="submit">Unban</button></form></td>
</tr>
{{else}}
<tr><td colspan="5"><span class="note">no active bans</span></td></tr>
{{end}}
</tbody>
</table>
</section>
<section class="panel"><h2>Recent episodes</h2>
<table class="list">
<thead><tr><th>Class</th><th>Key</th><th class="num">Rejected</th><th>First seen</th><th>Last seen</th></tr></thead>
+14 -4
View File
@@ -389,17 +389,27 @@ type BroadcastView struct {
ConnectorEnabled bool
}
// ThrottledView is the rate-limit observability page: the recent gateway-reported
// throttle episodes (in-memory, reset on restart) and the accounts currently
// carrying the high-rate flag. FlagThreshold and FlagWindow caption the active
// auto-flag tuning.
// ThrottledView is the rate-limit observability page: the temporary IP bans the
// gateway is currently enforcing, the recent gateway-reported throttle episodes
// (in-memory, reset on restart) and the accounts currently carrying the high-rate
// flag. FlagThreshold and FlagWindow caption the active auto-flag tuning.
type ThrottledView struct {
Bans []BanRow
Episodes []ThrottleEpisodeRow
Flagged []FlaggedAccountRow
FlagThreshold int
FlagWindow string
}
// BanRow is one temporary IP ban the gateway is enforcing, with its reason and its
// since/expiry timestamps; the row carries an unban action.
type BanRow struct {
IP string
Reason string
Since string
Expires string
}
// ThrottleEpisodeRow is one recently throttled limiter key. UserID links to the
// user card and is set only for the user class (the other classes key by IP).
type ThrottleEpisodeRow struct {
+92
View File
@@ -0,0 +1,92 @@
// Package banview mirrors the gateway's active IP bans for the admin console and
// collects operator unban requests for the gateway to apply. Like ratewatch it is
// in-memory, single-instance and resets on a backend restart by design — the
// gateway re-reports its active set on the next sync, and the durable effect (the
// ban itself) lives in the gateway, not here.
package banview
import (
"sort"
"sync"
"time"
)
// Ban is one active IP ban as reported by the gateway.
type Ban struct {
IP string
Reason string
Since time.Time
Expires time.Time
}
// View holds the last-reported active bans and the operator's pending unbans.
type View struct {
now func() time.Time
mu sync.Mutex
bans map[string]Ban // last reported active set, keyed by IP
unban map[string]struct{} // IPs an operator marked for unban
}
// New constructs an empty View.
func New() *View {
return &View{now: time.Now, bans: make(map[string]Ban), unban: make(map[string]struct{})}
}
// Ingest replaces the mirrored active set with the gateway's latest report,
// skipping entries with an empty IP or one that has already expired.
func (v *View) Ingest(active []Ban) {
now := v.now()
v.mu.Lock()
defer v.mu.Unlock()
v.bans = make(map[string]Ban, len(active))
for _, b := range active {
if b.IP == "" || !now.Before(b.Expires) {
continue
}
v.bans[b.IP] = b
}
}
// Recent returns the mirrored active bans, most recently banned first.
func (v *View) Recent() []Ban {
now := v.now()
v.mu.Lock()
defer v.mu.Unlock()
out := make([]Ban, 0, len(v.bans))
for _, b := range v.bans {
if now.Before(b.Expires) {
out = append(out, b)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Since.After(out[j].Since) })
return out
}
// RequestUnban records an operator request to lift the ban on ip; the gateway
// applies it on its next sync (so the console reflects it within the sync
// interval). An empty ip is ignored.
func (v *View) RequestUnban(ip string) {
if ip == "" {
return
}
v.mu.Lock()
defer v.mu.Unlock()
v.unban[ip] = struct{}{}
}
// DrainUnbans returns and clears the IPs operators have marked for unban since the
// previous drain. It returns nil when there are none.
func (v *View) DrainUnbans() []string {
v.mu.Lock()
defer v.mu.Unlock()
if len(v.unban) == 0 {
return nil
}
out := make([]string, 0, len(v.unban))
for ip := range v.unban {
out = append(out, ip)
}
clear(v.unban)
return out
}
+64
View File
@@ -0,0 +1,64 @@
package banview
import (
"testing"
"time"
)
func viewAt(clk *time.Time) *View {
v := New()
v.now = func() time.Time { return *clk }
return v
}
func TestIngestRecentDropsExpired(t *testing.T) {
clk := time.Date(2026, 6, 21, 12, 0, 0, 0, time.UTC)
v := viewAt(&clk)
v.Ingest([]Ban{
{IP: "1.1.1.1", Reason: "tripwire", Since: clk, Expires: clk.Add(time.Hour)},
{IP: "2.2.2.2", Reason: "rejections", Since: clk.Add(-2 * time.Hour), Expires: clk.Add(-time.Hour)}, // expired
{IP: "", Reason: "x", Since: clk, Expires: clk.Add(time.Hour)}, // empty IP
})
got := v.Recent()
if len(got) != 1 || got[0].IP != "1.1.1.1" || got[0].Reason != "tripwire" {
t.Fatalf("Recent = %+v, want one live ban for 1.1.1.1", got)
}
}
func TestIngestReplaces(t *testing.T) {
clk := time.Date(2026, 6, 21, 12, 0, 0, 0, time.UTC)
v := viewAt(&clk)
v.Ingest([]Ban{{IP: "1.1.1.1", Since: clk, Expires: clk.Add(time.Hour)}})
v.Ingest([]Ban{{IP: "2.2.2.2", Since: clk, Expires: clk.Add(time.Hour)}})
got := v.Recent()
if len(got) != 1 || got[0].IP != "2.2.2.2" {
t.Fatalf("Recent = %+v, want only the latest report (2.2.2.2)", got)
}
}
func TestRecentOrdersBySince(t *testing.T) {
clk := time.Date(2026, 6, 21, 12, 0, 0, 0, time.UTC)
v := viewAt(&clk)
v.Ingest([]Ban{
{IP: "old", Since: clk.Add(-10 * time.Minute), Expires: clk.Add(time.Hour)},
{IP: "new", Since: clk.Add(-1 * time.Minute), Expires: clk.Add(time.Hour)},
})
got := v.Recent()
if len(got) != 2 || got[0].IP != "new" || got[1].IP != "old" {
t.Fatalf("Recent order = %+v, want most recent first", got)
}
}
func TestUnbanRoundTrip(t *testing.T) {
clk := time.Date(2026, 6, 21, 12, 0, 0, 0, time.UTC)
v := viewAt(&clk)
v.RequestUnban("3.3.3.3")
v.RequestUnban("") // ignored
drained := v.DrainUnbans()
if len(drained) != 1 || drained[0] != "3.3.3.3" {
t.Fatalf("DrainUnbans = %v, want [3.3.3.3]", drained)
}
if again := v.DrainUnbans(); again != nil {
t.Fatalf("second DrainUnbans = %v, want nil (cleared)", again)
}
}
+12 -8
View File
@@ -63,6 +63,7 @@ type gameCache struct {
type cachedGame struct {
game *engine.Game
seats []Seat
variant string
lastAccess time.Time
}
@@ -71,24 +72,27 @@ func newGameCache(ttl time.Duration, now func() time.Time) *gameCache {
return &gameCache{entries: make(map[uuid.UUID]*cachedGame), ttl: ttl, now: now}
}
// get returns the live game for id and refreshes its idle timer, or (nil, false).
func (c *gameCache) get(id uuid.UUID) (*engine.Game, bool) {
// get returns the live game and its immutable seat list for id and refreshes its idle
// timer, or (nil, nil, false). The seats let a read check membership (and label seats)
// without re-loading the game from the store, since seats never change after a game starts.
func (c *gameCache) get(id uuid.UUID) (*engine.Game, []Seat, bool) {
c.mu.Lock()
defer c.mu.Unlock()
e, ok := c.entries[id]
if !ok {
return nil, false
return nil, nil, false
}
e.lastAccess = c.now()
return e.game, true
return e.game, e.seats, true
}
// put stores g as the live game for id. variant labels the entry so the active-
// games gauge can report counts by variant without inspecting engine internals.
func (c *gameCache) put(id uuid.UUID, g *engine.Game, variant string) {
// put stores g as the live game for id together with its seat list. variant labels the
// entry so the active-games gauge can report counts by variant without inspecting engine
// internals; seats are the game's immutable seat standings for the membership fast path.
func (c *gameCache) put(id uuid.UUID, g *engine.Game, variant string, seats []Seat) {
c.mu.Lock()
defer c.mu.Unlock()
c.entries[id] = &cachedGame{game: g, variant: variant, lastAccess: c.now()}
c.entries[id] = &cachedGame{game: g, seats: seats, variant: variant, lastAccess: c.now()}
}
// remove drops id from the cache (used on a finished game and after a failed
+3 -3
View File
@@ -94,8 +94,8 @@ func TestGameCacheEviction(t *testing.T) {
cur := time.Unix(1_700_000_000, 0)
cache := newGameCache(time.Hour, func() time.Time { return cur })
id := uuid.New()
cache.put(id, nil, "scrabble_en")
if _, ok := cache.get(id); !ok {
cache.put(id, nil, "scrabble_en", nil)
if _, _, ok := cache.get(id); !ok {
t.Fatal("game must be resident after put")
}
cur = cur.Add(30 * time.Minute)
@@ -104,7 +104,7 @@ func TestGameCacheEviction(t *testing.T) {
if n := cache.sweep(); n != 1 {
t.Errorf("sweep evicted %d, want 1", n)
}
if _, ok := cache.get(id); ok {
if _, _, ok := cache.get(id); ok {
t.Error("game must be evicted after idle TTL")
}
if cache.size() != 0 {
+27 -18
View File
@@ -287,12 +287,12 @@ func (svc *Service) Create(ctx context.Context, params CreateParams) (Game, erro
if err := svc.store.CreateGame(ctx, ins, seats, seeding.draws); err != nil {
return Game{}, err
}
svc.cache.put(id, g, params.Variant.String())
svc.metrics.recordStarted(ctx, params.Variant, params.VsAI)
created, err := svc.store.GetGame(ctx, id)
if err != nil {
return Game{}, err
}
svc.cache.put(id, g, params.Variant.String(), created.Seats)
// Honest-AI game seated with a robot: if the robot moves first, reply at once
// (the periodic driver is the fallback). No-op for every human-only game.
svc.triggerAI(created)
@@ -890,26 +890,35 @@ func (svc *Service) timeoutGame(ctx context.Context, gameID uuid.UUID, now time.
// EvaluatePlay previews a tentative play for a seated player against the current
// board without committing it: whether it is legal and what it would score.
func (svc *Service) EvaluatePlay(ctx context.Context, gameID, accountID uuid.UUID, tiles []engine.TileRecord) (EvalResult, error) {
pre, err := svc.store.GetGame(ctx, gameID)
if err != nil {
return EvalResult{}, err
}
if _, ok := pre.seatOf(accountID); !ok {
return EvalResult{}, ErrNotAPlayer
}
if pre.Status == StatusFinished {
return EvalResult{}, ErrFinished
}
unlock := svc.locks.lock(gameID)
defer unlock()
g, err := svc.liveGame(ctx, pre)
if err != nil {
return EvalResult{}, err
// Hot path: an active game stays cached — the engine game is mutated in place across
// moves and evicted only when it finishes — so on a hit the cached live game and its
// immutable seat list answer the membership check and the score with no DB read. This
// preview is fired on every tile placement, the hottest gameplay call at scale.
g, seats, ok := svc.cache.get(gameID)
if !ok {
// Cold path: load and validate from the store, then replay into the cache.
pre, err := svc.store.GetGame(ctx, gameID)
if err != nil {
return EvalResult{}, err
}
if pre.Status == StatusFinished {
return EvalResult{}, ErrFinished
}
if g, err = svc.liveGame(ctx, pre); err != nil {
return EvalResult{}, err
}
seats = pre.Seats
}
if !seatedIn(seats, accountID) {
return EvalResult{}, ErrNotAPlayer
}
validateStart := time.Now()
rec, err := g.EvaluatePlay(tiles)
svc.metrics.recordValidate(ctx, pre.Variant, validateStart)
svc.metrics.recordValidate(ctx, g.Variant(), validateStart)
if err != nil {
if errors.Is(err, engine.ErrIllegalPlay) {
return EvalResult{Valid: false}, nil
@@ -1359,7 +1368,7 @@ func (svc *Service) ExportGCG(ctx context.Context, gameID uuid.UUID) (string, er
// liveGame returns the live engine.Game for pre, rebuilding it from the journal
// on a cache miss. Callers must hold the per-game lock.
func (svc *Service) liveGame(ctx context.Context, pre Game) (*engine.Game, error) {
if g, ok := svc.cache.get(pre.ID); ok {
if g, _, ok := svc.cache.get(pre.ID); ok {
return g, nil
}
g, err := svc.replay(ctx, pre)
@@ -1374,7 +1383,7 @@ func (svc *Service) liveGame(ctx context.Context, pre Game) (*engine.Game, error
}
}
if !g.Over() {
svc.cache.put(pre.ID, g, pre.Variant.String())
svc.cache.put(pre.ID, g, pre.Variant.String(), pre.Seats)
}
return g, nil
}
+23 -17
View File
@@ -355,27 +355,33 @@ func (s *Store) ExpiredOpen(ctx context.Context, now time.Time) ([]OpenGame, err
// GetGame loads the games row joined with its seats (ordered by seat), or
// ErrNotFound.
func (s *Store) GetGame(ctx context.Context, id uuid.UUID) (Game, error) {
gstmt := postgres.SELECT(table.Games.AllColumns).
FROM(table.Games).
// One round-trip: the game joined with its seats. A LEFT JOIN keeps a (would-be)
// seatless game returning the game with no seats, exactly as the prior two-query
// version did; ORDER BY seat preserves seat order. The games columns repeat per seat
// row — cheap at 2-4 seats, and one round-trip instead of two, which matters because
// GetGame is the universal "load the game" step on every game operation.
stmt := postgres.SELECT(table.Games.AllColumns, table.GamePlayers.AllColumns).
FROM(table.Games.LEFT_JOIN(table.GamePlayers, table.GamePlayers.GameID.EQ(table.Games.GameID))).
WHERE(table.Games.GameID.EQ(postgres.UUID(id))).
LIMIT(1)
var grow model.Games
if err := gstmt.QueryContext(ctx, s.db, &grow); err != nil {
if errors.Is(err, qrm.ErrNoRows) {
return Game{}, ErrNotFound
}
ORDER_BY(table.GamePlayers.Seat.ASC())
var rows []struct {
model.Games
model.GamePlayers
}
if err := stmt.QueryContext(ctx, s.db, &rows); err != nil {
return Game{}, fmt.Errorf("game: get %s: %w", id, err)
}
sstmt := postgres.SELECT(table.GamePlayers.AllColumns).
FROM(table.GamePlayers).
WHERE(table.GamePlayers.GameID.EQ(postgres.UUID(id))).
ORDER_BY(table.GamePlayers.Seat.ASC())
var srows []model.GamePlayers
if err := sstmt.QueryContext(ctx, s.db, &srows); err != nil {
return Game{}, fmt.Errorf("game: get seats %s: %w", id, err)
if len(rows) == 0 {
return Game{}, ErrNotFound
}
return projectGame(grow, srows)
seats := make([]model.GamePlayers, 0, len(rows))
for i := range rows {
// Skip the phantom all-NULL seat row a LEFT JOIN yields for a seatless game.
if rows[i].GamePlayers.GameID == id {
seats = append(seats, rows[i].GamePlayers)
}
}
return projectGame(rows[0].Games, seats)
}
// GetGameVariant reads just a game's variant — a cheap single-column lookup the edge uses
+12
View File
@@ -184,6 +184,18 @@ func (g Game) seatOf(accountID uuid.UUID) (int, bool) {
return 0, false
}
// seatedIn reports whether accountID holds a seat in seats. It backs the read-side
// membership check against the cached, immutable seat list, so a hot read can skip
// loading the game from the store.
func seatedIn(seats []Seat, accountID uuid.UUID) bool {
for _, s := range seats {
if s.AccountID == accountID {
return true
}
}
return false
}
// MoveResult is the outcome of a committed transition: the decoded move and the
// post-move game, plus the actor's own refilled rack and the bag size after the draw
// (Rack/BagLen), so the mover renders the next state from the response without a
+12 -6
View File
@@ -118,10 +118,13 @@ func TestProvisionTelegramSeedsNewAccountOnly(t *testing.T) {
store := account.NewStore(testDB)
ext := "tg-" + uuid.NewString()
acc, err := store.ProvisionTelegram(ctx, ext, "ru-RU", "thehandle", "Иван")
acc, created, err := store.ProvisionTelegram(ctx, ext, "ru-RU", "thehandle", "Иван")
if err != nil {
t.Fatalf("provision telegram: %v", err)
}
if !created {
t.Error("created = false on first contact, want true")
}
if acc.PreferredLanguage != "ru" {
t.Errorf("PreferredLanguage = %q, want ru", acc.PreferredLanguage)
}
@@ -133,10 +136,13 @@ func TestProvisionTelegramSeedsNewAccountOnly(t *testing.T) {
}
// A later login with different fields returns the same account, unchanged.
again, err := store.ProvisionTelegram(ctx, ext, "en", "other", "Other")
again, created, err := store.ProvisionTelegram(ctx, ext, "en", "other", "Other")
if err != nil {
t.Fatalf("re-provision telegram: %v", err)
}
if created {
t.Error("created = true on a repeat login, want false")
}
if again.ID != acc.ID {
t.Errorf("re-provision id = %s, want %s", again.ID, acc.ID)
}
@@ -150,7 +156,7 @@ func TestProvisionTelegramSeedsNewAccountOnly(t *testing.T) {
// language CHECK.
func TestProvisionTelegramUnknownLanguageDefaults(t *testing.T) {
ctx := context.Background()
acc, err := account.NewStore(testDB).ProvisionTelegram(ctx, "tg-"+uuid.NewString(), "fr", "", "")
acc, _, err := account.NewStore(testDB).ProvisionTelegram(ctx, "tg-"+uuid.NewString(), "fr", "", "")
if err != nil {
t.Fatalf("provision telegram: %v", err)
}
@@ -166,7 +172,7 @@ func TestProvisionTelegramUnknownLanguageDefaults(t *testing.T) {
func TestHighRateFlagRoundTrip(t *testing.T) {
ctx := context.Background()
store := account.NewStore(testDB)
acc, err := store.ProvisionTelegram(ctx, "tg-"+uuid.NewString(), "en", "", "Player")
acc, _, err := store.ProvisionTelegram(ctx, "tg-"+uuid.NewString(), "en", "", "Player")
if err != nil {
t.Fatalf("provision telegram: %v", err)
}
@@ -222,7 +228,7 @@ func TestIdentityExternalID(t *testing.T) {
ctx := context.Background()
store := account.NewStore(testDB)
ext := "tg-" + uuid.NewString()
acc, err := store.ProvisionTelegram(ctx, ext, "en", "", "Tg User")
acc, _, err := store.ProvisionTelegram(ctx, ext, "en", "", "Tg User")
if err != nil {
t.Fatalf("provision telegram: %v", err)
}
@@ -247,7 +253,7 @@ func TestIdentityExternalID(t *testing.T) {
func TestNotificationsInAppOnlyRoundTrip(t *testing.T) {
ctx := context.Background()
store := account.NewStore(testDB)
acc, err := store.ProvisionTelegram(ctx, "tg-"+uuid.NewString(), "en", "", "Player")
acc, _, err := store.ProvisionTelegram(ctx, "tg-"+uuid.NewString(), "en", "", "Player")
if err != nil {
t.Fatalf("provision telegram: %v", err)
}
+1 -1
View File
@@ -222,7 +222,7 @@ func TestConsoleGameDetailRobotSchedule(t *testing.T) {
func TestConsoleThrottledViewAndFlagClear(t *testing.T) {
ctx := context.Background()
accounts := account.NewStore(testDB)
acc, err := accounts.ProvisionTelegram(ctx, "tg-"+uuid.NewString(), "en", "", "Throttled Player")
acc, _, err := accounts.ProvisionTelegram(ctx, "tg-"+uuid.NewString(), "en", "", "Throttled Player")
if err != nil {
t.Fatalf("provision: %v", err)
}
@@ -0,0 +1,315 @@
//go:build integration
package inttest
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
"github.com/google/uuid"
"go.uber.org/zap/zaptest"
"scrabble/backend/internal/account"
"scrabble/backend/internal/notify"
"scrabble/backend/internal/server"
"scrabble/backend/internal/session"
)
// chatAccessBody mirrors the backend's /internal/chat-access JSON for the test.
type chatAccessBody struct {
ExternalID string `json:"external_id"`
Registered bool `json:"registered"`
Eligible bool `json:"eligible"`
}
// chatAccess issues the gateway-internal chat-access query and asserts a 200.
func chatAccess(t *testing.T, srv *server.Server, body string) chatAccessBody {
t.Helper()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodPost, "/api/v1/internal/chat-access", strings.NewReader(body))
req.Header.Set("Content-Type", "application/json")
srv.Handler().ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("chat-access %s = %d: %s", body, rec.Code, rec.Body.String())
}
var b chatAccessBody
if err := json.Unmarshal(rec.Body.Bytes(), &b); err != nil {
t.Fatalf("decode chat-access: %v", err)
}
return b
}
// TestChatAccessResolver drives the gateway-internal eligibility resolver over HTTP:
// the registered/suspended/chat_muted truth table by Telegram identity and by account
// id, the suspension dominating the chat_muted role, an unknown identity reported
// unregistered, and an account with no Telegram identity carrying an empty external_id.
func TestChatAccessResolver(t *testing.T) {
ctx := context.Background()
accounts := account.NewStore(testDB)
srv := server.New(":0", server.Deps{Logger: zaptest.NewLogger(t), DB: testDB, Accounts: accounts})
ext := "tg-" + uuid.NewString()
acc, _, err := accounts.ProvisionTelegram(ctx, ext, "en", "", "Chatter")
if err != nil {
t.Fatalf("provision: %v", err)
}
id := acc.ID
byExt := func() chatAccessBody { return chatAccess(t, srv, `{"external_id":"`+ext+`"}`) }
byUser := func() chatAccessBody { return chatAccess(t, srv, `{"user_id":"`+id.String()+`"}`) }
// A registered, unsuspended, unmuted account is eligible by either address, and the
// account-id query resolves back to its Telegram identity.
if b := byExt(); !b.Registered || !b.Eligible || b.ExternalID != ext {
t.Fatalf("fresh by external_id = %+v, want registered+eligible+ext", b)
}
if b := byUser(); !b.Registered || !b.Eligible || b.ExternalID != ext {
t.Fatalf("fresh by user_id = %+v, want registered+eligible+ext", b)
}
// A suspension mutes; a lift restores.
if _, err := accounts.Suspend(ctx, id, nil, "", "", nil); err != nil {
t.Fatalf("suspend: %v", err)
}
if b := byExt(); !b.Registered || b.Eligible {
t.Fatalf("suspended = %+v, want registered but not eligible", b)
}
if err := accounts.LiftSuspension(ctx, id); err != nil {
t.Fatalf("lift: %v", err)
}
if b := byExt(); !b.Eligible {
t.Fatalf("after lift = %+v, want eligible", b)
}
// The chat_muted role mutes independently; a revoke restores.
if err := accounts.GrantRole(ctx, id, account.RoleChatMuted); err != nil {
t.Fatalf("grant chat_muted: %v", err)
}
if b := byExt(); !b.Registered || b.Eligible {
t.Fatalf("chat_muted = %+v, want registered but not eligible", b)
}
// Suspension dominates: while chat_muted is set, lifting a concurrent suspension
// must not re-grant chat (the role still mutes).
if _, err := accounts.Suspend(ctx, id, nil, "", "", nil); err != nil {
t.Fatalf("suspend over mute: %v", err)
}
if b := byExt(); b.Eligible {
t.Fatalf("suspended+muted = %+v, want not eligible", b)
}
if err := accounts.LiftSuspension(ctx, id); err != nil {
t.Fatalf("lift over mute: %v", err)
}
if b := byExt(); b.Eligible {
t.Fatalf("lifted but still muted = %+v, want not eligible", b)
}
if err := accounts.RevokeRole(ctx, id, account.RoleChatMuted); err != nil {
t.Fatalf("revoke chat_muted: %v", err)
}
if b := byExt(); !b.Eligible {
t.Fatalf("after revoke = %+v, want eligible", b)
}
// An unknown Telegram identity is unregistered (and thus left muted).
if b := chatAccess(t, srv, `{"external_id":"tg-missing-`+uuid.NewString()+`"}`); b.Registered || b.Eligible {
t.Fatalf("unknown identity = %+v, want neither registered nor eligible", b)
}
// An account with no Telegram identity (a guest) carries an empty external_id, so
// the gateway has nothing to gate.
guest := provisionGuest(t)
if b := chatAccess(t, srv, `{"user_id":"`+guest.String()+`"}`); b.ExternalID != "" || b.Registered {
t.Fatalf("guest by user_id = %+v, want empty external_id and not registered", b)
}
// A request naming neither address is a bad request.
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodPost, "/api/v1/internal/chat-access", strings.NewReader(`{}`))
req.Header.Set("Content-Type", "application/json")
srv.Handler().ServeHTTP(rec, req)
if rec.Code != http.StatusBadRequest {
t.Fatalf("empty query = %d, want 400", rec.Code)
}
}
// captureNotifier records every published intent so a test can assert which live
// events a console action emitted.
type captureNotifier struct {
mu sync.Mutex
intents []notify.Intent
}
func (c *captureNotifier) Publish(in ...notify.Intent) {
c.mu.Lock()
defer c.mu.Unlock()
c.intents = append(c.intents, in...)
}
// count returns how many intents of kind addressed to user were captured.
func (c *captureNotifier) count(user uuid.UUID, kind string) int {
c.mu.Lock()
defer c.mu.Unlock()
n := 0
for _, in := range c.intents {
if in.UserID == user && in.Kind == kind {
n++
}
}
return n
}
// TestChatAccessPublishedOnModeration drives the admin console and asserts each
// moderation action that can change chat eligibility — block, unblock, and the
// chat_muted role grant/revoke — emits the chat_access_changed signal the gateway
// turns into a chat-gate command.
func TestChatAccessPublishedOnModeration(t *testing.T) {
notifier := &captureNotifier{}
srv := server.New(":0", server.Deps{
Logger: zaptest.NewLogger(t),
DB: testDB,
Accounts: account.NewStore(testDB),
Games: newGameService(),
Registry: testRegistry,
DictDir: dictDir(),
Notifier: notifier,
})
h := srv.Handler()
id := provisionAccount(t)
base := "http://admin.test/_gm/users/" + id.String()
const origin = "http://admin.test"
steps := []struct {
name, path, body string
want string
}{
{"block", "/block", "duration=permanent", "Blocked"},
{"unblock", "/unblock", "", "Unblocked"},
{"grant chat_muted", "/grant-role", "role=chat_muted", "Role granted"},
{"revoke chat_muted", "/revoke-role", "role=chat_muted", "Role revoked"},
}
for i, s := range steps {
code, body := consoleDo(h, http.MethodPost, base+s.path, s.body, origin)
if code != http.StatusOK || !strings.Contains(body, s.want) {
t.Fatalf("%s = %d, has %q = %v", s.name, code, s.want, strings.Contains(body, s.want))
}
if got := notifier.count(id, notify.KindChatAccessChanged); got != i+1 {
t.Fatalf("after %s: chat_access_changed count = %d, want %d", s.name, got, i+1)
}
}
}
// TestChatAccessPublishedOnFirstRegistration checks that a Telegram first contact
// (the sessions/telegram endpoint creating the account) emits chat_access_changed —
// the re-grant for a user who joined the moderated chat before registering — and that
// a repeat login does not re-emit.
func TestChatAccessPublishedOnFirstRegistration(t *testing.T) {
notifier := &captureNotifier{}
srv := server.New(":0", server.Deps{
Logger: zaptest.NewLogger(t),
DB: testDB,
Accounts: account.NewStore(testDB),
Sessions: session.NewService(session.NewStore(testDB), session.NewCache()),
Notifier: notifier,
})
h := srv.Handler()
ext := "tg-" + uuid.NewString()
post := func() {
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodPost, "/api/v1/internal/sessions/telegram",
strings.NewReader(`{"external_id":"`+ext+`","language_code":"en","first_name":"Reg"}`))
req.Header.Set("Content-Type", "application/json")
h.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("telegram auth = %d: %s", rec.Code, rec.Body.String())
}
}
post()
acc, err := account.NewStore(testDB).AccountByIdentity(context.Background(), account.KindTelegram, ext)
if err != nil {
t.Fatalf("lookup: %v", err)
}
if got := notifier.count(acc.ID, notify.KindChatAccessChanged); got != 1 {
t.Fatalf("first registration: chat_access_changed count = %d, want 1", got)
}
// A repeat login (the account already exists) must not re-emit.
post()
if got := notifier.count(acc.ID, notify.KindChatAccessChanged); got != 1 {
t.Fatalf("repeat login: chat_access_changed count = %d, want still 1", got)
}
}
// TestSuspensionsExpiredBetween checks the sweeper's window query: a non-lifted
// temporary block whose expiry falls in the window is returned, while one outside the
// window, a permanent block, and a lifted block are not.
func TestSuspensionsExpiredBetween(t *testing.T) {
ctx := context.Background()
accounts := account.NewStore(testDB)
// A temporary block whose expiry already lapsed at a known instant.
tempID := provisionAccount(t)
expiry := time.Now().Add(-time.Hour).Truncate(time.Second)
if _, err := accounts.Suspend(ctx, tempID, &expiry, "", "", nil); err != nil {
t.Fatalf("suspend temp: %v", err)
}
contains := func(ids []uuid.UUID, want uuid.UUID) bool {
for _, id := range ids {
if id == want {
return true
}
}
return false
}
// A window straddling the expiry returns the account.
got, err := accounts.SuspensionsExpiredBetween(ctx, expiry.Add(-time.Minute), expiry.Add(time.Minute))
if err != nil {
t.Fatalf("expired between: %v", err)
}
if !contains(got, tempID) {
t.Fatalf("window over expiry missing the lapsed block %s", tempID)
}
// A window entirely after the expiry does not.
got, err = accounts.SuspensionsExpiredBetween(ctx, expiry.Add(time.Minute), expiry.Add(2*time.Minute))
if err != nil {
t.Fatalf("expired between (after): %v", err)
}
if contains(got, tempID) {
t.Fatalf("window after expiry should not return %s", tempID)
}
// A permanent block never appears, even in a wide window.
permID := provisionAccount(t)
if _, err := accounts.Suspend(ctx, permID, nil, "", "", nil); err != nil {
t.Fatalf("suspend perm: %v", err)
}
// A lifted block does not appear either. The block must still be in force when lifted
// (LiftSuspension only lifts in-force blocks), so its expiry is in the future and the
// wide window below still covers it — yet lifted_at excludes it.
liftID := provisionAccount(t)
liftExpiry := time.Now().Add(30 * time.Minute).Truncate(time.Second)
if _, err := accounts.Suspend(ctx, liftID, &liftExpiry, "", "", nil); err != nil {
t.Fatalf("suspend lift: %v", err)
}
if err := accounts.LiftSuspension(ctx, liftID); err != nil {
t.Fatalf("lift: %v", err)
}
wide, err := accounts.SuspensionsExpiredBetween(ctx, time.Now().Add(-2*time.Hour), time.Now().Add(time.Hour))
if err != nil {
t.Fatalf("expired between (wide): %v", err)
}
if contains(wide, permID) {
t.Fatalf("permanent block %s must not be reported as expired", permID)
}
if contains(wide, liftID) {
t.Fatalf("lifted block %s must not be reported as expired", liftID)
}
}
+6
View File
@@ -543,6 +543,12 @@ func TestEvaluatePlayPreview(t *testing.T) {
if bad.Valid {
t.Error("disconnected play must be invalid")
}
// A non-seated account cannot preview: with the game warm in the live cache, the
// membership check runs against the cached seat list (the hot path that skips GetGame).
if _, err := svc.EvaluatePlay(ctx, g.ID, provisionAccount(t), hint.Tiles); !errors.Is(err, game.ErrNotAPlayer) {
t.Errorf("evaluate by a non-player = %v, want ErrNotAPlayer", err)
}
}
// TestConcurrentSubmitSerialized confirms the per-game lock lets only one of two
@@ -38,7 +38,7 @@ func TestSuspensionGate(t *testing.T) {
Accounts: accounts,
})
acc, err := accounts.ProvisionTelegram(ctx, "tg-"+uuid.NewString(), "ru", "", "Blocked")
acc, _, err := accounts.ProvisionTelegram(ctx, "tg-"+uuid.NewString(), "ru", "", "Blocked")
if err != nil {
t.Fatalf("provision: %v", err)
}
+1 -1
View File
@@ -18,7 +18,7 @@ func TestUserListFilter(t *testing.T) {
st := account.NewStore(testDB)
uniq := uuid.NewString()
human, err := st.ProvisionTelegram(ctx, "tg-"+uniq, "en", "", "Zzqxhuman")
human, _, err := st.ProvisionTelegram(ctx, "tg-"+uniq, "en", "", "Zzqxhuman")
if err != nil {
t.Fatalf("provision human: %v", err)
}
+10
View File
@@ -216,6 +216,16 @@ func BannerChanged(userID uuid.UUID) Intent {
return Notification(userID, NotifyBanner)
}
// ChatAccessChanged signals that userID's eligibility to write in the moderated
// Telegram discussion chat may have changed (an admin block/unblock, a chat_muted
// grant/revoke, or a temporary block lapsing). It carries no payload: the gateway
// resolves the user's Telegram identity and current eligibility and pushes the
// resulting chat-gate command to the bot. Unlike the lobby notifications it is an
// infra signal — a distinct top-level kind, never an out-of-app rendered message.
func ChatAccessChanged(userID uuid.UUID) Intent {
return Intent{UserID: userID, Kind: KindChatAccessChanged, EventID: eventID()}
}
// eventID returns a best-effort correlation id for one emitted event.
func eventID() string {
if id, err := uuid.NewV7(); err == nil {
+7
View File
@@ -35,6 +35,13 @@ const (
// KindGameOver announces a finished game to each seated player, driving the
// out-of-app "game over" push.
KindGameOver = "game_over"
// KindChatAccessChanged signals that a player's eligibility to write in the
// moderated Telegram discussion chat may have changed (an admin block or unblock,
// a chat_muted grant or revoke, or a temporary block lapsing). It carries no
// payload and is never fanned out to in-app clients: the gateway consumes it to
// resolve the player's Telegram identity and current eligibility and push the
// resulting chat-gate command to the bot.
KindChatAccessChanged = "chat_access_changed"
)
// Notification sub-kinds carried in a KindNotification event payload; the client
+131
View File
@@ -0,0 +1,131 @@
package server
import (
"context"
"errors"
"net/http"
"github.com/gin-gonic/gin"
"github.com/google/uuid"
"scrabble/backend/internal/account"
"scrabble/backend/internal/notify"
)
// chatAccessRequest is the gateway's chat write-eligibility query, addressed either
// by Telegram identity (ExternalID — the join path, when the bot sees a user enter
// the chat) or by account id (UserID — the change path, resolving an emitted
// chat-access-changed event). Exactly one field is set.
type chatAccessRequest struct {
ExternalID string `json:"external_id"`
UserID string `json:"user_id"`
}
// chatAccessResponse is the resolved eligibility. ExternalID echoes the account's
// Telegram identity (empty when it has none — the gateway then has nothing to gate);
// Registered reports whether the lookup found an account at all; Eligible is the
// final gate the bot applies (registered and neither admin-suspended nor chat-muted).
type chatAccessResponse struct {
ExternalID string `json:"external_id"`
Registered bool `json:"registered"`
Eligible bool `json:"eligible"`
}
// handleChatAccess resolves whether a Telegram user may write in the moderated
// discussion chat. It is gateway-internal: the gateway's bot-link serves the bot's
// join-time query (by external_id) and resolves an emitted chat-access-changed event
// (by user_id) through it.
func (s *Server) handleChatAccess(c *gin.Context) {
var req chatAccessRequest
if err := c.ShouldBindJSON(&req); err != nil {
abortBadRequest(c, "invalid body")
return
}
switch {
case req.ExternalID != "":
s.respondChatAccessByExternalID(c, req.ExternalID)
case req.UserID != "":
s.respondChatAccessByUserID(c, req.UserID)
default:
abortBadRequest(c, "external_id or user_id required")
}
}
// respondChatAccessByExternalID answers the join-path query: an unknown identity is
// reported unregistered (and left muted); a known one carries its current eligibility.
func (s *Server) respondChatAccessByExternalID(c *gin.Context, externalID string) {
ctx := c.Request.Context()
resp := chatAccessResponse{ExternalID: externalID}
acc, err := s.accounts.AccountByIdentity(ctx, account.KindTelegram, externalID)
if errors.Is(err, account.ErrNotFound) {
c.JSON(http.StatusOK, resp)
return
}
if err != nil {
s.abortErr(c, err)
return
}
resp.Registered = true
eligible, err := s.chatEligible(ctx, acc.ID)
if err != nil {
s.abortErr(c, err)
return
}
resp.Eligible = eligible
c.JSON(http.StatusOK, resp)
}
// respondChatAccessByUserID answers the change-path query: an account with no
// Telegram identity carries an empty external_id (nothing for the gateway to gate);
// otherwise it carries the identity and the current eligibility.
func (s *Server) respondChatAccessByUserID(c *gin.Context, raw string) {
ctx := c.Request.Context()
uid, err := uuid.Parse(raw)
if err != nil {
abortBadRequest(c, "invalid user_id")
return
}
var resp chatAccessResponse
ext, err := s.accounts.IdentityExternalID(ctx, uid, account.KindTelegram)
if errors.Is(err, account.ErrNotFound) {
c.JSON(http.StatusOK, resp)
return
}
if err != nil {
s.abortErr(c, err)
return
}
resp.ExternalID = ext
resp.Registered = true
eligible, err := s.chatEligible(ctx, uid)
if err != nil {
s.abortErr(c, err)
return
}
resp.Eligible = eligible
c.JSON(http.StatusOK, resp)
}
// chatEligible reports whether the account may write in the moderated discussion
// chat: not currently admin-suspended and not holding the chat_muted role. A
// suspension dominates — it mutes regardless of the role. Registration is established
// by the caller's identity lookup.
func (s *Server) chatEligible(ctx context.Context, accountID uuid.UUID) (bool, error) {
if _, blocked, err := s.accounts.CurrentSuspension(ctx, accountID); err != nil {
return false, err
} else if blocked {
return false, nil
}
muted, err := s.accounts.HasRole(ctx, accountID, account.RoleChatMuted)
if err != nil {
return false, err
}
return !muted, nil
}
// publishChatAccessChange emits the chat-access-changed signal for the account, so
// the gateway re-resolves the player's chat eligibility and pushes the chat-gate
// command to the bot. Best-effort (notify.Nop when no notifier is wired).
func (s *Server) publishChatAccessChange(id uuid.UUID) {
s.notifier.Publish(notify.ChatAccessChanged(id))
}
+12
View File
@@ -37,11 +37,23 @@ func (s *Server) registerRoutes() {
// before delivering an out-of-app notification.
in.POST("/push-target", s.handlePushTarget)
}
if s.accounts != nil {
// Moderated-chat write eligibility for the Telegram bot: resolve a Telegram
// identity (the bot's join-time query) or an account id (a chat-access-changed
// event) to whether the user may write in the discussion chat. It needs only the
// account store, not the session service, so it registers independently.
s.internal.POST("/chat-access", s.handleChatAccess)
}
if s.ratewatch != nil {
// The gateway's periodic rate-limiter rejection summary: feeds the
// admin console's throttled view and the high-rate auto-flag.
s.internal.POST("/ratelimit/report", s.handleRateLimitReport)
}
if s.banview != nil {
// The gateway's periodic active-ban sync: feeds the admin console's
// active-bans panel and returns the operator's pending unbans.
s.internal.POST("/bans/sync", s.handleBanSync)
}
u := s.user
if s.accounts != nil {
u.GET("/profile", s.handleProfile)
@@ -66,6 +66,7 @@ func (s *Server) registerConsole(router *gin.Engine) {
gm.POST("/reasons/:id/update", s.consoleUpdateReason)
gm.POST("/reasons/:id/delete", s.consoleDeleteReason)
gm.GET("/throttled", s.consoleThrottled)
gm.POST("/bans/unban", s.consoleUnban)
gm.GET("/games", s.consoleGames)
gm.GET("/games/:id", s.consoleGameDetail)
gm.GET("/complaints", s.consoleComplaints)
@@ -874,6 +875,13 @@ func (s *Server) consoleThrottled(c *gin.Context) {
view.Episodes = append(view.Episodes, row)
}
}
if s.banview != nil {
for _, b := range s.banview.Recent() {
view.Bans = append(view.Bans, adminconsole.BanRow{
IP: b.IP, Reason: b.Reason, Since: fmtTime(b.Since), Expires: fmtTime(b.Expires),
})
}
}
flagged, err := s.accounts.ListFlaggedHighRate(ctx)
if err != nil {
s.consoleError(c, err)
@@ -887,6 +895,21 @@ func (s *Server) consoleThrottled(c *gin.Context) {
s.renderConsole(c, "throttled", "throttled", "Throttled", view)
}
// consoleUnban lifts a temporary IP ban — the operator's manual override. The
// gateway applies it on its next active-ban sync, so the ban clears within the
// sync interval rather than immediately.
func (s *Server) consoleUnban(c *gin.Context) {
ip := trimForm(c, "ip")
if ip == "" {
s.renderConsoleMessage(c, "Invalid", "an IP address is required", "/_gm/throttled")
return
}
if s.banview != nil {
s.banview.RequestUnban(ip)
}
s.renderConsoleMessage(c, "Unban requested", fmt.Sprintf("%s will be unbanned on the next gateway sync", ip), "/_gm/throttled")
}
// consoleClearHighRateFlag clears the soft high-rate marker — the operator's
// reversible review action.
func (s *Server) consoleClearHighRateFlag(c *gin.Context) {
@@ -964,6 +987,9 @@ func (s *Server) consoleBlockUser(c *gin.Context) {
s.consoleError(c, err)
return
}
// Re-evaluate the player's moderated-chat write access: a block mutes them in
// the discussion chat if they are currently in it.
s.publishChatAccessChange(id)
s.renderConsoleMessage(c, "Blocked", fmt.Sprintf("account blocked; %d game(s) forfeited", forfeited), back)
}
@@ -978,6 +1004,9 @@ func (s *Server) consoleUnblockUser(c *gin.Context) {
s.consoleError(c, err)
return
}
// Re-evaluate the player's moderated-chat write access: an unblock restores it
// (unless they are still chat-muted) for a member currently in the chat.
s.publishChatAccessChange(id)
s.renderConsoleMessage(c, "Unblocked", "the block was lifted; lost games are not restored", "/_gm/users/"+id.String())
}
@@ -248,6 +248,9 @@ func (s *Server) consoleGrantRole(c *gin.Context) {
if role == account.RoleNoBanner {
s.publishBannerChange(id)
}
if role == account.RoleChatMuted {
s.publishChatAccessChange(id)
}
s.renderConsoleMessage(c, "Role granted", "granted "+role, back)
}
@@ -270,6 +273,9 @@ func (s *Server) consoleRevokeRole(c *gin.Context) {
if role == account.RoleNoBanner {
s.publishBannerChange(id)
}
if role == account.RoleChatMuted {
s.publishChatAccessChange(id)
}
s.renderConsoleMessage(c, "Role revoked", "revoked "+role, back)
}
+7 -1
View File
@@ -35,11 +35,17 @@ func (s *Server) handleTelegramAuth(c *gin.Context) {
abortBadRequest(c, "external_id is required")
return
}
acc, err := s.accounts.ProvisionTelegram(c.Request.Context(), req.ExternalID, req.LanguageCode, req.Username, req.FirstName)
acc, created, err := s.accounts.ProvisionTelegram(c.Request.Context(), req.ExternalID, req.LanguageCode, req.Username, req.FirstName)
if err != nil {
s.abortErr(c, err)
return
}
if created {
// First registration: re-evaluate moderated-chat write access, so a user who
// joined the chat before registering is granted on the spot (no chat_member
// event fires on registration).
s.publishChatAccessChange(acc.ID)
}
s.mintSession(c, acc)
}
+48
View File
@@ -0,0 +1,48 @@
package server
import (
"net/http"
"time"
"github.com/gin-gonic/gin"
"scrabble/backend/internal/banview"
)
// banSyncRequest mirrors the gateway's active-ban report: every entry is one
// currently-enforced IP ban.
type banSyncRequest struct {
Active []banSyncEntry `json:"active"`
}
// banSyncEntry is one active ban in the sync request.
type banSyncEntry struct {
IP string `json:"ip"`
Reason string `json:"reason"`
Since time.Time `json:"since"`
Expires time.Time `json:"expires"`
}
// banSyncResponse returns the IPs an operator has marked for unban for the gateway
// to apply on its next sync.
type banSyncResponse struct {
Unban []string `json:"unban"`
}
// handleBanSync ingests the gateway's active-ban report into the ban view (the
// admin console's active-bans panel) and returns the operator's pending unbans.
// Internal, gateway-only: like the rate-limit report it trusts the network
// segment and carries no user identity.
func (s *Server) handleBanSync(c *gin.Context) {
var req banSyncRequest
if err := c.ShouldBindJSON(&req); err != nil {
abortBadRequest(c, "invalid ban sync")
return
}
bans := make([]banview.Ban, 0, len(req.Active))
for _, e := range req.Active {
bans = append(bans, banview.Ban{IP: e.IP, Reason: e.Reason, Since: e.Since, Expires: e.Expires})
}
s.banview.Ingest(bans)
c.JSON(http.StatusOK, banSyncResponse{Unban: s.banview.DrainUnbans()})
}
+7
View File
@@ -20,6 +20,7 @@ import (
"scrabble/backend/internal/account"
"scrabble/backend/internal/adminconsole"
"scrabble/backend/internal/ads"
"scrabble/backend/internal/banview"
"scrabble/backend/internal/connector"
"scrabble/backend/internal/engine"
"scrabble/backend/internal/feedback"
@@ -83,6 +84,10 @@ type Deps struct {
// admin console's throttled view + the high-rate auto-flag. A nil RateWatch
// disables the internal report endpoint and the console view.
RateWatch *ratewatch.Watch
// BanView mirrors the gateway's active IP bans for the admin console and
// collects operator unban requests. A nil BanView disables the internal
// ban-sync endpoint and the console's active-bans panel.
BanView *banview.View
// Ads is the advertising-banner domain service: campaign rotation feeding the
// profile.get banner block, plus the banner admin console section. A nil Ads
// omits the banner block and disables the banner console.
@@ -115,6 +120,7 @@ type Server struct {
dictDir string
connector *connector.Client
ratewatch *ratewatch.Watch
banview *banview.View
ads *ads.Service
notifier notify.Publisher
console *adminconsole.Renderer
@@ -164,6 +170,7 @@ func New(addr string, deps Deps) *Server {
dictDir: deps.DictDir,
connector: deps.Connector,
ratewatch: deps.RateWatch,
banview: deps.BanView,
ads: deps.Ads,
notifier: notifier,
http: &http.Server{Addr: addr, Handler: engine},
+4
View File
@@ -46,6 +46,10 @@ GRAFANA_ADMIN_PASSWORD=admin
AWG_CONF= # required; AmneziaWG sidecar config (the bot's Telegram egress)
TELEGRAM_BOT_TOKEN= # required
TELEGRAM_GAME_CHANNEL_ID=
TELEGRAM_CHAT_ID= # moderated discussion chat (channel's linked group); empty disables gating
TELEGRAM_PROMO_BOT_TOKEN= # optional standalone promo bot token; empty disables it
TELEGRAM_BOT_USERNAME= # main bot @username without the @ (promo message); required when the promo token is set
TELEGRAM_BOT_LINK= # main bot Mini App link for the promo button (reuse VITE_TELEGRAM_LINK); required when the promo token is set
TELEGRAM_MINIAPP_URL= # required
TELEGRAM_TEST_ENV=false
TELEGRAM_API_BASE_URL=
+69 -3
View File
@@ -17,11 +17,12 @@ operational reference for **every environment variable**.
| `backend` | built (`backend/Dockerfile`) | Domain service; bakes in the DAWG dictionaries; runs migrations at boot. |
| `postgres` | `postgres:17-alpine` | Database (named volume, `pg_isready` healthcheck). |
| `validator` | built (`platform/telegram/Dockerfile`, target `validator`) | Telegram HMAC validator (no VPN, no Bot API); internal gRPC at `validator:9091`. Game login depends only on this. |
| `vpn` + `bot` | sidecar + built (`platform/telegram/Dockerfile`, target `bot`) | Telegram bot; egresses through the AmneziaWG sidecar; holds no inbound port — dials the gateway bot-link (mTLS) at `gateway:9443`. |
| `vpn` + `bot` | sidecar + built (`platform/telegram/Dockerfile`, target `bot`) | Telegram bot, gated to the **`telegram-local`** profile; egresses through the AmneziaWG sidecar and dials the gateway bot-link (mTLS) at `gateway:9443`. The test contour activates the profile; the prod **main** host omits it and runs the bot standalone on its **own host** (`docker-compose.bot.yml`, no VPN — native Bot API egress). |
| `otelcol` | `otel/opentelemetry-collector-contrib` | OTLP/gRPC `:4317` → Prometheus scrape (`:9464`) + Tempo. |
| `prometheus` | `prom/prometheus` | Metrics, 15d retention. |
| `prometheus` | `prom/prometheus` | Metrics, 15d retention (7d in prod). |
| `tempo` | `grafana/tempo` | Traces, 72h retention. |
| `grafana` | `grafana/grafana` | Dashboards (provisioned), anonymous-admin behind caddy's `/_gm/grafana`. |
| `node_exporter` | `quay.io/prometheus/node-exporter` | Host CPU/memory/disk metrics (Prometheus job `node`); the OOM signal on the tight prod main host (2 vCPU / 1.9 GiB). |
Networking: inter-service traffic is on the private `internal` network
(project-scoped DNS); only `caddy` joins the shared external `edge` network so the
@@ -59,7 +60,6 @@ compose binds from this directory.
| Variable | Gitea kind | Purpose |
| --- | --- | --- |
| `POSTGRES_PASSWORD` | secret | Postgres password (also embedded in `BACKEND_POSTGRES_DSN`). |
| `AWG_CONF` | secret | AmneziaWG config for the VPN sidecar (the bot's only Telegram egress in the test contour). **Must not contain a `DNS=` line** — it hijacks the shared netns's resolv.conf and breaks the bot resolving `otelcol` / `gateway`. Without it, Docker's resolver handles `otelcol`, `gateway` and `api.telegram.org`. |
| `GM_BASICAUTH_HASH` | secret | bcrypt hash gating `/_gm` (admin console + Grafana). Generate with `docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'`. |
| `TELEGRAM_MINIAPP_URL` | variable | The Mini App URL the bot hands out in deep links / buttons. |
@@ -67,6 +67,13 @@ compose binds from this directory.
secret) and the bot (Bot API). It defaults to empty in compose, but both **fail at
boot** when it is empty.
**Conditionally — `AWG_CONF`** (secret): the AmneziaWG config for the VPN sidecar, needed
only when the `telegram-local` profile runs (the test contour and local runs with the
bot). It is **not** `:?`-guarded — compose interpolates profiled-out services too, so the
prod main host (no VPN) must not require it. It **must not contain a `DNS=` line** — that
hijacks the shared netns's resolv.conf and breaks the bot resolving `otelcol` / `gateway`;
without it Docker's resolver handles `otelcol`, `gateway` and `api.telegram.org`.
## Optional variables (with defaults)
| Variable | Gitea kind | Default | Purpose |
@@ -110,6 +117,65 @@ collector's / gateway's internal IP is fine (connected route), but its `AWG_CONF
which resolves `otelcol`, `gateway` and `api.telegram.org`. `GATEWAY_ADMIN_*` is
intentionally **unset** — caddy owns `/_gm` in the contour.
## Production rollout
Prod runs on **two hosts** (main = full stack + ACME on the domain; tg = the bot only,
native Bot API, no VPN), one-time provisioned by **[`ansible/`](ansible/)** (docker, a
non-sudo `deploy` user holding the CI key, key-only sshd, default-deny ufw, fail2ban).
Re-run `ansible/` after a host resize — it is idempotent.
**To roll out:** merge `development → master` (CI green), then run the **`prod-deploy`**
workflow manually (Gitea → Actions → prod-deploy → run from `master`, input
`confirm=deploy`). It builds + pushes the images to the registry, ships the
compose/config/certs/env over SSH, deploys the main host with `prod-deploy.sh` (rolling,
health-gated, **auto-rollback to the previous tag**), then the bot host, then probes the
public site. After `master` is green this workflow is the **only** thing that touches
prod — nothing auto-deploys there. It runs four visible jobs: **build → deploy-main →
deploy-bot → verify** (the per-service rolling shows in the deploy-main log).
**Versioning.** Each release is a git tag `vX.Y.Z` on `master`; the deploy stamps
`git describe --tags` into every image tag, every binary (`-ldflags``pkg/version`
the `service.version` telemetry attribute) and the SPA About screen. Tag the release
before running the deploy:
```sh
git tag -a v1.0.0 -m v1.0.0 && git push origin v1.0.0
```
**Manual rollback** (any time after a successful deploy). Run the **`prod-rollback`**
workflow (Gitea → Actions → prod-rollback, `confirm=rollback`). Leave `target_version`
blank to roll back to the previously deployed version (read from the host's
`PREVIOUS_TAG`), or set it to a release tag from the **Releases** page. It re-deploys
that already-published image rolling + health-gated — no rebuild, no DB migration
(image rollback is DB-safe under the expand-contract rule). The registry keeps every
release tag, so any prior release is reachable.
**Migrations** must be **expand-contract** (backward-compatible; goose is forward-only):
the automatic rollback is image-only and never restores the DB. A deploy that changes
`backend/internal/postgres/migrations/` opens a maintenance window — the backend (sole
writer) is stopped for a consistent `pg_dump` into `/opt/scrabble/dumps` before the new
backend migrates. **Manual DB restore** (only if a migration was destructive):
`docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA backend CASCADE'`,
then pipe the dump into the same `psql`, and redeploy the matching old tag.
**bot-link cert rotation:** regenerate (`deploy/gen-certs.sh /tmp/c --force`), reset the
five `PROD_BOTLINK_*` secrets from `/tmp/c`, and re-run the workflow — both hosts redeploy
together with the fresh CA.
**Sizing / monitoring:** the main host launches undersized (2 vCPU / 1.9 GiB); the prod
overlay trims limits + `GOMAXPROCS=2` + 7d Prometheus retention, and `node_exporter` feeds
host memory to Grafana (`/_gm/grafana/`). Watch host memory and resize at Selectel when
players arrive.
**`PROD_` Gitea set** (mirrors `TEST_`, mapped onto the unprefixed names above) — secrets:
`PROD_{POSTGRES_PASSWORD, GM_BASICAUTH_HASH, GRAFANA_ADMIN_PASSWORD, TELEGRAM_BOT_TOKEN,
TELEGRAM_PROMO_BOT_TOKEN, REGISTRY_PASSWORD, SSH_KEY, SSH_KNOWN_HOSTS, BOTLINK_CA,
BOTLINK_GATEWAY_CERT, BOTLINK_GATEWAY_KEY, BOTLINK_BOT_CERT, BOTLINK_BOT_KEY}`; variables:
`PROD_{REGISTRY_USER, MAIN_HOST, TG_HOST, CADDY_SITE_ADDRESS, GM_BASICAUTH_USER,
GRAFANA_ROOT_URL, LOG_LEVEL, DICT_VERSION, TELEGRAM_MINIAPP_URL, TELEGRAM_GAME_CHANNEL_ID,
TELEGRAM_CHAT_ID, TELEGRAM_BOT_USERNAME, VITE_TELEGRAM_BOT_ID, VITE_TELEGRAM_LINK,
VITE_TELEGRAM_GAME_CHANNEL_NAME}`.
## Host-side setup (outside this repo)
- **`edge` network** must exist on the host (`docker network create edge`).
+49
View File
@@ -0,0 +1,49 @@
# Prod host provisioning (Stage 18)
Idempotent Ansible that prepares the two production hosts. It installs Docker, a
non-sudo `deploy` service account, SSH hardening, a default-deny firewall,
fail2ban, unattended security upgrades and time sync. It does **not** deploy the
application — that is `.gitea/workflows/prod-deploy.yaml`'s job, running as the
`deploy` account this playbook creates.
Hosts are referenced by `~/.ssh/config` aliases (`scrabble-main-ops`,
`scrabble-tg-ops`), so no IPs or key paths live in the repo.
## Prerequisites (controller)
- `ansible` with the bundled collections (`community.general`, `community.docker`,
`ansible.posix`).
- The two hosts reachable as root via the ssh-config aliases, host keys already
accepted into `known_hosts` (`host_key_checking = True`).
## One-time: the CI deploy key
The CI prod-deploy workflow logs into the hosts as `deploy` using a dedicated
key. Generate it once on the controller, authorize its public half via the
playbook, and store its private half **only** in the Gitea `PROD_SSH_KEY` secret:
```sh
ssh-keygen -t ed25519 -N '' -C scrabble-ci-deploy \
-f ~/.ssh/scrabble_ci_deploy_ed25519
# private half -> Gitea secret PROD_SSH_KEY (set via API); never commit it
```
## Run
```sh
cd deploy/ansible
ansible-playbook site.yml
```
The playbook reads the public key from `~/.ssh/scrabble_ci_deploy_ed25519.pub` by
default; override with `-e deploy_ci_pubkey_path=/path/to/key.pub`. Re-running is
safe (idempotent) and survives a host resize.
## What each host gets
- **both** (`common`): docker-ce + compose plugin, `daemon.json` (live-restore,
10m×3 log rotation), `deploy` user (docker group, no sudo), key-only sshd,
`ufw` default-deny incoming + allow SSH, fail2ban sshd jail, unattended
upgrades, chrony, `/opt/scrabble/{config,certs,dumps,images}`.
- **main**: `ufw` opens 80/443/9443; the external `edge` docker network.
- **tg**: verifies direct `api.telegram.org` egress (the no-VPN assumption).
+11
View File
@@ -0,0 +1,11 @@
[defaults]
inventory = inventory.ini
roles_path = roles
interpreter_python = /usr/bin/python3
host_key_checking = True
stdout_callback = yaml
deprecation_warnings = False
retry_files_enabled = False
[ssh_connection]
pipelining = True
+21
View File
@@ -0,0 +1,21 @@
---
# Service account the CI prod-deploy workflow uses to drive docker on the hosts.
# Membership in the docker group is root-equivalent (docker socket access), which
# is all the deploy workflow needs; the account is deliberately not given sudo.
deploy_user: deploy
# Public half of the dedicated CI deploy SSH key, read from the controller at run
# time. The private half is generated on the controller during provisioning and
# stored ONLY in the Gitea PROD_SSH_KEY secret; it is never committed. Override the
# path with -e deploy_ci_pubkey_path=/path/to/key.pub if the key lives elsewhere.
deploy_ci_pubkey_path: "{{ lookup('env', 'HOME') }}/.ssh/scrabble_ci_deploy_ed25519.pub"
deploy_ci_pubkey: "{{ lookup('file', deploy_ci_pubkey_path) }}"
# Base directory the deploy workflow rsyncs compose files, config, certs and dumps
# into. Owned by deploy_user so the workflow needs no elevation.
scrabble_base_dir: /opt/scrabble
# Docker daemon json-file log rotation, mirroring the compose x-logging anchor so
# the host's own containers (and any ad-hoc runs) rotate identically.
docker_log_max_size: "10m"
docker_log_max_file: "3"
+19
View File
@@ -0,0 +1,19 @@
# Production inventory for Stage 18.
#
# Hosts resolve through the operator's ~/.ssh/config aliases, so HostName (public
# IP), User and IdentityFile live there — no IPs or key paths are committed here.
# scrabble-main-ops -> main stack host (public IP, domain erudit-game.ru)
# scrabble-tg-ops -> Telegram bot host (direct Bot API egress, no VPN)
[main]
scrabble-main-ops
[tg]
scrabble-tg-ops
[prod:children]
main
tg
[prod:vars]
ansible_user=root
@@ -0,0 +1,15 @@
---
- name: restart docker
ansible.builtin.service:
name: docker
state: restarted
- name: reload sshd
ansible.builtin.service:
name: ssh
state: reloaded
- name: restart fail2ban
ansible.builtin.service:
name: fail2ban
state: restarted
+167
View File
@@ -0,0 +1,167 @@
---
# Common baseline applied to both prod hosts: Docker engine, a non-sudo deploy
# service account, SSH hardening, a default-deny firewall, fail2ban, unattended
# security upgrades and time sync. Every task is idempotent.
- name: Install base packages
ansible.builtin.apt:
name:
- ca-certificates
- curl
- gnupg
- ufw
- fail2ban
- unattended-upgrades
- chrony
state: present
update_cache: true
cache_valid_time: 3600
# --- Docker engine (official repo; trixie is published upstream) ---------------
- name: Create apt keyring directory
ansible.builtin.file:
path: /etc/apt/keyrings
state: directory
mode: "0755"
- name: Install Docker apt GPG key
ansible.builtin.get_url:
url: https://download.docker.com/linux/debian/gpg
dest: /etc/apt/keyrings/docker.asc
mode: "0644"
- name: Add Docker apt repository
ansible.builtin.apt_repository:
repo: >-
deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc]
https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable
filename: docker
state: present
- name: Install Docker engine and the compose plugin
ansible.builtin.apt:
name:
- docker-ce
- docker-ce-cli
- containerd.io
- docker-buildx-plugin
- docker-compose-plugin
state: present
update_cache: true
- name: Configure the Docker daemon (live-restore + log rotation)
ansible.builtin.template:
src: daemon.json.j2
dest: /etc/docker/daemon.json
mode: "0644"
notify: restart docker
- name: Enable and start Docker
ansible.builtin.service:
name: docker
enabled: true
state: started
# --- Deploy service account ----------------------------------------------------
- name: Create the deploy service account
ansible.builtin.user:
name: "{{ deploy_user }}"
groups: docker
append: true
shell: /bin/bash
create_home: true
- name: Ensure the deploy .ssh directory
ansible.builtin.file:
path: "/home/{{ deploy_user }}/.ssh"
state: directory
owner: "{{ deploy_user }}"
group: "{{ deploy_user }}"
mode: "0700"
- name: Authorize the CI deploy SSH key (exclusive)
ansible.builtin.copy:
dest: "/home/{{ deploy_user }}/.ssh/authorized_keys"
content: "{{ deploy_ci_pubkey }}\n"
owner: "{{ deploy_user }}"
group: "{{ deploy_user }}"
mode: "0600"
# --- SSH hardening -------------------------------------------------------------
- name: Harden sshd (key-only auth)
ansible.builtin.template:
src: sshd-hardening.conf.j2
dest: /etc/ssh/sshd_config.d/10-scrabble-hardening.conf
mode: "0644"
validate: sshd -t -f %s
notify: reload sshd
# --- Firewall (default deny incoming) ------------------------------------------
# SSH is allowed before the policy flips so enabling ufw never locks us out.
- name: Allow SSH through the firewall
community.general.ufw:
rule: allow
name: OpenSSH
- name: Default-deny incoming, allow outgoing
community.general.ufw:
direction: "{{ item.direction }}"
policy: "{{ item.policy }}"
loop:
- { direction: incoming, policy: deny }
- { direction: outgoing, policy: allow }
- name: Enable the firewall
community.general.ufw:
state: enabled
# --- fail2ban ------------------------------------------------------------------
- name: Configure the fail2ban sshd jail
ansible.builtin.template:
src: jail.local.j2
dest: /etc/fail2ban/jail.local
mode: "0644"
notify: restart fail2ban
- name: Enable and start fail2ban
ansible.builtin.service:
name: fail2ban
enabled: true
state: started
# --- Unattended security upgrades + time sync ----------------------------------
- name: Enable unattended upgrades
ansible.builtin.copy:
dest: /etc/apt/apt.conf.d/20auto-upgrades
mode: "0644"
content: |
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
- name: Enable and start chrony
ansible.builtin.service:
name: chrony
enabled: true
state: started
# --- Deploy directories --------------------------------------------------------
- name: Create the scrabble base directories
ansible.builtin.file:
path: "{{ scrabble_base_dir }}/{{ item }}"
state: directory
owner: "{{ deploy_user }}"
group: "{{ deploy_user }}"
mode: "0750"
loop:
- ""
- config
- certs
- dumps
- images
@@ -0,0 +1,8 @@
{
"live-restore": true,
"log-driver": "json-file",
"log-opts": {
"max-size": "{{ docker_log_max_size }}",
"max-file": "{{ docker_log_max_file }}"
}
}
@@ -0,0 +1,9 @@
# Managed by Ansible (deploy/ansible).
[DEFAULT]
bantime = 1h
findtime = 10m
maxretry = 5
backend = systemd
[sshd]
enabled = true
@@ -0,0 +1,6 @@
# Managed by Ansible (deploy/ansible). Key-only authentication.
# root stays reachable by key (prohibit-password) for provisioning re-runs.
PasswordAuthentication no
PermitRootLogin prohibit-password
PubkeyAuthentication yes
KbdInteractiveAuthentication no
+18
View File
@@ -0,0 +1,18 @@
---
# Main stack host: public web + bot-link ports and the external 'edge' network
# the compose stack attaches caddy to.
- name: Open public web and bot-link ports
community.general.ufw:
rule: allow
port: "{{ item }}"
proto: tcp
loop:
- "80" # HTTP (ACME challenge + redirect to HTTPS)
- "443" # HTTPS (caddy edge)
- "9443" # bot-link mTLS (remote bot dials in; mutual TLS gates access)
- name: Ensure the external 'edge' docker network exists
community.docker.docker_network:
name: edge
state: present
+19
View File
@@ -0,0 +1,19 @@
---
# Telegram bot host: holds no inbound port beyond SSH (the bot dials out to the
# Bot API and into the main host's bot-link). We only verify direct Bot API
# egress here, since the "no VPN" decision depends on it.
- name: Verify direct Telegram Bot API egress (no VPN on this host)
ansible.builtin.uri:
url: https://api.telegram.org/
method: GET
status_code: [200, 301, 302, 401, 404] # any HTTP reply proves reachability
timeout: 10
register: tg_egress
failed_when: false
- name: Report Telegram reachability
ansible.builtin.debug:
msg: >-
api.telegram.org reachable:
{{ (tg_egress.status | default(0) | int) > 0 }} (status {{ tg_egress.status | default('none') }})
+31
View File
@@ -0,0 +1,31 @@
---
# Stage 18 host provisioning. Idempotent: safe to re-run after a host resize.
# Prepares hosts only (docker, hardening, service account, firewall); the
# application is deployed separately by .gitea/workflows/prod-deploy.yaml.
- name: Common baseline (both hosts)
hosts: prod
become: true
pre_tasks:
- name: Require a well-formed CI deploy public key
ansible.builtin.assert:
that:
- deploy_ci_pubkey | length > 0
- deploy_ci_pubkey is search('^(ssh|ecdsa)-')
fail_msg: >-
deploy_ci_pubkey is empty or malformed. Generate the key first
(see deploy/ansible/README.md) or override deploy_ci_pubkey_path.
roles:
- common
- name: Main stack host
hosts: main
become: true
roles:
- main
- name: Telegram bot host
hosts: tg
become: true
roles:
- tg
+20 -2
View File
@@ -38,10 +38,28 @@
}
}
# The game SPA and the Connect edge are served by the gateway.
# The game SPA and the Connect edge are served by the gateway. Strip any
# client-supplied X-Scrabble-Honeypot here so the gateway only ever honours the
# tag the honeypot block sets below (a client cannot self-tag a real request).
@gateway path /app /app/* /telegram /telegram/* /scrabble.edge.v1.Gateway/*
handle @gateway {
reverse_proxy gateway:8081
reverse_proxy gateway:8081 {
header_up -X-Scrabble-Honeypot
}
}
# Honeypot decoy paths: classic vulnerability-scanner bait no real client ever
# requests. Route them to the gateway tagged with X-Scrabble-Honeypot — the set
# replaces any client-supplied value — so it logs the scanner hit and (in prod)
# bans the source IP. (A delete + set in one block would not work: Caddy applies
# header_up deletions after sets, which would strip the tag we just set; the real
# endpoints instead strip the header in the @gateway block above.) Keep this list
# disjoint from every legitimate landing/app path.
@honeypot path /.env /.git /.git/* /.aws/* /wp-login.php /wp-admin /wp-admin/* /phpmyadmin /phpmyadmin/*
handle @honeypot {
reverse_proxy gateway:8081 {
header_up X-Scrabble-Honeypot 1
}
}
# Everything else — the public landing at / and any stray path — is static.
+53
View File
@@ -0,0 +1,53 @@
# Production Telegram bot host descriptor (standalone — NOT an overlay). Run only on
# the bot host:
# docker compose -f docker-compose.bot.yml up -d
#
# The bot egresses to the Bot API directly (no VPN sidecar) and dials the main host's
# published bot-link :9443 over mTLS. It exports no telemetry — otelcol lives on the
# main host and is unreachable from here — so observe it via `docker logs` on this host.
# Values come from the prod-deploy workflow (PROD_ secrets/variables); BOT_IMAGE is the
# pushed registry tag and BOTLINK_GATEWAY_ADDR is the main host's <ip>:9443.
name: scrabble-bot
services:
bot:
container_name: scrabble-telegram-bot
image: ${BOT_IMAGE:?set BOT_IMAGE to the registry tag}
restart: unless-stopped
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
environment:
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:?set TELEGRAM_BOT_TOKEN}
TELEGRAM_GAME_CHANNEL_ID: ${TELEGRAM_GAME_CHANNEL_ID:-}
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID:-}
TELEGRAM_PROMO_BOT_TOKEN: ${TELEGRAM_PROMO_BOT_TOKEN:-}
TELEGRAM_BOT_USERNAME: ${TELEGRAM_BOT_USERNAME:-}
TELEGRAM_BOT_LINK: ${TELEGRAM_BOT_LINK:-}
TELEGRAM_MINIAPP_URL: ${TELEGRAM_MINIAPP_URL:?set TELEGRAM_MINIAPP_URL}
# Real Bot API in prod (the test contour pins TELEGRAM_TEST_ENV=true instead).
TELEGRAM_TEST_ENV: "false"
TELEGRAM_API_BASE_URL: ${TELEGRAM_API_BASE_URL:-}
TELEGRAM_OWNS_UPDATES: "true"
# Dials the main host's published bot-link. ServerName stays `gateway` (the cert
# SAN), so TLS validation is independent of the dial address.
TELEGRAM_GATEWAY_ADDR: ${BOTLINK_GATEWAY_ADDR:?set BOTLINK_GATEWAY_ADDR (main:9443)}
TELEGRAM_BOTLINK_SERVER_NAME: gateway
TELEGRAM_BOTLINK_TLS_CERT: /certs/bot.crt
TELEGRAM_BOTLINK_TLS_KEY: /certs/bot.key
TELEGRAM_BOTLINK_TLS_CA: /certs/ca.crt
TELEGRAM_LOG_LEVEL: ${LOG_LEVEL:-info}
TELEGRAM_SERVICE_NAME: scrabble-telegram-bot
# No telemetry export: otelcol is on the main host, unreachable from here.
TELEGRAM_OTEL_TRACES_EXPORTER: none
TELEGRAM_OTEL_METRICS_EXPORTER: none
GOMAXPROCS: "1"
volumes:
- ${SCRABBLE_CONFIG_DIR:-.}/certs:/certs:ro
deploy:
resources:
limits:
cpus: "1.0"
memory: 256M
+98
View File
@@ -0,0 +1,98 @@
# Production main-host overlay, applied on top of docker-compose.yml on the main host:
# docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
#
# It (1) publishes caddy 80/443 — there is no host caddy in prod, so the contour caddy
# owns the edge and does its own ACME on CADDY_SITE_ADDRESS — and the gateway bot-link
# :9443 the remote bot dials in over mTLS; and (2) retunes the R7 limits down for the
# 2 vCPU / 1.9 GiB host (GOMAXPROCS=2, smaller memory caps, shorter Prometheus
# retention). The contour launches deliberately undersized at zero players; the added
# node_exporter + Grafana watch host memory so it can be resized at Selectel when
# traffic arrives.
#
# The bot + its VPN sidecar are absent here (the telegram-local profile is not
# activated); the prod bot runs on its own host from docker-compose.bot.yml.
services:
caddy:
ports:
- "80:80"
- "443:443"
deploy:
resources:
limits:
memory: 96M
gateway:
# Prod pulls the pushed image by tag instead of building locally; the base
# build: section stays dormant because the deploy always pulls first.
image: ${REGISTRY:?set REGISTRY}/scrabble-gateway:${TAG:?set TAG}
ports:
- "9443:9443"
environment:
# 2 vCPU host: align the Go scheduler with the cgroup quota (R7's 3 needs 3 cores).
GOMAXPROCS: "2"
deploy:
resources:
limits:
cpus: "2.0"
memory: 384M
backend:
image: ${REGISTRY:?set REGISTRY}/scrabble-backend:${TAG:?set TAG}
deploy:
resources:
limits:
memory: 384M
postgres:
deploy:
resources:
limits:
memory: 384M
validator:
image: ${REGISTRY:?set REGISTRY}/scrabble-telegram-validator:${TAG:?set TAG}
deploy:
resources:
limits:
memory: 96M
landing:
image: ${REGISTRY:?set REGISTRY}/scrabble-landing:${TAG:?set TAG}
deploy:
resources:
limits:
memory: 64M
otelcol:
deploy:
resources:
limits:
memory: 256M
prometheus:
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.retention.time=7d
deploy:
resources:
limits:
memory: 256M
tempo:
deploy:
resources:
limits:
memory: 384M
grafana:
deploy:
resources:
limits:
memory: 256M
postgres_exporter:
deploy:
resources:
limits:
memory: 64M
+56 -1
View File
@@ -71,6 +71,8 @@ services:
# Seed dictionary for a FRESH volume; the per-contour value comes from the
# deploy env (Gitea TEST_/PROD_DICT_VERSION). See the volume note below.
DICT_VERSION: ${DICT_VERSION:-v1.2.1}
# Build version stamped into the binary (git tag; see pkg/version).
VERSION: ${APP_VERSION:-dev}
restart: unless-stopped
logging: *default-logging
depends_on:
@@ -132,6 +134,8 @@ services:
VITE_TELEGRAM_GAME_CHANNEL_NAME: ${VITE_TELEGRAM_GAME_CHANNEL_NAME:-}
VITE_GATEWAY_URL: ${VITE_GATEWAY_URL:-}
VITE_APP_VERSION: ${APP_VERSION:-dev}
# Go binary version (the SPA's VITE_APP_VERSION is the same git tag).
VERSION: ${APP_VERSION:-dev}
restart: unless-stopped
logging: *default-logging
depends_on: [backend]
@@ -150,6 +154,13 @@ services:
GATEWAY_BOTLINK_TLS_CERT: /certs/gateway.crt
GATEWAY_BOTLINK_TLS_KEY: /certs/gateway.key
GATEWAY_BOTLINK_TLS_CA: /certs/ca.crt
# Anti-abuse IP ban (fail2ban-style), fed by rate-limit rejections and the
# honeypot/honeytoken. Off by default: it bans by client IP, which is only
# real in prod — the test contour arrives as one shared NAT address, so a ban
# there would be self-inflicted (the honeypot/honeytoken still log). Prod sets
# these from PROD_ inputs; GATEWAY_HONEYTOKEN is the planted bearer trap.
GATEWAY_ABUSE_BAN_ENABLED: ${GATEWAY_ABUSE_BAN_ENABLED:-false}
GATEWAY_HONEYTOKEN: ${GATEWAY_HONEYTOKEN:-}
GATEWAY_LOG_LEVEL: ${LOG_LEVEL:-info}
GATEWAY_SERVICE_NAME: scrabble-gateway
GATEWAY_OTEL_TRACES_EXPORTER: otlp
@@ -211,6 +222,8 @@ services:
context: ..
dockerfile: platform/telegram/Dockerfile
target: validator
args:
VERSION: ${APP_VERSION:-dev}
restart: unless-stopped
logging: *default-logging
environment:
@@ -233,14 +246,22 @@ services:
networks: [internal]
# --- Telegram bot (egress via the VPN sidecar in test; dials the gateway) ---
# vpn + bot are gated to the `telegram-local` profile: the test contour runs them
# locally (CI passes --profile telegram-local), the prod main host omits them, and
# the prod bot runs on its own host from deploy/docker-compose.bot.yml.
vpn:
container_name: scrabble-telegram-vpn
image: docker.iliadenisov.ru/developer/amneziawg-sidecar:latest
profiles: ["telegram-local"]
restart: unless-stopped
logging: *default-logging
privileged: true
environment:
AWG_CONF: ${AWG_CONF:?set AWG_CONF}
# Required by the vpn sidecar, which is gated to the telegram-local profile.
# Compose can't scope a `:?` guard to a profile (interpolation runs for
# profiled-out services too) and the prod main host has no VPN, so this is a soft
# default; the test contour always supplies TEST_AWG_CONF and the sidecar validates it.
AWG_CONF: ${AWG_CONF:-}
networks:
internal:
aliases: [telegram]
@@ -248,10 +269,13 @@ services:
bot:
container_name: scrabble-telegram-bot
image: scrabble-telegram-bot:latest
profiles: ["telegram-local"]
build:
context: ..
dockerfile: platform/telegram/Dockerfile
target: bot
args:
VERSION: ${APP_VERSION:-dev}
restart: unless-stopped
logging: *default-logging
depends_on: [vpn]
@@ -261,6 +285,17 @@ services:
# at boot; an empty value leaves the bot down while the rest of the contour comes up.
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-}
TELEGRAM_GAME_CHANNEL_ID: ${TELEGRAM_GAME_CHANNEL_ID:-}
# The moderated discussion chat (a channel's linked group) the bot gates write
# access in. Empty disables gating. The group must ALLOW sending by default — the bot
# only restricts (mutes the ineligible) — and the bot must be an admin there with the
# "Ban users" right; chat_member updates are delivered only to a chat admin.
TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID:-}
# The optional standalone promo bot (its own token) answering /start with a button
# into the main bot's app. Empty disables it; when set it needs the main bot's
# @username and the Mini App link (reused from the UI's VITE_TELEGRAM_LINK).
TELEGRAM_PROMO_BOT_TOKEN: ${TELEGRAM_PROMO_BOT_TOKEN:-}
TELEGRAM_BOT_USERNAME: ${TELEGRAM_BOT_USERNAME:-}
TELEGRAM_BOT_LINK: ${TELEGRAM_BOT_LINK:-}
TELEGRAM_MINIAPP_URL: ${TELEGRAM_MINIAPP_URL:?set TELEGRAM_MINIAPP_URL}
TELEGRAM_TEST_ENV: ${TELEGRAM_TEST_ENV:-false}
TELEGRAM_API_BASE_URL: ${TELEGRAM_API_BASE_URL:-}
@@ -426,6 +461,26 @@ services:
memory: 128M
networks: [internal]
# node_exporter exports host CPU/memory/disk metrics. The prod main host runs a tight
# 1.9 GiB budget, so host memory pressure — not just per-container docker_stats — is
# what warns before an OOM. Prometheus scrapes it at :9100 (see prometheus.yml).
node_exporter:
container_name: scrabble-node-exporter
image: quay.io/prometheus/node-exporter:v1.8.2
restart: unless-stopped
logging: *default-logging
command:
- --path.rootfs=/host
- --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host)($|/)
pid: host
volumes:
- /:/host:ro,rslave
deploy:
resources:
limits:
memory: 64M
networks: [internal]
networks:
internal:
name: scrabble-internal
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env bash
# Production main-host deploy driver. Runs ON the main host, invoked over SSH by
# .gitea/workflows/prod-deploy.yaml as the deploy user (which must already be
# `docker login`ed to the registry). It pulls the images at the new tag and rolls
# the stack ONE service at a time in dependency order (least -> most dependent),
# health-checking after each; any failure rolls the whole stack back to the
# previously deployed tag.
#
# A schema migration adds a maintenance window: the backend (the only writer) is
# stopped so a consistent pg_dump is taken before the new backend migrates forward.
# Image rollback alone is safe under the expand-contract migration rule, so the
# automatic rollback never touches the database; the dump is kept for a MANUAL
# restore if a migration turned out to be destructive (see deploy/prod/README.md).
#
# Required env (exported by the workflow over SSH):
# REGISTRY registry namespace, e.g. docker.iliadenisov.ru/developer
# TAG new image tag (the deployed git SHA)
# PREV_TAG previously deployed tag, or "none" on the first deploy
# MIGRATION "1" when the deploy carries a schema migration, else "0"
# Optional: COMPOSE_DIR ENV_FILE DUMP_DIR STATE_FILE POSTGRES_USER POSTGRES_DB
set -uo pipefail
# Runtime compose vars (POSTGRES_*, GM_*, GRAFANA_*, CADDY_*, TELEGRAM_*, REGISTRY,
# SCRABBLE_CONFIG_DIR, ...) come from a shell-sourceable env file the workflow writes
# with single-quoted values. Exporting them into the process environment lets compose
# interpolate ${...} without re-parsing the value — a plain --env-file would mangle the
# literal '$' in the bcrypt GM_BASICAUTH_HASH.
ENV_FILE="${ENV_FILE:-/opt/scrabble/env.sh}"
# shellcheck disable=SC1090
[ -f "$ENV_FILE" ] && . "$ENV_FILE"
REGISTRY="${REGISTRY:?REGISTRY required (env.sh)}"
TAG="${TAG:?TAG required}"
PREV_TAG="${PREV_TAG:-none}"
MIGRATION="${MIGRATION:-0}"
COMPOSE_DIR="${COMPOSE_DIR:-/opt/scrabble/compose}"
DUMP_DIR="${DUMP_DIR:-/opt/scrabble/dumps}"
STATE_FILE="${STATE_FILE:-/opt/scrabble/DEPLOYED_TAG}"
# The prior deployed tag, preserved on every successful deploy so prod-rollback can
# target "the previous version" with no operator input.
PREV_STATE_FILE="${PREV_STATE_FILE:-/opt/scrabble/PREVIOUS_TAG}"
PG_USER="${POSTGRES_USER:-scrabble}"
PG_DB="${POSTGRES_DB:-scrabble}"
cd "$COMPOSE_DIR" || { echo "compose dir $COMPOSE_DIR missing"; exit 1; }
export REGISTRY
# otelcol joins the host docker group to read the socket; the GID varies per host.
DOCKER_GID="$(getent group docker | cut -d: -f3)"
export DOCKER_GID
dc() { docker compose -f docker-compose.yml -f docker-compose.prod.yml "$@"; }
use_tag() { export TAG="$1"; }
# --- health probes (one-off containers on the contour networks, like CI) --------
_probe() { docker run --rm --network "$1" alpine:3.20 wget -q -T 5 -O /dev/null "$2"; }
health_backend() { for _ in $(seq 1 20); do _probe scrabble-internal http://backend:8080/readyz && return 0; sleep 3; done; return 1; }
health_landing() { for _ in $(seq 1 20); do _probe scrabble-internal http://landing:80/ && return 0; sleep 3; done; return 1; }
health_postgres() { for _ in $(seq 1 30); do [ "$(docker inspect -f '{{.State.Health.Status}}' scrabble-postgres 2>/dev/null)" = healthy ] && return 0; sleep 2; done; return 1; }
health_running() { # health_running <container>: running, not restarting, stable restart count
local n="$1" s r c1 c2
for _ in $(seq 1 20); do
s="$(docker inspect -f '{{.State.Status}}' "$n" 2>/dev/null || echo missing)"
r="$(docker inspect -f '{{.State.Restarting}}' "$n" 2>/dev/null || echo true)"
if [ "$s" = running ] && [ "$r" = false ]; then
c1="$(docker inspect -f '{{.RestartCount}}' "$n")"; sleep 5
c2="$(docker inspect -f '{{.RestartCount}}' "$n")"
[ "$c1" = "$c2" ] && return 0
fi
sleep 3
done
return 1
}
roll() { # roll <service> <health-cmd...>
local svc="$1"; shift
echo ">>> rolling $svc -> $TAG"
dc up -d --no-build --no-deps "$svc" || return 1
"$@" || { echo "!!! $svc failed health check"; return 1; }
echo "<<< $svc healthy"
}
rollback() {
echo "########## ROLLBACK -> $PREV_TAG ##########"
if [ "$PREV_TAG" = none ]; then
echo "no previous tag (first deploy): cannot roll back; leaving the stack up for inspection."
return
fi
use_tag "$PREV_TAG"
dc up -d --no-build --remove-orphans
echo "rolled back to $PREV_TAG."
[ "$MIGRATION" = 1 ] && echo "NOTE: the DB is forward-migrated; a pre-deploy dump is in $DUMP_DIR — restore manually ONLY if the migration was destructive (see deploy/README.md, prod runbook)."
}
commit_tag() {
# Record the just-deployed tag as current, preserving the prior one as previous.
[ -f "$STATE_FILE" ] && cp "$STATE_FILE" "$PREV_STATE_FILE"
echo "$TAG" > "$STATE_FILE"
}
mkdir -p "$DUMP_DIR"
echo "=== prod deploy: tag=$TAG prev=$PREV_TAG migration=$MIGRATION ==="
use_tag "$TAG"
dc pull
# First deploy: nothing to roll from; bring the whole stack up and gate on health.
if [ -z "$(docker ps -aq -f name=scrabble-backend)" ]; then
echo "first deploy: bringing the whole stack up"
dc up -d --no-build --remove-orphans || { echo "compose up failed"; exit 1; }
health_backend || { echo "backend not ready"; exit 1; }
health_landing || { echo "landing not ready"; exit 1; }
commit_tag
echo "first deploy healthy ($TAG)."
exit 0
fi
# Migration deploy: freeze writes and snapshot a consistent dump before migrating.
if [ "$MIGRATION" = 1 ]; then
echo "migration deploy: opening maintenance window (stopping the backend = the only writer)"
dc stop backend
dump="$DUMP_DIR/pre-$TAG-$(date +%Y%m%d-%H%M%S).sql"
if ! docker exec scrabble-postgres pg_dump -U "$PG_USER" -d "$PG_DB" -n backend > "$dump"; then
echo "pg_dump failed; restarting the old backend and aborting"
dc start backend
exit 1
fi
echo "consistent dump: $dump"
fi
# Roll one service at a time, least -> most dependent; any failure rolls everything back.
roll postgres health_postgres || { rollback; exit 1; }
roll backend health_backend || { rollback; exit 1; }
roll gateway health_running scrabble-gateway || { rollback; exit 1; }
roll landing health_landing || { rollback; exit 1; }
roll validator health_running scrabble-telegram-validator || { rollback; exit 1; }
roll caddy health_running scrabble-caddy || { rollback; exit 1; }
# Observability + node_exporter: bring up the remainder and pick up any config changes.
dc up -d --no-build --remove-orphans || { rollback; exit 1; }
# Final internal sanity before committing the new tag.
health_backend || { rollback; exit 1; }
commit_tag
echo "=== deploy healthy ($TAG) ==="
+5
View File
@@ -18,3 +18,8 @@ scrape_configs:
- job_name: postgres_exporter
static_configs:
- targets: ["postgres_exporter:9187"]
# Host-level metrics (memory/CPU/disk). Matters most on the prod main host's tight
# 1.9 GiB budget, where total host memory is the OOM-proximity signal.
- job_name: node
static_configs:
- targets: ["node_exporter:9100"]
+89 -16
View File
@@ -128,7 +128,11 @@ dropped). Horizontal scaling is explicit future work.
and GCG are unaffected** (they stay decoded concrete characters, §9.1).
- **gateway ↔ backend (sync)**: plain HTTP REST/JSON. The gateway injects
`X-User-ID` for authenticated requests; `backend` never re-derives identity
from the body.
from the body. Because every sync call targets the one backend host, the
gateway's REST client widens its keep-alive pool well past the stdlib default
of 2 idle connections per host; otherwise the per-request connection churn
exhausts ephemeral ports and burns gateway CPU under load (see
[`../loadtest/REPORT.md`](../loadtest/REPORT.md)).
- **backend → gateway (live)**: a single gRPC server-stream carries live events
(your-turn, opponent-moved, chat, nudge). The gateway bridges them to the
client's in-app stream while the app is open. Out-of-app delivery uses
@@ -824,7 +828,13 @@ the bot renders the message and skips the rest — so in-app-only sub-kinds like
block-state sync to the blocker) never become a platform push. Operator broadcasts
(`SendToUser` / `SendToGameChannel`, §10 admin) render in an **operator-chosen** language in
the console; the backend calls them on the **gateway's bot-link relay**, which forwards them
to the bot and **awaits its delivery ack** (so the console still reports delivered/not).
to the bot and **awaits its delivery ack** (so the console still reports delivered/not). Beyond
messages the same bot-link carries a **chat-gate control path** — a `ChatGate` command sets a user's
write access in the moderated discussion chat and the bot's unary `ResolveChatEligibility` resolves a
joiner's eligibility (neither renders a message; see *Moderated discussion chat* below). An optional
**standalone promo bot** runs in the bot container (`TELEGRAM_PROMO_BOT_TOKEN`): a second bot
answering `/start` with a URL button into the **main** bot's Mini App (`?startapp`, since a `web_app`
button would sign initData with the promo token); it is self-contained — no bot-link, no gateway.
Session-revocation events and cursor-based stream resume stay deferred (single-instance MVP).
A separate **advertising-banner** channel feeds the client's one-line strip (UI_DESIGN.md),
@@ -922,6 +932,28 @@ edits take effect on the next `profile.get` (open/reconnect/foreground), not mid
list/detail, cleared by the operator, **never an automatic ban** and never a request
gate. The Edge/UX dashboard graphs the aggregate request rate against the rejection
rate by class.
- **Temporary IP ban (prod-only):** with `GATEWAY_ABUSE_BAN_ENABLED` set, the gateway
enforces a fail2ban-style block keyed by client IP, fed by three signals: an IP that
sustains `GATEWAY_ABUSE_BAN_THRESHOLD` rate-limiter rejections within
`GATEWAY_ABUSE_BAN_WINDOW` (the IP-keyed public/email/admin classes — the user class
stays the soft-flag's concern, never the ban's), a **honeypot** decoy-path hit, and a
**honeytoken** (a planted bearer no real client holds, `GATEWAY_HONEYTOKEN`). A banned
IP is refused with **429** by an edge middleware (`abuseGuard`) before any work —
covering the Connect edge, the live stream and the static SPA/landing the per-op limiter
never gated. A rejection ban lasts `GATEWAY_ABUSE_BAN_DURATION`; a tripwire/honeytoken
hit is near-zero-false-positive and earns a longer fixed ban (1 h / 24 h). The ban is
**in-memory, single-instance and resets on restart**, like `ratewatch`; each ban
increments `gateway_abuse_banned_total` (`reason` = rejections/tripwire/honeytoken). The
decoy paths live only in the contour **caddy**, which tags them with `X-Scrabble-Honeypot`
(stripping any client-supplied value) and routes them to the gateway. It is **off by
default and only enabled in prod**: the ban keys by real client IP, which the shared-NAT
test contour does not expose (every client arrives as one address), so a ban there would
be self-inflicted — the honeypot/honeytoken still **log** in the contour, only the ban
*action* is gated. Operators see the active bans and lift them on the admin console's
**Throttled** page; the gateway syncs its active set to the backend every 30 s
(`POST /api/v1/internal/bans/sync`, network-trusted like the rejection report) and applies
the operator unbans the response returns, so a manual unban takes effect within the sync
interval.
- Unauthenticated `GET /healthz` (liveness) and `GET /readyz` (readiness — the
database answers a bounded ping and the session cache is warmed).
- The backend serves a **second listener** — a gRPC server
@@ -932,7 +964,7 @@ edits take effect on the next `profile.get` (open/reconnect/foreground), not mid
| Concern | Enforced by |
| --- | --- |
| Public rate limiting / anti-abuse | gateway (per-IP public/email/admin classes, per-user authenticated class; a request body cap of `GATEWAY_MAX_BODY_BYTES`; rejections are metered, summarised to the backend and surfaced in the admin console with a conservative reversible auto-flag — §11) |
| Public rate limiting / anti-abuse | gateway (per-IP public/email/admin classes, per-user authenticated class; a request body cap of `GATEWAY_MAX_BODY_BYTES`; rejections are metered, summarised to the backend and surfaced in the admin console with a conservative reversible auto-flag — §11). In prod a **temporary IP ban** (`GATEWAY_ABUSE_BAN_ENABLED`) blocks an IP that sustains rejections or trips a **honeypot** decoy path / **honeytoken**, refused with 429 before any work; operators lift bans from the console. Off in the shared-NAT test contour, where the client IP is not real (§11) |
| Telegram initData validation (bot-token HMAC) | the Telegram **validator**; the gateway delegates it over gRPC, so the bot token (the HMAC secret) lives only in the validator and the bot, never in the gateway |
| Session minting; email-code / guest validation | gateway (with backend) |
| Session → `user_id` resolution, `X-User-ID` injection | gateway |
@@ -965,9 +997,28 @@ revoked token would fail session resolution at the gateway *before* the gate, se
login instead of the blocked screen). A block instantly **forfeits** every active game the player
is in (the opponent wins, exactly as a resignation — the engine resigns off-turn) and cancels
their open matchmaking games; a temporary block lapses automatically once its expiry passes (no
sweeper the gate recomputes against `now`). No operator identity is recorded (shared
sweeper for the gate — it recomputes against `now`). No operator identity is recorded (shared
Basic-Auth).
**Moderated discussion chat.** A channel's linked discussion group is gated by the Telegram bot
(`TELEGRAM_CHAT_ID`). The group **allows sending by default** and the bot only **restricts**: Telegram
intersects the chat default with each user's permission, so a per-user grant can never exceed a
deny-by-default group — the gate must mute the ineligible, not grant the eligible. A user may write
while they are **registered and neither admin-suspended nor holding the chat-only `chat_muted` role**
(`eligible = registered AND NOT suspended AND NOT chat_muted` — the game suspension dominates); the bot
**mutes** an ineligible member and **un-mutes** an eligible one it had muted, leaving an already-allowed
eligible member untouched (it acts only when the current state differs, so it is idempotent and never
loops on its own change). A single backend resolver behind `POST /api/v1/internal/chat-access` answers
both directions: the bot's `ResolveChatEligibility` on a `chat_member` event (over the mTLS bot-link),
and a `chat_access_changed` event — emitted on a block/unblock, a `chat_muted` grant/revoke, a first
Telegram registration, or a temporary block lapsing (a dedicated `account.SuspensionSweeper`, since no
request fires then) — drives a `ChatGate` command the gateway pushes to the bot. The bot applies it
only to a member currently in the chat (a per-user `getChatMember` probe, since bots cannot list
members); the signal is idempotent and is never an in-app or out-of-app message. `chat_muted` is an
`account_roles` entry (an operator toggle in the console), so it needs no schema change. The bot
must be an administrator in the group with the **restrict-members** right and `chat_member` in its
allowed updates.
**Short numeric codes** (email confirm-codes and friend codes) are stored
only as SHA-256 hashes and are short-lived and single-use. The unauthenticated
email path carries a tight per-IP sub-limit (5 / 10 min); the **friend-code redeem**
@@ -1005,8 +1056,10 @@ plaintext relay (`GATEWAY_BOTLINK_RELAY_ADDR`) the backend admin console calls.
The full contour (`deploy/docker-compose.yml`) runs one `gateway`, one `backend`,
one Postgres, the static `landing`, the Telegram `validator` and `bot` (+ the bot's VPN
sidecar) and the **observability stack**
OTel Collector (OTLP/gRPC ingest → Prometheus metrics + Tempo traces) and Grafana
sidecar — the `bot`+`vpn` pair is gated to a `telegram-local` compose profile so the prod
main host can omit them) and the **observability stack**
OTel Collector (OTLP/gRPC ingest → Prometheus metrics + Tempo traces), a `node_exporter`
for host CPU/memory (the prod main host's OOM signal), and Grafana
with provisioned datasources and dashboards. All services export OTLP to the
collector; the bot shares the VPN sidecar's netns, so its `AWG_CONF` must not
carry a `DNS=` directive (that would hijack resolv.conf and stop it resolving
@@ -1030,16 +1083,36 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`):
generated by `deploy/gen-certs.sh` before `compose up`; the bot keeps its VPN sidecar
for Telegram egress and dials the gateway by its internal name, so the bot-link stays
on the internal network.
- **Prod**: a manual SSH deploy after `development → master`. There is no
host caddy, so the contour ships its own caddy terminating TLS — set
`CADDY_SITE_ADDRESS` to the domain and the caddy does its own ACME. The **bot runs
on a separate host** with native Telegram access (no VPN), deployed by SSH alongside
the main app (rolled together so the bot-link protocol versions never skew); the
gateway **publishes** the bot-link port and the certificates come from `PROD_`
secrets — a long-lived CA with leaves rotated by a scheduled job. The bot dials the
gateway's public bot-link endpoint and holds no inbound port; login is unaffected if
that host or the link is down. *(This prod wiring is the deferred final stage; the
code and the unified test contour land first — see `PRERELEASE.md`.)*
- **Prod**: a **manual** rollout — `.gitea/workflows/prod-deploy.yaml`, `workflow_dispatch`
only (from `master`, `confirm=deploy`), run after `development → master` is merged green.
It builds and pushes the images to the registry (`docker.iliadenisov.ru`), then deploys
over SSH onto **two hosts** provisioned by `deploy/ansible/` (docker, a non-sudo `deploy`
service account holding a dedicated CI key, key-only sshd, default-deny ufw, fail2ban):
the **main host** runs the full stack (`docker-compose.yml` + `docker-compose.prod.yml`),
the **bot host** runs only the bot (`docker-compose.bot.yml`, no VPN — native Bot API
egress, telemetry off). There is no host caddy, so the contour caddy terminates TLS —
`CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. The gateway **publishes**
the bot-link `:9443`; the remote bot dials it over mTLS (certs from `PROD_BOTLINK_*`,
ServerName `gateway`, so TLS validation is independent of the public dial address), holds
no inbound port, and login is unaffected if that host or the link is down.
`deploy/prod-deploy.sh` rolls the main stack **one service at a time in dependency order**
(postgres → backend → gateway → landing → validator → caddy), health-checking after each;
any failure **rolls the whole stack back to the previous image tag**. A **schema migration**
adds a maintenance window: the backend (the sole writer) is stopped for a consistent
`pg_dump` before the new backend migrates forward — image rollback stays DB-safe under the
expand-contract migration rule, and the dump is kept for a manual restore. The workflow runs
four visible jobs (build → deploy-main → deploy-bot → verify). Releases are git tags
`vX.Y.Z`; the version is stamped into the image tag, every binary (`-ldflags``pkg/version`
→ the `service.version` telemetry attribute) and the SPA About screen. A separate manual
**`prod-rollback`** workflow re-deploys any prior release tag (blank input = the previous
deployed version, tracked on the host) over the same rolling, health-gated path — image-only,
no DB migration. The main host is
intentionally **launch-sized** (2 vCPU / 1.9 GiB): the prod overlay trims the R7 limits
(`GOMAXPROCS=2`, smaller caps, 7d Prometheus retention) and a **node_exporter** feeds
host-memory metrics to Grafana so it can be resized reactively as players arrive.
`GATEWAY_ABUSE_BAN_ENABLED=true` in prod (the per-IP ban is meaningful only with real
client IPs). The `vpn`+`bot` pair is gated to a `telegram-local` compose profile the test
contour activates; the prod main host omits it.
## 14. CI & branches
+32 -4
View File
@@ -31,9 +31,15 @@ ephemeral guest. The gateway validates the credential once and mints a thin
session token; the backend resolves it to an internal `user_id`. A **Telegram Mini
App** launch authenticates from the platform's signed `initData`, themes the UI to
the Telegram colours, and — on first contact — seeds the new account's interface
language from the Telegram client. Telegram runs a **single bot**: every player uses
language from the Telegram client. If a launch cannot reach the backend (for example during a
deployment), the Mini App retries quietly and then shows a small "couldn't load" screen with a
**Retry** button, rather than dropping to the web sign-in, which has no place inside Telegram.
Telegram runs a **single bot**: every player uses
the same bot, and all of its chat and out-of-app notifications are written in the
player's own **interface language** (en/ru). Guests are session-only with restricted features
player's own **interface language** (en/ru). A separate optional **promo bot** can run alongside the
main one — its only job is to answer `/start` with a short message and a button that opens the
**main** bot's app, where the player picks their game variant; it is an onboarding entry point that
touches nothing else. Guests are session-only with restricted features
(auto-match only; no friends, stats or history); an abandoned guest that never
joined a game and has been idle past the retention window is garbage-collected. While the app is open the client
keeps a live stream and receives in-app updates in real time — the opponent's move,
@@ -53,6 +59,10 @@ reconnect), and pending reads resume on their own — the interface stays usable
flashing a red banner each time.
### Accounts, linking & merge
_Sign-in is currently provider-only, so the in-profile linking UI is temporarily hidden; it
returns once the anonymous `/app/` guest (whose upgrade path this is) ships. The flow below
describes it for when it does._
First platform contact auto-provisions a durable account. From the profile a player
links an email (via a confirm code) or their Telegram (via the web sign-in); a guest
who links their first identity becomes a durable account. The "already taken" status
@@ -65,6 +75,10 @@ account is kept and the guest's games move into it. A merge is blocked only whil
two accounts share a game still in progress.
### Lobby & matchmaking
On a cold open the lobby greets the player with a brief **loading splash** — Scrabble tiles
spelling **ЭРУДИТ / ЗАГРУЗКА / ОЖИДАНИЕ** as a small crossword — that clears the moment the
games list is ready, so the list never flashes an "empty" state on a slow connection.
The lobby lists **my games** and offers a bottom tab bar — new game, statistics, and a
**⚙️ settings** tab opening the settings hub (settings, profile, friends, about). The
**my games** list groups games into three
@@ -295,8 +309,14 @@ recently throttled users/IPs the gateway reported (an in-memory window — it re
a backend restart) and the accounts currently carrying the soft **high-rate flag**. An
account sustaining rejections past a tunable threshold is flagged automatically —
the marker is reversible, shown as a badge in the user list and on the user card, and
**never blocks play**; the operator reviews and clears it from the user card. There is
no automatic ban.
**never blocks play**; the operator reviews and clears it from the user card. The
account flag itself is never a ban. In **production** the same page also lists the
**active IP bans** the gateway is enforcing: a temporary block of a client IP that
floods the service past a threshold, or trips a hidden **honeypot** path or a planted
**honeytoken** — a high-confidence sign of a scanner or hostile bot, never a normal
player. Each ban shows its reason and expiry with an **Unban** action; bans auto-expire
and the operator can lift one early. IP bans are a production-only safeguard — the
shared test environment cannot tell its clients apart, so it does not enforce them.
The console also lets an operator **manually block** an account — the hard counterpart to the
soft high-rate flag. From the user card the operator blocks the account **permanently** or
@@ -310,6 +330,14 @@ plus the reason when one was given, and the app stops all background traffic wit
temporary block lifts itself when it expires; the operator can also **unblock** from the user card
at any time (games already lost stay lost).
Where the bot manages a channel's **linked discussion chat**, everyone may write by default and the
bot **mutes** a player who is **not registered** or is **blocked**, un-muting them once they register
or are unblocked. So an unregistered newcomer who comments is muted (the promo bot points them at the
game to register, after which the bot restores their voice), and a registered, unblocked player simply
writes. An operator can also **mute a player in the chat only** — a `chat_muted` role on the user card —
without a full account block; an account block mutes them in the chat regardless. Muting and unmuting
take effect for a player already in the chat; one who is not in it is unaffected until they next join.
From the user card the operator can also **top up a player's hint wallet**: an additive grant
(1100 hints per action) that raises the balance shown on the card. Grants are **raise-only**
the console can never lower a wallet (a player only loses hints by spending them in a game), so an
+33 -3
View File
@@ -32,9 +32,15 @@ top-1 подсказку, безлимитную проверку слова с
session-токен; backend сопоставляет его с внутренним `user_id`. Запуск **Telegram
Mini App** авторизует по подписанным `initData` платформы, перекрашивает интерфейс
в цвета Telegram и — при первом контакте — задаёт язык интерфейса нового аккаунта по
языку Telegram-клиента. Telegram держит **единого бота**: все игроки пользуются одним
языку Telegram-клиента. Если запуск не может достучаться до бэкенда (например, во время
деплоя), Mini App тихо повторяет попытки, а затем показывает небольшой экран «не удалось
загрузить» с кнопкой **Повторить**, вместо того чтобы сбрасывать на веб-вход, которому внутри
Telegram не место. Telegram держит **единого бота**: все игроки пользуются одним
и тем же ботом, а весь его чат и внеприложенческие уведомления пишутся на **языке
интерфейса** самого игрока (en/ru). Гость — только сессия, с урезанными функциями (только
интерфейса** самого игрока (en/ru). Рядом с основным может работать отдельный опциональный
**промо-бот** — его единственная задача отвечать на `/start` коротким сообщением и кнопкой,
открывающей приложение **основного** бота, где игрок выбирает нужный вариант игры; это точка входа
для онбординга, не затрагивающая больше ничего. Гость — только сессия, с урезанными функциями (только
авто-подбор; без друзей, статистики и истории); заброшенный гость, не вошедший ни
в одну игру и простаивавший дольше окна удержания, удаляется сборщиком. Пока приложение открыто, клиент
держит живой стрим и получает обновления в реальном времени — ход соперника, ваш ход,
@@ -54,6 +60,10 @@ Mini App** авторизует по подписанным `initData` плат
рабочим вместо красного баннера каждый раз.
### Аккаунты, привязка и слияние
_Вход сейчас только через провайдера, поэтому UI привязки в профиле временно скрыт; он
вернётся, когда появится анонимный `/app/`-гость (для апгрейда которого он и нужен). Описание
ниже — на этот случай._
Первый контакт с платформы заводит постоянный аккаунт. Из профиля игрок
привязывает email (по confirm-коду) или свой Telegram (через веб-вход); гость,
привязавший первую личность, становится постоянным аккаунтом. Факт «личность уже
@@ -66,6 +76,10 @@ Mini App** авторизует по подписанным `initData` плат
запрещено, только пока у аккаунтов есть общая незавершённая игра.
### Лобби и подбор
При холодном запуске лобби встречает игрока короткой **заставкой загрузки** — фишки Scrabble
складывают небольшой кроссворд из слов **ЭРУДИТ / ЗАГРУЗКА / ОЖИДАНИЕ** — и она исчезает, как
только список игр готов, поэтому на медленном соединении список не мигает «пустым» состоянием.
В лобби — список **мои игры** и нижний tab-bar (новая игра, статистика и вкладка
**⚙️ настройки**, открывающая хаб настроек — настройки, профиль, друзья, о программе).
Список **мои игры** разбит на три секции —
@@ -303,7 +317,14 @@ Telegram-identity) или **отправить пост в игровой кан
флагом**. Аккаунт, устойчиво превышающий настраиваемый порог отказов, помечается
автоматически — маркер обратим, виден бейджем в списке пользователей и на карточке
аккаунта и **никогда не блокирует игру**; оператор рассматривает и снимает его с
карточки пользователя. Автоматического бана нет.
карточки пользователя. Сам флаг аккаунта баном не является. В **проде** та же
страница дополнительно перечисляет **активные баны по IP**, которые применяет gateway:
временную блокировку IP клиента, превысившего порог наплыва, либо задевшего скрытую
**honeypot**-ловушку или подброшенный **honeytoken** — высокодостоверный признак
сканера или враждебного бота, но не нормального игрока. У каждого бана показаны причина
и срок, рядом действие **Unban**; баны истекают сами, а оператор может снять бан раньше.
Баны по IP — защита только для прода: общий тестовый контур не различает своих клиентов,
поэтому там не применяется.
Консоль также позволяет оператору **вручную заблокировать** аккаунт — жёсткий аналог мягкого
high-rate флага. С карточки пользователя оператор блокирует аккаунт **навсегда** или **до даты**
@@ -318,6 +339,15 @@ high-rate флага. С карточки пользователя операт
истечении срока; оператор также может **разблокировать** с карточки пользователя в любой момент
(уже проигранные партии не возвращаются).
Там, где бот ведёт **привязанный к каналу чат-обсуждение**, по умолчанию писать может каждый, а бот
**глушит** игрока, который **не зарегистрирован** или **заблокирован**, и снимает мьют, как только тот
зарегистрируется или будет разблокирован. То есть незарегистрированного новичка, написавшего в чат,
бот глушит (промо-бот направляет его в игру зарегистрироваться, после чего бот возвращает голос), а
зарегистрированный незаблокированный игрок просто пишет. Оператор также может **замьютить игрока только
в чате** — роль `chat_muted` на карточке пользователя — без полной блокировки аккаунта; блокировка
аккаунта всё равно мьютит его в чате. Мьют и размьют срабатывают для игрока, уже находящегося в чате;
того, кого в чате нет, это не затрагивает до его следующего входа.
С карточки пользователя оператор также может **пополнить кошелёк подсказок** игрока: аддитивное
начисление (1–100 подсказок за раз), которое **только увеличивает** баланс на карточке. Начисления
**только в плюс** — понизить кошелёк из консоли нельзя (игрок теряет подсказки только тратя их в
+3 -3
View File
@@ -133,9 +133,9 @@ tests or touching CI.
engine tests do). It is **not** part of the per-PR suite's behavioural assertions: it
runs ad hoc as a one-shot container against the contour, producing a trip report (bugs
+ a per-container resource profile) read off the **otelcol `docker_stats` +
postgres_exporter** Grafana dashboard on the contour. Two passes are recorded — the
early [`REPORT-R2.md`](../loadtest/REPORT-R2.md) and the final, tuned
[`REPORT-R7.md`](../loadtest/REPORT-R7.md). See [`../loadtest/README.md`](../loadtest/README.md).
postgres_exporter** Grafana dashboard on the contour. The findings — including the
`game.evaluate` hot-path model and the gateway→backend connection-pool fix — are written
up in [`REPORT.md`](../loadtest/REPORT.md). See [`../loadtest/README.md`](../loadtest/README.md).
- **User feedback**`internal/feedback` unit tests cover the attachment allow-list /
content-type and the channel normaliser; the UI covers `detectChannel`, the attachment gate and
the feedback wire round-trip (`channel` / `feedback` / `codec` tests) plus a Playwright e2e
+23
View File
@@ -20,6 +20,29 @@ the game** (`growNav`) does the nav bar grow to absorb spare height, so the stri
the title while the board and controls pin to the **bottom** for thumb reach. Every screen
except Login uses `Screen`.
## Loading splash (`components/Splash.svelte`)
On a **cold app open** the lobby is the landing screen, but its game list arrives over the
network — on a slow link the empty "no games yet" line would flash before the games load. A
full-screen **tile splash** covers that gap: it lays a small Scrabble crossword out of the
words **ЭРУДИТ** / **ЗАГРУЗКА** / **ОЖИДАНИЕ**, tile by tile, until the lobby's first load
settles, then removes itself to reveal the populated list. The tiles carry their **Эрудит
point values** (hardcoded in `lib/splash.ts`, since the alphabet table the board's
`valueForLetter` reads is not cached yet at boot) and mirror a placed board tile's look
(cream stock, bottom edge, drop shadow). The words form a 6×8 crossword: ЭРУДИТ horizontal,
ЗАГРУЗКА and ОЖИДАНИЕ vertical, crossing it through the shared **Р** and **Д** (laid once).
It is an **App-level overlay** shown while `routeIsLobby && !app.splashDone` (so it also
covers the session bootstrap; a deep-link to another screen is not covered). The lobby sets
`app.lobbyReady` when its first load settles (success **or** error). Each word is laid, then
**held ~0.25 s** so it stays readable, and only after that hold does the readiness check fire —
so a word never blinks away the instant it finishes. ЭРУДИТ lays + holds over ~1.25 s, then the
splash loops ЗАГРУЗКА → ОЖИДАНИЕ (clearing back to ЭРУДИТ between rounds) until ready, so even a
fast load shows it for ~1.25 s. Each tile **drops in** with a brief scale + fade. Under **reduced motion**
(or the mock build, to keep the Playwright smoke unblocked) it shows a static ЭРУДИТ and
dismisses as soon as the lobby is ready. The pure layout and timing live in `lib/splash.ts`
(unit-tested); `Splash.svelte` is the renderer.
## Navigation
- **Back**: a thin, compact `<` drawn from two rotated CSS borders (`Header.svelte`
+3 -1
View File
@@ -70,7 +70,9 @@ RUN rm gateway/internal/webui/dist/landing.html
# Reduce the workspace to what the gateway needs: gateway + pkg (loadtest is not in
# this context; its scrabble/gateway replace targets ./gateway, which is present here).
RUN go work edit -dropuse=./backend -dropuse=./platform/telegram -dropuse=./loadtest
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/gateway ./gateway/cmd/gateway
# VERSION (the deploy passes the git tag) is stamped into the binary via the linker.
ARG VERSION=dev
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags "-X scrabble/pkg/version.Version=${VERSION}" -o /out/gateway ./gateway/cmd/gateway
# --- runtime -----------------------------------------------------------------
FROM gcr.io/distroless/static-debian12:nonroot AS gateway
+19 -1
View File
@@ -23,7 +23,7 @@ proto/edge/v1/ # Connect envelope contract (committed generated Go)
internal/config/ # GATEWAY_* env config
internal/backendclient/ # typed REST client (+ X-User-ID) and push gRPC client
internal/session/ # in-memory session cache (LRU/TTL, backend fallback)
internal/ratelimit/ # token-bucket limiter (golang.org/x/time/rate) + the rejection tracker
internal/ratelimit/ # token-bucket limiter (golang.org/x/time/rate) + the rejection tracker + the temporary IP banlist
internal/connector/ # gRPC client to the Telegram connector (initData validate, out-of-app push) + routing
internal/push/ # live-event fan-out hub (per-user client streams)
internal/transcode/ # FlatBuffers<->REST bridge + message_type registry
@@ -89,6 +89,11 @@ validator (`ValidateLoginWidget`) and forward the trusted `external_id`. These
| `GATEWAY_SESSION_CACHE_MAX` | `50000` | cached session cap |
| `GATEWAY_PUSH_HEARTBEAT_INTERVAL` | `10s` | live-stream keep-alive (an immediate heartbeat also fires on open, under the ~15s edge idle timeout) |
| `GATEWAY_MAX_BODY_BYTES` | `1048576` | caps one request body and one Connect message read; an oversized Execute is refused with `resource_exhausted` |
| `GATEWAY_ABUSE_BAN_ENABLED` | `false` | enable the temporary IP ban (prod-only — keys by real client IP, off in the shared-NAT test contour) |
| `GATEWAY_ABUSE_BAN_THRESHOLD` | `100` | rate-limiter rejections within the window that ban an IP |
| `GATEWAY_ABUSE_BAN_WINDOW` | `2m` | rolling window the rejection strikes accumulate over |
| `GATEWAY_ABUSE_BAN_DURATION` | `15m` | length of a rejection-earned ban (tripwire 1h, honeytoken 24h are fixed) |
| `GATEWAY_HONEYTOKEN` | unset | planted bearer value; presenting it bans the caller and raises an alarm |
| `GATEWAY_SERVICE_NAME` | `scrabble-gateway` | OpenTelemetry `service.name` |
| `GATEWAY_OTEL_TRACES_EXPORTER` | `none` | `none`, `stdout` or `otlp` (gRPC; endpoint from `OTEL_EXPORTER_OTLP_*`) |
| `GATEWAY_OTEL_METRICS_EXPORTER` | `none` | `none`, `stdout` or `otlp` |
@@ -104,6 +109,19 @@ per-key rejection tracker every 30 s, emits a Warn summary per throttled key and
posts the report to the backend (`/api/v1/internal/ratelimit/report`), feeding
the admin console's throttled view and the high-rate auto-flag.
Temporary IP ban (prod-only, `GATEWAY_ABUSE_BAN_ENABLED`): a fail2ban-style block
keyed by client IP, fed by sustained rate-limiter rejections (the IP-keyed classes),
a **honeypot** decoy-path hit (the contour caddy tags decoy paths with
`X-Scrabble-Honeypot`), and a **honeytoken** (`GATEWAY_HONEYTOKEN`). A banned IP is
refused with 429 by the `abuseGuard` edge middleware before any work — covering the
Connect edge, the live stream and the static SPA/landing. Each ban increments
`gateway_abuse_banned_total{reason}` (`rejections`/`tripwire`/`honeytoken`). The ban
is in-memory (resets on restart); it is **off by default** because it keys by the real
client IP, which the shared-NAT test contour does not expose (detection still logs
there, only the ban action is gated). The gateway syncs its active set to the backend
every 30 s (`/api/v1/internal/bans/sync`) for the console's **Active IP bans** panel
and applies the operator unbans the response returns.
## Run
```sh
+76 -1
View File
@@ -68,6 +68,9 @@ const (
// throttleReportInterval is the cadence of the rate-limiter rejection
// summary: the Warn log per throttled key and the report to the backend.
throttleReportInterval = 30 * time.Second
// banSyncInterval is the cadence of the active-ban sync to the backend (which
// feeds the admin-console view) and the operator-unban pull.
banSyncInterval = 30 * time.Second
)
func main() {
@@ -119,6 +122,12 @@ func run(ctx context.Context, cfg config.Config, logger *zap.Logger) error {
sessions := session.NewCache(backend, cfg.SessionTTL, cfg.SessionCacheMax)
limiter := ratelimit.New()
tracker := ratelimit.NewTracker()
banlist := ratelimit.NewBanlist(ratelimit.BanConfig{
Enabled: cfg.Abuse.BanEnabled,
Threshold: cfg.Abuse.BanThreshold,
Window: cfg.Abuse.BanWindow,
Duration: cfg.Abuse.BanDuration,
})
hub := push.NewHub(0)
var validator transcode.TelegramValidator
@@ -138,7 +147,11 @@ func run(ctx context.Context, cfg config.Config, logger *zap.Logger) error {
// fire-and-forget; the backend admin relay (plaintext, internal) awaits the Ack.
var botHub *botlink.Hub
if cfg.BotLinkEnabled() {
botHub = botlink.NewHub(logger, tel.MeterProvider().Meter("scrabble/gateway/botlink"))
botHub = botlink.NewHub(logger, tel.MeterProvider().Meter("scrabble/gateway/botlink"),
func(ctx context.Context, externalID string) (bool, bool, error) {
r, rerr := backend.ChatEligibility(ctx, externalID)
return r.Registered, r.Eligible, rerr
})
tlsCfg, terr := mtls.ServerConfig(cfg.BotLink.CertFile, cfg.BotLink.KeyFile, cfg.BotLink.CAFile)
if terr != nil {
return terr
@@ -183,6 +196,8 @@ func run(ctx context.Context, cfg config.Config, logger *zap.Logger) error {
Sessions: sessions,
Limiter: limiter,
Tracker: tracker,
Banlist: banlist,
Honeytoken: cfg.Abuse.Honeytoken,
Hub: hub,
RateLimit: cfg.RateLimit,
Heartbeat: cfg.PushHeartbeatInterval,
@@ -197,6 +212,11 @@ func run(ctx context.Context, cfg config.Config, logger *zap.Logger) error {
go runPushPump(ctx, backend, hub, botHub, logger)
// Periodically summarise rate-limiter rejections (Warn log + backend report).
go runThrottleReporter(ctx, tracker, backend, logger)
// When the IP ban is enabled (prod), sync the active set to the backend (the
// admin-console view) and apply the operator unbans it returns.
if cfg.Abuse.BanEnabled {
go runBanSync(ctx, banlist, backend, logger)
}
public := &http.Server{Addr: cfg.HTTPAddr, Handler: edge.HTTPHandler(), ReadHeaderTimeout: readHeaderTimeout}
servers := []*namedServer{{name: "public", srv: public}}
@@ -298,6 +318,31 @@ func runThrottleReporter(ctx context.Context, tracker *ratelimit.Tracker, backen
}
}
// runBanSync periodically reports the gateway's active IP bans to the backend (the
// admin-console view) and applies the operator unbans it returns, until the
// context is done. A failed sync is logged and dropped — the next tick reports
// fresh state, and a missed unban is retried on it.
func runBanSync(ctx context.Context, banlist *ratelimit.Banlist, backend *backendclient.Client, logger *zap.Logger) {
ticker := time.NewTicker(banSyncInterval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
}
unban, err := backend.SyncBans(ctx, banlist.Active())
if err != nil {
logger.Warn("ban sync failed", zap.Error(err))
continue
}
for _, ip := range unban {
banlist.Unban(ip)
logger.Info("ban cleared by operator", zap.String("client_ip", ip))
}
}
}
// runPushPump keeps a backend push subscription open, forwarding every event to
// the hub and re-subscribing after the stream ends, until the context is done. For
// the out-of-app push kinds it also routes events whose recipient has no live
@@ -320,6 +365,15 @@ func runPushPump(ctx context.Context, backend *backendclient.Client, hub *push.H
}
break
}
// A chat-access-changed event is an infra signal, not an in-app event:
// resolve the recipient's Telegram identity and current eligibility and
// push the chat-gate command to the bot, without fanning it out to clients.
if ev.GetKind() == chatAccessChangedKind {
if bot != nil {
go deliverChatGate(ctx, backend, bot, ev.GetUserId(), logger)
}
continue
}
hub.Publish(push.Event{
UserID: ev.GetUserId(),
Kind: ev.GetKind(),
@@ -357,6 +411,27 @@ func deliverOutOfApp(ctx context.Context, backend *backendclient.Client, bot *bo
bot.Send(botlink.NotifyCommand(target.ExternalID, kind, payload, target.Language))
}
// chatAccessChangedKind is the backend event signalling that a player's moderated-chat
// write eligibility may have changed; the gateway turns it into a bot-link chat-gate
// command rather than an in-app event (it mirrors notify.KindChatAccessChanged).
const chatAccessChangedKind = "chat_access_changed"
// deliverChatGate resolves a chat-access-changed event to the recipient's Telegram
// identity and current eligibility and pushes the chat-gate command to the bot. It is
// best-effort: a recipient with no Telegram identity is skipped, and a resolve failure
// is logged and dropped (the next moderation action, or a re-join, re-applies the gate).
func deliverChatGate(ctx context.Context, backend *backendclient.Client, bot *botlink.Hub, userID string, logger *zap.Logger) {
res, err := backend.ChatAccessByUser(ctx, userID)
if err != nil {
logger.Warn("chat-gate resolve failed", zap.String("user_id", userID), zap.Error(err))
return
}
if res.ExternalID == "" {
return // no Telegram identity, nothing to gate
}
bot.Send(botlink.ChatGateCommand(res.ExternalID, res.Eligible))
}
// sleep waits for d or until ctx is cancelled, reporting whether it waited the
// full duration.
func sleep(ctx context.Context, d time.Duration) bool {
+28
View File
@@ -215,6 +215,34 @@ func (c *Client) PushTarget(ctx context.Context, userID string) (PushTargetResp,
return out, err
}
// ChatAccessResp is a user's moderated-chat write eligibility: ExternalID is their
// Telegram identity (empty when they have none, so the gateway has nothing to gate),
// Registered whether an account was found, and Eligible the final gate the bot applies
// (registered and neither admin-suspended nor chat-muted).
type ChatAccessResp struct {
ExternalID string `json:"external_id"`
Registered bool `json:"registered"`
Eligible bool `json:"eligible"`
}
// ChatEligibility resolves a Telegram identity to its moderated-chat write
// eligibility — the join path, when the bot sees a user enter the chat.
func (c *Client) ChatEligibility(ctx context.Context, externalID string) (ChatAccessResp, error) {
var out ChatAccessResp
err := c.do(ctx, http.MethodPost, "/api/v1/internal/chat-access", "", "",
map[string]string{"external_id": externalID}, &out)
return out, err
}
// ChatAccessByUser resolves an account id to its Telegram identity and current
// moderated-chat write eligibility — the change path, for a chat-access-changed event.
func (c *Client) ChatAccessByUser(ctx context.Context, userID string) (ChatAccessResp, error) {
var out ChatAccessResp
err := c.do(ctx, http.MethodPost, "/api/v1/internal/chat-access", "", "",
map[string]string{"user_id": userID}, &out)
return out, err
}
// GuestAuth provisions a guest account and mints a session.
func (c *Client) GuestAuth(ctx context.Context) (SessionResp, error) {
var out SessionResp
+38 -1
View File
@@ -22,6 +22,19 @@ import (
pushv1 "scrabble/pkg/proto/push/v1"
)
// backendMaxIdleConns sizes the REST keep-alive pool to the single backend host. The
// default transport caps idle connections per host at 2 (http.DefaultMaxIdleConnsPerHost),
// which — since every synchronous client call proxies to that one host — forces a fresh
// TCP connection (and a lingering TIME_WAIT socket) for almost every request under load.
// That connection churn burns gateway CPU and exhausts ephemeral ports at scale, all
// while the backend itself sits near-idle. Pooling the connections lets them be reused.
//
// The stress harness measured the effect at 500 concurrent players: the churn collapsed
// from ~26 500 TIME_WAIT sockets to ~0 and peak gateway CPU from ~1.75 to ~0.26 cores,
// with the pool settling at ~225 live connections. 512 keeps ~2x headroom over that
// observed peak so a burst never re-caps the pool. See loadtest/REPORT.md.
const backendMaxIdleConns = 512
// Client calls the backend's REST API and opens its push gRPC stream.
type Client struct {
baseURL string
@@ -41,9 +54,14 @@ func New(httpURL, grpcAddr string, timeout time.Duration) (*Client, error) {
if err != nil {
return nil, fmt.Errorf("backendclient: dial push %s: %w", grpcAddr, err)
}
// Clone the default transport (keeping its proxy, dialer and timeouts) and widen the
// idle pool so REST calls to the backend reuse connections instead of churning them.
transport := http.DefaultTransport.(*http.Transport).Clone()
transport.MaxIdleConns = backendMaxIdleConns
transport.MaxIdleConnsPerHost = backendMaxIdleConns
return &Client{
baseURL: strings.TrimRight(httpURL, "/"),
http: &http.Client{Timeout: timeout},
http: &http.Client{Timeout: timeout, Transport: transport},
conn: conn,
push: pushv1.NewPushClient(conn),
}, nil
@@ -137,3 +155,22 @@ func (c *Client) ReportRateLimited(ctx context.Context, windowSeconds int, entri
}{WindowSeconds: windowSeconds, Entries: entries}
return c.do(ctx, http.MethodPost, "/api/v1/internal/ratelimit/report", "", "", body, nil)
}
// SyncBans reports the gateway's currently-active IP bans to the backend and
// returns the IPs an operator has marked for unban since the previous sync. It is
// the ban mirror of ReportRateLimited plus the manual-unban backchannel: the
// backend renders the active set in the admin console and drains the operator's
// unban requests into the response. Like the rejection report it carries no user
// identity and rides the trusted internal segment.
func (c *Client) SyncBans(ctx context.Context, active []ratelimit.Ban) ([]string, error) {
body := struct {
Active []ratelimit.Ban `json:"active"`
}{Active: active}
var out struct {
Unban []string `json:"unban"`
}
if err := c.do(ctx, http.MethodPost, "/api/v1/internal/bans/sync", "", "", body, &out); err != nil {
return nil, err
}
return out.Unban, nil
}
@@ -0,0 +1,31 @@
package backendclient
import (
"net/http"
"testing"
"time"
)
// TestBackendTransportPoolsConnections guards the fix for the gateway->backend
// connection churn. Every synchronous client call proxies to the single backend host,
// so the REST client must widen the idle-connection pool past the default per-host cap
// of 2 (http.DefaultMaxIdleConnsPerHost) — otherwise almost every request under load
// opens a fresh TCP connection that then lingers in TIME_WAIT, burning gateway CPU and
// exhausting ephemeral ports. Reverting to the default transport (`&http.Client{...}`
// with no Transport) would silently reintroduce that, so assert the pool is widened.
func TestBackendTransportPoolsConnections(t *testing.T) {
c, err := New("http://backend.invalid", "localhost:9090", time.Second)
if err != nil {
t.Fatalf("New: %v", err)
}
defer func() { _ = c.Close() }()
tr, ok := c.http.Transport.(*http.Transport)
if !ok {
t.Fatalf("REST transport = %T, want a *http.Transport with a widened idle pool", c.http.Transport)
}
if tr.MaxIdleConnsPerHost <= http.DefaultMaxIdleConnsPerHost {
t.Errorf("MaxIdleConnsPerHost = %d, want > default %d (else per-call connection churn)",
tr.MaxIdleConnsPerHost, http.DefaultMaxIdleConnsPerHost)
}
}
@@ -46,3 +46,39 @@ func TestReportRateLimited(t *testing.T) {
t.Fatalf("backend received %+v, want window 30 + %+v", got, entries[0])
}
}
// TestSyncBans verifies the gateway reports its active bans to the backend's
// internal endpoint and returns the operator unban list the backend replies with.
func TestSyncBans(t *testing.T) {
var got struct {
Active []ratelimit.Ban `json:"active"`
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost || r.URL.Path != "/api/v1/internal/bans/sync" {
t.Errorf("call = %s %s, want POST /api/v1/internal/bans/sync", r.Method, r.URL.Path)
}
if err := json.NewDecoder(r.Body).Decode(&got); err != nil {
t.Errorf("decode sync: %v", err)
}
_, _ = w.Write([]byte(`{"unban":["203.0.113.9"]}`))
}))
defer srv.Close()
c, err := backendclient.New(srv.URL, "localhost:9090", 2*time.Second)
if err != nil {
t.Fatalf("backendclient: %v", err)
}
defer func() { _ = c.Close() }()
active := []ratelimit.Ban{{IP: "198.51.100.4", Reason: ratelimit.ReasonTripwire}}
unban, err := c.SyncBans(context.Background(), active)
if err != nil {
t.Fatalf("SyncBans: %v", err)
}
if len(got.Active) != 1 || got.Active[0].IP != "198.51.100.4" || got.Active[0].Reason != ratelimit.ReasonTripwire {
t.Fatalf("backend received active = %+v, want one tripwire ban for 198.51.100.4", got.Active)
}
if len(unban) != 1 || unban[0] != "203.0.113.9" {
t.Fatalf("unban = %v, want [203.0.113.9]", unban)
}
}
+12
View File
@@ -36,3 +36,15 @@ func SendToGameChannelCommand(text string) *botlinkv1.Command {
}},
}
}
// ChatGateCommand builds a chat-gate command that sets whether the Telegram user
// identified by externalID may write in the moderated discussion chat. The bot
// applies it only to a member currently in the chat (guarded on getChatMember).
func ChatGateCommand(externalID string, allow bool) *botlinkv1.Command {
return &botlinkv1.Command{
Payload: &botlinkv1.Command_ChatGate{ChatGate: &botlinkv1.ChatGateCommand{
ExternalId: externalID,
Allow: allow,
}},
}
}
+33 -6
View File
@@ -31,13 +31,21 @@ var ErrNoBot = errors.New("botlink: no bot connected")
// (at-most-once under backpressure).
const outboundBuffer = 64
// EligibilityResolver answers a Telegram identity's moderated-chat write eligibility
// for the bot's join-time ResolveChatEligibility query: registered reports whether the
// identity maps to an account, eligible is the final gate the bot acts on (registered
// and neither admin-suspended nor chat-muted). The gateway backs it with the backend
// chat-access endpoint.
type EligibilityResolver func(ctx context.Context, externalID string) (registered, eligible bool, err error)
// Hub registers connected bots and routes send commands to them. A single bot is
// expected today; the registry already holds a set so adding more later needs no
// rewrite.
type Hub struct {
botlinkv1.UnimplementedBotLinkServer
log *zap.Logger
log *zap.Logger
eligibility EligibilityResolver
mu sync.Mutex
links map[*link]struct{}
@@ -56,15 +64,18 @@ type link struct {
out chan *botlinkv1.ToBot
}
// NewHub builds a Hub. A nil meter disables metrics; a nil logger is tolerated.
func NewHub(log *zap.Logger, meter metric.Meter) *Hub {
// NewHub builds a Hub. resolve answers the bot's join-time chat-eligibility query
// (nil rejects it as unavailable). A nil meter disables metrics; a nil logger is
// tolerated.
func NewHub(log *zap.Logger, meter metric.Meter, resolve EligibilityResolver) *Hub {
if log == nil {
log = zap.NewNop()
}
h := &Hub{
log: log,
links: make(map[*link]struct{}),
pending: make(map[string]chan *botlinkv1.Ack),
log: log,
eligibility: resolve,
links: make(map[*link]struct{}),
pending: make(map[string]chan *botlinkv1.Ack),
}
if meter != nil {
h.connected, _ = meter.Int64UpDownCounter("botlink_connected_bots",
@@ -120,6 +131,22 @@ func (h *Hub) Link(stream grpc.BidiStreamingServer[botlinkv1.FromBot, botlinkv1.
}
}
// ResolveChatEligibility serves the bot's join-time query: whether the Telegram user
// identified in the request may write in the moderated discussion chat. It delegates
// to the configured resolver (the backend chat-access endpoint), unlike the streamed
// Commands it is a plain request/response over the same mTLS channel.
func (h *Hub) ResolveChatEligibility(ctx context.Context, req *botlinkv1.ChatEligibilityRequest) (*botlinkv1.ChatEligibilityResponse, error) {
if h.eligibility == nil {
return nil, status.Error(codes.Unavailable, "chat eligibility resolver not configured")
}
registered, eligible, err := h.eligibility(ctx, req.GetExternalId())
if err != nil {
h.log.Warn("resolve chat eligibility failed", zap.String("external_id", req.GetExternalId()), zap.Error(err))
return nil, status.Error(codes.Internal, "resolve chat eligibility")
}
return &botlinkv1.ChatEligibilityResponse{Registered: registered, Eligible: eligible}, nil
}
// register adds a connected bot.
func (h *Hub) register(l *link) {
h.mu.Lock()
+67 -3
View File
@@ -8,7 +8,9 @@ import (
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/status"
"google.golang.org/grpc/test/bufconn"
botlinkv1 "scrabble/pkg/proto/botlink/v1"
@@ -23,12 +25,18 @@ type fakeBot struct {
received chan *botlinkv1.Command
}
// startHub registers a Hub on an in-memory gRPC server and returns the hub plus a
// dialer for fake bots.
// startHub registers a Hub (no chat-eligibility resolver) on an in-memory gRPC
// server and returns the hub plus a dialer for fake bots.
func startHub(t *testing.T) (*Hub, func(t *testing.T) botlinkv1.BotLinkClient) {
return startHubWith(t, nil)
}
// startHubWith is startHub with an explicit chat-eligibility resolver, for the
// ResolveChatEligibility tests.
func startHubWith(t *testing.T, resolve EligibilityResolver) (*Hub, func(t *testing.T) botlinkv1.BotLinkClient) {
t.Helper()
lis := bufconn.Listen(1 << 20)
hub := NewHub(nil, nil)
hub := NewHub(nil, nil, resolve)
srv := grpc.NewServer()
botlinkv1.RegisterBotLinkServer(srv, hub)
go func() { _ = srv.Serve(lis) }()
@@ -162,6 +170,62 @@ func TestHubSendAsync(t *testing.T) {
}
}
func TestHubSendChatGate(t *testing.T) {
hub, dial := startHub(t)
ctx := t.Context()
bot := &fakeBot{ack: false}
bot.connect(t, ctx, hub, dial(t))
hub.Send(ChatGateCommand("42", true))
select {
case cmd := <-bot.received:
cg := cmd.GetChatGate()
if cg.GetExternalId() != "42" || !cg.GetAllow() {
t.Errorf("chat_gate = %+v, want external_id=42 allow=true", cg)
}
case <-time.After(time.Second):
t.Fatal("bot received no command")
}
}
func TestHubResolveChatEligibility(t *testing.T) {
var gotExt string
_, dial := startHubWith(t, func(_ context.Context, ext string) (bool, bool, error) {
gotExt = ext
return true, ext == "good", nil
})
client := dial(t)
ctx := t.Context()
resp, err := client.ResolveChatEligibility(ctx, &botlinkv1.ChatEligibilityRequest{ExternalId: "good"})
if err != nil {
t.Fatalf("ResolveChatEligibility: %v", err)
}
if gotExt != "good" {
t.Errorf("resolver external_id = %q, want good", gotExt)
}
if !resp.GetRegistered() || !resp.GetEligible() {
t.Errorf("resp = %+v, want registered+eligible", resp)
}
resp, err = client.ResolveChatEligibility(ctx, &botlinkv1.ChatEligibilityRequest{ExternalId: "muted"})
if err != nil {
t.Fatalf("ResolveChatEligibility(muted): %v", err)
}
if !resp.GetRegistered() || resp.GetEligible() {
t.Errorf("resp = %+v, want registered but not eligible", resp)
}
}
func TestHubResolveChatEligibilityUnconfigured(t *testing.T) {
_, dial := startHub(t) // nil resolver
client := dial(t)
_, err := client.ResolveChatEligibility(t.Context(), &botlinkv1.ChatEligibilityRequest{ExternalId: "x"})
if status.Code(err) != codes.Unavailable {
t.Fatalf("err = %v, want Unavailable", err)
}
}
func TestRelayServerNoBot(t *testing.T) {
hub, _ := startHub(t)
relay := NewRelayServer(hub, 200*time.Millisecond)
+1 -1
View File
@@ -94,7 +94,7 @@ func startMTLSHub(t *testing.T) (hub *Hub, addr, caFile, cliCert, cliKey string)
if err != nil {
t.Fatalf("listen: %v", err)
}
hub = NewHub(nil, nil)
hub = NewHub(nil, nil, nil)
srv := grpc.NewServer(grpc.Creds(credentials.NewTLS(tlsCfg)))
botlinkv1.RegisterBotLinkServer(srv, hub)
go func() { _ = srv.Serve(lis) }()
+68
View File
@@ -46,6 +46,8 @@ type Config struct {
MaxBodyBytes int
// RateLimit configures the in-memory anti-abuse limiter.
RateLimit RateLimitConfig
// Abuse configures the temporary IP ban and the honeytoken (prod-only).
Abuse AbuseConfig
// Telemetry configures the OpenTelemetry providers (shared bootstrap).
Telemetry pkgtel.Config
}
@@ -87,8 +89,32 @@ type RateLimitConfig struct {
EmailBurst int
}
// AbuseConfig configures the gateway's temporary IP ban (fail2ban-style) and the
// honeytoken trap. BanEnabled gates the ban action and is off by default: it is
// only safe where the real client IP is visible (i.e. in prod, not behind the
// shared-NAT test contour). Detection of honeypot/honeytoken hits is logged
// regardless of BanEnabled — only the ban action is gated.
type AbuseConfig struct {
// BanEnabled turns the IP ban on. Off by default (prod-only).
BanEnabled bool
// BanThreshold is the rate-limiter rejection count within BanWindow that bans
// a client IP.
BanThreshold int
// BanWindow is the rolling window the rejection strikes accumulate over.
BanWindow time.Duration
// BanDuration is the length of a rejection-earned ban (tripwire and honeytoken
// bans use their own, longer, fixed durations).
BanDuration time.Duration
// Honeytoken, when non-empty, is a planted bearer value: presenting it bans the
// caller and raises a high-severity alarm. Empty disables the trap.
Honeytoken string
}
// Defaults applied when the corresponding environment variable is unset.
const (
defaultAbuseBanThreshold = 100
defaultAbuseBanWindow = 2 * time.Minute
defaultAbuseBanDuration = 15 * time.Minute
defaultHTTPAddr = ":8081"
defaultLogLevel = "info"
defaultBackendHTTPURL = "http://localhost:8080"
@@ -120,6 +146,17 @@ func DefaultRateLimit() RateLimitConfig {
}
}
// DefaultAbuse returns the built-in anti-abuse settings: the ban disabled
// (prod-only) with the agreed thresholds, and no honeytoken.
func DefaultAbuse() AbuseConfig {
return AbuseConfig{
BanEnabled: false,
BanThreshold: defaultAbuseBanThreshold,
BanWindow: defaultAbuseBanWindow,
BanDuration: defaultAbuseBanDuration,
}
}
// Load reads the configuration from the environment, applies defaults, and
// validates the result.
func Load() (Config, error) {
@@ -134,6 +171,7 @@ func Load() (Config, error) {
ValidatorAddr: os.Getenv("GATEWAY_VALIDATOR_ADDR"),
SessionCacheMax: defaultSessionCacheMax,
RateLimit: DefaultRateLimit(),
Abuse: DefaultAbuse(),
BotLink: BotLinkConfig{
Addr: os.Getenv("GATEWAY_BOTLINK_ADDR"),
RelayAddr: os.Getenv("GATEWAY_BOTLINK_RELAY_ADDR"),
@@ -162,6 +200,19 @@ func Load() (Config, error) {
if c.MaxBodyBytes, err = envInt("GATEWAY_MAX_BODY_BYTES", DefaultMaxBodyBytes); err != nil {
return Config{}, err
}
c.Abuse.Honeytoken = os.Getenv("GATEWAY_HONEYTOKEN")
if c.Abuse.BanEnabled, err = envBool("GATEWAY_ABUSE_BAN_ENABLED", c.Abuse.BanEnabled); err != nil {
return Config{}, err
}
if c.Abuse.BanThreshold, err = envInt("GATEWAY_ABUSE_BAN_THRESHOLD", c.Abuse.BanThreshold); err != nil {
return Config{}, err
}
if c.Abuse.BanWindow, err = envDuration("GATEWAY_ABUSE_BAN_WINDOW", c.Abuse.BanWindow); err != nil {
return Config{}, err
}
if c.Abuse.BanDuration, err = envDuration("GATEWAY_ABUSE_BAN_DURATION", c.Abuse.BanDuration); err != nil {
return Config{}, err
}
if c.BotLink.SendTimeout, err = envDuration("GATEWAY_BOTLINK_SEND_TIMEOUT", defaultBotLinkSendTimeout); err != nil {
return Config{}, err
}
@@ -199,6 +250,9 @@ func (c Config) validate() error {
if c.MaxBodyBytes <= 0 {
return fmt.Errorf("config: GATEWAY_MAX_BODY_BYTES must be positive")
}
if c.Abuse.BanEnabled && (c.Abuse.BanThreshold <= 0 || c.Abuse.BanWindow <= 0 || c.Abuse.BanDuration <= 0) {
return fmt.Errorf("config: GATEWAY_ABUSE_BAN_THRESHOLD/_WINDOW/_DURATION must be positive when GATEWAY_ABUSE_BAN_ENABLED")
}
if c.BotLink.Addr != "" {
if c.BotLink.CertFile == "" || c.BotLink.KeyFile == "" || c.BotLink.CAFile == "" {
return fmt.Errorf("config: GATEWAY_BOTLINK_ADDR requires GATEWAY_BOTLINK_TLS_CERT, _KEY and _CA")
@@ -219,6 +273,20 @@ func envOr(key, fallback string) string {
return fallback
}
// envBool parses the environment variable named key as a bool, returning fallback
// when it is unset and an error when it is set but malformed.
func envBool(key string, fallback bool) (bool, error) {
v := os.Getenv(key)
if v == "" {
return fallback, nil
}
b, err := strconv.ParseBool(v)
if err != nil {
return false, fmt.Errorf("config: %s: %w", key, err)
}
return b, nil
}
// envInt parses the environment variable named key as an int, returning fallback
// when it is unset and an error when it is set but malformed.
func envInt(key string, fallback int) (int, error) {
+50
View File
@@ -2,6 +2,7 @@ package config
import (
"testing"
"time"
pkgtel "scrabble/pkg/telemetry"
)
@@ -45,3 +46,52 @@ func TestLoadMaxBodyBytes(t *testing.T) {
t.Fatal("Load: expected an error for a non-positive body cap, got nil")
}
}
// TestLoadAbuseDefaults verifies the anti-abuse ban defaults: disabled (prod-only),
// the agreed thresholds, and no honeytoken.
func TestLoadAbuseDefaults(t *testing.T) {
c, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
want := DefaultAbuse()
if c.Abuse != want {
t.Errorf("Abuse = %+v, want %+v", c.Abuse, want)
}
if c.Abuse.BanEnabled {
t.Error("ban must default to disabled (enabled only in prod)")
}
}
// TestLoadAbuseOverrides verifies the anti-abuse environment variables are parsed.
func TestLoadAbuseOverrides(t *testing.T) {
t.Setenv("GATEWAY_ABUSE_BAN_ENABLED", "true")
t.Setenv("GATEWAY_ABUSE_BAN_THRESHOLD", "50")
t.Setenv("GATEWAY_ABUSE_BAN_WINDOW", "90s")
t.Setenv("GATEWAY_ABUSE_BAN_DURATION", "30m")
t.Setenv("GATEWAY_HONEYTOKEN", "deadbeef")
c, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
want := AbuseConfig{
BanEnabled: true,
BanThreshold: 50,
BanWindow: 90 * time.Second,
BanDuration: 30 * time.Minute,
Honeytoken: "deadbeef",
}
if c.Abuse != want {
t.Errorf("Abuse = %+v, want %+v", c.Abuse, want)
}
}
// TestLoadAbuseRejectsBadThreshold verifies an enabled ban with a non-positive
// threshold fails validation.
func TestLoadAbuseRejectsBadThreshold(t *testing.T) {
t.Setenv("GATEWAY_ABUSE_BAN_ENABLED", "true")
t.Setenv("GATEWAY_ABUSE_BAN_THRESHOLD", "0")
if _, err := Load(); err == nil {
t.Fatal("Load: expected an error for an enabled ban with a zero threshold, got nil")
}
}
+219
View File
@@ -0,0 +1,219 @@
package connectsrv_test
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"time"
"connectrpc.com/connect"
"scrabble/gateway/internal/backendclient"
"scrabble/gateway/internal/config"
"scrabble/gateway/internal/connectsrv"
"scrabble/gateway/internal/push"
"scrabble/gateway/internal/ratelimit"
"scrabble/gateway/internal/session"
"scrabble/gateway/internal/transcode"
edgev1 "scrabble/gateway/proto/edge/v1"
"scrabble/gateway/proto/edge/v1/edgev1connect"
)
const honeypotHeader = "X-Scrabble-Honeypot"
// guardedEdge wires an edge with an explicit banlist and honeytoken over a fake
// backend, returning the front URL, a Connect client and a cleanup func.
func guardedEdge(t *testing.T, bl *ratelimit.Banlist, honeytoken string, limits config.RateLimitConfig, backendHandler http.HandlerFunc) (string, edgev1connect.GatewayClient, func()) {
t.Helper()
backendSrv := httptest.NewServer(backendHandler)
backend, err := backendclient.New(backendSrv.URL, "localhost:9090", 2*time.Second)
if err != nil {
t.Fatalf("backendclient: %v", err)
}
edge := connectsrv.NewServer(connectsrv.Deps{
Registry: transcode.NewRegistry(backend, nil),
Sessions: session.NewCache(backend, time.Minute, 100),
Limiter: ratelimit.New(),
Banlist: bl,
Honeytoken: honeytoken,
Hub: push.NewHub(0),
RateLimit: limits,
Heartbeat: 15 * time.Second,
})
edgeSrv := httptest.NewServer(edge.HTTPHandler())
client := edgev1connect.NewGatewayClient(http.DefaultClient, edgeSrv.URL)
return edgeSrv.URL, client, func() {
edgeSrv.Close()
_ = backend.Close()
backendSrv.Close()
}
}
// noRedirect is an HTTP client that surfaces a redirect instead of following it,
// so the test can tell a 308 (passed the guard) from a 429 (blocked).
func noRedirect() *http.Client {
return &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
}}
}
func enabledBanlist(threshold int) *ratelimit.Banlist {
return ratelimit.NewBanlist(ratelimit.BanConfig{
Enabled: true, Threshold: threshold, Window: time.Minute, Duration: time.Hour,
})
}
// TestAbuseGuardBlocksBannedIP verifies a banned client IP is refused with 429 at
// the HTTP layer, before any handler runs.
func TestAbuseGuardBlocksBannedIP(t *testing.T) {
bl := enabledBanlist(100)
bl.BanNow("127.0.0.1", ratelimit.ReasonTripwire)
url, _, cleanup := guardedEdge(t, bl, "", config.DefaultRateLimit(), func(w http.ResponseWriter, r *http.Request) {})
defer cleanup()
resp, err := noRedirect().Get(url + "/")
if err != nil {
t.Fatalf("get: %v", err)
}
_ = resp.Body.Close()
if resp.StatusCode != http.StatusTooManyRequests {
t.Fatalf("banned GET / = %d, want 429", resp.StatusCode)
}
}
// TestHoneypotHeaderTrips verifies a request carrying the honeypot header is 404'd
// and bans the client IP, so the next request is blocked.
func TestHoneypotHeaderTrips(t *testing.T) {
bl := enabledBanlist(100)
url, _, cleanup := guardedEdge(t, bl, "", config.DefaultRateLimit(), func(w http.ResponseWriter, r *http.Request) {
t.Error("backend must not be called for a honeypot hit")
})
defer cleanup()
req, _ := http.NewRequest(http.MethodGet, url+"/.env", nil)
req.Header.Set(honeypotHeader, "1")
resp, err := noRedirect().Do(req)
if err != nil {
t.Fatalf("honeypot get: %v", err)
}
_ = resp.Body.Close()
if resp.StatusCode != http.StatusNotFound {
t.Fatalf("honeypot hit = %d, want 404", resp.StatusCode)
}
if !bl.Banned("127.0.0.1") {
t.Fatal("a honeypot hit must ban the client IP")
}
follow, err := noRedirect().Get(url + "/")
if err != nil {
t.Fatalf("follow-up get: %v", err)
}
_ = follow.Body.Close()
if follow.StatusCode != http.StatusTooManyRequests {
t.Fatalf("post-trip GET / = %d, want 429", follow.StatusCode)
}
}
// TestHoneypotDetectsWithoutBanWhenDisabled verifies the prod-only gate: a disabled
// banlist still 404s the decoy (detection/logging) but bans nothing.
func TestHoneypotDetectsWithoutBanWhenDisabled(t *testing.T) {
bl := ratelimit.NewBanlist(ratelimit.BanConfig{}) // disabled
url, _, cleanup := guardedEdge(t, bl, "", config.DefaultRateLimit(), func(w http.ResponseWriter, r *http.Request) {})
defer cleanup()
req, _ := http.NewRequest(http.MethodGet, url+"/.env", nil)
req.Header.Set(honeypotHeader, "1")
resp, err := noRedirect().Do(req)
if err != nil {
t.Fatalf("honeypot get: %v", err)
}
_ = resp.Body.Close()
if resp.StatusCode != http.StatusNotFound {
t.Fatalf("decoy = %d, want 404", resp.StatusCode)
}
if bl.Banned("127.0.0.1") {
t.Fatal("a disabled banlist must not ban")
}
follow, err := noRedirect().Get(url + "/")
if err != nil {
t.Fatalf("follow-up: %v", err)
}
_ = follow.Body.Close()
if follow.StatusCode != http.StatusPermanentRedirect {
t.Fatalf("post-trip GET / = %d, want 308 (not banned)", follow.StatusCode)
}
}
// TestPublicRejectionStrikesBan verifies a public-class limiter rejection feeds the
// banlist: with a one-strike threshold the rejected IP is then banned.
func TestPublicRejectionStrikesBan(t *testing.T) {
bl := enabledBanlist(1)
limits := config.DefaultRateLimit()
limits.PublicPerMinute, limits.PublicBurst = 1, 1
_, client, cleanup := guardedEdge(t, bl, "", limits, func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"token":"tok","user_id":"u-1","is_guest":true,"display_name":"Guest"}`))
})
defer cleanup()
if _, err := client.Execute(context.Background(), connect.NewRequest(&edgev1.ExecuteRequest{MessageType: transcode.MsgAuthGuest})); err != nil {
t.Fatalf("first execute: %v", err)
}
_, err := client.Execute(context.Background(), connect.NewRequest(&edgev1.ExecuteRequest{MessageType: transcode.MsgAuthGuest}))
if connect.CodeOf(err) != connect.CodeResourceExhausted {
t.Fatalf("second execute code = %v, want ResourceExhausted", connect.CodeOf(err))
}
if !bl.Banned("127.0.0.1") {
t.Fatal("a public rejection must strike the banlist")
}
}
// TestUserRejectionDoesNotBan verifies the user limiter class (keyed by account id,
// not IP) does not feed the IP banlist — that path is the backend's soft flag.
func TestUserRejectionDoesNotBan(t *testing.T) {
bl := enabledBanlist(1)
limits := config.DefaultRateLimit()
limits.UserPerMinute, limits.UserBurst = 1, 1
_, client, cleanup := guardedEdge(t, bl, "", limits, func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/api/v1/internal/sessions/resolve":
_, _ = w.Write([]byte(`{"user_id":"u-1","is_guest":false}`))
case "/api/v1/user/feedback/unread":
_, _ = w.Write([]byte(`{"reply_unread":false}`))
default:
t.Errorf("unexpected backend path %s", r.URL.Path)
}
})
defer cleanup()
for i := range 2 {
req := connect.NewRequest(&edgev1.ExecuteRequest{MessageType: transcode.MsgFeedbackUnread})
req.Header().Set("Authorization", "Bearer tok")
_, err := client.Execute(context.Background(), req)
if i == 1 && connect.CodeOf(err) != connect.CodeResourceExhausted {
t.Fatalf("second execute code = %v, want ResourceExhausted", connect.CodeOf(err))
}
}
if len(bl.Active()) != 0 {
t.Fatalf("user-class rejection must not ban; active = %v", bl.Active())
}
}
// TestHoneytokenBansAndRejects verifies presenting the planted honeytoken bans the
// caller and returns the ordinary invalid-session error without a backend call.
func TestHoneytokenBansAndRejects(t *testing.T) {
bl := enabledBanlist(100)
_, client, cleanup := guardedEdge(t, bl, "s3cr3t-trap", config.DefaultRateLimit(), func(w http.ResponseWriter, r *http.Request) {
t.Error("backend must not be called for the honeytoken")
})
defer cleanup()
req := connect.NewRequest(&edgev1.ExecuteRequest{MessageType: transcode.MsgProfileGet})
req.Header().Set("Authorization", "Bearer s3cr3t-trap")
_, err := client.Execute(context.Background(), req)
if connect.CodeOf(err) != connect.CodeUnauthenticated {
t.Fatalf("honeytoken code = %v, want Unauthenticated", connect.CodeOf(err))
}
if !bl.Banned("127.0.0.1") {
t.Fatal("the honeytoken must ban the caller")
}
}
+13 -1
View File
@@ -26,6 +26,7 @@ var activeUserWindows = []struct {
type serverMetrics struct {
edge metric.Float64Histogram
rateLimited metric.Int64Counter
banned metric.Int64Counter
active *activeUsers
}
@@ -48,7 +49,12 @@ func newServerMetrics(meter metric.Meter) *serverMetrics {
if err != nil {
c, _ = noop.NewMeterProvider().Meter(meterName).Int64Counter("gateway_rate_limited_total")
}
m := &serverMetrics{edge: h, rateLimited: c, active: newActiveUsers()}
b, err := meter.Int64Counter("gateway_abuse_banned_total",
metric.WithDescription("Temporary IP bans applied at the edge, by reason (rejections, tripwire or honeytoken)."))
if err != nil {
b, _ = noop.NewMeterProvider().Meter(meterName).Int64Counter("gateway_abuse_banned_total")
}
m := &serverMetrics{edge: h, rateLimited: c, banned: b, active: newActiveUsers()}
gauge, err := meter.Int64ObservableGauge("active_users",
metric.WithDescription("Distinct accounts that performed an authenticated action within the window (in-memory, single gateway instance)."))
@@ -86,3 +92,9 @@ func (m *serverMetrics) recordActive(uid string) {
func (m *serverMetrics) recordRateLimited(ctx context.Context, class string) {
m.rateLimited.Add(ctx, 1, metric.WithAttributes(attribute.String("class", class)))
}
// recordBan counts one temporary IP ban under reason (rejections, tripwire or
// honeytoken).
func (m *serverMetrics) recordBan(ctx context.Context, reason string) {
m.banned.Add(ctx, 1, metric.WithAttributes(attribute.String("reason", reason)))
}
@@ -90,3 +90,41 @@ func TestRateLimitedMetric(t *testing.T) {
t.Errorf("rate_limited counts = %v, want user=2 public=1", counts)
}
}
// TestBannedMetric records ban events through a manual reader and asserts
// gateway_abuse_banned_total splits by reason.
func TestBannedMetric(t *testing.T) {
ctx := context.Background()
reader := sdkmetric.NewManualReader()
meter := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)).Meter("test")
m := newServerMetrics(meter)
m.recordBan(ctx, "tripwire")
m.recordBan(ctx, "tripwire")
m.recordBan(ctx, "honeytoken")
var rm metricdata.ResourceMetrics
if err := reader.Collect(ctx, &rm); err != nil {
t.Fatalf("collect: %v", err)
}
counts := map[string]int64{}
for _, sm := range rm.ScopeMetrics {
for _, md := range sm.Metrics {
if md.Name != "gateway_abuse_banned_total" {
continue
}
sum, ok := md.Data.(metricdata.Sum[int64])
if !ok {
t.Fatalf("gateway_abuse_banned_total is not an int64 sum")
}
for _, dp := range sum.DataPoints {
reason, _ := dp.Attributes.Value(attribute.Key("reason"))
counts[reason.AsString()] += dp.Value
}
}
}
if counts["tripwire"] != 2 || counts["honeytoken"] != 1 {
t.Errorf("banned counts = %v, want tripwire=2 honeytoken=1", counts)
}
}
+72 -7
View File
@@ -8,6 +8,7 @@ package connectsrv
import (
"context"
"crypto/subtle"
"errors"
"net"
"net/http"
@@ -34,6 +35,12 @@ import (
// heartbeatKind is the live-stream keep-alive event kind.
const heartbeatKind = "heartbeat"
// honeypotHeader marks a request the edge proxy routed from a honeypot decoy path;
// any request carrying it is treated as a scanner hit. The proxy strips any
// client-supplied value before setting its own, and a spoofed value only bans the
// spoofer, so trusting it is safe.
const honeypotHeader = "X-Scrabble-Honeypot"
// Limiter classes, the `class` attribute of gateway_rate_limited_total and the
// class field of the periodic rejection report.
const (
@@ -63,6 +70,8 @@ type Server struct {
sessions *session.Cache
limiter *ratelimit.Limiter
tracker *ratelimit.Tracker
banlist *ratelimit.Banlist
honeytoken string
hub *push.Hub
heartbeat time.Duration
log *zap.Logger
@@ -86,7 +95,13 @@ type Deps struct {
// Tracker accumulates limiter rejections for the periodic report; nil
// selects a private tracker (rejections are then only counted, never
// reported).
Tracker *ratelimit.Tracker
Tracker *ratelimit.Tracker
// Banlist enforces temporary IP bans on the hot path; nil selects a disabled
// (inert) banlist.
Banlist *ratelimit.Banlist
// Honeytoken, when non-empty, is the planted bearer value whose presentation
// bans the caller and raises a high-severity alarm.
Honeytoken string
Hub *push.Hub
RateLimit config.RateLimitConfig
Heartbeat time.Duration
@@ -116,6 +131,10 @@ func NewServer(d Deps) *Server {
if limiter == nil {
limiter = ratelimit.New()
}
banlist := d.Banlist
if banlist == nil {
banlist = ratelimit.NewBanlist(ratelimit.BanConfig{})
}
rl := d.RateLimit
if rl == (config.RateLimitConfig{}) {
rl = config.DefaultRateLimit()
@@ -125,6 +144,8 @@ func NewServer(d Deps) *Server {
sessions: d.Sessions,
limiter: limiter,
tracker: tracker,
banlist: banlist,
honeytoken: d.Honeytoken,
hub: d.Hub,
heartbeat: d.Heartbeat,
log: log,
@@ -172,9 +193,11 @@ func (s *Server) HTTPHandler() http.Handler {
mux.Handle("/telegram/", webui.Handler("/telegram/", "index.html"))
mux.Handle("/app/", webui.Handler("/app/", "index.html"))
mux.Handle("/", http.RedirectHandler("/app/", http.StatusPermanentRedirect))
// Every request body on the public listener is capped (the admin proxy POSTs
// included); the h2c server carries explicit stream/idle sizing.
return h2c.NewHandler(maxBodyHandler(s.maxBodyBytes, mux), &http2.Server{
// abuseGuard is the outermost wrap (right under h2c) so a banned IP or a
// honeypot hit is turned away before the body cap and the mux. Every request
// body on the public listener is then capped (the admin proxy POSTs included);
// the h2c server carries explicit stream/idle sizing.
return h2c.NewHandler(s.abuseGuard(maxBodyHandler(s.maxBodyBytes, mux)), &http2.Server{
MaxConcurrentStreams: h2cMaxConcurrentStreams,
IdleTimeout: h2cIdleTimeout,
})
@@ -189,6 +212,32 @@ func maxBodyHandler(limit int, next http.Handler) http.Handler {
})
}
// abuseGuard refuses a banned client IP with 429 before any work, and turns a
// honeypot decoy hit (the proxy-set honeypotHeader) into an instant ban plus a
// bland 404 that is indistinguishable from an ordinary miss. The ban check and the
// tripwire ban are inert on a disabled banlist (the prod-only gate); the tripwire
// hit is logged either way as scanner telemetry.
func (s *Server) abuseGuard(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ip := peerIP(r.RemoteAddr, r.Header)
if s.banlist.Banned(ip) {
http.Error(w, "banned", http.StatusTooManyRequests)
return
}
if r.Header.Get(honeypotHeader) != "" {
s.log.Warn("honeypot tripwire",
zap.String("path", r.URL.Path),
zap.String("client_ip", ip))
if s.banlist.BanNow(ip, ratelimit.ReasonTripwire) {
s.metrics.recordBan(r.Context(), string(ratelimit.ReasonTripwire))
}
http.NotFound(w, r)
return
}
next.ServeHTTP(w, r)
})
}
// Execute runs one unary operation. Domain failures are returned in the envelope
// (result_code != "ok", HTTP 200); only edge failures (rate limit, missing
// session, unknown type, internal) become Connect errors.
@@ -207,7 +256,7 @@ func (s *Server) Execute(ctx context.Context, req *connect.Request[edgev1.Execut
tr := transcode.Request{Payload: req.Msg.GetPayload(), ClientIP: clientIP}
if op.Auth {
uid, isGuest, err := s.resolve(ctx, req.Header())
uid, isGuest, err := s.resolve(ctx, req.Header(), clientIP)
if err != nil {
result = "unauthenticated"
return nil, err
@@ -263,7 +312,7 @@ func (s *Server) Execute(ctx context.Context, req *connect.Request[edgev1.Execut
// Subscribe streams the authenticated user's live events with a keep-alive
// heartbeat until the client disconnects.
func (s *Server) Subscribe(ctx context.Context, req *connect.Request[edgev1.SubscribeRequest], stream *connect.ServerStream[edgev1.Event]) error {
uid, _, err := s.resolve(ctx, req.Header())
uid, _, err := s.resolve(ctx, req.Header(), peerIP(req.Peer().Addr, req.Header()))
if err != nil {
return err
}
@@ -311,6 +360,12 @@ func (s *Server) Subscribe(ctx context.Context, req *connect.Request[edgev1.Subs
func (s *Server) noteRateLimited(ctx context.Context, class, key, msgType string) {
s.metrics.recordRateLimited(ctx, class)
s.tracker.Add(class, key)
// IP-keyed rejections (public, email, admin — the key is the client IP) feed
// the ban; the user class is keyed by account id and is the backend soft-flag's
// concern, not the IP ban's.
if class != classUser && s.banlist.Strike(key) {
s.metrics.recordBan(ctx, string(ratelimit.ReasonRejections))
}
s.log.Debug("rate limited",
zap.String("class", class),
zap.String("key", key),
@@ -344,11 +399,21 @@ func (s *Server) limitAdmin(next http.Handler) http.Handler {
// resolve extracts and resolves the Authorization bearer token to an account id
// and its guest flag, returning a Connect Unauthenticated error when it is missing
// or unknown.
func (s *Server) resolve(ctx context.Context, h http.Header) (string, bool, error) {
func (s *Server) resolve(ctx context.Context, h http.Header, clientIP string) (string, bool, error) {
token := bearerToken(h.Get("Authorization"))
if token == "" {
return "", false, connect.NewError(connect.CodeUnauthenticated, errMissingToken)
}
// The honeytoken is a planted value no real client holds: presenting it is a
// high-confidence intrusion signal, so ban the caller and raise the alarm, then
// return the ordinary invalid-session error so the trap stays indistinguishable.
if s.honeytoken != "" && subtle.ConstantTimeCompare([]byte(token), []byte(s.honeytoken)) == 1 {
s.log.Warn("honeytoken presented", zap.String("client_ip", clientIP))
if s.banlist.BanNow(clientIP, ratelimit.ReasonHoneytoken) {
s.metrics.recordBan(ctx, string(ratelimit.ReasonHoneytoken))
}
return "", false, connect.NewError(connect.CodeUnauthenticated, errInvalidSession)
}
uid, isGuest, err := s.sessions.Resolve(ctx, token)
if err != nil {
// An unknown or expired token (a backend 4xx) is the client's problem and
+211
View File
@@ -0,0 +1,211 @@
package ratelimit
import (
"sort"
"sync"
"time"
)
// Reason labels why a client IP was banned; it is the reason attribute of the
// gateway_abuse_banned_total metric and a field of the ban report to the backend.
type Reason string
const (
// ReasonRejections is a ban earned by sustained rate-limiter rejections: the IP
// accumulated BanConfig.Threshold strikes within BanConfig.Window.
ReasonRejections Reason = "rejections"
// ReasonTripwire is an instant ban from a honeypot decoy-path hit.
ReasonTripwire Reason = "tripwire"
// ReasonHoneytoken is an instant ban from a planted credential being presented.
ReasonHoneytoken Reason = "honeytoken"
)
// Ban durations for the high-confidence reasons. A rejection ban uses the
// configured BanConfig.Duration; a tripwire or honeytoken hit is near
// zero-false-positive, so it earns a markedly longer ban.
const (
tripwireBanDuration = time.Hour
honeytokenBanDuration = 24 * time.Hour
)
// BanConfig tunes the temporary IP ban. Enabled gates the whole mechanism — kept
// off where the real client IP is not visible (e.g. every client arriving as one
// shared NAT address); when false every method is inert.
type BanConfig struct {
// Enabled turns enforcement on. While false Strike/BanNow record nothing,
// Banned is always false and Active is empty.
Enabled bool
// Threshold is the strike count within Window that earns a rejection ban.
Threshold int
// Window is the rolling window the strikes accumulate over.
Window time.Duration
// Duration is the length of a rejection ban (tripwire/honeytoken use their own).
Duration time.Duration
}
// Ban is a snapshot of one active ban for the periodic report and the admin view.
// Its JSON shape is the gateway→backend ban-sync wire contract.
type Ban struct {
IP string `json:"ip"`
Reason Reason `json:"reason"`
Since time.Time `json:"since"`
Expires time.Time `json:"expires"`
}
// Banlist is the gateway's in-memory temporary IP ban: a fail2ban-style block fed
// by sustained rate-limiter rejections (Strike) and by instant honeypot /
// honeytoken hits (BanNow), enforced on the hot path by Banned and cleared either
// by lapse or by an operator Unban. Entries are swept lazily so an expired ban
// does not leak memory. Like the rate limiter it is single-instance and resets on
// restart by design.
type Banlist struct {
cfg BanConfig
now func() time.Time
mu sync.Mutex
entries map[string]*banEntry
lastSweep time.Time
}
// banEntry is one IP's ban state: a live ban (until set) or, before the
// threshold, only the recent strike times.
type banEntry struct {
reason Reason
since time.Time
until time.Time // zero while only accumulating strikes
strikes []time.Time // strike times within the window; pruned on each strike
}
// NewBanlist constructs a Banlist with cfg.
func NewBanlist(cfg BanConfig) *Banlist {
return &Banlist{cfg: cfg, now: time.Now, entries: make(map[string]*banEntry)}
}
// Strike records one rate-limiter rejection for ip and reports whether it earned a
// ban (Threshold strikes within Window). It is a no-op on a disabled banlist.
func (b *Banlist) Strike(ip string) bool {
if !b.cfg.Enabled {
return false
}
b.mu.Lock()
defer b.mu.Unlock()
now := b.now()
b.sweepLocked(now)
e := b.entries[ip]
if e == nil {
e = &banEntry{}
b.entries[ip] = e
}
if now.Before(e.until) {
return false // already banned (a banned IP normally never reaches here)
}
cutoff := now.Add(-b.cfg.Window)
kept := e.strikes[:0]
for _, t := range e.strikes {
if t.After(cutoff) {
kept = append(kept, t)
}
}
e.strikes = append(kept, now)
if len(e.strikes) >= b.cfg.Threshold {
e.reason = ReasonRejections
e.since = now
e.until = now.Add(b.cfg.Duration)
e.strikes = nil
return true
}
return false
}
// BanNow bans ip immediately for reason and reports whether a ban is now in effect
// (true on an enabled banlist, false when disabled).
func (b *Banlist) BanNow(ip string, reason Reason) bool {
if !b.cfg.Enabled {
return false
}
b.mu.Lock()
defer b.mu.Unlock()
now := b.now()
b.sweepLocked(now)
e := b.entries[ip]
if e == nil {
e = &banEntry{}
b.entries[ip] = e
}
e.reason = reason
e.since = now
e.until = now.Add(banDuration(reason, b.cfg.Duration))
e.strikes = nil
return true
}
// Banned reports whether ip is currently banned. It is always false on a disabled
// banlist.
func (b *Banlist) Banned(ip string) bool {
if !b.cfg.Enabled {
return false
}
b.mu.Lock()
defer b.mu.Unlock()
e := b.entries[ip]
return e != nil && b.now().Before(e.until)
}
// Unban clears any ban and accumulated strikes for ip.
func (b *Banlist) Unban(ip string) {
b.mu.Lock()
defer b.mu.Unlock()
delete(b.entries, ip)
}
// Active returns a snapshot of the currently-banned IPs, most recently banned
// first. It is empty on a disabled banlist.
func (b *Banlist) Active() []Ban {
if !b.cfg.Enabled {
return nil
}
b.mu.Lock()
defer b.mu.Unlock()
now := b.now()
out := make([]Ban, 0, len(b.entries))
for ip, e := range b.entries {
if now.Before(e.until) {
out = append(out, Ban{IP: ip, Reason: e.reason, Since: e.since, Expires: e.until})
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Since.After(out[j].Since) })
return out
}
// banDuration maps a reason to its ban length.
func banDuration(reason Reason, rejectionDuration time.Duration) time.Duration {
switch reason {
case ReasonTripwire:
return tripwireBanDuration
case ReasonHoneytoken:
return honeytokenBanDuration
default:
return rejectionDuration
}
}
// sweepLocked discards lapsed bans and stale strike-only entries, at most once per
// sweepInterval. The caller holds b.mu.
func (b *Banlist) sweepLocked(now time.Time) {
if now.Sub(b.lastSweep) < sweepInterval {
return
}
b.lastSweep = now
cutoff := now.Add(-b.cfg.Window)
for ip, e := range b.entries {
if !e.until.IsZero() {
if !now.Before(e.until) {
delete(b.entries, ip) // ban lapsed
}
continue
}
if len(e.strikes) == 0 || !e.strikes[len(e.strikes)-1].After(cutoff) {
delete(b.entries, ip) // strike-only entry gone stale
}
}
}
+143
View File
@@ -0,0 +1,143 @@
package ratelimit
import (
"testing"
"time"
)
// banlistAt builds an enabled banlist whose clock the test drives through clk.
func banlistAt(clk *time.Time, cfg BanConfig) *Banlist {
bl := NewBanlist(cfg)
bl.now = func() time.Time { return *clk }
return bl
}
func enabledCfg() BanConfig {
return BanConfig{Enabled: true, Threshold: 3, Window: time.Minute, Duration: 15 * time.Minute}
}
func TestBanlistStrikeThreshold(t *testing.T) {
clk := time.Date(2026, 6, 21, 12, 0, 0, 0, time.UTC)
bl := banlistAt(&clk, enabledCfg())
if bl.Strike("1.2.3.4") {
t.Fatal("first strike must not ban")
}
if bl.Strike("1.2.3.4") {
t.Fatal("second strike must not ban")
}
if bl.Banned("1.2.3.4") {
t.Fatal("must not be banned before the third strike")
}
if !bl.Strike("1.2.3.4") {
t.Fatal("third strike within the window must ban")
}
if !bl.Banned("1.2.3.4") {
t.Fatal("must be banned after the threshold strike")
}
}
func TestBanlistStrikeWindowResets(t *testing.T) {
clk := time.Date(2026, 6, 21, 12, 0, 0, 0, time.UTC)
bl := banlistAt(&clk, enabledCfg())
// Strikes spaced wider than the window never accumulate to a ban.
for i := range 5 {
if bl.Strike("9.9.9.9") {
t.Fatalf("strike %d should not ban: each falls outside the previous window", i)
}
clk = clk.Add(2 * time.Minute)
}
if bl.Banned("9.9.9.9") {
t.Fatal("strikes outside the rolling window must not ban")
}
}
func TestBanlistBanExpires(t *testing.T) {
clk := time.Date(2026, 6, 21, 12, 0, 0, 0, time.UTC)
bl := banlistAt(&clk, enabledCfg())
bl.BanNow("5.5.5.5", ReasonRejections)
if !bl.Banned("5.5.5.5") {
t.Fatal("must be banned right after BanNow")
}
clk = clk.Add(15*time.Minute + time.Second)
if bl.Banned("5.5.5.5") {
t.Fatal("ban must lapse after its duration")
}
}
func TestBanlistReasonDurations(t *testing.T) {
clk := time.Date(2026, 6, 21, 12, 0, 0, 0, time.UTC)
bl := banlistAt(&clk, enabledCfg())
bl.BanNow("a", ReasonTripwire) // 1h
bl.BanNow("b", ReasonHoneytoken) // 24h
clk = clk.Add(90 * time.Minute)
if bl.Banned("a") {
t.Fatal("tripwire ban (1h) must have lapsed after 90m")
}
if !bl.Banned("b") {
t.Fatal("honeytoken ban (24h) must still hold after 90m")
}
}
func TestBanlistUnban(t *testing.T) {
clk := time.Date(2026, 6, 21, 12, 0, 0, 0, time.UTC)
bl := banlistAt(&clk, enabledCfg())
bl.BanNow("7.7.7.7", ReasonTripwire)
bl.Unban("7.7.7.7")
if bl.Banned("7.7.7.7") {
t.Fatal("Unban must clear the ban")
}
}
func TestBanlistActiveSnapshot(t *testing.T) {
clk := time.Date(2026, 6, 21, 12, 0, 0, 0, time.UTC)
bl := banlistAt(&clk, enabledCfg())
bl.BanNow("a", ReasonTripwire)
bl.BanNow("b", ReasonHoneytoken)
active := bl.Active()
if len(active) != 2 {
t.Fatalf("Active = %d bans, want 2", len(active))
}
byIP := map[string]Ban{}
for _, b := range active {
byIP[b.IP] = b
}
if byIP["a"].Reason != ReasonTripwire || byIP["b"].Reason != ReasonHoneytoken {
t.Fatalf("Active reasons = %+v", byIP)
}
if !byIP["a"].Expires.After(byIP["a"].Since) {
t.Fatal("Expires must be after Since")
}
clk = clk.Add(2 * time.Hour) // tripwire (1h) lapses, honeytoken (24h) holds
if got := len(bl.Active()); got != 1 {
t.Fatalf("Active after 2h = %d, want 1 (only honeytoken)", got)
}
}
func TestBanlistDisabledIsInert(t *testing.T) {
clk := time.Date(2026, 6, 21, 12, 0, 0, 0, time.UTC)
cfg := enabledCfg()
cfg.Enabled = false
bl := banlistAt(&clk, cfg)
for range 10 {
if bl.Strike("1.1.1.1") {
t.Fatal("disabled banlist must never ban via Strike")
}
}
if bl.BanNow("2.2.2.2", ReasonHoneytoken) {
t.Fatal("disabled banlist must never ban via BanNow")
}
if bl.Banned("1.1.1.1") || bl.Banned("2.2.2.2") {
t.Fatal("disabled banlist must report nothing banned")
}
if len(bl.Active()) != 0 {
t.Fatal("disabled banlist must expose no active bans")
}
}
+12 -8
View File
@@ -15,10 +15,12 @@ and prints a trip-report summary. It stays in the repo for repeats.
2. **Drive** (edge protocol over h2c): assembles real 24 player games via the
invitation flow (`invitation.create``invitation.accept`, no robots), then runs
each player's turn loop — poll `game.state`, replay `game.history`, generate a legal
**mid-ranked** move with the embedded `scrabble-solver`, and `game.submit_play`
(or pass/exchange). A fraction of turns exercise nudge / chat / check-word / draft /
profile-update / stats. Each player also holds a live `Subscribe` stream. The
moderate ramp is **50 → 200 → 500** concurrent players, ~12 min per step.
**mid-ranked** move with the embedded `scrabble-solver`, **compose it tile by tile with
the debounced `game.evaluate` preview a real client fires** (the hottest gameplay call),
persist a `draft.save`, and `game.submit_play` (or pass/exchange). A fraction of turns
exercise nudge / chat / check-word / draft / profile-update / stats. Each player also
holds a live `Subscribe` stream. The moderate ramp is **50 → 200 → 500** concurrent
players, ~12 min per step. `--eval=false` drops the evaluate model for an A/B baseline.
3. **Hammer**: drives `games.list` from one account far above the per-user rate limit
to verify the limiter holds (`rate_limited` results) and measure its cost.
4. **Report**: per-operation latency percentiles, throughput, result-code breakdown,
@@ -72,6 +74,8 @@ Key `run` flags (env in parentheses):
| `--games-per-player` | `0` (random 35) | target concurrent games per player |
| `--tick` | `800ms` | per-player op cadence (keeps a player under the per-user limit) |
| `--secondary-prob` | `0.08` | chance per tick of a non-move op |
| `--eval` | `true` | model the per-tile `game.evaluate` preview (the gameplay hot path); `false` reproduces the pre-evaluate harness |
| `--eval-recon` | `1` | extra full-composition evaluate re-previews per play (reconsideration), beyond one per placed tile |
| `--hammer-workers` / `--hammer-dur` | `20` / `15s` | gateway-hammer (0 workers disables) |
| `--reset` / `--cleanup` | `false` | delete harness rows before / after the run |
@@ -93,11 +97,11 @@ runs unconditionally. Use an **absolute** path (here via `$PWD`): `go test ./loa
runs each package from its own directory, so a relative `BACKEND_DICT_DIR` would not
resolve.
## Trip reports
## Trip report
The two stress passes are written up in the repo: the early pass in
[`REPORT-R2.md`](REPORT-R2.md) and the final, tuned pass in
[`REPORT-R7.md`](REPORT-R7.md).
The stress findings — the final run, the `game.evaluate` hot-path model, the
gateway→backend connection-pool fix, and the revised sizing — are written up in
[`REPORT.md`](REPORT.md).
## Caveat
-162
View File
@@ -1,162 +0,0 @@
# R2 — early stress-run trip report
The early stress pass for `PRERELEASE.md` R2. It exercises the system through the
**edge protocol** with the `scrabble/loadtest` harness, to surface logic/concurrency
bugs and capture a resource baseline that feeds R3 (edge hardening), R6 (refactor) and
R7 (final tuning). Pass bar: **diagnostic** — the run "passes" by completing without the
harness crashing; findings are recorded below, not gated.
## Method
- **Driver:** the `scrabble/loadtest` module, run as a one-shot container on the
`scrabble-internal` docker network (reaching `postgres:5432` and `gateway:8081`
directly, bypassing the host→gateway hairpin).
- **Seed:** 10 000 durable + 1 000 guest accounts with pre-created sessions written
directly to Postgres (token hash matches `backend/internal/session`), so the driver
authenticates without the per-IP-limited auth ops.
- **Games:** assembled through the real **invitation** flow (`invitation.create`
`invitation.accept`), 24 players each, no robots; variants spread over
scrabble_en / scrabble_ru / erudit_ru.
- **Play:** each virtual player holds a live `Subscribe` stream and, per tick, polls
`game.state`, replays `game.history` and submits a **mid-ranked** legal move generated
locally by the embedded `scrabble-solver` (the edge carries no board), or
passes/exchanges; a fraction exercise nudge / chat / check-word / draft / profile /
stats. A separate **gateway-hammer** floods `games.list` from one account.
- **Scale:** moderate ramp **50 → 200 → 500** concurrent players, 10 min/step (the
agreed moderate profile; harness and contour share this host's CPU).
- **Resource capture:** `docker stats` (docker API) sampled every 28 s for per-container
CPU/memory; Prometheus for edge latency/throughput, `postgres_exporter` internals and
per-service Go runtime metrics.
## Run configuration
```
loadtest run --durable 10000 --guest 1000 --steps 50,200,500 --step-dur 10m \
--tick 800ms --hammer-workers 20 --hammer-dur 15s --cleanup
```
Date: 2026-06-09. Contour: the R1-baseline schema, freshly deployed with the R2
exporters. Seeded population removed by `--cleanup` afterwards.
## Findings
### Validated (fixed within R2)
- **Harness draft payload.** `draft.save` first returned `bad_request`: the backend
draft DTO's `rack_order` is a string (the harness sent `[]`). Fixed → `ok`.
- **Harness profile marker.** `profile.update` first returned `invalid_profile`: the
editable-display-name validator (`backend/internal/account/profile.go`) forbids digits
and colons, but the seed marker was `lt:…`. Switched the marker to a distinctive
letters-only string → `ok`. Cleanup still matches it.
### By-design behaviour (correctly exercised, not bugs)
- **`chat_not_your_turn`** — chat is gated to the sender's turn
(`backend/internal/social/chat.go`); off-turn posts are correctly rejected.
- **`nudge_own_turn`** — you nudge the player whose turn it is, so a nudge on your own
turn is correctly rejected. The harness nudges/chats at random ticks, so a share of
these codes is expected.
### Observability gap (key R7 input)
- **cAdvisor yields only the root cgroup on the contour host.** Its docker factory
registers, but per-container init fails — `failed to identify the read-write layer ID
… /rootfs/var/lib/docker/image/overlayfs/…: no such file or directory` — because this
host's `/var/lib/docker` is a **separate XFS mount** not visible under cAdvisor's
`/rootfs` bind (the existing galaxy deployment on the same host has the same
limitation). So the **Scrabble — Resources** dashboard's per-container panels are empty
here, and per-container CPU/RSS for this run was captured via `docker stats` instead.
Postgres internals (`postgres_exporter`) and per-service Go runtime metrics
(`go_*` by `service_name`) work. **Recommendation for R7:** adopt the otelcol
**`docker_stats`** receiver (already the contrib image) — it reads per-container stats
via the docker API with no cgroup dependency — and/or run the final pass on hardware
where cAdvisor resolves containers. (Decision to confirm with the owner.)
### Run results
The ramp ran clean to 500 players with no harness crash, no deadlock and
`stream errors: 0`; cleanup removed all 11 000 seeded accounts (and their ~941 games).
- **Ramp:** step 1 = 50 players / 90 games, step 2 = 200 / 282, step 3 = 500 / 569.
- **Volume (30 min):** 1.20 M total edge calls, 659 req/s average. Real gameplay at
scale: **48 870 committed plays**, 52 772 `your_turn` + 159 631 `opponent_moved`
events, **2 798 games finished**.
- **Latency under load (peak, step 3):** `game.state` p50 ≈ 100 ms, p90/p99 in the
200500 ms buckets, max 849 ms; `game.submit_play` similar (p99 ≤ 500 ms, max 490 ms).
Lobby ops stayed fast (invitation/games.list p99 ≤ 10 ms).
- **Rate limiter holds.** The gateway-hammer sent 522 667 `games.list` from one account;
**522 486 (99.97 %) were `rate_limited`**, only 135 `ok` (the burst). Rejections are
cheap — p99 = 2 ms — and the gateway sustained ~16 k req/s of rejections during the
flood. The per-user limiter behaves as designed (R3 input: the cost is negligible).
**Top finding — `transport_error` under saturation.** At 500 players ~14 % of
`game.state` calls (72 429 / 519 067) and a few % of the other ops returned a Connect
`transport_error` (not a domain code). It correlates with the CPU saturation below: the
backend/gateway are pinned near one core each while the host also runs the 86 %-core
harness, so the edge sheds load (resets/timeouts) at the knee. It is **amplified by a
harness artifact** — all 500 virtual players multiplex over a *single* shared
`http2.Transport`, so 500 persistent `Subscribe` streams plus Execute calls press on one
HTTP/2 connection's concurrent-stream limit; real clients each use their own connection.
**Actions:** R7 harness — give each player (or a pool) its own transport, and run on
hardware not shared with the contour; R3 — confirm the gateway's h2c
`MaxConcurrentStreams` and edge timeouts are sized for many persistent streams.
**Minor findings:**
- `unauthenticated` on a tiny share (188 / 519 067 `game.state`, ~0.04 %) — transient
session-resolve failures under load; worth a glance in R3 but not material.
- one `internal` on `game.pass` (1 / 4 788).
- `game_finished` dominates `chat.nudge`/`chat.post` (≈ 3 900 each): the harness keeps
secondary ops on games that already ended. Harness refinement — drop finished games
from the rotation (R7).
- `nudge_own_turn` / `chat_not_your_turn` / `nudge_too_soon` are the expected turn/rate
gates, correctly exercised.
## Resource baseline
Per-container peak during step 3 (500 players), from `docker stats`:
| container | peak CPU | memory |
|-----------|---------:|-------:|
| scrabble-backend | **99 %** (~1 core) | 91 MiB |
| scrabble-gateway | **93 %** | 76 MiB |
| scrabble-postgres | **90 %** | 69 MiB |
| scrabble-loadtest (harness) | **86 %** | 42 MiB |
| scrabble-otelcol | 10 % | 110 MiB |
| scrabble-tempo | 9 % | 446 MiB |
| prometheus / postgres-exporter | ~0 % | 46 / 16 MiB |
- **The contour is CPU-bound at 500 concurrent players:** backend, gateway and Postgres
each saturate ~1 core (single-instance MVP config), so the system draws ~3 cores at
this scale; memory is modest (≤ 100 MiB per Go service). This is the sizing input for
R7 (pool sizes, GOMAXPROCS, container limits) and the prod cutover.
- **Caveat:** the harness itself peaked at **86 % of a core** on the *same host*, so the
step-3 latency and `transport_error` figures are pessimistic — the contour competed
with the generator for CPU. A clean ceiling needs separate hardware (R7).
- **Postgres:** peak 28 backend connections, ~5 581 commits/s at the peak, **100 % cache
hit ratio** (no disk reads) — the DB was comfortable; CPU, not I/O, is its limit here.
- **Goroutines:** backend 638, gateway **1 698** (it holds the 500 `Subscribe` streams +
per-request goroutines), telegram 49 — all stable, no leak across the ramp.
## Recommendations feeding later phases
- **R3 (edge hardening):** the per-user limiter holds (99.97 % rejected, p99 2 ms) — add
the per-IP body-size cap on top. Investigate the **~14 % `transport_error` on
`game.state` at 500 players**: confirm the gateway h2c `MaxConcurrentStreams` and edge
read/write timeouts are sized for many persistent `Subscribe` streams, and glance at the
~0.04 % transient `unauthenticated` resolves under load.
- **R6 (refactor):** no logic bug forced a code change beyond the two harness-payload
fixes; the run surfaced no deadlock or goroutine leak across the ramp.
- **R7 (final tuning + stress):** (1) fix the per-container observability gap — adopt the
otelcol `docker_stats` receiver so Grafana shows per-container CPU/RSS on the contour;
(2) refine the harness — per-player/pooled transports and dropping finished games from
the rotation — and run on hardware **not** shared with the contour; (3) size pools /
GOMAXPROCS / container limits from the CPU-bound peak (~1 core each for backend, gateway,
Postgres at 500 players).
## Re-running
See [`README.md`](README.md). Briefly, from the repo root:
```sh
docker build -f loadtest/Dockerfile -t scrabble-loadtest .
docker run --rm --name scrabble-loadtest --network scrabble-internal \
-e POSTGRES_PASSWORD=… scrabble-loadtest run # add --reset on a re-run
```
The harness stays in the repo for the R7 repeat.
-212
View File
@@ -1,212 +0,0 @@
# R7 — final stress-run trip report
The final pre-release stress pass for [`PRERELEASE.md`](../PRERELEASE.md) R7. It re-runs
the R2 harness (`scrabble/loadtest`) against the **final, refactored system** on a
freshly redeployed contour, to confirm the system holds at scale and to settle the
resource sizing (container limits, `GOMAXPROCS`, pools, rate limits, log levels) before
the Stage 18 prod cutover. Pass bar: **diagnostic + a tuning decision** — the run
"passes" by completing cleanly; the per-container resource profile drives the tuning
recorded below. Companion to the early pass, [`REPORT-R2.md`](REPORT-R2.md).
## What changed since the R2 pass
- **Harness — per-player transports.** Each virtual player now owns its `edge.Client`
(its own `http2.Transport` / h2c connection carrying both its `Subscribe` stream and
its `Execute` calls), instead of all players multiplexing over one shared transport.
R2 traced the ~14 % `transport_error` on `game.state` at 500 players to that single
shared connection's stream limit; per-player connections mirror real clients and
remove the artifact, so this pass measures the system, not the harness.
- **Harness — drop finished games.** `playTurn` reports a finished game and the player
drops it from its rotation, so secondary ops stop hitting `game_finished` on ended
games (the other R2 harness finding).
- **Observability — otelcol `docker_stats`.** cAdvisor (which resolves only the root
cgroup on this host — separate-XFS `/var/lib/docker`) is replaced by the otelcol
`docker_stats` receiver, reading per-container CPU/memory/network from the Docker API.
Per-container panels now populate on the contour host. (`api_version` pinned to 1.44;
the daemon's minimum is 1.40.)
- **Contour — container limits + `GOMAXPROCS`.** `deploy.resources.limits` now bound
every service; the Go services pin `GOMAXPROCS` to their CPU limit so the runtime
matches the cgroup quota. Starting values were generous over the R2 peak; this pass
validates them and settles the agreed sizing (below).
## Method
Unchanged from R2 except for the per-player transports and the dropped-finished-games
refinement above:
- **Driver:** the `scrabble/loadtest` module, run as a one-shot container on the
`scrabble-internal` docker network (reaching `postgres:5432` / `gateway:8081`
directly), capped at `--cpus 3` so the contour keeps the host's spare cores.
- **Seed:** 10 000 durable + 1 000 guest accounts with pre-created sessions written
straight to Postgres (token hash matches `backend/internal/session`).
- **Games:** assembled through the real **invitation** flow, 24 players each, no
robots; variants over scrabble_en / scrabble_ru / erudit_ru.
- **Play:** each player holds a live `Subscribe` stream and, per tick, polls
`game.state`, replays `game.history` and submits a **mid-ranked** legal move generated
locally by the embedded `scrabble-solver`, or passes / exchanges; a fraction exercise
nudge / chat / check-word / draft / profile / stats. A separate **gateway-hammer**
floods `games.list` from one account.
- **Scale:** the same moderate ramp **50 → 200 → 500** concurrent players, 10 min/step.
- **Resource capture:** `docker stats` (docker API) sampled every ~20 s for per-container
CPU/memory; the otelcol **`docker_stats`** receiver → Prometheus → the Grafana
**Scrabble — Resources** dashboard for the same per-container series; `postgres_exporter`
internals and per-service Go runtime metrics.
## Run configuration
```
docker run --rm --cpus=3 --name scrabble-loadtest --network scrabble-internal \
-e POSTGRES_PASSWORD=… scrabble-loadtest \
run --durable 10000 --guest 1000 --steps 50,200,500 --step-dur 10m \
--tick 800ms --hammer-workers 20 --hammer-dur 15s --reset --cleanup
```
Date: 2026-06-10. Contour: the R1-baseline schema, freshly redeployed with the R7
container limits / `GOMAXPROCS` (backend/gateway/postgres capped at 2 cores + 512 MiB,
`GOMAXPROCS=2`) and the `docker_stats` observability. Seeded population removed by
`--cleanup` afterwards.
## Findings
The ramp ran clean to 500 players — no harness crash, no deadlock, `stream errors: 0`
and cleanup removed all 11 000 seeded accounts.
- **Volume (1827 s):** 821 680 edge calls (449.7 req/s incl. the hammer). Real gameplay
at scale: **50 916 committed plays**, 4 817 passes, 2 931 games finished; 165 755
`opponent_moved` + 54 864 `your_turn` events.
- **The per-player transport fix worked.** `game.state` returned `transport_error` on
**3 173 / 127 403 = 2.49 %** of calls — down from R2's ~14 % on the same step. Other
ops were lower still (`game.history` 0.43 %, `game.submit_play` 0.28 %). The residual
is the gateway bursting into its 2-core cap (see the profile below), not the harness.
- **Dropping finished games worked.** `game_finished` on `chat.nudge` / `chat.post` fell
to **35 / 36** (R2: ≈ 3 900 each) — secondary ops no longer hammer ended games.
- **The limiter holds.** The gateway-hammer sent 565 152 `games.list`; **564 979
(99.97 %) were `rate_limited`** (154 ok burst, 19 deadline), p99 = 2 ms, ~309 req/s of
rejections sustained — unchanged from R2.
- **Latency (peak):** `game.state` p50 ≈ 100 ms, p99 in the 2000 ms bucket (max 2549 ms);
`game.submit_play` p50 100 / p99 1000 ms bucket. Lobby ops stayed fast
(invitation / games.list p99 ≤ 10 ms). The p99 tail correlates with the gateway
burst-throttling, not the backend (which stayed at ~0.85 core).
## Resource profile
Per-container peak during step 3 (500 players), with the R7 starting limits in force
(backend/gateway/postgres capped at 2 cores / 512 MiB). Two CPU columns: `docker stats`
samples a ~1 s window (catches bursts); the otelcol `docker_stats` receiver averages over
its 30 s collection interval (smooths them) — they agree within sampling error, which
validates the new observability path.
| container | CPU burst (1 s) | CPU sustained (30 s) | CPU cap | mem peak | mem cap |
|-----------|----------------:|---------------------:|--------:|---------:|--------:|
| scrabble-gateway | **217 %** (at cap) | ~145 % | 200 % | 167 MiB | 512 MiB |
| scrabble-postgres | 138 % | ~153 % | 200 % | 117 MiB | 512 MiB |
| scrabble-backend | 85 % | ~89 % | 200 % | 116 MiB | 512 MiB |
| scrabble-tempo | 33 % | — | (none) | **1024 MiB** (at cap) | 1024 MiB |
| scrabble-otelcol | 11 % | — | (none) | 131 MiB | 512 MiB |
| scrabble-loadtest (harness) | 157 % | — | 300 % | 369 MiB | — |
- **The gateway is the binding constraint.** With one h2c connection per player it draws
~1.45 cores sustained and **bursts to its 2-core cap** at 500 players, throttling
briefly — the source of the 2.49 % `transport_error`. R2 saw only ~0.93 core because
all 500 players shared one connection; the +~0.5 core is the realistic per-connection
overhead (500 separate HTTP/2 connections). This is a sizing fact, not a regression.
- **backend is over-provisioned** (~0.85 core vs a 2-core cap); **postgres** (~1.4 cores)
has headroom; both stayed ≤ 120 MiB.
- **tempo reached its 1 GiB memory cap** (R2: 446 MiB) — an OOM risk under sustained
tracing.
- **Postgres backends peaked at 28**, with the backend pool at its `MaxOpenConns=25` cap.
Cache hit stayed ~100 % (no disk reads); CPU, not I/O, is the limit.
- **docker log volume (30 min):** backend 14.2 MiB, gateway 4.6 MiB, postgres 0.04 MiB —
the backend's per-request latency line at info dominates, and json-file logs had no
rotation.
## Tuning applied
Agreed from the profile (all in `deploy/docker-compose.yml`; no code change — the pool
is already env-driven):
| knob | from | to | why |
|------|------|----|-----|
| gateway CPU + `GOMAXPROCS` | 2 cores / 2 | **3 cores / 3** | it bursts into the 2-core cap at 500 players (the 2.49 % `transport_error`); 3 absorbs the bursts |
| tempo memory | 1 GiB | **2 GiB** | it reached the 1 GiB cap (OOM risk) |
| backend `MAX_OPEN_CONNS` | 25 | **40** | the pool sat at its 25-conn cap at peak; headroom trims the p99 tail |
| docker logs | unbounded | **json-file 10m × 3** | bound the ~14 MiB / 30 min backend log; level stays `info` |
Left as-is: backend / postgres at 2 cores / 512 MiB (peak ~0.85 / ~1.4 cores — headroom
is cheap on the shared host); the per-user rate limiter and `h2cMaxConcurrentStreams=250`
(per-connection now, ~1 stream each — ample) and cache TTLs (no pressure observed).
### Validation re-run
Re-running the **same gradual ramp** (50 → 200 → 500) on the tuned contour confirms the
fix:
- **`game.state` `transport_error` fell to 0.72 %** (853 / 119 051), down from 2.49 % at
2 cores. The latency tail also improved — p99 in the 1000 ms bucket, max 1220 ms (was
the 2000 ms bucket, max 2549 ms).
- The **gateway peaked at ~2 cores** (≈196 % on the 30 s gauge) — now comfortably **under
the 3-core cap**, so it no longer throttles. backend ~1 core, postgres ~1.3 cores.
- **tempo peaked at ~1.27 GiB** — under the new 2 GiB cap (it would have OOM-ed at 1 GiB).
- Drop-finished still holds (`game_finished` on chat 41/42); the limiter still rejects
99.97 % of the hammer at p99 2 ms; `stream errors: 0`.
A separate **burst stress** (a single 100 → 500 jump — 400 players connecting at once)
**pegged the gateway at 3 cores** (≈296 % sustained) and pushed `game.state`
`transport_error` to 9.27 %. The gateway is **connection-CPU-bound and bursty**: average
load is ~1 core, but a mass-simultaneous connection storm saturates whatever single-node
cap it is given. Real arrivals are gradual (the canonical run), where 3 cores has
headroom; the lever for a true arrival spike is **horizontal scaling**, not more cores per
node — carried into the prod recommendation below.
## Prod-sizing recommendation (Stage 18)
The contour is **CPU-bound and gateway-led** at 500 concurrent players. Carry these to the
prod contour env (the same compose, `PROD_*` values):
- **gateway: ≥ 3 cores** per ~500 concurrent players, `GOMAXPROCS` pinned to the limit —
it scales with the **connection count**, not just the request rate; beyond one node's
worth, scale the gateway **horizontally** rather than vertically.
- **backend: ~12 cores**, pool 40 — comfortable; the work is light per request.
- **postgres: ~2 cores / ≥ 512 MiB** — ~1.4 cores at 500 players, 100 % cache hit.
- **tempo: ≥ 2 GiB**; the Go services run under ~170 MiB (256 MiB would suffice, 512 is
safe); pin `GOMAXPROCS` to each CPU limit; keep json-file rotation.
- Memory is not the constraint anywhere; CPU is.
### VPS / VDS sizing (single-host contour)
The whole contour (the app + the observability stack) runs on one host via
`docker-compose`. The tiers below are grounded in the R7 profile (**≈5.5 cores / ≈2.5 GiB
RAM peak at 500 concurrent players**; ≈0.5 GiB idle) and the **measured** on-disk
footprint: prod images ≈2.4 GB; the Tempo volume **3.1 GB at 72 h** retention; Prometheus
≈12 GB at 15 d; the game DB 23 MiB and growing with history. CPU and disk grow; RAM has
the most slack.
| tier | CPU | RAM | disk | handles |
|------|-----|-----|------|---------|
| **Minimum** | 2 cores | 2 GiB | 20 GiB | ~up to ~150 concurrent; lower the compose limits (gateway 1.5 / backend·postgres 1 / tempo 1 GiB) to fit the box |
| **Average** (reasonable load) | 4 cores | 4 GiB | 40 GiB | ~300400 concurrent comfortably; the tested 500 with occasional gateway burst-throttling |
| **Maximum** (worry-free) | 8 cores | 8 GiB | 80 GiB | 500+ concurrent with full gateway burst headroom (its 3-core cap) + room to grow; the compose limits fit as-is |
- The per-service limits in `docker-compose.yml` are tuned for the **Average/Maximum**
target (the gateway alone caps at 3 cores). On the **Minimum** tier, scale them down to
match the host or the caps over-subscribe it.
- **Disk is dominated by observability retention + DB growth.** Tempo (72 h traces) and
Prometheus (15 d metrics) are the main levers — shorten the windows (or move Tempo to
object storage) to cut disk; Postgres grows with game history, so budget for months of
it; container logs are already capped (json-file 10m × 3 ≈ 30 MiB each).
- **RAM** rarely binds: the contour peaks ≈2.5 GiB at 500 players and the sum of all
configured limits is ≈5.6 GiB, so 8 GiB never strains.
- Beyond one host's worth of players, scale the **gateway horizontally** (it is
connection-CPU-bound) rather than ordering an ever-bigger box.
## Re-running
See [`README.md`](README.md). Briefly, from the repo root:
```sh
docker build -f loadtest/Dockerfile -t scrabble-loadtest .
docker run --rm --cpus=3 --name scrabble-loadtest --network scrabble-internal \
-e POSTGRES_PASSWORD=… scrabble-loadtest run --reset --cleanup
```
The harness stays in the repo for future repeats.
+194
View File
@@ -0,0 +1,194 @@
# loadtest — stress trip report
The pre-release stress write-up for [`PRERELEASE.md`](../PRERELEASE.md). It drives the
`scrabble/loadtest` harness against a freshly redeployed test contour to confirm the
system holds at scale and to settle resource sizing before the prod cutover. The harness
stays in the repo for repeats; see [`README.md`](README.md) for how to run it.
This report supersedes the earlier per-phase notes. The harness has been through three
passes: an early diagnostic, a tuning pass that sized container limits / `GOMAXPROCS`, and
this final pass — which **added the per-tile `game.evaluate` preview to the model** (the
hottest real gameplay call, previously unmodelled) and, with it, surfaced and fixed the
**gateway→backend connection-pool bottleneck** described below. The numbers here are from
that final pass.
## What it models
The harness seeds a large account population with pre-created sessions directly in
Postgres, then drives virtual players through the **gateway edge protocol** (h2c) in real
games assembled via the invitation flow. Each player owns its own `edge.Client` (its own
h2c connection, like a real client), holds a live `Subscribe` stream, and per tick polls
`game.state`, replays `game.history`, generates a legal **mid-ranked** move with the
embedded `scrabble-solver`, and submits it (or passes/exchanges). A fraction of ticks
exercise nudge / chat / check-word / draft / profile / stats. A separate **gateway-hammer**
floods `games.list` to verify the rate limiter.
### The evaluate hot path (this pass)
A real client previews every tentative play as the user arranges tiles: the UI fires a
debounced `game.evaluate` (legality + score) on each placement change while it is the
player's turn. Over a single composed word that is **several evaluate calls per turn**
far more than the one `submit_play` — so `game.evaluate` is the single hottest gameplay
request at scale. The earlier passes did not model it at all (they submitted directly),
which understated the real load.
This pass models it: when a player composes a play of *K* newly-placed tiles, it fires one
`evaluate` per landed tile (a growing prefix of the tiles), plus a small number of
full-composition re-previews for reconsideration, spaced by a human-paced gap (the client's
250 ms debounce), then one `draft.save`, then `submit_play`. `--eval=false` reproduces the
pre-evaluate harness for an A/B baseline; `--eval-recon` tunes the reconsideration count.
`game.check_word` is a *different*, manual "look this word up" panel (throttled, on demand)
— not the per-tile call — and is exercised separately as a secondary op.
## Final run (eval-on, after the connection-pool fix)
Contour: backend / postgres capped at 2 cores / 512 MiB (`GOMAXPROCS=2`), gateway at
3 cores / 512 MiB (`GOMAXPROCS=3`), per the tuned `deploy/docker-compose.yml`. Gradual ramp
**50 → 200 → 500** concurrent players, 4 min/step, `--tick 800ms`, gateway-hammer on. The
harness ran as a one-shot container on `scrabble-internal`, capped at `--cpus 3`. The DB was
wiped before the run (`DROP SCHEMA backend CASCADE`); the seeded population was removed by
`--cleanup` afterwards.
Per-operation results at the 500-player peak (740 s, gameplay rows; the hammer row is the
limiter probe):
| operation | count | req/s | p50 | p99 | max | notes |
|-----------|------:|------:|----:|----:|----:|-------|
| game.evaluate | 85 721 | 115.9 | 1 ms | 200 ms | 193 ms | **the hot path** — all ok |
| game.state | 115 926 | 156.7 | 100 ms | 200 ms | 260 ms | transport_error 86 (0.07 %) |
| game.history | 22 258 | 30.1 | 5 ms | 100 ms | 195 ms | all ok |
| draft.save | 23 031 | 31.1 | 2 ms | 200 ms | 194 ms | all ok |
| game.submit_play | 21 704 | 29.3 | 1 ms | 200 ms | 274 ms | ok 3 902; not_your_turn / illegal_play are concurrent-play races (see caveat) |
| hammer:games.list | 522 756 | 706.7 | 1 ms | 2 ms | 53 ms | **99.97 % rate_limited** — limiter holds |
- **Volume:** 802 200 total edge calls (1 084 req/s incl. the hammer; ~377 req/s of real
gameplay). `stream errors: 0`. Live events: 11 199 `opponent_moved`, 4 153 `your_turn`.
- **`game.evaluate` is the dominant gameplay write-path call** at ~116 req/s — second only
to the `game.state` poll — and it is cheap: p50 1 ms, effectively zero errors. The backend
serves it straight from the in-memory live-game cache; on a warm hit it skips the database
entirely (see *Postgres read path* below, which halved its p99 to 100 ms).
- **Latency stayed healthy** under the heavier evaluate load: every gameplay op p99 ≤ 200 ms.
- **The limiter holds** unchanged: 99.97 % of the hammer rejected at p99 2 ms.
### Peak CPU (500 players)
| container | CPU peak | cap |
|-----------|---------:|----:|
| scrabble-postgres | **165 %** (~1.65 cores) | 200 % |
| scrabble-backend | 77 % (~0.77 core) | 200 % |
| scrabble-gateway | **26 %** (~0.26 core) | 300 % |
| scrabble-loadtest (harness) | 42 % | 300 % |
Memory stayed modest everywhere (Go services ≤ ~90 MiB). **Postgres is now the busiest
service** — it has headroom (1.65 of 2 cores) but is the scaling axis. The gateway, after
the fix below, is near-idle.
## The headline finding: gateway→backend connection churn
The gateway proxies every synchronous client call to the single backend host over REST.
Its backend HTTP client used the default transport, whose **`MaxIdleConnsPerHost` is 2**
(`http.DefaultMaxIdleConnsPerHost`). So the gateway kept only **2** keep-alive connections
to the backend and opened — then closed — a fresh TCP connection for almost every other
call. Measured at the gateway's network namespace:
| | gateway→backend sockets |
|---|---|
| before (eval-on, 500 players) | **TIME_WAIT ≈ 26 500**, ESTABLISHED 2 |
| after (eval-on, 500 players) | TIME_WAIT ≈ 0 (steady state), **ESTABLISHED ≈ 225 (reused)** |
26 500 TIME_WAIT sockets is the connection **churn**: ~440 new connections per second,
each a full TCP handshake + teardown, the socket then lingering 60 s. That count sits right
under the ~28 000 ephemeral-port ceiling — the latent cliff that produced the residual
`transport_error` the earlier passes chased on the *client* side (h2c streams) but never
eliminated, because the real cause was here, on the *backend* side.
The fix is one custom `http.Transport` with a wide idle pool
(`gateway/internal/backendclient/client.go`, `backendMaxIdleConns`). Before / after, same
eval-on workload at 500 players:
| metric | before fix | after fix |
|--------|-----------:|----------:|
| gateway→backend TIME_WAIT | ~26 500 | **~0** |
| gateway CPU peak | **175 %** (~1.75 cores) | **26 %** (~0.26 core) |
| game.state p99 | 500 ms | 200 ms |
**The churn was burning ~1.5 gateway cores of pure connection setup/teardown.** Removing it
cut peak gateway CPU ~7× and erased the port-exhaustion cliff. The backend and postgres CPU
are unchanged — they do the real work; only the gateway's wasted overhead disappeared. The
pool settles at ~225 live connections at 500 players; the constant is set to 512 for ~2×
headroom.
## Sizing — why the old "≈150 concurrent / 2-core" figure was a bug, not a floor
The earlier tuning pass concluded the gateway was the binding constraint — "size it for
≥ 3 cores per 500 players, scale it horizontally" — and the single-host "minimum" tier
topped out near ~150 concurrent. **That was sizing around the connection-churn bug.** The
gateway drew ~1.753 cores not from proxying work but from churning backend connections;
the backend behind it sat near-idle the whole time.
With the churn fixed, at **500 concurrent players** the app draws roughly:
- **gateway ≈ 0.26 core** (was ~3) — no longer the constraint,
- **backend ≈ 0.77 core**,
- **postgres ≈ 1.65 cores** — now the busiest, with headroom,
**2.7 app cores total** (down from the ~5.5-core contour peak the tuning pass recorded,
*and* under a heavier, more realistic workload that now includes `game.evaluate`). Postgres,
not the gateway, is the scaling axis.
Revised single-host guidance (app + co-resident observability stack on one box):
| tier | CPU | RAM | handles |
|------|-----|-----|---------|
| **Minimum** | 2 cores | 2 GiB | comfortably the low hundreds of concurrent — the gateway no longer eats cores; postgres + the observability stack set the limit |
| **Average** | 4 cores | 4 GiB | 500 concurrent with headroom |
| **Maximum** | 8 cores | 8 GiB | 500+ with full burst headroom and room to grow |
The gateway's compose limit can drop well below its old 3 cores; it is now connection-pool
bound, not connection-CPU bound. Memory was never the constraint. Disk is still dominated
by observability retention (Tempo, Prometheus) + DB growth — unchanged from before.
## Postgres read path (warm-cache optimization)
Following this pass, `game.evaluate` no longer reads the database on the hot path. An
active game is already resident in the in-memory live-game cache (mutated in place across
moves, evicted only on finish), so the preview answers its seat-membership check from the
cached immutable seat list and scores against the cached engine game — **no `GetGame` on a
warm hit**. `GetGame` itself was also folded from two round-trips (game, then seats) into a
single `LEFT JOIN`. Measured at 500 players, **`game.evaluate` p99 halved (200 → 100 ms)**
and the per-operation query count dropped.
It did **not** cut postgres CPU, and the measurement says why: postgres is **write-bound**,
not read-bound. `pg_stat_user_tables` puts the cost in the per-move `CommitMove`
transaction (a `game_moves` insert plus `games` / `game_players` updates), the debounced
`game_drafts` upserts (~60 k in one run), and the journal replays — not the cheap, indexed,
fully-cached `GetGame` lookups this change removed (one re-run even committed 28 % more
plays, whose extra writes masked the saved reads). Postgres also runs with headroom
(~1.5 of 2 cores), and the gateway fix freed ~3 cores on the box, so the lever if postgres
ever caps is **more cores** (it is CPU-bound, not I/O), not riskier write-path surgery. So
this change is a latency / query-volume win, deliberately not a DB-CPU one.
## Caveat — harness fidelity
The harness's `not_your_turn` and `illegal_play` on `submit_play` are concurrent-play
artifacts, not system errors: it generates a move from a locally replayed board, and a
fast opponent (or a transport hiccup) can move between the state fetch and the submit,
leaving the move out of turn or illegal on the now-changed board. A real client previews
with `evaluate` and only submits a legal, in-turn play. These rejections are cheap domain
outcomes (HTTP-ok with a stable code) and do not change the request *load*, which is what
the run measures. The harness also shares the host CPU with the contour (capped with
`--cpus`); a fully isolated ceiling on separate hardware remains future work.
## Re-running
From the repo root:
```sh
docker build -f loadtest/Dockerfile -t scrabble-loadtest .
docker run --rm --cpus=3 --name scrabble-loadtest --network scrabble-internal \
-e POSTGRES_PASSWORD="$TEST_POSTGRES_PASSWORD" scrabble-loadtest run --reset --cleanup
```
`--eval=false` reproduces the pre-evaluate baseline for comparison. The authoritative hard
reset of the contour DB remains `DROP SCHEMA backend CASCADE` + a backend restart.
+3
View File
@@ -73,6 +73,8 @@ func cmdRun(ctx context.Context, log *slog.Logger, args []string) error {
gpp := fs.Int("games-per-player", 0, "target concurrent games per player (0 => random 3..5)")
tick := fs.Duration("tick", 800*time.Millisecond, "per-player operation cadence")
secProb := fs.Float64("secondary-prob", 0.08, "chance per tick of a non-move operation")
eval := fs.Bool("eval", true, "model the per-tile evaluate preview (the realistic gameplay hot path); --eval=false reproduces the pre-evaluate harness for an A/B baseline")
evalRecon := fs.Int("eval-recon", 1, "extra full-composition evaluate re-previews per play (reconsideration), beyond one per placed tile")
hammerWorkers := fs.Int("hammer-workers", 20, "gateway-hammer concurrent callers (0 disables)")
hammerDur := fs.Duration("hammer-dur", 15*time.Second, "gateway-hammer duration")
reset := fs.Bool("reset", false, "delete prior harness rows before seeding")
@@ -117,6 +119,7 @@ func cmdRun(ctx context.Context, log *slog.Logger, args []string) error {
cfg := scenario.RealisticConfig{
Steps: steps, StepDur: *stepDur, GamesPerPlayer: *gpp,
Tick: *tick, SecondaryProb: *secProb,
Eval: *eval, EvalRecon: *evalRecon,
}
if err := drv.RunRealistic(ctx, pool, cfg); err != nil && !errors.Is(err, context.Canceled) {
return err
+1
View File
@@ -24,6 +24,7 @@ const (
msgSubmitPlay = "game.submit_play"
msgPass = "game.pass"
msgExchange = "game.exchange"
msgEvaluate = "game.evaluate"
msgState = "game.state"
msgHistory = "game.history"
msgGamesList = "games.list"
+27
View File
@@ -63,6 +63,33 @@ func submitPlay(gameID string, tiles []PlayTile) []byte {
return b.FinishedBytes()
}
// evalReq builds an EvalRequest payload (game id plus the tentative newly-placed tiles).
// It mirrors submitPlay's shape — the backend infers the play's orientation the same way —
// so a preview previews exactly what submitting those tiles would score.
func evalReq(gameID string, tiles []PlayTile) []byte {
b := flatbuffers.NewBuilder(256)
gid := b.CreateString(gameID)
offs := make([]flatbuffers.UOffsetT, len(tiles))
for i, t := range tiles {
fb.PlayTileStart(b)
fb.PlayTileAddRow(b, int32(t.Row))
fb.PlayTileAddCol(b, int32(t.Col))
fb.PlayTileAddLetter(b, t.Letter)
fb.PlayTileAddBlank(b, t.Blank)
offs[i] = fb.PlayTileEnd(b)
}
fb.EvalRequestStartTilesVector(b, len(offs))
for i := len(offs) - 1; i >= 0; i-- {
b.PrependUOffsetT(offs[i])
}
tilesVec := b.EndVector(len(offs))
fb.EvalRequestStart(b)
fb.EvalRequestAddGameId(b, gid)
fb.EvalRequestAddTiles(b, tilesVec)
b.Finish(fb.EvalRequestEnd(b))
return b.FinishedBytes()
}
// exchange builds an ExchangeRequest payload swapping the listed rack tiles (alphabet
// indices; 255 a blank).
func exchange(gameID string, tiles []byte) []byte {
+9
View File
@@ -53,6 +53,15 @@ func (c *Client) Exchange(ctx context.Context, token, gameID string, tiles []byt
return decodeMoveResultGame(r.Payload), r.Code, nil
}
// Evaluate previews a tentative play's legality and score without committing it. It is
// the per-tile composition call a real client fires (debounced) on every change while
// arranging a word, so it is the hottest gameplay request at scale. The harness records
// only the result code and latency; an illegal preview is a successful "ok" call.
func (c *Client) Evaluate(ctx context.Context, token, gameID string, tiles []PlayTile) (string, error) {
r, err := c.execute(ctx, token, msgEvaluate, evalReq(gameID, tiles))
return r.Code, err
}
// Nudge prods the opponent whose turn it is.
func (c *Client) Nudge(ctx context.Context, token, gameID string) (string, error) {
r, err := c.execute(ctx, token, msgNudge, gameAction(gameID))
+76 -6
View File
@@ -42,19 +42,35 @@ type RealisticConfig struct {
GamesPerPlayer int // target concurrent games per player; 0 => random 3..5
Tick time.Duration // per-player operation cadence (keeps a player under the per-user limit)
SecondaryProb float64 // chance per tick of a non-move operation
Eval bool // model the per-tile evaluate preview (the gameplay hot path); false reproduces the pre-evaluate harness
EvalRecon int // extra full-composition evaluate re-previews per play, beyond one per placed tile
}
// DefaultRealistic returns the moderate ramp: 50 -> 200
// -> 500 concurrent players, ~12 minutes per step, ~1 op/s per player.
// -> 500 concurrent players, ~12 minutes per step, ~1 op/s per player, with the
// per-tile evaluate preview modelled (the realistic hot path).
func DefaultRealistic() RealisticConfig {
return RealisticConfig{
Steps: []int{50, 200, 500},
StepDur: 12 * time.Minute,
Tick: 800 * time.Millisecond,
SecondaryProb: 0.08,
Eval: true,
EvalRecon: 1,
}
}
// evalGapBase and evalGapSpan bound the modelled pause between successive tile
// placements: the client's 250 ms debounce coalesces faster drags into a single
// evaluate, so a thoughtful player's previews are spaced by a gap drawn from
// [base, base+span] — wide enough that a normal composition stays under the per-user
// rate limit, the way a real one does (the limiter's cost is measured by the hammer,
// not by self-inflicted rejections here).
const (
evalGapBase = 250 * time.Millisecond
evalGapSpan = 500 * time.Millisecond
)
// RunRealistic runs the staged ramp. Each step activates more players (drawn from the
// seeded pool), assembles a cohort of games for them and starts their turn loops; the
// loops run until the whole ramp ends. Players from earlier steps keep playing, so
@@ -128,7 +144,7 @@ func (d *Driver) playerLoop(ctx context.Context, p seed.Account, games []*Game,
d.secondaryOp(ctx, c, p, g, rng)
continue
}
if d.playTurn(ctx, c, p, g, rng) {
if d.playTurn(ctx, c, p, g, cfg, rng) {
active = slices.DeleteFunc(active, func(x *Game) bool { return x == g })
gi = 0
if len(active) == 0 {
@@ -161,10 +177,10 @@ func (d *Driver) subscribeLoop(ctx context.Context, c *edge.Client, p seed.Accou
}
// playTurn plays one turn in g over the player's client when it is the player's
// move: fetch state, replay history, pick a legal move and submit it (or exchange /
// pass). It reports whether the game has finished, so the caller can drop it from the
// rotation.
func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g *Game, rng *rand.Rand) (finished bool) {
// move: fetch state, replay history, pick a legal move, compose it (the per-tile
// evaluate previews a real client fires) and submit it (or exchange / pass). It reports
// whether the game has finished, so the caller can drop it from the rotation.
func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g *Game, cfg RealisticConfig, rng *rand.Rand) (finished bool) {
seat := g.seatOf(p.ID.String())
if seat < 0 {
return false
@@ -196,6 +212,7 @@ func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g
}
switch action.Kind {
case "play":
d.composePlay(ctx, c, p, g, action.Tiles, cfg, rng)
t0 = time.Now()
_, code, _ := c.SubmitPlay(ctx, p.Token, g.ID, action.Tiles)
d.rec.Record("game.submit_play", code, time.Since(t0))
@@ -211,6 +228,59 @@ func (d *Driver) playTurn(ctx context.Context, c *edge.Client, p seed.Account, g
return false
}
// composePlay models a player arranging the chosen play tile by tile before committing:
// the debounced evaluate preview the real client fires on each placement (a growing prefix
// of the tiles), a few full-composition re-previews for reconsideration (recall a tile, try
// another spot), and the single draft persistence the client debounces out. evaluate is the
// hottest gameplay request at scale, so omitting it (the pre-evaluate harness) understated
// the load; cfg.Eval false reproduces that baseline for an A/B comparison. Every step
// honours ctx, so end-of-run cancellation never blocks on a sleep or an in-flight preview.
func (d *Driver) composePlay(ctx context.Context, c *edge.Client, p seed.Account, g *Game, tiles []edge.PlayTile, cfg RealisticConfig, rng *rand.Rand) {
if !cfg.Eval || len(tiles) == 0 {
return
}
// One evaluate per landed tile: the growing prefix mirrors the client re-previewing
// after each placement (an early prefix is often illegal, which is still a successful
// "ok" round trip — exactly the backend work a real composition triggers).
for n := 1; n <= len(tiles); n++ {
if !jitterSleep(ctx, rng, evalGapBase, evalGapSpan) {
return
}
t0 := time.Now()
code, _ := c.Evaluate(ctx, p.Token, g.ID, tiles[:n])
d.rec.Record("game.evaluate", code, time.Since(t0))
}
for r := 0; r < cfg.EvalRecon; r++ {
if !jitterSleep(ctx, rng, evalGapBase, evalGapSpan) {
return
}
t0 := time.Now()
code, _ := c.Evaluate(ctx, p.Token, g.ID, tiles)
d.rec.Record("game.evaluate", code, time.Since(t0))
}
// The client persists the in-progress composition (debounced to one upsert). Its opaque
// JSON content does not affect the call's cost, so a minimal valid shape stands in.
t0 := time.Now()
code, _ := c.DraftSave(ctx, p.Token, g.ID, `{"rack_order":"","board_tiles":[]}`)
d.rec.Record("draft.save", code, time.Since(t0))
}
// jitterSleep pauses for a randomised gap in [base, base+span], modelling the human pause
// between tile placements that the client's debounce coalesces into one evaluate. It
// returns false if ctx is cancelled during the wait, so a composition unwinds promptly at
// end of run.
func jitterSleep(ctx context.Context, rng *rand.Rand, base, span time.Duration) bool {
d := base + time.Duration(rng.Int63n(int64(span)+1))
t := time.NewTimer(d)
defer t.Stop()
select {
case <-ctx.Done():
return false
case <-t.C:
return true
}
}
// secondaryOp exercises one of the non-move edge operations the plan calls out, so
// the run touches nudge / chat / check-word / draft / profile / stats too, over the
// player's own client.
+219 -22
View File
@@ -224,6 +224,7 @@ type Command struct {
// *Command_Notify
// *Command_SendToUser
// *Command_SendToChannel
// *Command_ChatGate
Payload isCommand_Payload `protobuf_oneof:"payload"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
@@ -300,6 +301,15 @@ func (x *Command) GetSendToChannel() *v1.SendToGameChannelRequest {
return nil
}
func (x *Command) GetChatGate() *ChatGateCommand {
if x != nil {
if x, ok := x.Payload.(*Command_ChatGate); ok {
return x.ChatGate
}
}
return nil
}
type isCommand_Payload interface {
isCommand_Payload()
}
@@ -316,12 +326,18 @@ type Command_SendToChannel struct {
SendToChannel *v1.SendToGameChannelRequest `protobuf:"bytes,4,opt,name=send_to_channel,json=sendToChannel,proto3,oneof"`
}
type Command_ChatGate struct {
ChatGate *ChatGateCommand `protobuf:"bytes,5,opt,name=chat_gate,json=chatGate,proto3,oneof"`
}
func (*Command_Notify) isCommand_Payload() {}
func (*Command_SendToUser) isCommand_Payload() {}
func (*Command_SendToChannel) isCommand_Payload() {}
func (*Command_ChatGate) isCommand_Payload() {}
// Ack reports the outcome of the Command with command_id. delivered mirrors the
// connector delivery semantics (false when the kind is not rendered out-of-app, the
// user never started the bot, or no channel is configured); error carries an
@@ -386,6 +402,166 @@ func (x *Ack) GetError() string {
return ""
}
// ChatGateCommand sets a Telegram user's write access in the moderated discussion
// chat. external_id is the user's Telegram identity (as in the backend identities
// table); allow grants the right to write when true and revokes it when false. The
// bot applies it only to a user currently in the chat — it guards on getChatMember,
// so a command for an absent user is a no-op. The gateway emits one whenever the
// user's eligibility may have changed: an admin block or unblock, a chat_muted
// grant or revoke, or a temporary block lapsing.
type ChatGateCommand struct {
state protoimpl.MessageState `protogen:"open.v1"`
ExternalId string `protobuf:"bytes,1,opt,name=external_id,json=externalId,proto3" json:"external_id,omitempty"`
Allow bool `protobuf:"varint,2,opt,name=allow,proto3" json:"allow,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *ChatGateCommand) Reset() {
*x = ChatGateCommand{}
mi := &file_botlink_v1_botlink_proto_msgTypes[5]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *ChatGateCommand) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*ChatGateCommand) ProtoMessage() {}
func (x *ChatGateCommand) ProtoReflect() protoreflect.Message {
mi := &file_botlink_v1_botlink_proto_msgTypes[5]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use ChatGateCommand.ProtoReflect.Descriptor instead.
func (*ChatGateCommand) Descriptor() ([]byte, []int) {
return file_botlink_v1_botlink_proto_rawDescGZIP(), []int{5}
}
func (x *ChatGateCommand) GetExternalId() string {
if x != nil {
return x.ExternalId
}
return ""
}
func (x *ChatGateCommand) GetAllow() bool {
if x != nil {
return x.Allow
}
return false
}
// ChatEligibilityRequest asks whether the Telegram user identified by external_id
// may write in the moderated discussion chat.
type ChatEligibilityRequest struct {
state protoimpl.MessageState `protogen:"open.v1"`
ExternalId string `protobuf:"bytes,1,opt,name=external_id,json=externalId,proto3" json:"external_id,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *ChatEligibilityRequest) Reset() {
*x = ChatEligibilityRequest{}
mi := &file_botlink_v1_botlink_proto_msgTypes[6]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *ChatEligibilityRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*ChatEligibilityRequest) ProtoMessage() {}
func (x *ChatEligibilityRequest) ProtoReflect() protoreflect.Message {
mi := &file_botlink_v1_botlink_proto_msgTypes[6]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use ChatEligibilityRequest.ProtoReflect.Descriptor instead.
func (*ChatEligibilityRequest) Descriptor() ([]byte, []int) {
return file_botlink_v1_botlink_proto_rawDescGZIP(), []int{6}
}
func (x *ChatEligibilityRequest) GetExternalId() string {
if x != nil {
return x.ExternalId
}
return ""
}
// ChatEligibilityResponse is the eligibility answer. registered reports whether the
// external_id maps to an account at all; eligible is the final gate the bot acts on
// (registered and neither admin-suspended nor chat-muted).
type ChatEligibilityResponse struct {
state protoimpl.MessageState `protogen:"open.v1"`
Registered bool `protobuf:"varint,1,opt,name=registered,proto3" json:"registered,omitempty"`
Eligible bool `protobuf:"varint,2,opt,name=eligible,proto3" json:"eligible,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *ChatEligibilityResponse) Reset() {
*x = ChatEligibilityResponse{}
mi := &file_botlink_v1_botlink_proto_msgTypes[7]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *ChatEligibilityResponse) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*ChatEligibilityResponse) ProtoMessage() {}
func (x *ChatEligibilityResponse) ProtoReflect() protoreflect.Message {
mi := &file_botlink_v1_botlink_proto_msgTypes[7]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use ChatEligibilityResponse.ProtoReflect.Descriptor instead.
func (*ChatEligibilityResponse) Descriptor() ([]byte, []int) {
return file_botlink_v1_botlink_proto_rawDescGZIP(), []int{7}
}
func (x *ChatEligibilityResponse) GetRegistered() bool {
if x != nil {
return x.Registered
}
return false
}
func (x *ChatEligibilityResponse) GetEligible() bool {
if x != nil {
return x.Eligible
}
return false
}
var File_botlink_v1_botlink_proto protoreflect.FileDescriptor
const file_botlink_v1_botlink_proto_rawDesc = "" +
@@ -400,22 +576,36 @@ const file_botlink_v1_botlink_proto_rawDesc = "" +
"\x05Hello\x12\x1f\n" +
"\vinstance_id\x18\x01 \x01(\tR\n" +
"instanceId\x12!\n" +
"\fowns_updates\x18\x02 \x01(\bR\vownsUpdates\"\x99\x02\n" +
"\fowns_updates\x18\x02 \x01(\bR\vownsUpdates\"\xde\x02\n" +
"\aCommand\x12\x1d\n" +
"\n" +
"command_id\x18\x01 \x01(\tR\tcommandId\x12=\n" +
"\x06notify\x18\x02 \x01(\v2#.scrabble.telegram.v1.NotifyRequestH\x00R\x06notify\x12K\n" +
"\fsend_to_user\x18\x03 \x01(\v2'.scrabble.telegram.v1.SendToUserRequestH\x00R\n" +
"sendToUser\x12X\n" +
"\x0fsend_to_channel\x18\x04 \x01(\v2..scrabble.telegram.v1.SendToGameChannelRequestH\x00R\rsendToChannelB\t\n" +
"\x0fsend_to_channel\x18\x04 \x01(\v2..scrabble.telegram.v1.SendToGameChannelRequestH\x00R\rsendToChannel\x12C\n" +
"\tchat_gate\x18\x05 \x01(\v2$.scrabble.botlink.v1.ChatGateCommandH\x00R\bchatGateB\t\n" +
"\apayload\"X\n" +
"\x03Ack\x12\x1d\n" +
"\n" +
"command_id\x18\x01 \x01(\tR\tcommandId\x12\x1c\n" +
"\tdelivered\x18\x02 \x01(\bR\tdelivered\x12\x14\n" +
"\x05error\x18\x03 \x01(\tR\x05error2O\n" +
"\x05error\x18\x03 \x01(\tR\x05error\"H\n" +
"\x0fChatGateCommand\x12\x1f\n" +
"\vexternal_id\x18\x01 \x01(\tR\n" +
"externalId\x12\x14\n" +
"\x05allow\x18\x02 \x01(\bR\x05allow\"9\n" +
"\x16ChatEligibilityRequest\x12\x1f\n" +
"\vexternal_id\x18\x01 \x01(\tR\n" +
"externalId\"U\n" +
"\x17ChatEligibilityResponse\x12\x1e\n" +
"\n" +
"registered\x18\x01 \x01(\bR\n" +
"registered\x12\x1a\n" +
"\beligible\x18\x02 \x01(\bR\beligible2\xc4\x01\n" +
"\aBotLink\x12D\n" +
"\x04Link\x12\x1c.scrabble.botlink.v1.FromBot\x1a\x1a.scrabble.botlink.v1.ToBot(\x010\x01B)Z'scrabble/pkg/proto/botlink/v1;botlinkv1b\x06proto3"
"\x04Link\x12\x1c.scrabble.botlink.v1.FromBot\x1a\x1a.scrabble.botlink.v1.ToBot(\x010\x01\x12s\n" +
"\x16ResolveChatEligibility\x12+.scrabble.botlink.v1.ChatEligibilityRequest\x1a,.scrabble.botlink.v1.ChatEligibilityResponseB)Z'scrabble/pkg/proto/botlink/v1;botlinkv1b\x06proto3"
var (
file_botlink_v1_botlink_proto_rawDescOnce sync.Once
@@ -429,31 +619,37 @@ func file_botlink_v1_botlink_proto_rawDescGZIP() []byte {
return file_botlink_v1_botlink_proto_rawDescData
}
var file_botlink_v1_botlink_proto_msgTypes = make([]protoimpl.MessageInfo, 5)
var file_botlink_v1_botlink_proto_msgTypes = make([]protoimpl.MessageInfo, 8)
var file_botlink_v1_botlink_proto_goTypes = []any{
(*FromBot)(nil), // 0: scrabble.botlink.v1.FromBot
(*ToBot)(nil), // 1: scrabble.botlink.v1.ToBot
(*Hello)(nil), // 2: scrabble.botlink.v1.Hello
(*Command)(nil), // 3: scrabble.botlink.v1.Command
(*Ack)(nil), // 4: scrabble.botlink.v1.Ack
(*v1.NotifyRequest)(nil), // 5: scrabble.telegram.v1.NotifyRequest
(*v1.SendToUserRequest)(nil), // 6: scrabble.telegram.v1.SendToUserRequest
(*v1.SendToGameChannelRequest)(nil), // 7: scrabble.telegram.v1.SendToGameChannelRequest
(*ChatGateCommand)(nil), // 5: scrabble.botlink.v1.ChatGateCommand
(*ChatEligibilityRequest)(nil), // 6: scrabble.botlink.v1.ChatEligibilityRequest
(*ChatEligibilityResponse)(nil), // 7: scrabble.botlink.v1.ChatEligibilityResponse
(*v1.NotifyRequest)(nil), // 8: scrabble.telegram.v1.NotifyRequest
(*v1.SendToUserRequest)(nil), // 9: scrabble.telegram.v1.SendToUserRequest
(*v1.SendToGameChannelRequest)(nil), // 10: scrabble.telegram.v1.SendToGameChannelRequest
}
var file_botlink_v1_botlink_proto_depIdxs = []int32{
2, // 0: scrabble.botlink.v1.FromBot.hello:type_name -> scrabble.botlink.v1.Hello
4, // 1: scrabble.botlink.v1.FromBot.ack:type_name -> scrabble.botlink.v1.Ack
3, // 2: scrabble.botlink.v1.ToBot.command:type_name -> scrabble.botlink.v1.Command
5, // 3: scrabble.botlink.v1.Command.notify:type_name -> scrabble.telegram.v1.NotifyRequest
6, // 4: scrabble.botlink.v1.Command.send_to_user:type_name -> scrabble.telegram.v1.SendToUserRequest
7, // 5: scrabble.botlink.v1.Command.send_to_channel:type_name -> scrabble.telegram.v1.SendToGameChannelRequest
0, // 6: scrabble.botlink.v1.BotLink.Link:input_type -> scrabble.botlink.v1.FromBot
1, // 7: scrabble.botlink.v1.BotLink.Link:output_type -> scrabble.botlink.v1.ToBot
7, // [7:8] is the sub-list for method output_type
6, // [6:7] is the sub-list for method input_type
6, // [6:6] is the sub-list for extension type_name
6, // [6:6] is the sub-list for extension extendee
0, // [0:6] is the sub-list for field type_name
2, // 0: scrabble.botlink.v1.FromBot.hello:type_name -> scrabble.botlink.v1.Hello
4, // 1: scrabble.botlink.v1.FromBot.ack:type_name -> scrabble.botlink.v1.Ack
3, // 2: scrabble.botlink.v1.ToBot.command:type_name -> scrabble.botlink.v1.Command
8, // 3: scrabble.botlink.v1.Command.notify:type_name -> scrabble.telegram.v1.NotifyRequest
9, // 4: scrabble.botlink.v1.Command.send_to_user:type_name -> scrabble.telegram.v1.SendToUserRequest
10, // 5: scrabble.botlink.v1.Command.send_to_channel:type_name -> scrabble.telegram.v1.SendToGameChannelRequest
5, // 6: scrabble.botlink.v1.Command.chat_gate:type_name -> scrabble.botlink.v1.ChatGateCommand
0, // 7: scrabble.botlink.v1.BotLink.Link:input_type -> scrabble.botlink.v1.FromBot
6, // 8: scrabble.botlink.v1.BotLink.ResolveChatEligibility:input_type -> scrabble.botlink.v1.ChatEligibilityRequest
1, // 9: scrabble.botlink.v1.BotLink.Link:output_type -> scrabble.botlink.v1.ToBot
7, // 10: scrabble.botlink.v1.BotLink.ResolveChatEligibility:output_type -> scrabble.botlink.v1.ChatEligibilityResponse
9, // [9:11] is the sub-list for method output_type
7, // [7:9] is the sub-list for method input_type
7, // [7:7] is the sub-list for extension type_name
7, // [7:7] is the sub-list for extension extendee
0, // [0:7] is the sub-list for field type_name
}
func init() { file_botlink_v1_botlink_proto_init() }
@@ -469,6 +665,7 @@ func file_botlink_v1_botlink_proto_init() {
(*Command_Notify)(nil),
(*Command_SendToUser)(nil),
(*Command_SendToChannel)(nil),
(*Command_ChatGate)(nil),
}
type x struct{}
out := protoimpl.TypeBuilder{
@@ -476,7 +673,7 @@ func file_botlink_v1_botlink_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_botlink_v1_botlink_proto_rawDesc), len(file_botlink_v1_botlink_proto_rawDesc)),
NumEnums: 0,
NumMessages: 5,
NumMessages: 8,
NumExtensions: 0,
NumServices: 1,
},
+34
View File
@@ -20,6 +20,13 @@ service BotLink {
// Link opens the single bot <-> gateway stream. The first client message is
// Hello; thereafter the client sends one Ack per received Command.
rpc Link(stream FromBot) returns (stream ToBot);
// ResolveChatEligibility answers whether the Telegram user identified by
// external_id may write in the moderated discussion chat: registered with an
// account and neither admin-suspended nor chat-muted. The bot calls it over the
// same mTLS channel when a user joins the chat, to decide whether to grant the
// write permission. Delivery of the answer is request/response (not best-effort).
rpc ResolveChatEligibility(ChatEligibilityRequest) returns (ChatEligibilityResponse);
}
// FromBot is a message the bot sends to the gateway: the opening Hello, then one
@@ -53,6 +60,7 @@ message Command {
scrabble.telegram.v1.NotifyRequest notify = 2;
scrabble.telegram.v1.SendToUserRequest send_to_user = 3;
scrabble.telegram.v1.SendToGameChannelRequest send_to_channel = 4;
ChatGateCommand chat_gate = 5;
}
}
@@ -65,3 +73,29 @@ message Ack {
bool delivered = 2;
string error = 3;
}
// ChatGateCommand sets a Telegram user's write access in the moderated discussion
// chat. external_id is the user's Telegram identity (as in the backend identities
// table); allow grants the right to write when true and revokes it when false. The
// bot applies it only to a user currently in the chat it guards on getChatMember,
// so a command for an absent user is a no-op. The gateway emits one whenever the
// user's eligibility may have changed: an admin block or unblock, a chat_muted
// grant or revoke, or a temporary block lapsing.
message ChatGateCommand {
string external_id = 1;
bool allow = 2;
}
// ChatEligibilityRequest asks whether the Telegram user identified by external_id
// may write in the moderated discussion chat.
message ChatEligibilityRequest {
string external_id = 1;
}
// ChatEligibilityResponse is the eligibility answer. registered reports whether the
// external_id maps to an account at all; eligible is the final gate the bot acts on
// (registered and neither admin-suspended nor chat-muted).
message ChatEligibilityResponse {
bool registered = 1;
bool eligible = 2;
}
+51 -2
View File
@@ -26,7 +26,8 @@ import (
const _ = grpc.SupportPackageIsVersion9
const (
BotLink_Link_FullMethodName = "/scrabble.botlink.v1.BotLink/Link"
BotLink_Link_FullMethodName = "/scrabble.botlink.v1.BotLink/Link"
BotLink_ResolveChatEligibility_FullMethodName = "/scrabble.botlink.v1.BotLink/ResolveChatEligibility"
)
// BotLinkClient is the client API for BotLink service.
@@ -41,6 +42,12 @@ type BotLinkClient interface {
// Link opens the single bot <-> gateway stream. The first client message is
// Hello; thereafter the client sends one Ack per received Command.
Link(ctx context.Context, opts ...grpc.CallOption) (grpc.BidiStreamingClient[FromBot, ToBot], error)
// ResolveChatEligibility answers whether the Telegram user identified by
// external_id may write in the moderated discussion chat: registered with an
// account and neither admin-suspended nor chat-muted. The bot calls it over the
// same mTLS channel when a user joins the chat, to decide whether to grant the
// write permission. Delivery of the answer is request/response (not best-effort).
ResolveChatEligibility(ctx context.Context, in *ChatEligibilityRequest, opts ...grpc.CallOption) (*ChatEligibilityResponse, error)
}
type botLinkClient struct {
@@ -64,6 +71,16 @@ func (c *botLinkClient) Link(ctx context.Context, opts ...grpc.CallOption) (grpc
// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
type BotLink_LinkClient = grpc.BidiStreamingClient[FromBot, ToBot]
func (c *botLinkClient) ResolveChatEligibility(ctx context.Context, in *ChatEligibilityRequest, opts ...grpc.CallOption) (*ChatEligibilityResponse, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(ChatEligibilityResponse)
err := c.cc.Invoke(ctx, BotLink_ResolveChatEligibility_FullMethodName, in, out, cOpts...)
if err != nil {
return nil, err
}
return out, nil
}
// BotLinkServer is the server API for BotLink service.
// All implementations must embed UnimplementedBotLinkServer
// for forward compatibility.
@@ -76,6 +93,12 @@ type BotLinkServer interface {
// Link opens the single bot <-> gateway stream. The first client message is
// Hello; thereafter the client sends one Ack per received Command.
Link(grpc.BidiStreamingServer[FromBot, ToBot]) error
// ResolveChatEligibility answers whether the Telegram user identified by
// external_id may write in the moderated discussion chat: registered with an
// account and neither admin-suspended nor chat-muted. The bot calls it over the
// same mTLS channel when a user joins the chat, to decide whether to grant the
// write permission. Delivery of the answer is request/response (not best-effort).
ResolveChatEligibility(context.Context, *ChatEligibilityRequest) (*ChatEligibilityResponse, error)
mustEmbedUnimplementedBotLinkServer()
}
@@ -89,6 +112,9 @@ type UnimplementedBotLinkServer struct{}
func (UnimplementedBotLinkServer) Link(grpc.BidiStreamingServer[FromBot, ToBot]) error {
return status.Errorf(codes.Unimplemented, "method Link not implemented")
}
func (UnimplementedBotLinkServer) ResolveChatEligibility(context.Context, *ChatEligibilityRequest) (*ChatEligibilityResponse, error) {
return nil, status.Errorf(codes.Unimplemented, "method ResolveChatEligibility not implemented")
}
func (UnimplementedBotLinkServer) mustEmbedUnimplementedBotLinkServer() {}
func (UnimplementedBotLinkServer) testEmbeddedByValue() {}
@@ -117,13 +143,36 @@ func _BotLink_Link_Handler(srv interface{}, stream grpc.ServerStream) error {
// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
type BotLink_LinkServer = grpc.BidiStreamingServer[FromBot, ToBot]
func _BotLink_ResolveChatEligibility_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(ChatEligibilityRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(BotLinkServer).ResolveChatEligibility(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: BotLink_ResolveChatEligibility_FullMethodName,
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(BotLinkServer).ResolveChatEligibility(ctx, req.(*ChatEligibilityRequest))
}
return interceptor(ctx, in, info, handler)
}
// BotLink_ServiceDesc is the grpc.ServiceDesc for BotLink service.
// It's only intended for direct use with grpc.RegisterService,
// and not to be introspected or modified (even as a copy)
var BotLink_ServiceDesc = grpc.ServiceDesc{
ServiceName: "scrabble.botlink.v1.BotLink",
HandlerType: (*BotLinkServer)(nil),
Methods: []grpc.MethodDesc{},
Methods: []grpc.MethodDesc{
{
MethodName: "ResolveChatEligibility",
Handler: _BotLink_ResolveChatEligibility_Handler,
},
},
Streams: []grpc.StreamDesc{
{
StreamName: "Link",
+13 -3
View File
@@ -29,6 +29,8 @@ import (
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/trace"
"scrabble/pkg/version"
)
// Exporter selectors supported per signal.
@@ -95,9 +97,7 @@ func New(ctx context.Context, cfg Config) (*Runtime, error) {
return nil, err
}
res, err := resource.New(ctx, resource.WithAttributes(
attribute.String("service.name", cfg.ServiceName),
))
res, err := serviceResource(ctx, cfg)
if err != nil {
return nil, fmt.Errorf("telemetry: build resource: %w", err)
}
@@ -122,6 +122,16 @@ func New(ctx context.Context, cfg Config) (*Runtime, error) {
return &Runtime{tracerProvider: tracerProvider, meterProvider: meterProvider}, nil
}
// serviceResource builds the OpenTelemetry resource describing this service: its
// service.name and the service.version stamped into the binary at build time
// (pkg/version, set from the git tag by the deploy).
func serviceResource(ctx context.Context, cfg Config) (*resource.Resource, error) {
return resource.New(ctx, resource.WithAttributes(
attribute.String("service.name", cfg.ServiceName),
attribute.String("service.version", version.Version),
))
}
// TracerProvider returns the runtime tracer provider, or the global one when r is
// not initialised.
func (r *Runtime) TracerProvider() trace.TracerProvider {
+21
View File
@@ -4,6 +4,8 @@ import (
"context"
"testing"
"time"
"scrabble/pkg/version"
)
// TestConfigValidate covers the supported and rejected exporter selections.
@@ -82,3 +84,22 @@ func TestNilRuntime(t *testing.T) {
t.Errorf("nil runtime Shutdown: %v", err)
}
}
// TestServiceResource checks the resource carries service.name and the embedded
// service.version (pkg/version, stamped at build time).
func TestServiceResource(t *testing.T) {
res, err := serviceResource(context.Background(), DefaultConfig("svc"))
if err != nil {
t.Fatalf("serviceResource: %v", err)
}
attrs := map[string]string{}
for _, kv := range res.Attributes() {
attrs[string(kv.Key)] = kv.Value.AsString()
}
if attrs["service.name"] != "svc" {
t.Errorf("service.name = %q, want svc", attrs["service.name"])
}
if attrs["service.version"] != version.Version {
t.Errorf("service.version = %q, want %q", attrs["service.version"], version.Version)
}
}
+10
View File
@@ -0,0 +1,10 @@
// Package version exposes the build version stamped into every Scrabble service
// binary. The default is "dev"; release builds override it through the linker
// (`go build -ldflags "-X scrabble/pkg/version.Version=<value>"`), wired from the
// VERSION build-arg in each service Dockerfile, which the deploy sets to the git
// tag (`git describe --tags`). It surfaces as the OpenTelemetry service.version
// resource attribute (see pkg/telemetry) and the SPA About screen.
package version
// Version is the build version, "dev" unless overridden at link time.
var Version = "dev"
+4 -2
View File
@@ -19,8 +19,10 @@ COPY platform/telegram ./platform/telegram
# Reduce the workspace to what the platform needs: only pkg + platform/telegram.
RUN go work edit -dropuse=./backend -dropuse=./gateway -dropuse=./loadtest -dropreplace=scrabble/gateway@v0.0.0 -dropreplace=scrabble-solver
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/validator ./platform/telegram/cmd/validator
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/bot ./platform/telegram/cmd/bot
# VERSION (the deploy passes the git tag) is stamped into both binaries via the linker.
ARG VERSION=dev
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags "-X scrabble/pkg/version.Version=${VERSION}" -o /out/validator ./platform/telegram/cmd/validator
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags "-X scrabble/pkg/version.Version=${VERSION}" -o /out/bot ./platform/telegram/cmd/bot
# --- validator (home) --------------------------------------------------------
FROM gcr.io/distroless/static-debian12:nonroot AS validator
+28 -1
View File
@@ -42,6 +42,27 @@ Telegram identity to an account from a browser. Both map a rejection to gRPC
launch button; a deep-link payload routes the launch to a game / invitation / friend
code. This is **self-contained** — the bot never calls back into the game, so `/start`
onboarding works even when the game is down.
- **Moderated-chat gating.** When `TELEGRAM_CHAT_ID` names a channel's linked discussion
group, the bot gates who may write there. The group **allows sending by default** (a
human setting) and the bot only **restricts** — Telegram intersects the chat default with
each user's permission, so a per-user grant cannot exceed a deny-by-default group, and the
gate must mute the ineligible rather than grant the eligible. On a `chat_member` event the
bot asks the gateway (`ResolveChatEligibility`) and **mutes** a member who is not registered
or is admin-suspended or `chat_muted`, **un-mutes** an eligible one it had muted, and leaves
an already-allowed eligible member untouched (it acts only when the state differs, so it is
idempotent and skips its own change). When an operator blocks/unblocks an account, toggles
its `chat_muted` role, or a user first registers, the gateway pushes a `ChatGate` command and
the bot applies it — but only to a member currently in the chat (it probes one user with
`getChatMember`, since bots cannot list members). The bot must be an **administrator** there
with the **"Ban users"** right (the Bot API `can_restrict_members`), and it subscribes to
`chat_member` updates, which Telegram delivers only to a chat admin.
- **Promo bot (optional).** When `TELEGRAM_PROMO_BOT_TOKEN` is set, the container also
runs a **second, standalone** bot whose only job is to answer `/start` with a localized
message and a button that opens the **main** bot's Mini App. The button is a **URL** to
the main bot's direct link (`TELEGRAM_BOT_LINK`, the same link the UI uses) with
`?startapp` — a `web_app` button would launch under the promo bot's identity (its token
would sign the initData), which the main bot's validator rejects. It is fully
self-contained: no bot-link, no gateway, no game.
- **Rate limiting.** Outbound sends are throttled (`TELEGRAM_SEND_RATE_PER_SECOND`,
default 25) to respect the Bot API flood limits.
@@ -57,7 +78,9 @@ parsing is Telegram-specific.
gateway also implements `SendToUser` / `SendToGameChannel` as the backend's admin
relay.
- `pkg/proto/botlink/v1`, service `BotLink` — the reverse bidi stream the **bot** dials
on the gateway (`Hello` / `Command` / `Ack`). Generated Go is committed under `pkg`.
on the gateway (`Hello` / `Command` / `Ack`), now also carrying a `ChatGateCommand` (set
a user's chat write access) and a unary `ResolveChatEligibility` (the bot's join-time
query) over the same mTLS channel. Generated Go is committed under `pkg`.
## Deep-link scheme
@@ -101,6 +124,10 @@ Bot (`cmd/bot`):
| `TELEGRAM_BOTLINK_SERVER_NAME` | — (required) | the gateway certificate's expected SNI / CN |
| `TELEGRAM_BOTLINK_TLS_CERT` / `_KEY` / `_CA` | — (required) | the bot client cert, its key, and the CA that signs the gateway server cert |
| `TELEGRAM_GAME_CHANNEL_ID` | — | the bot's game channel chat id for `SendToGameChannel` |
| `TELEGRAM_CHAT_ID` | — | the moderated discussion chat id (a channel's linked group); empty disables chat gating |
| `TELEGRAM_PROMO_BOT_TOKEN` | — | the optional standalone promo bot's token; empty disables it |
| `TELEGRAM_BOT_USERNAME` | — | the main bot's @username without the @ (promo message); required when the promo bot runs |
| `TELEGRAM_BOT_LINK` | — | the main bot's Mini App link for the promo button (the UI's `VITE_TELEGRAM_LINK`); required when the promo bot runs |
| `TELEGRAM_OWNS_UPDATES` | `true` | run the exclusive `getUpdates` long-poll (one bot per token) |
| `TELEGRAM_SEND_RATE_PER_SECOND` | `25` | outbound Bot API send cap (0 disables) |
| `TELEGRAM_INSTANCE_ID` | hostname | bot identity reported to the gateway |
+42 -2
View File
@@ -22,6 +22,7 @@ import (
"scrabble/platform/telegram/internal/bot"
"scrabble/platform/telegram/internal/botlink"
"scrabble/platform/telegram/internal/config"
"scrabble/platform/telegram/internal/promobot"
)
// telemetryShutdownTimeout bounds the OpenTelemetry flush during process exit.
@@ -70,6 +71,7 @@ func run(ctx context.Context, cfg config.BotConfig, logger *zap.Logger) error {
TestEnv: cfg.TestEnv,
MiniAppURL: cfg.MiniAppURL,
SendRatePerSecond: cfg.SendRatePerSecond,
ChatID: cfg.ChatID,
}, logger)
if err != nil {
return err
@@ -80,19 +82,52 @@ func run(ctx context.Context, cfg config.BotConfig, logger *zap.Logger) error {
return err
}
exec := botlink.NewExecutor(b, cfg.GameChannelID, logger)
client := botlink.NewClient(botlink.ClientConfig{
client, err := botlink.NewClient(botlink.ClientConfig{
GatewayAddr: cfg.BotLink.GatewayAddr,
InstanceID: cfg.BotLink.InstanceID,
OwnsUpdates: cfg.OwnsUpdates,
Creds: credentials.NewTLS(tlsCfg),
ReconnectDelay: cfg.BotLink.ReconnectDelay,
}, exec, logger)
if err != nil {
return err
}
defer func() { _ = client.Close() }()
// The chat-join eligibility query rides the same bot-link connection; wire it into
// the bot after the client is built — the late binding that breaks the bot <->
// client construction cycle.
b.SetEligibilityResolver(client.ResolveChatEligibility)
// The optional standalone promo bot: a second bot (its own token) that only answers
// /start with a button opening the main bot's Mini App. It is self-contained — no
// bot-link, no gateway — so onboarding works even when the game is down.
var promo *promobot.Bot
if cfg.PromoBotToken != "" {
// The promo bot is auxiliary and shares this process with the main bot, so its
// construction failure (a bad or unreachable promo token — tgbot.New validates it
// with getMe) is logged and the promo bot is skipped, never fatal: it must not
// take the main game bot down with it.
promo, err = promobot.New(promobot.Config{
Token: cfg.PromoBotToken,
APIBaseURL: cfg.APIBaseURL,
TestEnv: cfg.TestEnv,
BotUsername: cfg.BotUsername,
BotLinkURL: cfg.BotLinkURL,
SendRatePerSecond: cfg.SendRatePerSecond,
}, logger)
if err != nil {
logger.Error("promo bot disabled: construction failed (the main bot is unaffected)", zap.Error(err))
promo = nil
}
}
logger.Info("telegram bot starting",
zap.String("gateway", cfg.BotLink.GatewayAddr),
zap.String("miniapp_url", cfg.MiniAppURL),
zap.Bool("owns_updates", cfg.OwnsUpdates),
zap.Bool("test_env", cfg.TestEnv))
zap.Bool("test_env", cfg.TestEnv),
zap.Bool("chat_gating", cfg.ChatID != 0),
zap.Bool("promo_bot", promo != nil))
var wg sync.WaitGroup
// The long-poll holds the exclusive getUpdates lease (one bot per token); a bot
@@ -105,6 +140,11 @@ func run(ctx context.Context, cfg config.BotConfig, logger *zap.Logger) error {
logger.Error("bot-link client stopped", zap.Error(err))
}
})
// The promo bot runs its own getUpdates long-poll on its own token (no 409 with
// the main bot's lease).
if promo != nil {
wg.Go(func() { promo.Run(ctx) })
}
<-ctx.Done()
wg.Wait()
+256 -2
View File
@@ -8,6 +8,7 @@ package bot
import (
"context"
"net/url"
"strconv"
"strings"
tgbot "github.com/go-telegram/bot"
@@ -30,8 +31,19 @@ type Config struct {
// Telegram Bot API flood limits; 0 disables the limiter. The burst equals the
// per-second rate.
SendRatePerSecond int
// ChatID is the moderated discussion chat the bot gates write access in; 0
// disables chat gating (and the chat_member long-poll subscription). Gating needs
// the bot to be an administrator there with the restrict-members right.
ChatID int64
}
// EligibilityResolver answers whether the Telegram user identified by externalID
// (the decimal user id) may write in the moderated chat: registered and neither
// admin-suspended nor chat-muted. The bot calls it when a user joins the chat. It is
// late-bound (SetEligibilityResolver) because it is backed by the bot-link client,
// which is built after the bot.
type EligibilityResolver func(ctx context.Context, externalID string) (eligible bool, err error)
// Bot wraps a Telegram Bot API client and the Mini App launch URL.
type Bot struct {
api *tgbot.Bot
@@ -40,6 +52,14 @@ type Bot struct {
// limiter throttles outbound sends to stay under the Bot API flood limits; nil
// disables throttling.
limiter *rate.Limiter
// chatID is the moderated discussion chat (0 disables gating).
chatID int64
// botID is the bot's own Telegram user id (resolved at startup); it skips the
// chat_member updates the bot's own restrict actions generate — the grant loop guard.
botID int64
// eligibility resolves a joining user's chat write eligibility; nil leaves a
// joiner muted (fail-closed) until it is wired.
eligibility EligibilityResolver
}
// New builds the bot wrapper, registering the /start handler and a default handler
@@ -49,15 +69,25 @@ func New(cfg Config, log *zap.Logger) (*Bot, error) {
if log == nil {
log = zap.NewNop()
}
t := &Bot{miniAppURL: cfg.MiniAppURL, log: log}
t := &Bot{miniAppURL: cfg.MiniAppURL, log: log, chatID: cfg.ChatID}
if cfg.SendRatePerSecond > 0 {
t.limiter = rate.NewLimiter(rate.Limit(cfg.SendRatePerSecond), cfg.SendRatePerSecond)
}
opts := []tgbot.Option{
tgbot.WithDefaultHandler(t.handleStart),
tgbot.WithDefaultHandler(t.handleUpdate),
tgbot.WithMessageTextHandler("/start", tgbot.MatchTypePrefix, t.handleStart),
}
if cfg.ChatID != 0 {
// chat_member updates are off by default; subscribe explicitly (alongside
// messages) so the bot sees joins in the moderated chat. The bot must also be an
// administrator there for Telegram to deliver them.
opts = append(opts, tgbot.WithAllowedUpdates(tgbot.AllowedUpdates{
models.AllowedUpdateMessage,
models.AllowedUpdateMyChatMember,
models.AllowedUpdateChatMember,
}))
}
if cfg.TestEnv {
// Route to the Bot API test environment (.../bot<token>/test/METHOD).
opts = append(opts, tgbot.UseTestEnvironment())
@@ -90,9 +120,39 @@ func (t *Bot) Run(ctx context.Context) {
}); err != nil {
t.log.Warn("set menu button failed", zap.Error(err))
}
if t.chatID != 0 {
t.logChatAdminStatus(ctx)
}
t.api.Start(ctx)
}
// logChatAdminStatus checks, at startup, whether the bot can actually gate the
// moderated chat — it must be an administrator there with the restrict-members
// ("Ban users") right, or Telegram delivers no chat_member updates and restricts
// fail. It logs a prominent warning when the prerequisite is missing (the common
// misconfiguration), so the cause is visible without reproducing a join.
func (t *Bot) logChatAdminStatus(ctx context.Context) {
me, err := t.api.GetMe(ctx)
if err != nil {
t.log.Warn("chat self-check: getMe failed", zap.Error(err))
return
}
t.botID = me.ID
m, err := t.api.GetChatMember(ctx, &tgbot.GetChatMemberParams{ChatID: t.chatID, UserID: me.ID})
if err != nil {
t.log.Warn("chat gating self-check failed: the bot cannot read the chat — is it added and is TELEGRAM_CHAT_ID the discussion group id?",
zap.Int64("chat_id", t.chatID), zap.Error(err))
return
}
canRestrict := m.Type == models.ChatMemberTypeAdministrator && m.Administrator.CanRestrictMembers
if !canRestrict {
t.log.Warn(`chat gating WILL NOT WORK: the bot must be an administrator with the restrict-members ("Ban users") right`,
zap.Int64("chat_id", t.chatID), zap.String("bot_status", string(m.Type)))
return
}
t.log.Info("chat gating ready: bot is an admin with the restrict-members right", zap.Int64("chat_id", t.chatID))
}
// Notify sends a notification message with a Mini App launch button that opens the
// app at startParam (empty opens the lobby).
func (t *Bot) Notify(ctx context.Context, chatID int64, text, buttonText, startParam string) error {
@@ -131,6 +191,13 @@ func (t *Bot) handleStart(ctx context.Context, api *tgbot.Bot, update *models.Up
if update.Message == nil {
return
}
// Reply only in a private chat: the Mini App launch button is an inline web_app
// button, which Telegram permits only in private chats — replying to a group message
// (the bot is an admin in the moderated chat and now receives its messages) fails with
// BUTTON_TYPE_INVALID. In the group the bot only manages permissions, it never chats.
if update.Message.Chat.Type != models.ChatTypePrivate {
return
}
startParam := startPayload(update.Message.Text)
if _, err := api.SendMessage(ctx, &tgbot.SendMessageParams{
ChatID: update.Message.Chat.ID,
@@ -176,3 +243,190 @@ func startPayload(text string) string {
}
return strings.TrimSpace(strings.TrimPrefix(text, cmd))
}
// handleUpdate is the default-handler dispatcher: a chat-member change in the
// moderated chat drives the write-access gate; anything else is treated as a message
// and gets the Mini App launch reply.
func (t *Bot) handleUpdate(ctx context.Context, api *tgbot.Bot, update *models.Update) {
if update.ChatMember != nil {
t.handleChatMember(ctx, update.ChatMember)
return
}
t.handleStart(ctx, api, update)
}
// SetEligibilityResolver wires the chat-eligibility resolver after construction (the
// bot-link client backing it is built after the bot).
func (t *Bot) SetEligibilityResolver(resolve EligibilityResolver) {
t.eligibility = resolve
}
// handleChatMember keeps a chat member's write access in sync with their eligibility.
// The chat allows sending by default, so the bot mutes an ineligible member (not
// registered, or admin-suspended, or chat_muted) and restores an eligible one it had
// muted; an eligible member that can already send is left untouched. It acts only when
// the current state differs from the desired one, so it is idempotent and does not
// re-act on its own change; a resolve failure makes no change.
func (t *Bot) handleChatMember(ctx context.Context, cm *models.ChatMemberUpdated) {
user := chatMemberUser(cm.NewChatMember)
var uid int64
if user != nil {
uid = user.ID
}
// Log every chat_member update the bot receives: the one place to see whether
// Telegram delivers joins, for which chat, the transition, who performed it, and the
// new member's send/membership state.
canSend, isMember := restrictedSendState(cm.NewChatMember)
t.log.Debug("chat_member update",
zap.Int64("chat_id", cm.Chat.ID),
zap.Int64("configured_chat_id", t.chatID),
zap.Int64("user_id", uid),
zap.Int64("actor_id", cm.From.ID),
zap.String("old_status", string(cm.OldChatMember.Type)),
zap.String("new_status", string(cm.NewChatMember.Type)),
zap.Bool("new_can_send", canSend),
zap.Bool("new_is_member", isMember))
if t.chatID == 0 || cm.Chat.ID != t.chatID {
return
}
if user == nil || user.IsBot {
return
}
// Loop guard: the bot's own restrict re-fires a chat_member update whose performer is
// the bot; skip those so a grant never re-triggers itself.
if t.botID != 0 && cm.From.ID == t.botID {
return
}
// The chat allows sending by default and the bot only restricts: Telegram intersects
// the chat default with the per-user permission, so a per-user grant cannot exceed a
// deny-by-default — the gate must mute the ineligible, not grant the eligible.
// Determine whether the user is in the chat and can currently send: a plain member
// follows the permissive default; a restricted member can send only with
// CanSendMessages, and only while a member.
var inChat, currentlyCanSend bool
switch cm.NewChatMember.Type {
case models.ChatMemberTypeMember:
inChat, currentlyCanSend = true, true
case models.ChatMemberTypeRestricted:
inChat, currentlyCanSend = isMember, canSend
default:
return // left / kicked / administrator / owner — not a member to gate
}
if !inChat {
return
}
if t.eligibility == nil {
t.log.Warn("chat access: eligibility resolver not wired", zap.Int64("user_id", uid))
return
}
eligible, err := t.eligibility(ctx, strconv.FormatInt(user.ID, 10))
if err != nil {
t.log.Warn("chat access eligibility failed", zap.Int64("user_id", user.ID), zap.Error(err))
return
}
t.log.Debug("chat access evaluated",
zap.Int64("user_id", user.ID), zap.Bool("eligible", eligible), zap.Bool("can_send", currentlyCanSend))
// Desired: an eligible user may send, an ineligible one may not. Act only when the
// current state differs — idempotent, a no-op for the common eligible member, and it
// keeps the bot from re-acting on its own change.
if eligible == currentlyCanSend {
return
}
if err := t.setChatWrite(ctx, user.ID, eligible); err != nil {
t.log.Warn("set chat write failed",
zap.Int64("user_id", user.ID), zap.Bool("can_send", eligible), zap.Error(err))
return
}
t.log.Info("chat access applied", zap.Int64("user_id", user.ID), zap.Bool("can_send", eligible))
}
// ApplyChatGate applies a chat-gate command (an admin block/unblock or chat_muted
// change relayed by the gateway): it sets the user's write access, but only when they
// are currently in the chat. Bots cannot list members, so it probes the single user
// with getChatMember and is a no-op when they are absent (left/kicked) or an
// administrator (who cannot be restricted). It reports whether a restriction was
// applied.
func (t *Bot) ApplyChatGate(ctx context.Context, userID int64, allow bool) (bool, error) {
if t.chatID == 0 {
return false, nil
}
member, err := t.api.GetChatMember(ctx, &tgbot.GetChatMemberParams{ChatID: t.chatID, UserID: userID})
if err != nil {
return false, err
}
switch member.Type {
case models.ChatMemberTypeMember, models.ChatMemberTypeRestricted:
if err := t.setChatWrite(ctx, userID, allow); err != nil {
return false, err
}
t.log.Info("chat gate applied", zap.Int64("user_id", userID), zap.Bool("allow", allow))
return true, nil
default:
t.log.Debug("chat gate: user not in chat, skipped", zap.Int64("user_id", userID), zap.String("status", string(member.Type)))
return false, nil // absent, or an admin/owner who cannot be restricted
}
}
// setChatWrite restricts the user in the moderated chat to either the full send
// permission set (allow) or none (mute); the non-send permissions stay at their
// default-deny either way.
func (t *Bot) setChatWrite(ctx context.Context, userID int64, allow bool) error {
perms := models.ChatPermissions{}
if allow {
perms = chatWritePerms()
}
_, err := t.api.RestrictChatMember(ctx, &tgbot.RestrictChatMemberParams{
ChatID: t.chatID,
UserID: userID,
Permissions: &perms,
})
return err
}
// chatWritePerms grants a member the ability to send every kind of message; the
// non-send permissions stay denied.
func chatWritePerms() models.ChatPermissions {
return models.ChatPermissions{
CanSendMessages: true,
CanSendAudios: true,
CanSendDocuments: true,
CanSendPhotos: true,
CanSendVideos: true,
CanSendVideoNotes: true,
CanSendVoiceNotes: true,
CanSendPolls: true,
CanSendOtherMessages: true,
CanAddWebPagePreviews: true,
}
}
// restrictedSendState returns a restricted member's text-send permission and whether
// they are currently a member of the chat; (false, false) for any non-restricted
// status (the fields exist only on the restricted variant).
func restrictedSendState(m models.ChatMember) (canSend, isMember bool) {
if m.Type == models.ChatMemberTypeRestricted && m.Restricted != nil {
return m.Restricted.CanSendMessages, m.Restricted.IsMember
}
return false, false
}
// chatMemberUser returns the user a ChatMember refers to across the union variants,
// or nil for an unrecognised type.
func chatMemberUser(m models.ChatMember) *models.User {
switch m.Type {
case models.ChatMemberTypeOwner:
return m.Owner.User
case models.ChatMemberTypeAdministrator:
return &m.Administrator.User
case models.ChatMemberTypeMember:
return m.Member.User
case models.ChatMemberTypeRestricted:
return m.Restricted.User
case models.ChatMemberTypeLeft:
return m.Left.User
case models.ChatMemberTypeBanned:
return m.Banned.User
}
return nil
}

Some files were not shown because too many files have changed in this diff Show More