scrabble-game/deploy/README.md

deploy

The full Scrabble contour: backend + gateway + the static landing + Postgres + the Telegram validator + bot (the bot with a VPN sidecar) + the observability stack (OTel Collector → Prometheus + Tempo → Grafana), fronted by a caddy that owns a single /_gm Basic-Auth (the admin console + Grafana). Topology and the decision record are in ../docs/ARCHITECTURE.md §13; this file is the operational reference for every environment variable.

Services

Service	Image	Role
caddy	caddy:2-alpine	Edge proxy (alias scrabble on edge): single /_gm Basic-Auth → admin console + Grafana; /app/, /telegram/ + the Connect path → gateway; the catch-all (incl. /) → landing. TLS per CADDY_SITE_ADDRESS.
gateway	built (gateway/Dockerfile, target gateway)	Public edge; serves the embedded game SPA at /app/ + /telegram/; Connect-RPC edge. / redirects to /app/.
landing	built (gateway/Dockerfile, target landing)	Static landing page at / (caddy:2-alpine + the shared Vite build, deploy/landing/Caddyfile); absorbs stray public paths.
backend	built (backend/Dockerfile)	Domain service; bakes in the DAWG dictionaries; runs migrations at boot.
postgres	postgres:17-alpine	Database (named volume, pg_isready healthcheck).
renderer	built (renderer/Dockerfile)	Finished-game image-render sidecar (Node + skia-canvas running the shared ui/src/lib/gameimage.ts); internal-only HTTP at renderer:8090, called by the backend for the PNG export artifact.
validator	built (platform/telegram/Dockerfile, target validator)	Telegram HMAC validator (no VPN, no Bot API); internal gRPC at validator:9091. Game login depends only on this.
vpn + bot	sidecar + built (platform/telegram/Dockerfile, target bot)	Telegram bot, gated to the telegram-local profile; egresses through the AmneziaWG sidecar and dials the gateway bot-link (mTLS) at gateway:9443. The test contour activates the profile; the prod main host omits it and runs the bot standalone on its own host (docker-compose.bot.yml, no VPN — native Bot API egress).
otelcol	otel/opentelemetry-collector-contrib	OTLP/gRPC :4317 → Prometheus scrape (:9464) + Tempo.
prometheus	prom/prometheus	Metrics, 15d retention (7d in prod).
tempo	grafana/tempo	Traces, 72h retention.
grafana	grafana/grafana	Dashboards (provisioned), anonymous-admin behind caddy's /_gm/grafana.
node_exporter	quay.io/prometheus/node-exporter	Host CPU/memory/disk metrics (Prometheus job node); the OOM signal on the tight prod main host (2 vCPU / 1.9 GiB).

Networking: inter-service traffic is on the private internal network (project-scoped DNS); only caddy joins the shared external edge network so the host caddy can reach it at scrabble:80. edge must already exist on the host (docker network create edge).

Run it

Locally — copy the template, fill the required values, bring it up:

cp deploy/.env.example deploy/.env     # then edit deploy/.env
docker network create edge             # once, if it does not exist
cd deploy && docker compose up -d --build

In CI (the test contour) — .gitea/workflows/ci.yaml's deploy job maps the Gitea TEST_-prefixed secrets/variables onto the unprefixed names below and runs docker compose up -d --build on the runner host. The prod deploy maps the PROD_ set the same way. So a Gitea secret named TEST_POSTGRES_PASSWORD feeds the compose's POSTGRES_PASSWORD, etc.

Three naming classes in Gitea:

• Per-contour (TEST_/PROD_<NAME>) — values that differ between the contours (bots, hosts, public origin, log level, email senders): the common case below.
• Shared (one unprefixed <NAME>, no prefix) — values identical on every contour, stored once: DICT_VERSION, SMTP_RELAY_HOST/PORT/TLS/USER/PASS, GRAFANA_SMTP_PORT, VITE_VK_APP_LINK, VITE_VK_APP_ID, GATEWAY_VK_APP_SECRET, GATEWAY_VK_ID_CLIENT_SECRET (one Selectel relay + one pair of VK apps for all contours).
• Derived — not stored at all; the deploy computes them from PUBLIC_BASE_URL (deploy/write-prod-env.sh, and the ci.yaml deploy step): TELEGRAM_MINIAPP_URL (+ /telegram/), GRAFANA_ROOT_URL (+ /_gm/grafana/), VITE_VK_ID_REDIRECT_URL (+ /app/). Set them directly only for a local .env run.

The deploy job also seeds the config files (caddy, otelcol, prometheus, tempo, grafana) to a stable host path ($HOME/.scrabble-deploy) and sets SCRABBLE_CONFIG_DIR to it before up. The runner's checkout is an ephemeral act workspace that is removed after the job — binding config straight from it would dangle the mounts in the long-lived containers (Grafana would log no such file or directory). Locally SCRABBLE_CONFIG_DIR defaults to ., so the compose binds from this directory.

Required variables

docker compose aborts immediately if any of these is unset (they use :?):

Variable	Gitea kind	Purpose
POSTGRES_PASSWORD	secret	Postgres password (also embedded in BACKEND_POSTGRES_DSN).
GM_BASICAUTH_HASH	secret	bcrypt hash gating /_gm (admin console + Grafana). Generate with docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'.
TELEGRAM_MINIAPP_URL	derived	The Mini App URL the bot hands out in deep links / buttons. The deploy derives PUBLIC_BASE_URL + /telegram/; set it directly only for a local run (compose still :?-requires it).
EXPORT_SIGN_KEY	secret	HMAC key signing the public finished-game export download URLs (/dl/*). Generate with openssl rand -base64 32.
BACKEND_ROBOKASSA_MERCHANT_LOGIN	secret	Robokassa shop login for the direct RUB rail. Empty leaves the rail off.
BACKEND_ROBOKASSA_PASSWORD1 / …_PASSWORD2	secret	Robokassa pass phrases: Password1 signs the launch request, Password2 signs/verifies the Result callback. Use the shop's test pair while …_ROBOKASSA_TEST=1, the live pair for real money. (Password3 — Robokassa's JWT-invoice API — is unused.)
BACKEND_ROBOKASSA_TEST	variable	1 runs test payments against the test pass phrases (no real money); empty/0 is live. A variable, not a secret, so go-live is a flag flip + redeploy, not a secret rotation.

Plus the bot token — TELEGRAM_BOT_TOKEN (secret), shared by the validator (HMAC secret) and the bot (Bot API). It defaults to empty in compose, but both fail at boot when it is empty.

Conditionally — AWG_CONF (secret): the AmneziaWG config for the VPN sidecar, needed only when the telegram-local profile runs (the test contour and local runs with the bot). It is not :?-guarded — compose interpolates profiled-out services too, so the prod main host (no VPN) must not require it. It must not contain a DNS= line — that hijacks the shared netns's resolv.conf and breaks the bot resolving otelcol / gateway; without it Docker's resolver handles otelcol, gateway and api.telegram.org.

Optional variables (with defaults)

Variable	Gitea kind	Default	Purpose
POSTGRES_DB	variable	scrabble	Database name.
POSTGRES_USER	variable	scrabble	Database user.
DICT_VERSION	variable (shared)	v1.3.1	scrabble-dictionary release tag baked into the backend image as the seed for a fresh volume (build-arg). A live contour changes dictionary through the admin console, not this; on a seeded volume a changed value is ignored (the recorded .seed_version marker wins — the seed-drift guard, ARCHITECTURE.md §5). One shared unprefixed Gitea variable seeds both contours and pins the CI test suite.
LOG_LEVEL	variable	info	Shared log level for backend / gateway / validator / bot (debug|info|warn|error).
CADDY_SITE_ADDRESS	variable	:80	Caddy site address. Test: :80 (host caddy terminates TLS). Prod: a domain, so caddy does its own ACME.
GM_BASICAUTH_USER	variable	gm	Username for the /_gm Basic-Auth.
GRAFANA_ROOT_URL	derived	/_gm/grafana/	Grafana root URL (sub-path serving). The deploy derives PUBLIC_BASE_URL + /_gm/grafana/; set the full https://<domain>/_gm/grafana/ only for a local run.
GRAFANA_ADMIN_PASSWORD	secret	admin	Grafana admin password. Low impact (the login form is disabled, access is anonymous-admin behind caddy) but set it anyway.
TELEGRAM_GAME_CHANNEL_ID	variable	(empty)	The bot's game-channel id; empty/0 disables channel posts.
TELEGRAM_TEST_ENV	pinned	false	true routes the bot through Telegram's test environment (.../bot<token>/test/METHOD). The CI test contour pins this to true in ci.yaml (the contour is the test environment) — it is not a Gitea variable. Set it in .env for a local run; prod leaves it false.
TELEGRAM_API_BASE_URL	variable	(empty)	Override the Bot API host (a mock/self-hosted server); empty = https://api.telegram.org.
VITE_TELEGRAM_BOT_ID	variable	(empty)	UI build-arg: numeric bot id for the web Login Widget.
VITE_TELEGRAM_LINK	variable	(empty)	UI build-arg: friend-invite Mini App link (full URL, https://t.me/<bot>/<app> — <app> is the Mini App short name from BotFather).
VITE_TELEGRAM_GAME_CHANNEL_NAME	variable	(empty)	UI build-arg: the landing "Play in Telegram" link, the bot's game channel (e.g. https://t.me/Erudit_Game).
VITE_VK_APP_LINK	variable (shared)	(empty)	UI build-arg: the landing "Play on VK" link, the VK Mini App (full URL, https://vk.com/app<id>). One VK Mini App for all contours.
VITE_VK_APP_ID	variable (shared)	(empty)	VK ID "Web" app id (client_id) for VK web-login linking. Baked into the SPA authorize URL and reused as the gateway's GATEWAY_VK_ID_APP_ID. Empty disables the link.vk.* ops. One VK ID "Web" app for all contours.
VITE_VK_ID_REDIRECT_URL	derived	(empty)	VK ID trusted redirect URL — must match the app's registered redirect exactly. The deploy derives PUBLIC_BASE_URL + /app/ (used by the SPA and as the gateway's exchange redirect_uri); set it only for a local run.
GATEWAY_VK_APP_SECRET	secret (shared)	(empty)	The VK Mini App's protected key (client_secret): the gateway verifies the launch-parameter signature in-process under it (offline HMAC, no VK API call). Empty disables the VK auth path (auth.vk). One VK Mini App for all contours.
GATEWAY_VK_ID_CLIENT_SECRET	secret (shared)	(empty)	The VK ID "Web" app's protected key (client_secret) for the gateway's server-side confidential code exchange. A SEPARATE VK app from GATEWAY_VK_APP_SECRET (the Mini App). One VK ID "Web" app for all contours.
GATEWAY_HONEYTOKEN	secret	(empty)	Planted honeytoken bearer value: presenting it earns a 24h IP ban + a high-severity alarm where the IP ban is on (prod), or logs + a gateway_abuse_banned_total{reason="honeytoken"} metric (test, ban off). Plant the value somewhere an attacker would find it; empty disables the trap. Per-contour TEST_/PROD_GATEWAY_HONEYTOKEN.
GATEWAY_BLOCKLIST_ENABLED	variable	false	Enable the community IP blocklist at the edge (prod-only, same real-client-IP reason as the ban). Requires GATEWAY_BLOCKLIST_URL. Opt-in via PROD_GATEWAY_BLOCKLIST_ENABLED after verifying the feed.
GATEWAY_BLOCKLIST_URL	variable	(empty)	The curated CIDR feed to fetch (Spamhaus DROP). Required when enabled; refreshed every few hours and dropped fail-open once stale (48h). PROD_GATEWAY_BLOCKLIST_URL.
GATEWAY_BLOCKLIST_ALLOW	variable	(empty)	Comma-separated never-block set (CIDRs / bare IPs — own infra, monitoring) the feed can never block. PROD_GATEWAY_BLOCKLIST_ALLOW.
GATEWAY_MIN_CLIENT_VERSION	variable	(empty)	Minimum client build the gateway will serve — the native client-version gate (ARCHITECTURE.md §2). Empty ⇒ dormant (every build served, the web default). Set it to the release vMAJOR.MINOR.PATCH in the same rollout that ships an incompatible wire change, so an older bundled APK is turned away with an update required signal instead of failing blind. Validated at load; a non-empty unparseable value fails startup.
VITE_GATEWAY_URL	variable	(empty)	UI build-arg: gateway origin; empty = same-origin (the usual single-origin deploy).
SMTP_RELAY_HOST	variable (shared)	(empty)	Selectel SMTP relay host for confirm-code email. Empty leaves the backend on the log mailer (email disabled) — the contour still boots. One relay for every contour (limit 100 msgs / 5 min).
SMTP_RELAY_PORT	variable (shared)	465	Relay port. No client certificate is needed (the server cert is validated against the system roots).
SMTP_RELAY_TLS	variable (shared)	(empty)	Transport security: ssl (implicit TLS) or starttls. Empty derives it from the port (implicit on 465, STARTTLS otherwise); required for a relay on a non-standard port (e.g. Selectel's 1127 = SSL, 1126 = STARTTLS).
SMTP_RELAY_USER	secret (shared)	(empty)	Relay SMTP AUTH username.
SMTP_RELAY_PASS	secret (shared)	(empty)	Relay SMTP AUTH password.
SMTP_RELAY_FROM	variable	no-reply@localhost	Sender address. Must use the prod domain (no-reply@erudit-game.ru) — Selectel only accepts the verified sender domain — so it is the same on every contour.
PUBLIC_BASE_URL	variable	(empty)	Canonical public https origin for links in the email (the contour's own URL, e.g. https://erudit-game.ru). Required by the backend whenever SMTP_RELAY_HOST is set — it is never derived from a request Host header (anti-injection).
SMTP_RELAY_ADMIN_FROM	variable	(empty)	Backend operator-alert sender (new feedback / word complaints), distinct from the confirm-code From. Empty (with ADMIN_EMAIL) disables the alert worker.
ADMIN_EMAIL	variable	(empty)	Backend operator-alert recipient(s); several comma-separated addresses allowed.
SMTP_RELAY_SERVICE_FROM	variable	(empty)	Grafana infra-alert sender address.
SERVICE_EMAIL	variable	(empty)	Grafana infra-alert recipient(s); comma-separated allowed (read by the provisioned contact point via $__env{SERVICE_EMAIL}).
GRAFANA_SMTP_PORT	variable (shared)	(empty)	STARTTLS port Grafana dials on SMTP_RELAY_HOST (its client can't do the backend's implicit TLS), e.g. Selectel's 1126. Reuses SMTP_RELAY_HOST/USER/PASS.
GF_SMTP_ENABLED	variable	false	Set true to enable Grafana alert emails.

The six VITE_* are build-args baked into the gateway and landing images at build time (both targets share one UI build stage — keep the args identical so it is built once), so changing them requires a rebuild (--build), not just a restart.

Fixed internal wiring (not operator-set)

These are hard-wired in docker-compose.yml (no ${...}), pointing the services at each other on the internal network — listed here so they are not mistaken for missing config: BACKEND_POSTGRES_DSN (→ postgres, search_path=backend), GATEWAY_BACKEND_HTTP_URL/_GRPC_ADDR (→ backend), GATEWAY_VALIDATOR_ADDR (→ validator:9091), BACKEND_CONNECTOR_ADDR (→ the gateway bot-link relay gateway:9092), BACKEND_RENDERER_URL (→ renderer:8090, the image-render sidecar), the bot's TELEGRAM_GATEWAY_ADDR (→ gateway:9443, mTLS) with the GATEWAY_BOTLINK_* / TELEGRAM_BOTLINK_* cert paths under /certs (the mTLS material is generated by deploy/gen-certs.sh, gitignored, regenerated each deploy), and all services' *_OTEL_*_EXPORTER=otlp → OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317 (_INSECURE=true). The bot shares the VPN sidecar's netns: routing to the collector's / gateway's internal IP is fine (connected route), but its AWG_CONF must not set a DNS= directive — that hijacks resolv.conf and breaks resolving otelcol / gateway ("produced zero addresses"); without it the netns uses Docker's resolver, which resolves otelcol, gateway and api.telegram.org. GATEWAY_ADMIN_* is intentionally unset — caddy owns /_gm in the contour.

Bumping the dictionary version

The dictionary ships as a versioned release artifact (scrabble-dawg-vX.Y.Z.tar.gz) from scrabble-dictionary. The tag is a build-time input with no default in the images. It is a single Gitea repo variable — DICT_VERSION (unprefixed, shared across contours) — so a release bump is one edit:

• the CI test jobs read it via .gitea/workflows/ci.yaml's top-level env.DICT_VERSION: ${{ vars.DICT_VERSION }} (the unit/integration jobs download that dawg), and
• both deploy jobs feed the same variable to compose as the DICT_VERSION build-arg that bakes a fresh volume's seed.

For local builds set DICT_VERSION in deploy/.env (template: .env.example); a bare docker build needs --build-arg DICT_VERSION=vX.Y.Z. The Dockerfiles and compose carry no default — a missing value fails loudly instead of baking a stale tag.

Bumping the seed is a no-op on a live volume (the .seed_version marker wins — the seed-drift guard). A running contour/prod moves to a new release through the admin console /_gm/dictionary (upload the tarball, preview the per-variant diff, confirm); in-flight games keep their pinned version, new games use the new one (ARCHITECTURE.md §5).

Production rollout

Prod runs on two hosts (main = full stack + ACME on the domain; tg = the bot only, native Bot API, no VPN), one-time provisioned by ansible/ (docker, a non-sudo deploy user holding the CI key, key-only sshd, default-deny ufw, fail2ban). Re-run ansible/ after a host resize — it is idempotent.

To roll out: merge development → master (CI green), then run the prod-deploy workflow manually (Gitea → Actions → prod-deploy → run from master, input confirm=deploy). It builds + pushes the images to the registry, ships the compose/config/certs/env over SSH, deploys the main host with prod-deploy.sh (rolling, health-gated, auto-rollback to the previous tag; caddy is force-recreated on its roll so a bind-mounted Caddyfile change applies — its image is pinned and admin is off, so neither a new tag nor a hot reload would pick it up), then the bot host, then probes the public site. After master is green this workflow is the only thing that touches prod — nothing auto-deploys there. It runs four visible jobs: build → deploy-main → deploy-bot → verify (the per-service rolling shows in the deploy-main log).

Maintenance page. During the roll (and any migration window) prod-deploy.sh raises a flag the edge caddy serves a static 503 "технические работы" page from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm (Grafana) stays reachable. The flag is cleared on any exit (success, a health failure + rollback, or an error) by a shell trap, so it can never stick on. It arms from this feature's own deploy onward (the caddy carrying the gate must be live first). This is not a zero-downtime deploy — the backend is a single stateful instance (in-memory game state + push hub, no Redis), so its swap still blips (clients auto-reconnect the live stream); the page just makes the window graceful instead of raw 502s.

Versioning. Each release is a git tag vX.Y.Z on master; the deploy stamps git describe --tags into every image tag, every binary (-ldflags → pkg/version → the service.version telemetry attribute) and the SPA About screen. Tag the release before running the deploy:

git tag -a v1.0.0 -m v1.0.0 && git push origin v1.0.0

Manual rollback (any time after a successful deploy). Run the prod-rollback workflow (Gitea → Actions → prod-rollback, confirm=rollback). Leave target_version blank to roll back to the previously deployed version (read from the host's PREVIOUS_TAG), or set it to a release tag from the Releases page. It re-deploys that already-published image rolling + health-gated — no rebuild, no DB migration (image rollback is DB-safe under the expand-contract rule). The registry keeps every release tag, so any prior release is reachable.

Migrations must be expand-contract (backward-compatible; goose is forward-only): the automatic rollback is image-only and never restores the DB. A deploy that changes backend/internal/postgres/migrations/ opens a maintenance window — the backend (sole writer) is stopped for a consistent whole-database pg_dump into /opt/scrabble/dumps (it covers backend and payments) before the new backend migrates. Manual DB restore (only if a migration was destructive): drop the app schemas in the same instance and pipe the dump back — docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA IF EXISTS backend CASCADE; DROP SCHEMA IF EXISTS payments CASCADE;', then docker exec -i scrabble-postgres psql -U scrabble -d scrabble < the-dump.sql, and redeploy the matching old tag. This dump is a belt-and-braces net for a bad migration; point-in-time recovery (below) is the primary recovery path once armed, and the only one that survives losing the host.

Android app build & release (RuStore)

The standalone Android app is built by a manual workflow, never automatically, and is separate from the web/prod rollout — a signed APK uploaded to RuStore by hand. Full plan: ../ANDROID_PLAN.md.

• Trigger: Actions → android-build → Run workflow from master, input confirm=build (mirrors prod-deploy). Tag the release first (git tag vX.Y.Z on master) — the workflow refuses anything but a clean vMAJOR.MINOR.PATCH and derives versionCode = MA*1_000_000 + MI*1_000 + PA, versionName = MA.MI.PA. Watch it green with python3 ~/.claude/bin/gitea-ci-watch.py.
• Output: a release APK run artifact — signed when the signing secrets are present, an unsigned release APK otherwise (a dry run still proves the pipeline).
• Runner (host-executor): JDK 21 comes from setup-java; the Android SDK is host-provisioned — install it once (sdkmanager 'platform-tools' 'platforms;android-36' 'build-tools;36.0.0') and grant the runner user read+exec (sudo chmod -R a+rX /opt/android-sdk). Override the path with the ANDROID_SDK_DIR variable. The workflow's Verify the host Android SDK step fails fast with the fix if the SDK is missing or unreadable.
• Release keystore (create once, back up off-host — losing it means the app can never be updated): keytool -genkeypair -v -keystore erudit-release.jks -alias erudit -keyalg RSA -keysize 4096 -validity 10000, then set the Gitea secrets ANDROID_KEYSTORE_BASE64 (base64 -w0 erudit-release.jks), ANDROID_KEYSTORE_PASSWORD, ANDROID_KEY_ALIAS, ANDROID_KEY_PASSWORD. Set the ANDROID_RUSTORE_URL variable to the store listing once published (empty until then — the in-app update button no-ops).
• Wire-break discipline: the prod deploy that ships an incompatible wire change also bumps GATEWAY_MIN_CLIENT_VERSION to that release (see Optional variables), so an old installed APK is turned away cleanly instead of failing blind.
• Upload: download the artifact and upload it to RuStore by hand (no automated RuStore-API upload in the MVP).

Point-in-time recovery (PITR)

The main host archives Postgres continuously with pgBackRest to Selectel S3 (encrypted at rest, path-style addressing) so the database can be restored to any moment — protecting the money ledger and the game state against corruption or host loss. It is the primary recovery path; the migration-window pg_dump above is the secondary net. Live on the prod main host since v1.13.0 (armed + restore-drilled 2026-07-09).

Shape. A daily full base backup (a systemd timer on the main host, 04:00) plus continuous WAL archived by Postgres archive_command (a segment is forced at least every 5 minutes, so the recovery point is never more than a few minutes behind). Retention is 30 days (repo1-retention-full=30); the repository is AES-256-CBC encrypted. This is main-host only — the test contour never archives.

Wiring. pgBackRest ships inside the DB image (deploy/postgres/Dockerfile); the repository + S3 credentials + cipher are the PGBACKREST_* environment on the postgres service in docker-compose.prod.yml, rendered from the PROD_ Gitea set by write-prod-env.sh. Archiving is gated by PGBACKREST_ARCHIVE_MODE (default off), so shipping or redeploying this stack does not start archiving — the artifact is inert until armed, which is why an un-armed prod deploy can never pile WAL onto the disk. The base-backup timer is provisioned by the Ansible main role behind pitr_enabled (also default off). Two Grafana alerts watch health: WAL archiving failing (pg_stat_archiver_failed_count rising) and WAL archiving stalled (last archive over 30 min old while pg_wal_size_bytes is growing — the pg_wal-growth guard keeps an idle database, which archives nothing because it writes nothing, from false-triggering during quiet hours) — both absent/NaN-safe, so they stay quiet until archiving is armed.

Assessment (owner-reviewed; the gate before the first real money). Measured on prod pg_stat_wal: WAL is generated at ~0.77 MB/day and the database is ~9.6 MB. At 30-day retention on Selectel S3 (~2 ₽/GB·month) the archive is under ~0.3 GB → well under 1 ₽/month (compression halves it again); request volume is trivial. Performance impact is negligible: archive-push moves tiny compressed segments, and the daily full base backup is small (the whole cluster is ~32 MB → ~3.7 MB compressed in the repo) and checkpoint-bound (~1.5 min wall, minimal CPU/I-O) on the 2 vCPU host. Revisit if traffic grows ~100× (watch node_exporter during a base backup).

Arming (owner-coordinated, once, before real payments). Ships disarmed; to turn it on:

1. Owner (Selectel + Gitea): create an S3 bucket (name lowercase, no dots/underscores) and an S3 access key/secret; pick a repository cipher passphrase and store it apart from the S3 keys (losing it makes the archive unrecoverable). Set the Gitea PROD_ set — variables PROD_PGBACKREST_S3_ENDPOINT (the S3 host only — no https://, no port, no bucket) / _S3_BUCKET / _S3_REGION, an optional _S3_PORT (default 443, set only for a non-standard provider port), and PROD_PGBACKREST_ARCHIVE_MODE=on; secrets PROD_PGBACKREST_S3_KEY (the S3 access key) / _S3_KEY_SECRET (its paired secret key, shown once at creation) / _CIPHER_PASS (a fresh repository passphrase, e.g. openssl rand -base64 48).
2. Promote development → master, tag, and run prod-deploy. The roll recreates postgres with archive_mode=on behind the maintenance page. (Archive pushes fail harmlessly for the minute until step 3 creates the repository — the WAL is retained, not lost.)
3. On the main host, create the repository, verify archiving, take the first base backup. Run pgBackRest as the postgres OS user (-u postgres — lock-dir/PGDATA consistency with archive-push) and connect to the DB as the superuser role (--pg1-user=scrabble, the POSTGRES_USER, not postgres); it inherits the container's PGBACKREST_* env:

   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble stanza-create
   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check
   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble --type=full backup
   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble info   # (info takes no --pg1-user)
   

   The few archive-push failures logged between archive_mode=on and stanza-create are the expected transient (WAL retained, not lost); clear the counter afterwards with docker exec -u postgres scrabble-postgres psql -U scrabble -d scrabble -c "SELECT pg_stat_reset_shared('archiver');" so it does not trip the failing-archive alert.
4. Enable the daily timer: ansible-playbook site.yml --limit main -e pitr_enabled=true (installs + starts pgbackrest-backup.timer on the main host).
5. Confirm in Grafana that the two archiving alerts are green (last_archive_age now tracks a real number, failed_count flat).

Restore drill (proves recoverability; re-run after arming and after any major change). On an isolated, one-shot target (a throwaway VM or a container with no ingress), pgBackRest, the matching Postgres major, an empty PGDATA, and the same PGBACKREST_* environment:

# Throwaway container from the DB image (carries pgBackRest), the live PGBACKREST_* env, an
# empty PGDATA and no app network; restore, recover, verify, then wipe (-v drops the data).
IMG=$(docker inspect -f '{{.Config.Image}}' scrabble-postgres)
docker exec scrabble-postgres env | grep '^PGBACKREST_' > /tmp/drill.env; echo POSTGRES_PASSWORD=drill >> /tmp/drill.env
docker run -d --name pitr-drill --env-file /tmp/drill.env --entrypoint sleep "$IMG" infinity
docker exec pitr-drill sh -c 'rm -rf /var/lib/postgresql/data/*; chown -R postgres:postgres /var/lib/postgresql/data; chmod 700 /var/lib/postgresql/data'
# --type=immediate = to the base backup's consistency point; --type=time "--target=<ts>" for PITR
docker exec -u postgres pitr-drill pgbackrest --stanza=scrabble --pg1-path=/var/lib/postgresql/data --type=immediate restore
docker exec -u postgres pitr-drill pg_ctl -D /var/lib/postgresql/data -w start
docker exec -u postgres pitr-drill psql -U scrabble -d scrabble -c "SELECT count(*) FROM backend.accounts;"  # verify data intact
docker rm -fv pitr-drill; rm -f /tmp/drill.env   # wipe the restored real data

The target holds real money + personal data while it exists — keep it network-isolated and destroy it (wipe PGDATA + the instance) afterwards. Record each drill (date, target timestamp, outcome) here:

Date	Target	Result
2026-07-09	latest (--type=immediate)	PASS — v1.13.0 arming: 31.9 MB cluster restored from the encrypted S3 repo (3.7 MB compressed), recovered to consistency, backend.accounts intact; drill instance wiped.

bot-link cert rotation: regenerate (deploy/gen-certs.sh /tmp/c --force), reset the five PROD_BOTLINK_* secrets from /tmp/c, and re-run the workflow — both hosts redeploy together with the fresh CA.

Sizing / monitoring: the main host launches undersized (2 vCPU / 1.9 GiB); the prod overlay trims limits + GOMAXPROCS=2 + 7d Prometheus retention, and node_exporter feeds host memory to Grafana (/_gm/grafana/). Watch host memory and resize at Selectel when players arrive. The per-container memory caps are enforced (Compose v2 → cgroup), so no one service can eat all RAM, but they overcommit the host (~2.8 GiB of caps vs 1.9 GiB) — a 1 GiB swap file (Ansible common role, swap_size, vm.swappiness=10) cushions a simultaneous spike into swap instead of the kernel OOM-killer. The host_mem_low Grafana alert fires under 10% available.

Shared Gitea set (one unprefixed entry each, used by every contour) — secrets: GATEWAY_VK_APP_SECRET, GATEWAY_VK_ID_CLIENT_SECRET, SMTP_RELAY_USER, SMTP_RELAY_PASS; variables: DICT_VERSION, SMTP_RELAY_HOST, SMTP_RELAY_PORT, SMTP_RELAY_TLS, GRAFANA_SMTP_PORT, VITE_VK_APP_LINK, VITE_VK_APP_ID. Derived from PUBLIC_BASE_URL at deploy (not stored): TELEGRAM_MINIAPP_URL, GRAFANA_ROOT_URL, VITE_VK_ID_REDIRECT_URL.

PROD_ Gitea set (per-contour, mirrors TEST_, mapped onto the unprefixed names above) — secrets: PROD_{POSTGRES_PASSWORD, GM_BASICAUTH_HASH, GRAFANA_ADMIN_PASSWORD, TELEGRAM_BOT_TOKEN, TELEGRAM_PROMO_BOT_TOKEN, EXPORT_SIGN_KEY, GATEWAY_HONEYTOKEN, REGISTRY_PASSWORD, SSH_KEY, SSH_KNOWN_HOSTS, BOTLINK_CA, BOTLINK_GATEWAY_CERT, BOTLINK_GATEWAY_KEY, BOTLINK_BOT_CERT, BOTLINK_BOT_KEY, PGBACKREST_S3_KEY, PGBACKREST_S3_KEY_SECRET, PGBACKREST_CIPHER_PASS}; variables: PROD_{REGISTRY_USER, MAIN_HOST, TG_HOST, CADDY_SITE_ADDRESS, GM_BASICAUTH_USER, LOG_LEVEL, TELEGRAM_GAME_CHANNEL_ID, TELEGRAM_CHAT_ID, TELEGRAM_SUPPORT_CHAT_ID, TELEGRAM_BOT_USERNAME, VITE_TELEGRAM_BOT_ID, VITE_TELEGRAM_LINK, VITE_TELEGRAM_GAME_CHANNEL_NAME, SMTP_RELAY_FROM, PUBLIC_BASE_URL, SMTP_RELAY_ADMIN_FROM, ADMIN_EMAIL, SMTP_RELAY_SERVICE_FROM, SERVICE_EMAIL, GF_SMTP_ENABLED, PGBACKREST_S3_ENDPOINT, PGBACKREST_S3_PORT, PGBACKREST_S3_BUCKET, PGBACKREST_S3_REGION, PGBACKREST_ARCHIVE_MODE}. The test contour uses the same names under TEST_, minus the prod-only infra (MAIN_HOST/TG_HOST/REGISTRY_*/SSH_*/BOTLINK_* and the PGBACKREST_* PITR set, which is prod-only — the test contour never archives) and plus TEST_AWG_CONF (the bot's VPN egress).

Host-side setup (outside this repo)

• edge network must exist on the host (docker network create edge).
• Host caddy route <domain> → scrabble:80 (the in-compose caddy serves HTTP in the test contour; the host caddy terminates TLS). Not needed on prod, where the contour caddy owns TLS (set CADDY_SITE_ADDRESS to the domain).
• Branch protection requires the single status check CI / gate. The unit / integration / ui jobs are path-conditional (they skip when their code did not change), and the always-running gate job aggregates them (passing when each succeeded or was skipped), so a skipped job never blocks a merge. See ../CLAUDE.md "Branching & CI".