scrabble-game/deploy

deploy

The full Scrabble contour: backend + gateway + the static landing + Postgres + the Telegram validator + bot (the bot with a VPN sidecar) + the observability stack (OTel Collector → Prometheus + Tempo → Grafana), fronted by a caddy that owns a single /_gm Basic-Auth (the admin console + Grafana). Topology and the decision record are in ../docs/ARCHITECTURE.md §13; this file is the operational reference for every environment variable.

Services

Service	Image	Role
caddy	caddy:2-alpine	Edge proxy (alias scrabble on edge): single /_gm Basic-Auth → admin console + Grafana; /app/, /telegram/ + the Connect path → gateway; the catch-all (incl. /) → landing. TLS per CADDY_SITE_ADDRESS.
gateway	built (gateway/Dockerfile, target gateway)	Public edge; serves the embedded game SPA at /app/ + /telegram/; Connect-RPC edge. / redirects to /app/.
landing	built (gateway/Dockerfile, target landing)	Static landing page at / (caddy:2-alpine + the shared Vite build, deploy/landing/Caddyfile); absorbs stray public paths.
backend	built (backend/Dockerfile)	Domain service; bakes in the DAWG dictionaries; runs migrations at boot.
postgres	postgres:17-alpine	Database (named volume, pg_isready healthcheck).
renderer	built (renderer/Dockerfile)	Finished-game image-render sidecar (Node + skia-canvas running the shared ui/src/lib/gameimage.ts); internal-only HTTP at renderer:8090, called by the backend for the PNG export artifact.
validator	built (platform/telegram/Dockerfile, target validator)	Telegram HMAC validator (no VPN, no Bot API); internal gRPC at validator:9091. Game login depends only on this.
vpn + bot	sidecar + built (platform/telegram/Dockerfile, target bot)	Telegram bot, gated to the telegram-local profile; egresses through the AmneziaWG sidecar and dials the gateway bot-link (mTLS) at gateway:9443. The test contour activates the profile; the prod main host omits it and runs the bot standalone on its own host (docker-compose.bot.yml, no VPN — native Bot API egress).
otelcol	otel/opentelemetry-collector-contrib	OTLP/gRPC :4317 → Prometheus scrape (:9464) + Tempo.
prometheus	prom/prometheus	Metrics, 15d retention (7d in prod).
tempo	grafana/tempo	Traces, 72h retention.
grafana	grafana/grafana	Dashboards (provisioned), anonymous-admin behind caddy's /_gm/grafana.
node_exporter	quay.io/prometheus/node-exporter	Host CPU/memory/disk metrics (Prometheus job node); the OOM signal on the tight prod main host (2 vCPU / 1.9 GiB).

Networking: inter-service traffic is on the private internal network (project-scoped DNS); only caddy joins the shared external edge network so the host caddy can reach it at scrabble:80. edge must already exist on the host (docker network create edge).

Run it

Locally — copy the template, fill the required values, bring it up:

cp deploy/.env.example deploy/.env     # then edit deploy/.env
docker network create edge             # once, if it does not exist
cd deploy && docker compose up -d --build

In CI (the test contour) — .gitea/workflows/ci.yaml's deploy job maps the Gitea TEST_-prefixed secrets/variables onto the unprefixed names below and runs docker compose up -d --build on the runner host. The prod deploy maps the PROD_ set the same way. So a Gitea secret named TEST_POSTGRES_PASSWORD feeds the compose's POSTGRES_PASSWORD, etc.

Three naming classes in Gitea:

• Per-contour (TEST_/PROD_<NAME>) — values that differ between the contours (bots, hosts, public origin, log level, email senders): the common case below.
• Shared (one unprefixed <NAME>, no prefix) — values identical on every contour, stored once: DICT_VERSION, SMTP_RELAY_HOST/PORT/TLS/USER/PASS, GRAFANA_SMTP_PORT, VITE_VK_APP_LINK, VITE_VK_APP_ID, GATEWAY_VK_APP_SECRET, GATEWAY_VK_ID_CLIENT_SECRET (one Selectel relay + one pair of VK apps for all contours).
• Derived — not stored at all; the deploy computes them from PUBLIC_BASE_URL (deploy/write-prod-env.sh, and the ci.yaml deploy step): TELEGRAM_MINIAPP_URL (+ /telegram/), GRAFANA_ROOT_URL (+ /_gm/grafana/), VITE_VK_ID_REDIRECT_URL (+ /app/). Set them directly only for a local .env run.

The deploy job also seeds the config files (caddy, otelcol, prometheus, tempo, grafana) to a stable host path ($HOME/.scrabble-deploy) and sets SCRABBLE_CONFIG_DIR to it before up. The runner's checkout is an ephemeral act workspace that is removed after the job — binding config straight from it would dangle the mounts in the long-lived containers (Grafana would log no such file or directory). Locally SCRABBLE_CONFIG_DIR defaults to ., so the compose binds from this directory.

Required variables

docker compose aborts immediately if any of these is unset (they use :?):

Variable	Gitea kind	Purpose
POSTGRES_PASSWORD	secret	Postgres password (also embedded in BACKEND_POSTGRES_DSN).
GM_BASICAUTH_HASH	secret	bcrypt hash gating /_gm (admin console + Grafana). Generate with docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'.
TELEGRAM_MINIAPP_URL	derived	The Mini App URL the bot hands out in deep links / buttons. The deploy derives PUBLIC_BASE_URL + /telegram/; set it directly only for a local run (compose still :?-requires it).
EXPORT_SIGN_KEY	secret	HMAC key signing the public finished-game export download URLs (/dl/*). Generate with openssl rand -base64 32.
BACKEND_ROBOKASSA_MERCHANT_LOGIN	secret	Robokassa shop login for the direct RUB rail. Empty leaves the rail off.
BACKEND_ROBOKASSA_PASSWORD1 / …_PASSWORD2	secret	Robokassa pass phrases: Password1 signs the launch request, Password2 signs/verifies the Result callback. Use the shop's test pair while …_ROBOKASSA_TEST=1, the live pair for real money. (Password3 — Robokassa's JWT-invoice API — is unused.)
BACKEND_ROBOKASSA_TEST	variable	1 runs test payments against the test pass phrases (no real money); empty/0 is live. A variable, not a secret, so go-live is a flag flip + redeploy, not a secret rotation.

Plus the bot token — TELEGRAM_BOT_TOKEN (secret), shared by the validator (HMAC secret) and the bot (Bot API). It defaults to empty in compose, but both fail at boot when it is empty.

Conditionally — AWG_CONF (secret): the AmneziaWG config for the VPN sidecar, needed only when the telegram-local profile runs (the test contour and local runs with the bot). It is not :?-guarded — compose interpolates profiled-out services too, so the prod main host (no VPN) must not require it. It must not contain a DNS= line — that hijacks the shared netns's resolv.conf and breaks the bot resolving otelcol / gateway; without it Docker's resolver handles otelcol, gateway and api.telegram.org.

Optional variables (with defaults)

Variable	Gitea kind	Default	Purpose
POSTGRES_DB	variable	scrabble	Database name.
POSTGRES_USER	variable	scrabble	Database user.
DICT_VERSION	variable (shared)	v1.3.1	scrabble-dictionary release tag baked into the backend image as the seed for a fresh volume (build-arg). A live contour changes dictionary through the admin console, not this; on a seeded volume a changed value is ignored (the recorded .seed_version marker wins — the seed-drift guard, ARCHITECTURE.md §5). One shared unprefixed Gitea variable seeds both contours and pins the CI test suite.
LOG_LEVEL	variable	info	Shared log level for backend / gateway / validator / bot (debug|info|warn|error).
CADDY_SITE_ADDRESS	variable	:80	Caddy site address. Test: :80 (host caddy terminates TLS). Prod: a domain, so caddy does its own ACME.
GM_BASICAUTH_USER	variable	gm	Username for the /_gm Basic-Auth.
GRAFANA_ROOT_URL	derived	/_gm/grafana/	Grafana root URL (sub-path serving). The deploy derives PUBLIC_BASE_URL + /_gm/grafana/; set the full https://<domain>/_gm/grafana/ only for a local run.
GRAFANA_ADMIN_PASSWORD	secret	admin	Grafana admin password. Low impact (the login form is disabled, access is anonymous-admin behind caddy) but set it anyway.
TELEGRAM_GAME_CHANNEL_ID	variable	(empty)	The bot's game-channel id; empty/0 disables channel posts.
TELEGRAM_TEST_ENV	pinned	false	true routes the bot through Telegram's test environment (.../bot<token>/test/METHOD). The CI test contour pins this to true in ci.yaml (the contour is the test environment) — it is not a Gitea variable. Set it in .env for a local run; prod leaves it false.
TELEGRAM_API_BASE_URL	variable	(empty)	Override the Bot API host (a mock/self-hosted server); empty = https://api.telegram.org.
VITE_TELEGRAM_BOT_ID	variable	(empty)	UI build-arg: numeric bot id for the web Login Widget.
VITE_TELEGRAM_LINK	variable	(empty)	UI build-arg: friend-invite Mini App link (full URL, https://t.me/<bot>/<app> — <app> is the Mini App short name from BotFather).
VITE_TELEGRAM_GAME_CHANNEL_NAME	variable	(empty)	UI build-arg: the landing "Play in Telegram" link, the bot's game channel (e.g. https://t.me/Erudit_Game).
VITE_VK_APP_LINK	variable (shared)	(empty)	UI build-arg: the landing "Play on VK" link, the VK Mini App (full URL, https://vk.com/app<id>). One VK Mini App for all contours.
VITE_VK_APP_ID	variable (shared)	(empty)	VK ID "Web" app id (client_id) for VK web-login linking. Baked into the SPA authorize URL and reused as the gateway's GATEWAY_VK_ID_APP_ID. Empty disables the link.vk.* ops. One VK ID "Web" app for all contours.
VITE_VK_ID_REDIRECT_URL	derived	(empty)	VK ID trusted redirect URL — must match the app's registered redirect exactly. The deploy derives PUBLIC_BASE_URL + /app/ (used by the SPA and as the gateway's exchange redirect_uri); set it only for a local run.
GATEWAY_VK_APP_SECRET	secret (shared)	(empty)	The VK Mini App's protected key (client_secret): the gateway verifies the launch-parameter signature in-process under it (offline HMAC, no VK API call). Empty disables the VK auth path (auth.vk). One VK Mini App for all contours.
GATEWAY_VK_ID_CLIENT_SECRET	secret (shared)	(empty)	The VK ID "Web" app's protected key (client_secret) for the gateway's server-side confidential code exchange. A SEPARATE VK app from GATEWAY_VK_APP_SECRET (the Mini App). One VK ID "Web" app for all contours.
GATEWAY_HONEYTOKEN	secret	(empty)	Planted honeytoken bearer value: presenting it earns a 24h IP ban + a high-severity alarm where the IP ban is on (prod), or logs + a gateway_abuse_banned_total{reason="honeytoken"} metric (test, ban off). Plant the value somewhere an attacker would find it; empty disables the trap. Per-contour TEST_/PROD_GATEWAY_HONEYTOKEN.
VITE_GATEWAY_URL	variable	(empty)	UI build-arg: gateway origin; empty = same-origin (the usual single-origin deploy).
SMTP_RELAY_HOST	variable (shared)	(empty)	Selectel SMTP relay host for confirm-code email. Empty leaves the backend on the log mailer (email disabled) — the contour still boots. One relay for every contour (limit 100 msgs / 5 min).
SMTP_RELAY_PORT	variable (shared)	465	Relay port. No client certificate is needed (the server cert is validated against the system roots).
SMTP_RELAY_TLS	variable (shared)	(empty)	Transport security: ssl (implicit TLS) or starttls. Empty derives it from the port (implicit on 465, STARTTLS otherwise); required for a relay on a non-standard port (e.g. Selectel's 1127 = SSL, 1126 = STARTTLS).
SMTP_RELAY_USER	secret (shared)	(empty)	Relay SMTP AUTH username.
SMTP_RELAY_PASS	secret (shared)	(empty)	Relay SMTP AUTH password.
SMTP_RELAY_FROM	variable	no-reply@localhost	Sender address. Must use the prod domain (no-reply@erudit-game.ru) — Selectel only accepts the verified sender domain — so it is the same on every contour.
PUBLIC_BASE_URL	variable	(empty)	Canonical public https origin for links in the email (the contour's own URL, e.g. https://erudit-game.ru). Required by the backend whenever SMTP_RELAY_HOST is set — it is never derived from a request Host header (anti-injection).
SMTP_RELAY_ADMIN_FROM	variable	(empty)	Backend operator-alert sender (new feedback / word complaints), distinct from the confirm-code From. Empty (with ADMIN_EMAIL) disables the alert worker.
ADMIN_EMAIL	variable	(empty)	Backend operator-alert recipient(s); several comma-separated addresses allowed.
SMTP_RELAY_SERVICE_FROM	variable	(empty)	Grafana infra-alert sender address.
SERVICE_EMAIL	variable	(empty)	Grafana infra-alert recipient(s); comma-separated allowed (read by the provisioned contact point via $__env{SERVICE_EMAIL}).
GRAFANA_SMTP_PORT	variable (shared)	(empty)	STARTTLS port Grafana dials on SMTP_RELAY_HOST (its client can't do the backend's implicit TLS), e.g. Selectel's 1126. Reuses SMTP_RELAY_HOST/USER/PASS.
GF_SMTP_ENABLED	variable	false	Set true to enable Grafana alert emails.

The six VITE_* are build-args baked into the gateway and landing images at build time (both targets share one UI build stage — keep the args identical so it is built once), so changing them requires a rebuild (--build), not just a restart.

Fixed internal wiring (not operator-set)

These are hard-wired in docker-compose.yml (no ${...}), pointing the services at each other on the internal network — listed here so they are not mistaken for missing config: BACKEND_POSTGRES_DSN (→ postgres, search_path=backend), GATEWAY_BACKEND_HTTP_URL/_GRPC_ADDR (→ backend), GATEWAY_VALIDATOR_ADDR (→ validator:9091), BACKEND_CONNECTOR_ADDR (→ the gateway bot-link relay gateway:9092), BACKEND_RENDERER_URL (→ renderer:8090, the image-render sidecar), the bot's TELEGRAM_GATEWAY_ADDR (→ gateway:9443, mTLS) with the GATEWAY_BOTLINK_* / TELEGRAM_BOTLINK_* cert paths under /certs (the mTLS material is generated by deploy/gen-certs.sh, gitignored, regenerated each deploy), and all services' *_OTEL_*_EXPORTER=otlp → OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317 (_INSECURE=true). The bot shares the VPN sidecar's netns: routing to the collector's / gateway's internal IP is fine (connected route), but its AWG_CONF must not set a DNS= directive — that hijacks resolv.conf and breaks resolving otelcol / gateway ("produced zero addresses"); without it the netns uses Docker's resolver, which resolves otelcol, gateway and api.telegram.org. GATEWAY_ADMIN_* is intentionally unset — caddy owns /_gm in the contour.

Bumping the dictionary version

The dictionary ships as a versioned release artifact (scrabble-dawg-vX.Y.Z.tar.gz) from scrabble-dictionary. The tag is a build-time input with no default in the images. It is a single Gitea repo variable — DICT_VERSION (unprefixed, shared across contours) — so a release bump is one edit:

• the CI test jobs read it via .gitea/workflows/ci.yaml's top-level env.DICT_VERSION: ${{ vars.DICT_VERSION }} (the unit/integration jobs download that dawg), and
• both deploy jobs feed the same variable to compose as the DICT_VERSION build-arg that bakes a fresh volume's seed.

For local builds set DICT_VERSION in deploy/.env (template: .env.example); a bare docker build needs --build-arg DICT_VERSION=vX.Y.Z. The Dockerfiles and compose carry no default — a missing value fails loudly instead of baking a stale tag.

Bumping the seed is a no-op on a live volume (the .seed_version marker wins — the seed-drift guard). A running contour/prod moves to a new release through the admin console /_gm/dictionary (upload the tarball, preview the per-variant diff, confirm); in-flight games keep their pinned version, new games use the new one (ARCHITECTURE.md §5).

Production rollout

Prod runs on two hosts (main = full stack + ACME on the domain; tg = the bot only, native Bot API, no VPN), one-time provisioned by ansible/ (docker, a non-sudo deploy user holding the CI key, key-only sshd, default-deny ufw, fail2ban). Re-run ansible/ after a host resize — it is idempotent.

To roll out: merge development → master (CI green), then run the prod-deploy workflow manually (Gitea → Actions → prod-deploy → run from master, input confirm=deploy). It builds + pushes the images to the registry, ships the compose/config/certs/env over SSH, deploys the main host with prod-deploy.sh (rolling, health-gated, auto-rollback to the previous tag; caddy is force-recreated on its roll so a bind-mounted Caddyfile change applies — its image is pinned and admin is off, so neither a new tag nor a hot reload would pick it up), then the bot host, then probes the public site. After master is green this workflow is the only thing that touches prod — nothing auto-deploys there. It runs four visible jobs: build → deploy-main → deploy-bot → verify (the per-service rolling shows in the deploy-main log).

Maintenance page. During the roll (and any migration window) prod-deploy.sh raises a flag the edge caddy serves a static 503 "технические работы" page from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm (Grafana) stays reachable. The flag is cleared on any exit (success, a health failure + rollback, or an error) by a shell trap, so it can never stick on. It arms from this feature's own deploy onward (the caddy carrying the gate must be live first). This is not a zero-downtime deploy — the backend is a single stateful instance (in-memory game state + push hub, no Redis), so its swap still blips (clients auto-reconnect the live stream); the page just makes the window graceful instead of raw 502s.

Versioning. Each release is a git tag vX.Y.Z on master; the deploy stamps git describe --tags into every image tag, every binary (-ldflags → pkg/version → the service.version telemetry attribute) and the SPA About screen. Tag the release before running the deploy:

git tag -a v1.0.0 -m v1.0.0 && git push origin v1.0.0

Manual rollback (any time after a successful deploy). Run the prod-rollback workflow (Gitea → Actions → prod-rollback, confirm=rollback). Leave target_version blank to roll back to the previously deployed version (read from the host's PREVIOUS_TAG), or set it to a release tag from the Releases page. It re-deploys that already-published image rolling + health-gated — no rebuild, no DB migration (image rollback is DB-safe under the expand-contract rule). The registry keeps every release tag, so any prior release is reachable.

Migrations must be expand-contract (backward-compatible; goose is forward-only): the automatic rollback is image-only and never restores the DB. A deploy that changes backend/internal/postgres/migrations/ opens a maintenance window — the backend (sole writer) is stopped for a consistent whole-database pg_dump into /opt/scrabble/dumps (it covers backend and payments) before the new backend migrates. Manual DB restore (only if a migration was destructive): drop the app schemas in the same instance and pipe the dump back — docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA IF EXISTS backend CASCADE; DROP SCHEMA IF EXISTS payments CASCADE;', then docker exec -i scrabble-postgres psql -U scrabble -d scrabble < the-dump.sql, and redeploy the matching old tag. This dump is a belt-and-braces net for a bad migration; point-in-time recovery (below) is the primary recovery path once armed, and the only one that survives losing the host.

Point-in-time recovery (PITR)

The main host archives Postgres continuously with pgBackRest to Selectel S3 (encrypted at rest, path-style addressing) so the database can be restored to any moment — protecting the money ledger and the game state against corruption or host loss. It is the primary recovery path; the migration-window pg_dump above is the secondary net. Live on the prod main host since v1.13.0 (armed + restore-drilled 2026-07-09).

Shape. A daily full base backup (a systemd timer on the main host, 04:00) plus continuous WAL archived by Postgres archive_command (a segment is forced at least every 5 minutes, so the recovery point is never more than a few minutes behind). Retention is 30 days (repo1-retention-full=30); the repository is AES-256-CBC encrypted. This is main-host only — the test contour never archives.

Wiring. pgBackRest ships inside the DB image (deploy/postgres/Dockerfile); the repository + S3 credentials + cipher are the PGBACKREST_* environment on the postgres service in docker-compose.prod.yml, rendered from the PROD_ Gitea set by write-prod-env.sh. Archiving is gated by PGBACKREST_ARCHIVE_MODE (default off), so shipping or redeploying this stack does not start archiving — the artifact is inert until armed, which is why an un-armed prod deploy can never pile WAL onto the disk. The base-backup timer is provisioned by the Ansible main role behind pitr_enabled (also default off). Two Grafana alerts watch health: WAL archiving failing (pg_stat_archiver_failed_count rising) and WAL archiving stalled (last archive over 30 min old while pg_wal_size_bytes is growing — the pg_wal-growth guard keeps an idle database, which archives nothing because it writes nothing, from false-triggering during quiet hours) — both absent/NaN-safe, so they stay quiet until archiving is armed.

Assessment (owner-reviewed; the gate before the first real money). Measured on prod pg_stat_wal: WAL is generated at ~0.77 MB/day and the database is ~9.6 MB. At 30-day retention on Selectel S3 (~2 ₽/GB·month) the archive is under ~0.3 GB → well under 1 ₽/month (compression halves it again); request volume is trivial. Performance impact is negligible: archive-push moves tiny compressed segments, and the daily full base backup is small (the whole cluster is ~32 MB → ~3.7 MB compressed in the repo) and checkpoint-bound (~1.5 min wall, minimal CPU/I-O) on the 2 vCPU host. Revisit if traffic grows ~100× (watch node_exporter during a base backup).

Arming (owner-coordinated, once, before real payments). Ships disarmed; to turn it on:

1. Owner (Selectel + Gitea): create an S3 bucket (name lowercase, no dots/underscores) and an S3 access key/secret; pick a repository cipher passphrase and store it apart from the S3 keys (losing it makes the archive unrecoverable). Set the Gitea PROD_ set — variables PROD_PGBACKREST_S3_ENDPOINT (the S3 host only — no https://, no port, no bucket) / _S3_BUCKET / _S3_REGION, an optional _S3_PORT (default 443, set only for a non-standard provider port), and PROD_PGBACKREST_ARCHIVE_MODE=on; secrets PROD_PGBACKREST_S3_KEY (the S3 access key) / _S3_KEY_SECRET (its paired secret key, shown once at creation) / _CIPHER_PASS (a fresh repository passphrase, e.g. openssl rand -base64 48).
2. Promote development → master, tag, and run prod-deploy. The roll recreates postgres with archive_mode=on behind the maintenance page. (Archive pushes fail harmlessly for the minute until step 3 creates the repository — the WAL is retained, not lost.)
3. On the main host, create the repository, verify archiving, take the first base backup. Run pgBackRest as the postgres OS user (-u postgres — lock-dir/PGDATA consistency with archive-push) and connect to the DB as the superuser role (--pg1-user=scrabble, the POSTGRES_USER, not postgres); it inherits the container's PGBACKREST_* env:

   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble stanza-create
   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check
   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble --type=full backup
   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble info   # (info takes no --pg1-user)
   

   The few archive-push failures logged between archive_mode=on and stanza-create are the expected transient (WAL retained, not lost); clear the counter afterwards with docker exec -u postgres scrabble-postgres psql -U scrabble -d scrabble -c "SELECT pg_stat_reset_shared('archiver');" so it does not trip the failing-archive alert.
4. Enable the daily timer: ansible-playbook site.yml --limit main -e pitr_enabled=true (installs + starts pgbackrest-backup.timer on the main host).
5. Confirm in Grafana that the two archiving alerts are green (last_archive_age now tracks a real number, failed_count flat).

Restore drill (proves recoverability; re-run after arming and after any major change). On an isolated, one-shot target (a throwaway VM or a container with no ingress), pgBackRest, the matching Postgres major, an empty PGDATA, and the same PGBACKREST_* environment:

# Throwaway container from the DB image (carries pgBackRest), the live PGBACKREST_* env, an
# empty PGDATA and no app network; restore, recover, verify, then wipe (-v drops the data).
IMG=$(docker inspect -f '{{.Config.Image}}' scrabble-postgres)
docker exec scrabble-postgres env | grep '^PGBACKREST_' > /tmp/drill.env; echo POSTGRES_PASSWORD=drill >> /tmp/drill.env
docker run -d --name pitr-drill --env-file /tmp/drill.env --entrypoint sleep "$IMG" infinity
docker exec pitr-drill sh -c 'rm -rf /var/lib/postgresql/data/*; chown -R postgres:postgres /var/lib/postgresql/data; chmod 700 /var/lib/postgresql/data'
# --type=immediate = to the base backup's consistency point; --type=time "--target=<ts>" for PITR
docker exec -u postgres pitr-drill pgbackrest --stanza=scrabble --pg1-path=/var/lib/postgresql/data --type=immediate restore
docker exec -u postgres pitr-drill pg_ctl -D /var/lib/postgresql/data -w start
docker exec -u postgres pitr-drill psql -U scrabble -d scrabble -c "SELECT count(*) FROM backend.accounts;"  # verify data intact
docker rm -fv pitr-drill; rm -f /tmp/drill.env   # wipe the restored real data

The target holds real money + personal data while it exists — keep it network-isolated and destroy it (wipe PGDATA + the instance) afterwards. Record each drill (date, target timestamp, outcome) here:

Date	Target	Result
2026-07-09	latest (--type=immediate)	PASS — v1.13.0 arming: 31.9 MB cluster restored from the encrypted S3 repo (3.7 MB compressed), recovered to consistency, backend.accounts intact; drill instance wiped.

bot-link cert rotation: regenerate (deploy/gen-certs.sh /tmp/c --force), reset the five PROD_BOTLINK_* secrets from /tmp/c, and re-run the workflow — both hosts redeploy together with the fresh CA.

Sizing / monitoring: the main host launches undersized (2 vCPU / 1.9 GiB); the prod overlay trims limits + GOMAXPROCS=2 + 7d Prometheus retention, and node_exporter feeds host memory to Grafana (/_gm/grafana/). Watch host memory and resize at Selectel when players arrive. The per-container memory caps are enforced (Compose v2 → cgroup), so no one service can eat all RAM, but they overcommit the host (~2.8 GiB of caps vs 1.9 GiB) — a 1 GiB swap file (Ansible common role, swap_size, vm.swappiness=10) cushions a simultaneous spike into swap instead of the kernel OOM-killer. The host_mem_low Grafana alert fires under 10% available.

Shared Gitea set (one unprefixed entry each, used by every contour) — secrets: GATEWAY_VK_APP_SECRET, GATEWAY_VK_ID_CLIENT_SECRET, SMTP_RELAY_USER, SMTP_RELAY_PASS; variables: DICT_VERSION, SMTP_RELAY_HOST, SMTP_RELAY_PORT, SMTP_RELAY_TLS, GRAFANA_SMTP_PORT, VITE_VK_APP_LINK, VITE_VK_APP_ID. Derived from PUBLIC_BASE_URL at deploy (not stored): TELEGRAM_MINIAPP_URL, GRAFANA_ROOT_URL, VITE_VK_ID_REDIRECT_URL.

PROD_ Gitea set (per-contour, mirrors TEST_, mapped onto the unprefixed names above) — secrets: PROD_{POSTGRES_PASSWORD, GM_BASICAUTH_HASH, GRAFANA_ADMIN_PASSWORD, TELEGRAM_BOT_TOKEN, TELEGRAM_PROMO_BOT_TOKEN, EXPORT_SIGN_KEY, GATEWAY_HONEYTOKEN, REGISTRY_PASSWORD, SSH_KEY, SSH_KNOWN_HOSTS, BOTLINK_CA, BOTLINK_GATEWAY_CERT, BOTLINK_GATEWAY_KEY, BOTLINK_BOT_CERT, BOTLINK_BOT_KEY, PGBACKREST_S3_KEY, PGBACKREST_S3_KEY_SECRET, PGBACKREST_CIPHER_PASS}; variables: PROD_{REGISTRY_USER, MAIN_HOST, TG_HOST, CADDY_SITE_ADDRESS, GM_BASICAUTH_USER, LOG_LEVEL, TELEGRAM_GAME_CHANNEL_ID, TELEGRAM_CHAT_ID, TELEGRAM_SUPPORT_CHAT_ID, TELEGRAM_BOT_USERNAME, VITE_TELEGRAM_BOT_ID, VITE_TELEGRAM_LINK, VITE_TELEGRAM_GAME_CHANNEL_NAME, SMTP_RELAY_FROM, PUBLIC_BASE_URL, SMTP_RELAY_ADMIN_FROM, ADMIN_EMAIL, SMTP_RELAY_SERVICE_FROM, SERVICE_EMAIL, GF_SMTP_ENABLED, PGBACKREST_S3_ENDPOINT, PGBACKREST_S3_PORT, PGBACKREST_S3_BUCKET, PGBACKREST_S3_REGION, PGBACKREST_ARCHIVE_MODE}. The test contour uses the same names under TEST_, minus the prod-only infra (MAIN_HOST/TG_HOST/REGISTRY_*/SSH_*/BOTLINK_* and the PGBACKREST_* PITR set, which is prod-only — the test contour never archives) and plus TEST_AWG_CONF (the bot's VPN egress).

Host-side setup (outside this repo)

• edge network must exist on the host (docker network create edge).
• Host caddy route <domain> → scrabble:80 (the in-compose caddy serves HTTP in the test contour; the host caddy terminates TLS). Not needed on prod, where the contour caddy owns TLS (set CADDY_SITE_ADDRESS to the domain).
• Branch protection requires the single status check CI / gate. The unit / integration / ui jobs are path-conditional (they skip when their code did not change), and the always-running gate job aggregates them (passing when each succeeded or was skipped), so a skipped job never blocks a merge. See ../CLAUDE.md "Branching & CI".