scrabble-game/deploy/README.md

# deploy

The full Scrabble contour: `backend` + `gateway` + the static `landing` + Postgres +
the Telegram `validator` + `bot` (the bot with a VPN sidecar) + the observability stack
(OTel Collector → Prometheus + Tempo → Grafana), fronted by a **caddy** that owns a single
`/_gm` Basic-Auth (the admin console + Grafana). Topology and the decision record are in
[`../docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md) §13; this file is the
operational reference for **every environment variable**.

## Services

| Service | Image | Role |
| --- | --- | --- |
| `caddy` | `caddy:2-alpine` | Edge proxy (alias `scrabble` on `edge`): single `/_gm` Basic-Auth → admin console + Grafana; `/app/`, `/telegram/` + the Connect path → gateway; the catch-all (incl. `/`) → landing. TLS per `CADDY_SITE_ADDRESS`. |
| `gateway` | built (`gateway/Dockerfile`, target `gateway`) | Public edge; serves the embedded game SPA at `/app/` + `/telegram/`; Connect-RPC edge. `/` redirects to `/app/`. |
| `landing` | built (`gateway/Dockerfile`, target `landing`) | Static landing page at `/` (caddy:2-alpine + the shared Vite build, `deploy/landing/Caddyfile`); absorbs stray public paths. |
| `backend` | built (`backend/Dockerfile`) | Domain service; bakes in the DAWG dictionaries; runs migrations at boot. |
| `postgres` | `postgres:17-alpine` | Database (named volume, `pg_isready` healthcheck). |
| `renderer` | built (`renderer/Dockerfile`) | Finished-game image-render sidecar (Node + skia-canvas running the shared `ui/src/lib/gameimage.ts`); internal-only HTTP at `renderer:8090`, called by the backend for the PNG export artifact. |
| `validator` | built (`platform/telegram/Dockerfile`, target `validator`) | Telegram HMAC validator (no VPN, no Bot API); internal gRPC at `validator:9091`. Game login depends only on this. |
| `vpn` + `bot` | sidecar + built (`platform/telegram/Dockerfile`, target `bot`) | Telegram bot, gated to the **`telegram-local`** profile; egresses through the AmneziaWG sidecar and dials the gateway bot-link (mTLS) at `gateway:9443`. The test contour activates the profile; the prod **main** host omits it and runs the bot standalone on its **own host** (`docker-compose.bot.yml`, no VPN — native Bot API egress). |
| `otelcol` | `otel/opentelemetry-collector-contrib` | OTLP/gRPC `:4317` → Prometheus scrape (`:9464`) + Tempo. |
| `prometheus` | `prom/prometheus` | Metrics, 15d retention (7d in prod). |
| `tempo` | `grafana/tempo` | Traces, 72h retention. |
| `grafana` | `grafana/grafana` | Dashboards (provisioned), anonymous-admin behind caddy's `/_gm/grafana`. |
| `node_exporter` | `quay.io/prometheus/node-exporter` | Host CPU/memory/disk metrics (Prometheus job `node`); the OOM signal on the tight prod main host (2 vCPU / 1.9 GiB). |

Networking: inter-service traffic is on the private `internal` network
(project-scoped DNS); only `caddy` joins the shared external `edge` network so the
host caddy can reach it at `scrabble:80`. `edge` must already exist on the host
(`docker network create edge`).

## Run it

**Locally** — copy the template, fill the required values, bring it up:

```sh
cp deploy/.env.example deploy/.env     # then edit deploy/.env
docker network create edge             # once, if it does not exist
cd deploy && docker compose up -d --build
```

**In CI** (the test contour) — `.gitea/workflows/ci.yaml`'s `deploy` job maps the
Gitea **`TEST_`-prefixed** secrets/variables onto the unprefixed names below and
runs `docker compose up -d --build` on the runner host. The prod deploy maps the
**`PROD_`** set the same way. So a Gitea secret named `TEST_POSTGRES_PASSWORD`
feeds the compose's `POSTGRES_PASSWORD`, etc.

Three naming classes in Gitea:
- **Per-contour** (`TEST_`/`PROD_<NAME>`) — values that differ between the contours
  (bots, hosts, public origin, log level, email senders): the common case below.
- **Shared** (one unprefixed `<NAME>`, no prefix) — values identical on every contour,
  stored once: `DICT_VERSION`, `SMTP_RELAY_HOST`/`PORT`/`TLS`/`USER`/`PASS`,
  `GRAFANA_SMTP_PORT`, `VITE_VK_APP_LINK`, `VITE_VK_APP_ID`, `GATEWAY_VK_APP_SECRET`,
  `GATEWAY_VK_ID_CLIENT_SECRET` (one Selectel relay + one pair of VK apps for all contours).
- **Derived** — not stored at all; the deploy computes them from `PUBLIC_BASE_URL`
  (`deploy/write-prod-env.sh`, and the `ci.yaml` deploy step): `TELEGRAM_MINIAPP_URL`
  (`+ /telegram/`), `GRAFANA_ROOT_URL` (`+ /_gm/grafana/`), `VITE_VK_ID_REDIRECT_URL`
  (`+ /app/`). Set them directly only for a local `.env` run.

The deploy job also **seeds the config files** (`caddy`, `otelcol`, `prometheus`,
`tempo`, `grafana`) to a stable host path (`$HOME/.scrabble-deploy`) and sets
`SCRABBLE_CONFIG_DIR` to it before `up`. The runner's checkout is an ephemeral act
workspace that is removed after the job — binding config straight from it would
dangle the mounts in the long-lived containers (Grafana would log
`no such file or directory`). Locally `SCRABBLE_CONFIG_DIR` defaults to `.`, so the
compose binds from this directory.

## Required variables

`docker compose` aborts immediately if any of these is unset (they use `:?`):

| Variable | Gitea kind | Purpose |
| --- | --- | --- |
| `POSTGRES_PASSWORD` | secret | Postgres password (also embedded in `BACKEND_POSTGRES_DSN`). |
| `GM_BASICAUTH_HASH` | secret | bcrypt hash gating `/_gm` (admin console + Grafana). Generate with `docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'`. |
| `TELEGRAM_MINIAPP_URL` | derived | The Mini App URL the bot hands out in deep links / buttons. The deploy derives `PUBLIC_BASE_URL + /telegram/`; set it directly only for a local run (compose still `:?`-requires it). |
| `EXPORT_SIGN_KEY` | secret | HMAC key signing the public finished-game export download URLs (`/dl/*`). Generate with `openssl rand -base64 32`. |
| `BACKEND_ROBOKASSA_MERCHANT_LOGIN` | secret | Robokassa shop login for the direct RUB rail. Empty leaves the rail off. |
| `BACKEND_ROBOKASSA_PASSWORD1` / `…_PASSWORD2` | secret | Robokassa pass phrases: Password1 signs the launch request, Password2 signs/verifies the Result callback. Use the shop's **test** pair while `…_ROBOKASSA_TEST=1`, the **live** pair for real money. (Password3 — Robokassa's JWT-invoice API — is unused.) |
| `BACKEND_ROBOKASSA_TEST` | **variable** | `1` runs test payments against the test pass phrases (no real money); empty/`0` is live. A variable, not a secret, so go-live is a flag flip + redeploy, not a secret rotation. |
| `BACKEND_ROBOKASSA_{WEB,ANDROID}_{MERCHANT_LOGIN,PASSWORD1,PASSWORD2,TEST}` | secret / variable | Multi-shop direct rail (D42): per-channel Robokassa shops. The legacy `BACKEND_ROBOKASSA_*` seeds the **web** shop; `…_WEB_*` overrides it, `…_ANDROID_*` adds the **android** (RuStore) shop. Each credits the one `direct` wallet; the cabinet Result URL per shop is `/pay/robokassa/result/{web,android}`. Empty login ⇒ that channel unconfigured (order routing falls back to the web shop). |

**Plus the bot token** — `TELEGRAM_BOT_TOKEN` (secret), shared by the validator (HMAC
secret) and the bot (Bot API). It defaults to empty in compose, but both **fail at
boot** when it is empty.

**Conditionally — `AWG_CONF`** (secret): the AmneziaWG config for the VPN sidecar, needed
only when the `telegram-local` profile runs (the test contour and local runs with the
bot). It is **not** `:?`-guarded — compose interpolates profiled-out services too, so the
prod main host (no VPN) must not require it. It **must not contain a `DNS=` line** — that
hijacks the shared netns's resolv.conf and breaks the bot resolving `otelcol` / `gateway`;
without it Docker's resolver handles `otelcol`, `gateway` and `api.telegram.org`.

## Optional variables (with defaults)

| Variable | Gitea kind | Default | Purpose |
| --- | --- | --- | --- |
| `POSTGRES_DB` | variable | `scrabble` | Database name. |
| `POSTGRES_USER` | variable | `scrabble` | Database user. |
| `DICT_VERSION` | variable (shared) | `v1.3.1` | `scrabble-dictionary` release tag baked into the backend image as the **seed for a fresh volume** (build-arg). A live contour changes dictionary through the admin console, not this; on a seeded volume a changed value is ignored (the recorded `.seed_version` marker wins — the seed-drift guard, ARCHITECTURE.md §5). One shared unprefixed Gitea variable seeds both contours and pins the CI test suite. |
| `LOG_LEVEL` | variable | `info` | Shared log level for backend / gateway / validator / bot (`debug\|info\|warn\|error`). |
| `CADDY_SITE_ADDRESS` | variable | `:80` | Caddy site address. Test: `:80` (host caddy terminates TLS). Prod: a domain, so caddy does its own ACME. |
| `GM_BASICAUTH_USER` | variable | `gm` | Username for the `/_gm` Basic-Auth. |
| `GRAFANA_ROOT_URL` | derived | `/_gm/grafana/` | Grafana root URL (sub-path serving). The deploy derives `PUBLIC_BASE_URL + /_gm/grafana/`; set the full `https://<domain>/_gm/grafana/` only for a local run. |
| `GRAFANA_ADMIN_PASSWORD` | secret | `admin` | Grafana admin password. Low impact (the login form is disabled, access is anonymous-admin behind caddy) but set it anyway. |
| `TELEGRAM_GAME_CHANNEL_ID` | variable | _(empty)_ | The bot's game-channel id; empty/`0` disables channel posts. |
| `TELEGRAM_TEST_ENV` | _pinned_ | `false` | `true` routes the bot through Telegram's test environment (`.../bot<token>/test/METHOD`). **The CI test contour pins this to `true` in `ci.yaml`** (the contour is the test environment) — it is not a Gitea variable. Set it in `.env` for a local run; prod leaves it `false`. |
| `TELEGRAM_API_BASE_URL` | variable | _(empty)_ | Override the Bot API host (a mock/self-hosted server); empty = `https://api.telegram.org`. |
| `VITE_TELEGRAM_BOT_ID` | variable | _(empty)_ | UI build-arg: numeric bot id for the web Login Widget. |
| `VITE_TELEGRAM_LINK` | variable | _(empty)_ | UI build-arg: friend-invite Mini App link (full URL, `https://telegram.me/<bot>/<app>` — `<app>` is the Mini App short name from BotFather). |
| `VITE_TELEGRAM_GAME_CHANNEL_NAME` | variable | _(empty)_ | UI build-arg: the landing "Play in Telegram" link, the bot's game channel (e.g. `https://telegram.me/Erudit_Game`). |
| `VITE_VK_APP_LINK` | variable (shared) | _(empty)_ | UI build-arg: the landing "Play on VK" link, the VK Mini App (full URL, `https://vk.com/app<id>`). One VK Mini App for all contours. |
| `VITE_VK_APP_ID` | variable (shared) | _(empty)_ | VK ID "Web" app id (`client_id`) for VK web-login linking. Baked into the SPA authorize URL and reused as the gateway's `GATEWAY_VK_ID_APP_ID`. Empty disables the `link.vk.*` ops. One VK ID "Web" app for all contours. |
| `VITE_VK_ID_REDIRECT_URL` | derived | _(empty)_ | VK ID trusted redirect URL — must match the app's registered redirect exactly. The deploy derives `PUBLIC_BASE_URL + /app/` (used by the SPA and as the gateway's exchange `redirect_uri`); set it only for a local run. |
| `GATEWAY_VK_APP_SECRET` | secret (shared) | _(empty)_ | The VK Mini App's protected key (`client_secret`): the gateway verifies the launch-parameter signature in-process under it (offline HMAC, no VK API call). Empty disables the VK auth path (`auth.vk`). One VK Mini App for all contours. |
| `GATEWAY_VK_ID_CLIENT_SECRET` | secret (shared) | _(empty)_ | The VK ID "Web" app's protected key (`client_secret`) for the gateway's server-side confidential code exchange. A SEPARATE VK app from `GATEWAY_VK_APP_SECRET` (the Mini App). One VK ID "Web" app for all contours. |
| `GATEWAY_HONEYTOKEN` | secret | _(empty)_ | Planted honeytoken bearer value: presenting it earns a 24h IP ban + a high-severity alarm where the IP ban is on (prod), or logs + a `gateway_abuse_banned_total{reason="honeytoken"}` metric (test, ban off). Plant the value somewhere an attacker would find it; empty disables the trap. Per-contour `TEST_`/`PROD_GATEWAY_HONEYTOKEN`. |
| `GATEWAY_BLOCKLIST_ENABLED` | variable | `false` | Enable the community IP blocklist at the edge (prod-only, same real-client-IP reason as the ban). Requires `GATEWAY_BLOCKLIST_URL`. Opt-in via `PROD_GATEWAY_BLOCKLIST_ENABLED` after verifying the feed. |
| `GATEWAY_BLOCKLIST_URL` | variable | _(empty)_ | The curated CIDR feed to fetch (Spamhaus DROP). Required when enabled; refreshed every few hours and dropped fail-open once stale (48h). `PROD_GATEWAY_BLOCKLIST_URL`. |
| `GATEWAY_BLOCKLIST_ALLOW` | variable | _(empty)_ | Comma-separated never-block set (CIDRs / bare IPs — own infra, monitoring) the feed can never block. `PROD_GATEWAY_BLOCKLIST_ALLOW`. |
| `GATEWAY_MIN_CLIENT_VERSION` | variable | _(empty)_ | **Hard** tier of the client-version gate (ARCHITECTURE.md §2). Minimum client build the gateway will serve. Empty ⇒ **dormant** (every build served, the web default). Set it to the release `vMAJOR.MINOR.PATCH` in the **same** rollout that ships an incompatible wire change, so an older bundled APK is turned away with an *update required* signal instead of failing blind — the client then degrades to an offline "Update / Play offline" notice, not a hard lockout. Validated at load; a non-empty unparseable value fails startup. |
| `GATEWAY_RECOMMENDED_CLIENT_VERSION` | variable | _(empty)_ | **Soft** tier of the client-version gate (ARCHITECTURE.md §2). A build at or above `GATEWAY_MIN_CLIENT_VERSION` but **below** this is served normally, but every gated `Execute` response carries an additive `X-Update-Recommended: 1` header ⇒ the client shows a dismissable *update available* nudge (play continues). Empty ⇒ off. Validated at load: unparseable, or **below** `GATEWAY_MIN_CLIENT_VERSION`, fails startup. Bump it (ahead of `MIN`) to nudge upgrades before a hard cut-over. |
| `VITE_GATEWAY_URL` | variable | _(empty)_ | UI build-arg: gateway origin; empty = same-origin (the usual single-origin deploy). |
| `SMTP_RELAY_HOST` | variable (shared) | _(empty)_ | Selectel SMTP relay host for confirm-code email. Empty leaves the backend on the log mailer (email disabled) — the contour still boots. One relay for every contour (limit 100 msgs / 5 min). |
| `SMTP_RELAY_PORT` | variable (shared) | `465` | Relay port. No client certificate is needed (the server cert is validated against the system roots). |
| `SMTP_RELAY_TLS` | variable (shared) | _(empty)_ | Transport security: `ssl` (implicit TLS) or `starttls`. Empty derives it from the port (implicit on `465`, STARTTLS otherwise); required for a relay on a non-standard port (e.g. Selectel's `1127` = SSL, `1126` = STARTTLS). |
| `SMTP_RELAY_USER` | secret (shared) | _(empty)_ | Relay SMTP AUTH username. |
| `SMTP_RELAY_PASS` | secret (shared) | _(empty)_ | Relay SMTP AUTH password. |
| `SMTP_RELAY_FROM` | variable | `no-reply@localhost` | Sender address. **Must use the prod domain** (`no-reply@erudit-game.ru`) — Selectel only accepts the verified sender domain — so it is the same on every contour. |
| `PUBLIC_BASE_URL` | variable | _(empty)_ | Canonical public https origin for links in the email (the contour's own URL, e.g. `https://erudit-game.ru`). **Required by the backend whenever `SMTP_RELAY_HOST` is set** — it is never derived from a request Host header (anti-injection). |
| `SMTP_RELAY_ADMIN_FROM` | variable | _(empty)_ | Backend operator-alert sender (new feedback / word complaints), distinct from the confirm-code From. Empty (with `ADMIN_EMAIL`) disables the alert worker. |
| `ADMIN_EMAIL` | variable | _(empty)_ | Backend operator-alert recipient(s); several comma-separated addresses allowed. |
| `SMTP_RELAY_SERVICE_FROM` | variable | _(empty)_ | Grafana infra-alert sender address. |
| `SERVICE_EMAIL` | variable | _(empty)_ | Grafana infra-alert recipient(s); comma-separated allowed (read by the provisioned contact point via `$__env{SERVICE_EMAIL}`). |
| `GRAFANA_SMTP_PORT` | variable (shared) | _(empty)_ | STARTTLS port Grafana dials on `SMTP_RELAY_HOST` (its client can't do the backend's implicit TLS), e.g. Selectel's `1126`. Reuses `SMTP_RELAY_HOST`/`USER`/`PASS`. |
| `GF_SMTP_ENABLED` | variable | `false` | Set `true` to enable Grafana alert emails. |

The six `VITE_*` are **build-args** baked into the gateway and landing images at
build time (both targets share one UI build stage — keep the args identical so it is
built once), so changing them requires a rebuild (`--build`), not just a restart.

## Fixed internal wiring (not operator-set)

These are hard-wired in `docker-compose.yml` (no `${...}`), pointing the services
at each other on the `internal` network — listed here so they are not mistaken for
missing config: `BACKEND_POSTGRES_DSN` (→ `postgres`, `search_path=backend`),
`GATEWAY_BACKEND_HTTP_URL`/`_GRPC_ADDR` (→ `backend`),
`GATEWAY_VALIDATOR_ADDR` (→ `validator:9091`), `BACKEND_CONNECTOR_ADDR` (→ the gateway
bot-link relay `gateway:9092`), `BACKEND_RENDERER_URL` (→ `renderer:8090`, the image-render
sidecar), the bot's `TELEGRAM_GATEWAY_ADDR` (→ `gateway:9443`,
mTLS) with the `GATEWAY_BOTLINK_*` / `TELEGRAM_BOTLINK_*` cert paths under `/certs` (the
mTLS material is generated by `deploy/gen-certs.sh`, gitignored, regenerated each
deploy), and all services' `*_OTEL_*_EXPORTER=otlp` →
`OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317`
(`_INSECURE=true`). The bot shares the VPN sidecar's netns: routing to the
collector's / gateway's internal IP is fine (connected route), but its `AWG_CONF` must
**not** set a `DNS=` directive — that hijacks resolv.conf and breaks resolving `otelcol`
/ `gateway` ("produced zero addresses"); without it the netns uses Docker's resolver,
which resolves `otelcol`, `gateway` and `api.telegram.org`. `GATEWAY_ADMIN_*` is
intentionally **unset** — caddy owns `/_gm` in the contour.

## Bumping the dictionary version

The dictionary ships as a versioned **release artifact** (`scrabble-dawg-vX.Y.Z.tar.gz`) from
[`scrabble-dictionary`](https://gitea.iliadenisov.ru/developer/scrabble-dictionary). The tag is
a build-time input with **no default** in the images. It is a single Gitea repo variable —
**`DICT_VERSION`** (unprefixed, shared across contours) — so a release bump is **one edit**:

- the CI test jobs read it via `.gitea/workflows/ci.yaml`'s top-level
  `env.DICT_VERSION: ${{ vars.DICT_VERSION }}` (the unit/integration jobs download that dawg), and
- both deploy jobs feed the same variable to `compose` as the `DICT_VERSION` build-arg that
  bakes a **fresh** volume's seed.

For local builds set `DICT_VERSION` in `deploy/.env` (template: `.env.example`); a bare
`docker build` needs `--build-arg DICT_VERSION=vX.Y.Z`. The Dockerfiles and `compose` carry no
default — a missing value fails loudly instead of baking a stale tag.

Bumping the seed is a **no-op on a live volume** (the `.seed_version` marker wins — the
seed-drift guard). A running contour/prod moves to a new release **through the admin console**
`/_gm/dictionary` (upload the tarball, preview the per-variant diff, confirm); in-flight games
keep their pinned version, new games use the new one (ARCHITECTURE.md §5).

## Production rollout

Prod runs on **two hosts** (main = full stack + ACME on the domain; tg = the bot only,
native Bot API, no VPN), one-time provisioned by **[`ansible/`](ansible/)** (docker, a
non-sudo `deploy` user holding the CI key, key-only sshd, default-deny ufw, fail2ban).
Re-run `ansible/` after a host resize — it is idempotent.

**To roll out:** merge `development → master` (CI green), then run the **`prod-deploy`**
workflow manually (Gitea → Actions → prod-deploy → run from `master`, input
`confirm=deploy`). It builds + pushes the images to the registry, ships the
compose/config/certs/env over SSH, deploys the main host with `prod-deploy.sh` (rolling,
health-gated, **auto-rollback to the previous tag**; caddy is force-recreated on its roll so
a bind-mounted `Caddyfile` change applies — its image is pinned and admin is off, so neither a
new tag nor a hot reload would pick it up), then the bot host, then probes the
public site. After `master` is green this workflow is the **only** thing that touches
prod — nothing auto-deploys there. It runs four visible jobs: **build → deploy-main →
deploy-bot → verify** (the per-service rolling shows in the deploy-main log).

**Maintenance page.** During the roll (and any migration window) `prod-deploy.sh` raises a
flag the edge caddy serves a static 503 "технические работы" page from
(`deploy/caddy/maintenance.html`), for the user-facing routes only — `/_gm` (Grafana) stays
reachable. The flag is cleared on any exit (success, a health failure + rollback, or an
error) by a shell trap, so it can never stick on. It arms from this feature's own deploy
onward (the caddy carrying the gate must be live first). This is **not** a zero-downtime
deploy — the backend is a single stateful instance (in-memory game state + push hub, no
Redis), so its swap still blips (clients auto-reconnect the live stream); the page just
makes the window graceful instead of raw 502s.

**Versioning.** Each release is a git tag `vX.Y.Z` on `master`; the deploy stamps
`git describe --tags` into every image tag, every binary (`-ldflags` → `pkg/version` →
the `service.version` telemetry attribute) and the SPA About screen. Tag the release
before running the deploy:

```sh
git tag -a v1.0.0 -m v1.0.0 && git push origin v1.0.0
```

**Manual rollback** (any time after a successful deploy). Run the **`prod-rollback`**
workflow (Gitea → Actions → prod-rollback, `confirm=rollback`). Leave `target_version`
blank to roll back to the previously deployed version (read from the host's
`PREVIOUS_TAG`), or set it to a release tag from the **Releases** page. It re-deploys
that already-published image rolling + health-gated — no rebuild, no DB migration
(image rollback is DB-safe under the expand-contract rule). The registry keeps every
release tag, so any prior release is reachable.

**Migrations** must be **expand-contract** (backward-compatible; goose is forward-only):
the automatic rollback is image-only and never restores the DB. A deploy that changes
`backend/internal/postgres/migrations/` opens a maintenance window — the backend (sole
writer) is stopped for a consistent **whole-database** `pg_dump` into `/opt/scrabble/dumps`
(it covers `backend` **and** `payments`) before the new backend migrates. **Manual DB
restore** (only if a migration was destructive): drop the app schemas in the same instance
and pipe the dump back —
`docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA IF EXISTS backend CASCADE; DROP SCHEMA IF EXISTS payments CASCADE;'`,
then `docker exec -i scrabble-postgres psql -U scrabble -d scrabble < the-dump.sql`, and
redeploy the matching old tag. This dump is a belt-and-braces net for a bad migration;
**point-in-time recovery** (below) is the primary recovery path once armed, and the only one
that survives losing the host.

## Android app build & release (RuStore)

The standalone Android app is built by a **manual** workflow, never automatically, and is separate from the
web/prod rollout — a signed APK uploaded to RuStore by hand. The build/release runbook follows.

- **Trigger:** `Actions → android-build → Run workflow` from **`master`**, input `confirm=build` (mirrors
  `prod-deploy`). **Tag the release first** (`git tag vX.Y.Z` on `master`) — the workflow refuses anything
  but a clean `vMAJOR.MINOR.PATCH` and derives `versionCode = MA*1_000_000 + MI*1_000 + PA`,
  `versionName = MA.MI.PA`. Watch it green with `python3 ~/.claude/bin/gitea-ci-watch.py`.
- **Output:** a `release APK` run artifact — **signed** when the signing secrets are present, an
  **unsigned** release APK otherwise (a dry run still proves the pipeline).
- **Runner (host-executor):** JDK 21 comes from `setup-java`; the **Android SDK is host-provisioned** —
  install it once (`sdkmanager 'platform-tools' 'platforms;android-36' 'build-tools;36.0.0'`) and grant the
  `runner` user read+exec (`sudo chmod -R a+rX /opt/android-sdk`). Override the path with the
  `ANDROID_SDK_DIR` variable. The workflow's `Verify the host Android SDK` step fails fast with the fix if
  the SDK is missing or unreadable.
- **Release keystore** (create once, **back up off-host** — losing it means the app can never be updated):
  `keytool -genkeypair -v -keystore erudit-release.jks -alias erudit -keyalg RSA -keysize 4096 -validity 10000`,
  then set the Gitea **secrets** `ANDROID_RUSTORE_KEYSTORE_BASE64` (`base64 -w0 erudit-release.jks`),
  `ANDROID_RUSTORE_KEYSTORE_PASSWORD`, `ANDROID_RUSTORE_KEY_ALIAS`, `ANDROID_RUSTORE_KEY_PASSWORD`. Set the `ANDROID_RUSTORE_URL`
  variable to the store listing once published (empty until then — the in-app update button no-ops).
- **Wire-break discipline:** the prod deploy that ships an incompatible wire change **also** bumps
  `GATEWAY_MIN_CLIENT_VERSION` to that release (see Optional variables), so an old installed APK is turned
  away cleanly instead of failing blind.
- **Upload:** download the artifact and upload it to RuStore by hand (no automated RuStore-API upload in the MVP).

## Point-in-time recovery (PITR)

The main host archives Postgres continuously with **pgBackRest** to **Selectel S3**
(encrypted at rest, path-style addressing) so the database can be restored to any moment —
protecting the money ledger and the game state against corruption or host loss. It is the
primary recovery path; the migration-window `pg_dump` above is the secondary net.
**Live on the prod main host since v1.13.0** (armed + restore-drilled 2026-07-09).

**Shape.** A daily full **base backup** (a systemd timer on the main host, `04:00`) plus
**continuous WAL** archived by Postgres `archive_command` (a segment is forced at least every
5 minutes, so the recovery point is never more than a few minutes behind). Retention is **30
days** (`repo1-retention-full=30`); the repository is AES-256-CBC encrypted. This is main-host
only — the test contour never archives.

**Wiring.** pgBackRest ships inside the DB image (`deploy/postgres/Dockerfile`); the
repository + S3 credentials + cipher are the `PGBACKREST_*` environment on the postgres
service in `docker-compose.prod.yml`, rendered from the `PROD_` Gitea set by
`write-prod-env.sh`. Archiving is gated by **`PGBACKREST_ARCHIVE_MODE` (default `off`)**, so
shipping or redeploying this stack does **not** start archiving — the artifact is inert until
armed, which is why an un-armed prod deploy can never pile WAL onto the disk. The base-backup
timer is provisioned by the Ansible `main` role behind `pitr_enabled` (also default off). Two
Grafana alerts watch health: `WAL archiving failing` (`pg_stat_archiver_failed_count` rising)
and `WAL archiving stalled` (last archive over 30 min old **while `pg_wal_size_bytes` is growing**
— the pg_wal-growth guard keeps an idle database, which archives nothing because it writes nothing,
from false-triggering during quiet hours) — both absent/NaN-safe, so they stay quiet until
archiving is armed.

**Assessment (owner-reviewed; the gate before the first real money).** Measured on prod
`pg_stat_wal`: WAL is generated at **~0.77 MB/day** and the database is **~9.6 MB**. At
30-day retention on Selectel S3 (~2 ₽/GB·month) the archive is **under ~0.3 GB → well under
1 ₽/month** (compression halves it again); request volume is trivial. Performance impact is
**negligible**: archive-push moves tiny compressed segments, and the daily full base backup is
small (the whole cluster is ~32 MB → ~3.7 MB compressed in the repo) and checkpoint-bound
(~1.5 min wall, minimal CPU/I-O) on the 2 vCPU host. Revisit if traffic grows ~100× (watch
`node_exporter` during a base backup).

**Arming (owner-coordinated, once, before real payments).** Ships disarmed; to turn it on:

1. **Owner (Selectel + Gitea):** create an S3 bucket (name **lowercase, no dots/underscores**)
   and an S3 access key/secret; pick a repository **cipher passphrase** and store it **apart
   from the S3 keys** (losing it makes the archive unrecoverable). Set the Gitea `PROD_` set —
   variables `PROD_PGBACKREST_S3_ENDPOINT` (the S3 **host only** — no `https://`, no port, no
   bucket) / `_S3_BUCKET` / `_S3_REGION`, an optional `_S3_PORT` (default 443, set only for a
   non-standard provider port), and `PROD_PGBACKREST_ARCHIVE_MODE=on`; secrets
   `PROD_PGBACKREST_S3_KEY` (the S3 **access key**) / `_S3_KEY_SECRET` (its paired **secret
   key**, shown once at creation) / `_CIPHER_PASS` (a fresh repository passphrase, e.g.
   `openssl rand -base64 48`).
2. Promote `development → master`, tag, and run **prod-deploy**. The roll recreates postgres
   with `archive_mode=on` behind the maintenance page. (Archive pushes fail harmlessly for the
   minute until step 3 creates the repository — the WAL is retained, not lost.)
3. On the main host, create the repository, verify archiving, take the first base backup. Run
   pgBackRest **as the `postgres` OS user** (`-u postgres` — lock-dir/PGDATA consistency with
   `archive-push`) and connect to the DB **as the superuser role** (`--pg1-user=scrabble`, the
   `POSTGRES_USER`, not `postgres`); it inherits the container's `PGBACKREST_*` env:
   ```sh
   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble stanza-create
   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check
   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble --type=full backup
   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble info   # (info takes no --pg1-user)
   ```
   The few `archive-push` failures logged between `archive_mode=on` and `stanza-create` are the
   expected transient (WAL retained, not lost); clear the counter afterwards with
   `docker exec -u postgres scrabble-postgres psql -U scrabble -d scrabble -c "SELECT pg_stat_reset_shared('archiver');"`
   so it does not trip the failing-archive alert.
4. Enable the daily timer: `ansible-playbook site.yml --limit main -e pitr_enabled=true`
   (installs + starts `pgbackrest-backup.timer` on the main host).
5. Confirm in Grafana that the two archiving alerts are green (`last_archive_age` now tracks a
   real number, `failed_count` flat).

**Restore drill (proves recoverability; re-run after arming and after any major change).** On
an **isolated, one-shot** target (a throwaway VM or a container with no ingress), pgBackRest,
the matching Postgres major, an empty `PGDATA`, and the same `PGBACKREST_*` environment:

```sh
# Throwaway container from the DB image (carries pgBackRest), the live PGBACKREST_* env, an
# empty PGDATA and no app network; restore, recover, verify, then wipe (-v drops the data).
IMG=$(docker inspect -f '{{.Config.Image}}' scrabble-postgres)
docker exec scrabble-postgres env | grep '^PGBACKREST_' > /tmp/drill.env; echo POSTGRES_PASSWORD=drill >> /tmp/drill.env
docker run -d --name pitr-drill --env-file /tmp/drill.env --entrypoint sleep "$IMG" infinity
docker exec pitr-drill sh -c 'rm -rf /var/lib/postgresql/data/*; chown -R postgres:postgres /var/lib/postgresql/data; chmod 700 /var/lib/postgresql/data'
# --type=immediate = to the base backup's consistency point; --type=time "--target=<ts>" for PITR
docker exec -u postgres pitr-drill pgbackrest --stanza=scrabble --pg1-path=/var/lib/postgresql/data --type=immediate restore
docker exec -u postgres pitr-drill pg_ctl -D /var/lib/postgresql/data -w start
docker exec -u postgres pitr-drill psql -U scrabble -d scrabble -c "SELECT count(*) FROM backend.accounts;"  # verify data intact
docker rm -fv pitr-drill; rm -f /tmp/drill.env   # wipe the restored real data
```

The target holds **real money + personal data** while it exists — keep it network-isolated and
**destroy it (wipe `PGDATA` + the instance) afterwards**. Record each drill (date, target
timestamp, outcome) here:

| Date | Target | Result |
| --- | --- | --- |
| 2026-07-09 | latest (`--type=immediate`) | PASS — v1.13.0 arming: 31.9 MB cluster restored from the encrypted S3 repo (3.7 MB compressed), recovered to consistency, `backend.accounts` intact; drill instance wiped. |

**bot-link cert rotation:** regenerate (`deploy/gen-certs.sh /tmp/c --force`), reset the
five `PROD_BOTLINK_*` secrets from `/tmp/c`, and re-run the workflow — both hosts redeploy
together with the fresh CA.

**Sizing / monitoring:** the main host launches undersized (2 vCPU / 1.9 GiB); the prod
overlay trims limits + `GOMAXPROCS=2` + 7d Prometheus retention, and `node_exporter` feeds
host memory to Grafana (`/_gm/grafana/`). Watch host memory and resize at Selectel when
players arrive. The per-container memory caps are enforced (Compose v2 → cgroup), so no one
service can eat all RAM, but they **overcommit** the host (~2.8 GiB of caps vs 1.9 GiB) — a
**1 GiB swap file** (Ansible `common` role, `swap_size`, `vm.swappiness=10`) cushions a
simultaneous spike into swap instead of the kernel OOM-killer. The `host_mem_low` Grafana
alert fires under 10% available.

**Shared Gitea set** (one unprefixed entry each, used by every contour) — secrets:
`GATEWAY_VK_APP_SECRET, GATEWAY_VK_ID_CLIENT_SECRET, SMTP_RELAY_USER, SMTP_RELAY_PASS`;
variables: `DICT_VERSION, SMTP_RELAY_HOST, SMTP_RELAY_PORT, SMTP_RELAY_TLS, GRAFANA_SMTP_PORT,
VITE_VK_APP_LINK, VITE_VK_APP_ID`. **Derived from `PUBLIC_BASE_URL` at deploy** (not stored):
`TELEGRAM_MINIAPP_URL, GRAFANA_ROOT_URL, VITE_VK_ID_REDIRECT_URL`.

**`PROD_` Gitea set** (per-contour, mirrors `TEST_`, mapped onto the unprefixed names above)
— secrets:
`PROD_{POSTGRES_PASSWORD, GM_BASICAUTH_HASH, GRAFANA_ADMIN_PASSWORD, TELEGRAM_BOT_TOKEN,
TELEGRAM_PROMO_BOT_TOKEN, EXPORT_SIGN_KEY, GATEWAY_HONEYTOKEN, REGISTRY_PASSWORD, SSH_KEY,
SSH_KNOWN_HOSTS, BOTLINK_CA, BOTLINK_GATEWAY_CERT, BOTLINK_GATEWAY_KEY, BOTLINK_BOT_CERT,
BOTLINK_BOT_KEY, PGBACKREST_S3_KEY, PGBACKREST_S3_KEY_SECRET, PGBACKREST_CIPHER_PASS}`;
variables:
`PROD_{REGISTRY_USER, MAIN_HOST, TG_HOST, CADDY_SITE_ADDRESS, GM_BASICAUTH_USER, LOG_LEVEL,
TELEGRAM_GAME_CHANNEL_ID, TELEGRAM_CHAT_ID, TELEGRAM_SUPPORT_CHAT_ID, TELEGRAM_BOT_USERNAME,
VITE_TELEGRAM_BOT_ID, VITE_TELEGRAM_LINK, VITE_TELEGRAM_GAME_CHANNEL_NAME, SMTP_RELAY_FROM,
PUBLIC_BASE_URL, SMTP_RELAY_ADMIN_FROM, ADMIN_EMAIL, SMTP_RELAY_SERVICE_FROM, SERVICE_EMAIL,
GF_SMTP_ENABLED, PGBACKREST_S3_ENDPOINT, PGBACKREST_S3_PORT, PGBACKREST_S3_BUCKET,
PGBACKREST_S3_REGION, PGBACKREST_ARCHIVE_MODE}`. The test contour uses the same names under `TEST_`, minus the
prod-only infra (`MAIN_HOST`/`TG_HOST`/`REGISTRY_*`/`SSH_*`/`BOTLINK_*` and the `PGBACKREST_*`
PITR set, which is prod-only — the test contour never archives) and plus `TEST_AWG_CONF` (the
bot's VPN egress).

## Host-side setup (outside this repo)

- **`edge` network** must exist on the host (`docker network create edge`).
- **Host caddy** route `<domain> → scrabble:80` (the in-compose caddy serves HTTP
  in the test contour; the host caddy terminates TLS). Not needed on prod, where the
  contour caddy owns TLS (set `CADDY_SITE_ADDRESS` to the domain).
- **Branch protection** requires the single status check `CI / gate`.
  The `unit` / `integration` / `ui` jobs are path-conditional (they skip when their
  code did not change), and the always-running `gate` job aggregates them (passing
  when each succeeded or was skipped), so a skipped job never blocks a merge. See
  [`../CLAUDE.md`](../CLAUDE.md) "Branching & CI".