feat(deploy): continuous WAL archiving to S3 for point-in-time recovery

Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.

- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
  in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
  repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
  write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
  disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
  test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
  silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
  deploy/README.md; record the measured cost/perf assessment.

deploy/README.md

@@ -220,10 +220,86 @@ release tag, so any prior release is reachable.
 **Migrations** must be **expand-contract** (backward-compatible; goose is forward-only):
 the automatic rollback is image-only and never restores the DB. A deploy that changes
 `backend/internal/postgres/migrations/` opens a maintenance window — the backend (sole
-writer) is stopped for a consistent `pg_dump` into `/opt/scrabble/dumps` before the new
-backend migrates. **Manual DB restore** (only if a migration was destructive):
-`docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA backend CASCADE'`,
-then pipe the dump into the same `psql`, and redeploy the matching old tag.
+writer) is stopped for a consistent **whole-database** `pg_dump` into `/opt/scrabble/dumps`
+(it covers `backend` **and** `payments`) before the new backend migrates. **Manual DB
+restore** (only if a migration was destructive): drop the app schemas in the same instance
+and pipe the dump back —
+`docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA IF EXISTS backend CASCADE; DROP SCHEMA IF EXISTS payments CASCADE;'`,
+then `docker exec -i scrabble-postgres psql -U scrabble -d scrabble < the-dump.sql`, and
+redeploy the matching old tag. This dump is a belt-and-braces net for a bad migration;
+**point-in-time recovery** (below) is the primary recovery path once armed, and the only one
+that survives losing the host.
+
+## Point-in-time recovery (PITR)
+
+The main host archives Postgres continuously with **pgBackRest** to **Selectel S3**
+(encrypted at rest, path-style addressing) so the database can be restored to any moment —
+protecting the money ledger and the game state against corruption or host loss. It is the
+primary recovery path; the migration-window `pg_dump` above is the secondary net.
+
+**Shape.** A daily full **base backup** (a systemd timer on the main host, `04:00`) plus
+**continuous WAL** archived by Postgres `archive_command` (a segment is forced at least every
+5 minutes, so the recovery point is never more than a few minutes behind). Retention is **30
+days** (`repo1-retention-full=30`); the repository is AES-256-CBC encrypted. This is main-host
+only — the test contour never archives.
+
+**Wiring.** pgBackRest ships inside the DB image (`deploy/postgres/Dockerfile`); the
+repository + S3 credentials + cipher are the `PGBACKREST_*` environment on the postgres
+service in `docker-compose.prod.yml`, rendered from the `PROD_` Gitea set by
+`write-prod-env.sh`. Archiving is gated by **`PGBACKREST_ARCHIVE_MODE` (default `off`)**, so
+shipping or redeploying this stack does **not** start archiving — the artifact is inert until
+armed, which is why an un-armed prod deploy can never pile WAL onto the disk. The base-backup
+timer is provisioned by the Ansible `main` role behind `pitr_enabled` (also default off). Two
+Grafana alerts watch health: `WAL archiving failing` (`pg_stat_archiver_failed_count` rising)
+and `WAL archiving stalled` (`pg_stat_archiver_last_archive_age` over 30 min) — both
+absent/NaN-safe, so they stay quiet until archiving is armed.
+
+**Assessment (owner-reviewed; the gate before the first real money).** Measured on prod
+`pg_stat_wal`: WAL is generated at **~0.77 MB/day** and the database is **~9.6 MB**. At
+30-day retention on Selectel S3 (~2 ₽/GB·month) the archive is **under ~0.3 GB → well under
+1 ₽/month** (compression halves it again); request volume is trivial. Performance impact is
+**negligible**: archive-push moves tiny compressed segments, and the daily full base backup
+is a ~10 MB, sub-second job on the 2 vCPU host. Revisit both if traffic grows ~100× (watch
+`node_exporter` during a base backup).
+
+**Arming (owner-coordinated, once, before real payments).** Ships disarmed; to turn it on:
+
+1. **Owner (Selectel + Gitea):** create an S3 bucket (name **lowercase, no dots/underscores**)
+   and an S3 access key/secret; pick a repository **cipher passphrase** and store it **apart
+   from the S3 keys** (losing it makes the archive unrecoverable). Set the Gitea `PROD_` set —
+   variables `PROD_PGBACKREST_S3_ENDPOINT` / `_S3_BUCKET` / `_S3_REGION` and
+   `PROD_PGBACKREST_ARCHIVE_MODE=on`; secrets `PROD_PGBACKREST_S3_KEY` / `_S3_KEY_SECRET` /
+   `_CIPHER_PASS`.
+2. Promote `development → master`, tag, and run **prod-deploy**. The roll recreates postgres
+   with `archive_mode=on` behind the maintenance page. (Archive pushes fail harmlessly for the
+   minute until step 3 creates the repository — the WAL is retained, not lost.)
+3. On the main host, create the repository, take the first base backup, and verify:
+   ```sh
+   docker exec scrabble-postgres pgbackrest --stanza=scrabble stanza-create
+   docker exec scrabble-postgres pgbackrest --stanza=scrabble --type=full backup
+   docker exec scrabble-postgres pgbackrest --stanza=scrabble check
+   ```
+4. Enable the daily timer: `ansible-playbook site.yml -e pitr_enabled=true` (installs + starts
+   `pgbackrest-backup.timer` on the main host).
+5. Confirm in Grafana that the two archiving alerts are green (`last_archive_age` now tracks a
+   real number, `failed_count` flat).
+
+**Restore drill (proves recoverability; re-run after arming and after any major change).** On
+an **isolated, one-shot** target (a throwaway VM or a container with no ingress), pgBackRest,
+the matching Postgres major, an empty `PGDATA`, and the same `PGBACKREST_*` environment:
+
+```sh
+pgbackrest --stanza=scrabble --type=time "--target=YYYY-MM-DD HH:MM:SS+00" --delta restore
+# start Postgres; it replays WAL to the target; then verify a known row / the ledger tail
+```
+
+The target holds **real money + personal data** while it exists — keep it network-isolated and
+**destroy it (wipe `PGDATA` + the instance) afterwards**. Record each drill (date, target
+timestamp, outcome) here:
+
+| Date | Target timestamp | Result |
+| --- | --- | --- |
+| _pending first arming_ | — | — |
 
 **bot-link cert rotation:** regenerate (`deploy/gen-certs.sh /tmp/c --force`), reset the
 five `PROD_BOTLINK_*` secrets from `/tmp/c`, and re-run the workflow — both hosts redeploy
@@ -249,14 +325,17 @@ VITE_VK_APP_LINK, VITE_VK_APP_ID`. **Derived from `PUBLIC_BASE_URL` at deploy**
 `PROD_{POSTGRES_PASSWORD, GM_BASICAUTH_HASH, GRAFANA_ADMIN_PASSWORD, TELEGRAM_BOT_TOKEN,
 TELEGRAM_PROMO_BOT_TOKEN, EXPORT_SIGN_KEY, GATEWAY_HONEYTOKEN, REGISTRY_PASSWORD, SSH_KEY,
 SSH_KNOWN_HOSTS, BOTLINK_CA, BOTLINK_GATEWAY_CERT, BOTLINK_GATEWAY_KEY, BOTLINK_BOT_CERT,
-BOTLINK_BOT_KEY}`; variables:
+BOTLINK_BOT_KEY, PGBACKREST_S3_KEY, PGBACKREST_S3_KEY_SECRET, PGBACKREST_CIPHER_PASS}`;
+variables:
 `PROD_{REGISTRY_USER, MAIN_HOST, TG_HOST, CADDY_SITE_ADDRESS, GM_BASICAUTH_USER, LOG_LEVEL,
 TELEGRAM_GAME_CHANNEL_ID, TELEGRAM_CHAT_ID, TELEGRAM_SUPPORT_CHAT_ID, TELEGRAM_BOT_USERNAME,
 VITE_TELEGRAM_BOT_ID, VITE_TELEGRAM_LINK, VITE_TELEGRAM_GAME_CHANNEL_NAME, SMTP_RELAY_FROM,
 PUBLIC_BASE_URL, SMTP_RELAY_ADMIN_FROM, ADMIN_EMAIL, SMTP_RELAY_SERVICE_FROM, SERVICE_EMAIL,
-GF_SMTP_ENABLED}`. The test contour uses the same names under `TEST_`, minus the prod-only
-infra (`MAIN_HOST`/`TG_HOST`/`REGISTRY_*`/`SSH_*`/`BOTLINK_*`, which the test deploy generates
-or runs locally) and plus `TEST_AWG_CONF` (the bot's VPN egress).
+GF_SMTP_ENABLED, PGBACKREST_S3_ENDPOINT, PGBACKREST_S3_BUCKET, PGBACKREST_S3_REGION,
+PGBACKREST_ARCHIVE_MODE}`. The test contour uses the same names under `TEST_`, minus the
+prod-only infra (`MAIN_HOST`/`TG_HOST`/`REGISTRY_*`/`SSH_*`/`BOTLINK_*` and the `PGBACKREST_*`
+PITR set, which is prod-only — the test contour never archives) and plus `TEST_AWG_CONF` (the
+bot's VPN egress).
 
 ## Host-side setup (outside this repo)