feat(deploy): continuous WAL archiving to S3 for point-in-time recovery

Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.

- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
  in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
  repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
  write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
  disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
  test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
  silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
  deploy/README.md; record the measured cost/perf assessment.

PLAN.md

@@ -33,7 +33,7 @@ status — without re-deriving decisions.
 | E1 | Trusted platform signal | 1 | DONE |
 | E2 | Currency + benefit core | 1 | DONE |
 | E3 | Wallet UI | 1 | DONE |
-| E4 | Durability (PITR) | 2 | TODO |
+| E4 | Durability (PITR) | 2 | WIP |
 | E5 | Payment intake | 2 | TODO |
 | E6 | Ads | 2 | TODO |
 | E7 | Admin & reports | 2 | TODO |
@@ -446,40 +446,57 @@ noun agreement). Svelte whitespace/`$state` naming gotchas apply.
 
 ## E4 — Durability (PITR)
 
-**Status:** TODO · **Release 2** · depends on: E0 (schema exists) · mechanics: PAYMENTS §14.
+**Status:** WIP — repo artifacts landed; **prod arming pending** (owner-coordinated, before
+E5). · **Release 2** · depends on: E0 (schema exists) · mechanics: PAYMENTS §14 (D4).
 
 **Goal.** Continuous WAL archiving with point-in-time recovery, armed **before the first
 real money** is accepted (E5 prod). Protects both money and game data.
 
-**Work.**
+**Locked decisions (this stage).**
 
-- Add WAL archiving to the prod Postgres (pgBackRest or WAL-G): base backups + continuous
-  WAL to a second host or object storage. Wire into `deploy/` (compose/prod overlay +
-  ansible `deploy/ansible/`), config + secrets under the `PROD_` prefix.
-- Restore runbook in `deploy/README.md`: base + WAL replay to a timestamp; test the restore
-  on a scratch target.
-- Keep the existing migration-time `pg_dump` (`deploy/prod-deploy.sh`) as belt-and-braces.
+- **Tool: pgBackRest.** **Destination: Selectel S3** object storage (encrypted AES-256-CBC,
+  path-style addressing), **prod main host only** — the test contour never archives. (D4 left
+  the tool + destination open; resolved here. Warm replica stays deferred, per D4.)
+- **Retention 30 days**, **daily full base backup** (a systemd timer) + continuous WAL
+  (`archive_command`, a 5-minute forced switch bounds the recovery point). Assessment
+  (owner-reviewed prod gate): measured prod WAL **~0.77 MB/day**, DB **~9.6 MB** → archive
+  **< 1 ₽/month** on Selectel S3, **negligible** perf impact. Recorded in `deploy/README.md`.
+- **Archiving ships gated OFF** (`PGBACKREST_ARCHIVE_MODE` on the DB, `pitr_enabled` for the
+  timer — both default off), so the merged/redeployed artifact is inert until armed and an
+  un-armed prod deploy **cannot** pile WAL onto the disk.
 
-**Mandatory pre-prod assessment (owner-required, gates the Release 2 prod rollout).**
-Before the first money goes live, produce and record here:
+**Work (landed in the artifacts PR into `development`).**
 
-1. **Disk-storage cost estimate** for the WAL archive + base backups (retention window ×
-   WAL volume; account for both hosts / object-storage pricing). This can change hosting
-   sizing.
-2. **PG performance-degradation assessment** from continuous archiving (write amplification,
-   archive_command latency, backup I/O contention) on the prod-sized instance.
+- pgBackRest in the DB image (`deploy/postgres/Dockerfile`); postgres becomes a built + pushed
+  image (prod overlay `image:` + `prod-deploy.yaml` build/push list).
+- Repository config + `archive_mode` via `PGBACKREST_*` env on the prod-overlay postgres
+  service; secrets/vars rendered by `deploy/write-prod-env.sh` from the `PROD_PGBACKREST_*`
+  Gitea set; **parity added to `prod-rollback.yaml`** (a rollback must not disarm archiving).
+- Daily base-backup systemd timer (Ansible `main` role, gated by `pitr_enabled`).
+- Two Grafana alerts on the exporter's `pg_stat_archiver` metrics (failing / stalled),
+  absent/NaN-safe on the contour.
+- Belt-and-braces `pg_dump` fixed to dump the **whole DB** (`deploy/prod-deploy.sh` was
+  `-n backend`, silently excluding `payments`); manual-restore runbook updated.
+- Full PITR runbook + arming sequence + recorded assessment in `deploy/README.md`.
 
-Neither blocks building the plan, but both **must** be completed and reviewed before the
-prod release — surface the numbers to the owner (they may change host settings).
+**Prod arming (SEPARATE — owner-coordinated, before E5; NOT the artifacts PR).** Owner creates
+the Selectel S3 bucket + the `PROD_PGBACKREST_*` secrets/variables (incl. `ARCHIVE_MODE=on`);
+then promote `development → master` → `prod-deploy` (archive_mode on behind the maintenance
+window), `pgbackrest stanza-create` + first base backup + `check`, re-run Ansible with
+`-e pitr_enabled=true`, and run the restore drill on an isolated one-shot target. Exact steps:
+`deploy/README.md` (point-in-time recovery — arming).
 
-**Tests.** Restore drill on a scratch instance (base + WAL → target timestamp) documented
-and passing. No app-level tests.
+**Tests.** Restore drill on an isolated one-shot instance (base + WAL → target timestamp),
+recorded in `deploy/README.md`. No app-level tests. Local verification: `docker compose config`
+valid + the custom PG image builds (archiving inert on the contour).
 
-**Done-criteria.** WAL archiving live on prod PG; a timed restore verified; cost + perf
-assessment recorded and owner-reviewed; runbook in `deploy/README.md`.
+**Done-criteria.** Artifacts merged with archiving inert. **Fully done** at arming: WAL
+archiving live on prod PG; a timed restore verified; cost + perf assessment reviewed; runbook
+current in `deploy/README.md`.
 
-**Notes/risks.** The tg host is weak (1 vCPU) — if the archive lands there, watch
-contention. Migrations stay expand-contract so image rollback remains DB-safe alongside PITR.
+**Notes/risks.** The repository **cipher passphrase is unrecoverable if lost** — stored apart
+from the S3 keys. Enabling `archive_mode` restarts postgres (rides the prod-deploy maintenance
+window). Migrations stay expand-contract so image rollback remains DB-safe alongside PITR.
 
 ---