fix(deploy): run the base-backup timer's pgBackRest as the postgres role

docker exec defaults to root, so the systemd base-backup timer connected to the
database as role "root" (then "postgres") — neither exists; the superuser role is
POSTGRES_USER (scrabble). Run the timer's pgBackRest as the postgres OS user (-u postgres,
for lock-dir/PGDATA consistency with archive-push) and connect with --pg1-user=scrabble.
archive_command (run by the postgres server process) was already correct.

Also record the point-in-time-recovery arming + restore drill in deploy/README.md
(correct the runbook commands to the same invocation, fill the drill log) and mark the
durability work done in PLAN.md.

PLAN.md

@@ -33,7 +33,7 @@ status — without re-deriving decisions.
 | E1 | Trusted platform signal | 1 | DONE |
 | E2 | Currency + benefit core | 1 | DONE |
 | E3 | Wallet UI | 1 | DONE |
-| E4 | Durability (PITR) | 2 | WIP |
+| E4 | Durability (PITR) | 2 | DONE |
 | E5 | Payment intake | 2 | TODO |
 | E6 | Ads | 2 | TODO |
 | E7 | Admin & reports | 2 | TODO |
@@ -446,8 +446,8 @@ noun agreement). Svelte whitespace/`$state` naming gotchas apply.
 
 ## E4 — Durability (PITR)
 
-**Status:** WIP — repo artifacts landed; **prod arming pending** (owner-coordinated, before
+**Status:** DONE — armed + restore-drilled on prod (v1.13.0, 2026-07-09). · **Release 2** ·
-E5). · **Release 2** · depends on: E0 (schema exists) · mechanics: PAYMENTS §14 (D4).
+depends on: E0 (schema exists) · mechanics: PAYMENTS §14 (D4).
 
 **Goal.** Continuous WAL archiving with point-in-time recovery, armed **before the first
 real money** is accepted (E5 prod). Protects both money and game data.
@@ -479,24 +479,26 @@ real money** is accepted (E5 prod). Protects both money and game data.
   `-n backend`, silently excluding `payments`); manual-restore runbook updated.
 - Full PITR runbook + arming sequence + recorded assessment in `deploy/README.md`.
 
-**Prod arming (SEPARATE — owner-coordinated, before E5; NOT the artifacts PR).** Owner creates
+**Prod arming (completed 2026-07-09 with the v1.13.0 release).** Owner created the Selectel S3
-the Selectel S3 bucket + the `PROD_PGBACKREST_*` secrets/variables (incl. `ARCHIVE_MODE=on`);
+bucket (`erudite`, ru-6) + the `PROD_PGBACKREST_*` secrets/variables (incl. `ARCHIVE_MODE=on`);
-then promote `development → master` → `prod-deploy` (archive_mode on behind the maintenance
+promoted `development → master` → `prod-deploy` (archive_mode on behind the maintenance window),
-window), `pgbackrest stanza-create` + first base backup + `check`, re-run Ansible with
+then `stanza-create` + first base backup (31.9 MB cluster → 3.7 MB in the repo) + `check`, the
-`-e pitr_enabled=true`, and run the restore drill on an isolated one-shot target. Exact steps:
+Ansible `-e pitr_enabled=true` timer, and a restore drill on an isolated one-shot target (data
-`deploy/README.md` (point-in-time recovery — arming).
+intact, then wiped). Exact steps: `deploy/README.md` (point-in-time recovery — arming).
 
 **Tests.** Restore drill on an isolated one-shot instance (base + WAL → target timestamp),
 recorded in `deploy/README.md`. No app-level tests. Local verification: `docker compose config`
 valid + the custom PG image builds (archiving inert on the contour).
 
-**Done-criteria.** Artifacts merged with archiving inert. **Fully done** at arming: WAL
+**Done-criteria (met).** WAL archiving live on prod PG (v1.13.0); a restore verified on an
-archiving live on prod PG; a timed restore verified; cost + perf assessment reviewed; runbook
+isolated one-shot target; cost + perf assessment reviewed; runbook current in
-current in `deploy/README.md`.
+`deploy/README.md`.
 
 **Notes/risks.** The repository **cipher passphrase is unrecoverable if lost** — stored apart
-from the S3 keys. Enabling `archive_mode` restarts postgres (rides the prod-deploy maintenance
+from the S3 keys. Manual/timer pgBackRest runs use `docker exec -u postgres … --pg1-user=scrabble`
-window). Migrations stay expand-contract so image rollback remains DB-safe alongside PITR.
+(docker exec is root; the DB superuser role is `scrabble`, not `postgres`) — the systemd timer
++ the runbook carry this. Enabling `archive_mode` restarts postgres (rode the prod-deploy
+maintenance window). Migrations stay expand-contract so image rollback remains DB-safe with PITR.
 
 ---
 

deploy/README.md

@@ -236,6 +236,7 @@ The main host archives Postgres continuously with **pgBackRest** to **Selectel S
 (encrypted at rest, path-style addressing) so the database can be restored to any moment —
 protecting the money ledger and the game state against corruption or host loss. It is the
 primary recovery path; the migration-window `pg_dump` above is the secondary net.
+**Live on the prod main host since v1.13.0** (armed + restore-drilled 2026-07-09).
 
 **Shape.** A daily full **base backup** (a systemd timer on the main host, `04:00`) plus
 **continuous WAL** archived by Postgres `archive_command` (a segment is forced at least every
@@ -258,8 +259,9 @@ absent/NaN-safe, so they stay quiet until archiving is armed.
 `pg_stat_wal`: WAL is generated at **~0.77 MB/day** and the database is **~9.6 MB**. At
 30-day retention on Selectel S3 (~2 ₽/GB·month) the archive is **under ~0.3 GB → well under
 1 ₽/month** (compression halves it again); request volume is trivial. Performance impact is
-**negligible**: archive-push moves tiny compressed segments, and the daily full base backup
+**negligible**: archive-push moves tiny compressed segments, and the daily full base backup is
-is a ~10 MB, sub-second job on the 2 vCPU host. Revisit both if traffic grows ~100× (watch
+small (the whole cluster is ~32 MB → ~3.7 MB compressed in the repo) and checkpoint-bound
+(~1.5 min wall, minimal CPU/I-O) on the 2 vCPU host. Revisit if traffic grows ~100× (watch
 `node_exporter` during a base backup).
 
 **Arming (owner-coordinated, once, before real payments).** Ships disarmed; to turn it on:
@@ -276,14 +278,22 @@ is a ~10 MB, sub-second job on the 2 vCPU host. Revisit both if traffic grows ~1
 2. Promote `development → master`, tag, and run **prod-deploy**. The roll recreates postgres
    with `archive_mode=on` behind the maintenance page. (Archive pushes fail harmlessly for the
    minute until step 3 creates the repository — the WAL is retained, not lost.)
-3. On the main host, create the repository, take the first base backup, and verify:
+3. On the main host, create the repository, verify archiving, take the first base backup. Run
+   pgBackRest **as the `postgres` OS user** (`-u postgres` — lock-dir/PGDATA consistency with
+   `archive-push`) and connect to the DB **as the superuser role** (`--pg1-user=scrabble`, the
+   `POSTGRES_USER`, not `postgres`); it inherits the container's `PGBACKREST_*` env:
    ```sh
-   docker exec scrabble-postgres pgbackrest --stanza=scrabble stanza-create
+   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble stanza-create
-   docker exec scrabble-postgres pgbackrest --stanza=scrabble --type=full backup
+   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check
-   docker exec scrabble-postgres pgbackrest --stanza=scrabble check
+   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble --type=full backup
+   docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble info   # (info takes no --pg1-user)
    ```
-4. Enable the daily timer: `ansible-playbook site.yml -e pitr_enabled=true` (installs + starts
+   The few `archive-push` failures logged between `archive_mode=on` and `stanza-create` are the
-   `pgbackrest-backup.timer` on the main host).
+   expected transient (WAL retained, not lost); clear the counter afterwards with
+   `docker exec -u postgres scrabble-postgres psql -U scrabble -d scrabble -c "SELECT pg_stat_reset_shared('archiver');"`
+   so it does not trip the failing-archive alert.
+4. Enable the daily timer: `ansible-playbook site.yml --limit main -e pitr_enabled=true`
+   (installs + starts `pgbackrest-backup.timer` on the main host).
 5. Confirm in Grafana that the two archiving alerts are green (`last_archive_age` now tracks a
    real number, `failed_count` flat).
 
@@ -292,17 +302,26 @@ an **isolated, one-shot** target (a throwaway VM or a container with no ingress)
 the matching Postgres major, an empty `PGDATA`, and the same `PGBACKREST_*` environment:
 
 ```sh
-pgbackrest --stanza=scrabble --type=time "--target=YYYY-MM-DD HH:MM:SS+00" --delta restore
+# Throwaway container from the DB image (carries pgBackRest), the live PGBACKREST_* env, an
-# start Postgres; it replays WAL to the target; then verify a known row / the ledger tail
+# empty PGDATA and no app network; restore, recover, verify, then wipe (-v drops the data).
+IMG=$(docker inspect -f '{{.Config.Image}}' scrabble-postgres)
+docker exec scrabble-postgres env | grep '^PGBACKREST_' > /tmp/drill.env; echo POSTGRES_PASSWORD=drill >> /tmp/drill.env
+docker run -d --name pitr-drill --env-file /tmp/drill.env --entrypoint sleep "$IMG" infinity
+docker exec pitr-drill sh -c 'rm -rf /var/lib/postgresql/data/*; chown -R postgres:postgres /var/lib/postgresql/data; chmod 700 /var/lib/postgresql/data'
+# --type=immediate = to the base backup's consistency point; --type=time "--target=<ts>" for PITR
+docker exec -u postgres pitr-drill pgbackrest --stanza=scrabble --pg1-path=/var/lib/postgresql/data --type=immediate restore
+docker exec -u postgres pitr-drill pg_ctl -D /var/lib/postgresql/data -w start
+docker exec -u postgres pitr-drill psql -U scrabble -d scrabble -c "SELECT count(*) FROM backend.accounts;"  # verify data intact
+docker rm -fv pitr-drill; rm -f /tmp/drill.env   # wipe the restored real data
 ```
 
 The target holds **real money + personal data** while it exists — keep it network-isolated and
 **destroy it (wipe `PGDATA` + the instance) afterwards**. Record each drill (date, target
 timestamp, outcome) here:
 
-| Date | Target timestamp | Result |
+| Date | Target | Result |
 | --- | --- | --- |
-| _pending first arming_ | — | — |
+| 2026-07-09 | latest (`--type=immediate`) | PASS — v1.13.0 arming: 31.9 MB cluster restored from the encrypted S3 repo (3.7 MB compressed), recovered to consistency, `backend.accounts` intact; drill instance wiped. |
 
 **bot-link cert rotation:** regenerate (`deploy/gen-certs.sh /tmp/c --force`), reset the
 five `PROD_BOTLINK_*` secrets from `/tmp/c`, and re-run the workflow — both hosts redeploy

deploy/ansible/roles/main/templates/pgbackrest-backup.service.j2

@@ -10,6 +10,8 @@ Requires=docker.service
 
 [Service]
 Type=oneshot
-# docker exec runs as the image's postgres user and inherits the container's PGBACKREST_*
+# docker exec defaults to root, so run as the postgres OS user (-u postgres) for lock-dir and
-# environment (the S3 repository + cipher), so no repository config is needed on the host.
+# PGDATA consistency with archive-push, and connect to the database as the superuser role via
-ExecStart=/usr/bin/docker exec scrabble-postgres pgbackrest --stanza=scrabble --type=full backup
+# --pg1-user: that role is the POSTGRES_USER (scrabble), not "postgres". It inherits the
+# container's PGBACKREST_* environment (the S3 repository + cipher), so no host-side config.
+ExecStart=/usr/bin/docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble --type=full backup