fix(deploy): run the base-backup timer's pgBackRest as the postgres role
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m9s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m46s

docker exec defaults to root, so the systemd base-backup timer connected to the
database as role "root" (then "postgres") — neither exists; the superuser role is
POSTGRES_USER (scrabble). Run the timer's pgBackRest as the postgres OS user (-u postgres,
for lock-dir/PGDATA consistency with archive-push) and connect with --pg1-user=scrabble.
archive_command (run by the postgres server process) was already correct.

Also record the point-in-time-recovery arming + restore drill in deploy/README.md
(correct the runbook commands to the same invocation, fill the drill log) and mark the
durability work done in PLAN.md.
This commit is contained in:
Ilia Denisov
2026-07-09 02:03:59 +02:00
parent 850884d077
commit 7123ba439d
3 changed files with 52 additions and 29 deletions
+16 -14
View File
@@ -33,7 +33,7 @@ status — without re-deriving decisions.
| E1 | Trusted platform signal | 1 | DONE |
| E2 | Currency + benefit core | 1 | DONE |
| E3 | Wallet UI | 1 | DONE |
| E4 | Durability (PITR) | 2 | WIP |
| E4 | Durability (PITR) | 2 | DONE |
| E5 | Payment intake | 2 | TODO |
| E6 | Ads | 2 | TODO |
| E7 | Admin & reports | 2 | TODO |
@@ -446,8 +446,8 @@ noun agreement). Svelte whitespace/`$state` naming gotchas apply.
## E4 — Durability (PITR)
**Status:** WIP — repo artifacts landed; **prod arming pending** (owner-coordinated, before
E5). · **Release 2** · depends on: E0 (schema exists) · mechanics: PAYMENTS §14 (D4).
**Status:** DONE — armed + restore-drilled on prod (v1.13.0, 2026-07-09). · **Release 2** ·
depends on: E0 (schema exists) · mechanics: PAYMENTS §14 (D4).
**Goal.** Continuous WAL archiving with point-in-time recovery, armed **before the first
real money** is accepted (E5 prod). Protects both money and game data.
@@ -479,24 +479,26 @@ real money** is accepted (E5 prod). Protects both money and game data.
`-n backend`, silently excluding `payments`); manual-restore runbook updated.
- Full PITR runbook + arming sequence + recorded assessment in `deploy/README.md`.
**Prod arming (SEPARATE — owner-coordinated, before E5; NOT the artifacts PR).** Owner creates
the Selectel S3 bucket + the `PROD_PGBACKREST_*` secrets/variables (incl. `ARCHIVE_MODE=on`);
then promote `development → master` → `prod-deploy` (archive_mode on behind the maintenance
window), `pgbackrest stanza-create` + first base backup + `check`, re-run Ansible with
`-e pitr_enabled=true`, and run the restore drill on an isolated one-shot target. Exact steps:
`deploy/README.md` (point-in-time recovery — arming).
**Prod arming (completed 2026-07-09 with the v1.13.0 release).** Owner created the Selectel S3
bucket (`erudite`, ru-6) + the `PROD_PGBACKREST_*` secrets/variables (incl. `ARCHIVE_MODE=on`);
promoted `development → master` → `prod-deploy` (archive_mode on behind the maintenance window),
then `stanza-create` + first base backup (31.9 MB cluster → 3.7 MB in the repo) + `check`, the
Ansible `-e pitr_enabled=true` timer, and a restore drill on an isolated one-shot target (data
intact, then wiped). Exact steps: `deploy/README.md` (point-in-time recovery — arming).
**Tests.** Restore drill on an isolated one-shot instance (base + WAL → target timestamp),
recorded in `deploy/README.md`. No app-level tests. Local verification: `docker compose config`
valid + the custom PG image builds (archiving inert on the contour).
**Done-criteria.** Artifacts merged with archiving inert. **Fully done** at arming: WAL
archiving live on prod PG; a timed restore verified; cost + perf assessment reviewed; runbook
current in `deploy/README.md`.
**Done-criteria (met).** WAL archiving live on prod PG (v1.13.0); a restore verified on an
isolated one-shot target; cost + perf assessment reviewed; runbook current in
`deploy/README.md`.
**Notes/risks.** The repository **cipher passphrase is unrecoverable if lost** — stored apart
from the S3 keys. Enabling `archive_mode` restarts postgres (rides the prod-deploy maintenance
window). Migrations stay expand-contract so image rollback remains DB-safe alongside PITR.
from the S3 keys. Manual/timer pgBackRest runs use `docker exec -u postgres … --pg1-user=scrabble`
(docker exec is root; the DB superuser role is `scrabble`, not `postgres`) — the systemd timer
+ the runbook carry this. Enabling `archive_mode` restarts postgres (rode the prod-deploy
maintenance window). Migrations stay expand-contract so image rollback remains DB-safe with PITR.
---