docker exec defaults to root, so the systemd base-backup timer connected to the
database as role "root" (then "postgres") — neither exists; the superuser role is
POSTGRES_USER (scrabble). Run the timer's pgBackRest as the postgres OS user (-u postgres,
for lock-dir/PGDATA consistency with archive-push) and connect with --pg1-user=scrabble.
archive_command (run by the postgres server process) was already correct.
Also record the point-in-time-recovery arming + restore drill in deploy/README.md
(correct the runbook commands to the same invocation, fill the drill log) and mark the
durability work done in PLAN.md.
Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.
- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
deploy/README.md; record the measured cost/perf assessment.
Idempotent playbooks under deploy/ansible/ prepare both production hosts:
docker-ce + compose plugin, a non-sudo deploy service account holding the CI
deploy key, key-only sshd, default-deny ufw, fail2ban, unattended upgrades and
chrony. The main host also opens 80/443/9443 and creates the external edge
network; the tg host verifies direct Bot API egress (the no-VPN assumption).
The application is deployed separately by the prod-deploy workflow (later
phase), running as the deploy account this playbook provisions.
Ilia Denisov