fix(deploy): run the base-backup timer's pgBackRest as the postgres role
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m9s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m46s

docker exec defaults to root, so the systemd base-backup timer connected to the
database as role "root" (then "postgres") — neither exists; the superuser role is
POSTGRES_USER (scrabble). Run the timer's pgBackRest as the postgres OS user (-u postgres,
for lock-dir/PGDATA consistency with archive-push) and connect with --pg1-user=scrabble.
archive_command (run by the postgres server process) was already correct.

Also record the point-in-time-recovery arming + restore drill in deploy/README.md
(correct the runbook commands to the same invocation, fill the drill log) and mark the
durability work done in PLAN.md.
This commit is contained in:
Ilia Denisov
2026-07-09 02:03:59 +02:00
parent 850884d077
commit 7123ba439d
3 changed files with 52 additions and 29 deletions
+31 -12
View File
@@ -236,6 +236,7 @@ The main host archives Postgres continuously with **pgBackRest** to **Selectel S
(encrypted at rest, path-style addressing) so the database can be restored to any moment —
protecting the money ledger and the game state against corruption or host loss. It is the
primary recovery path; the migration-window `pg_dump` above is the secondary net.
**Live on the prod main host since v1.13.0** (armed + restore-drilled 2026-07-09).
**Shape.** A daily full **base backup** (a systemd timer on the main host, `04:00`) plus
**continuous WAL** archived by Postgres `archive_command` (a segment is forced at least every
@@ -258,8 +259,9 @@ absent/NaN-safe, so they stay quiet until archiving is armed.
`pg_stat_wal`: WAL is generated at **~0.77 MB/day** and the database is **~9.6 MB**. At
30-day retention on Selectel S3 (~2 ₽/GB·month) the archive is **under ~0.3 GB → well under
1 ₽/month** (compression halves it again); request volume is trivial. Performance impact is
**negligible**: archive-push moves tiny compressed segments, and the daily full base backup
is a ~10 MB, sub-second job on the 2 vCPU host. Revisit both if traffic grows ~100× (watch
**negligible**: archive-push moves tiny compressed segments, and the daily full base backup is
small (the whole cluster is ~32 MB → ~3.7 MB compressed in the repo) and checkpoint-bound
(~1.5 min wall, minimal CPU/I-O) on the 2 vCPU host. Revisit if traffic grows ~100× (watch
`node_exporter` during a base backup).
**Arming (owner-coordinated, once, before real payments).** Ships disarmed; to turn it on:
@@ -276,14 +278,22 @@ is a ~10 MB, sub-second job on the 2 vCPU host. Revisit both if traffic grows ~1
2. Promote `development → master`, tag, and run **prod-deploy**. The roll recreates postgres
with `archive_mode=on` behind the maintenance page. (Archive pushes fail harmlessly for the
minute until step 3 creates the repository — the WAL is retained, not lost.)
3. On the main host, create the repository, take the first base backup, and verify:
3. On the main host, create the repository, verify archiving, take the first base backup. Run
pgBackRest **as the `postgres` OS user** (`-u postgres` — lock-dir/PGDATA consistency with
`archive-push`) and connect to the DB **as the superuser role** (`--pg1-user=scrabble`, the
`POSTGRES_USER`, not `postgres`); it inherits the container's `PGBACKREST_*` env:
```sh
docker exec scrabble-postgres pgbackrest --stanza=scrabble stanza-create
docker exec scrabble-postgres pgbackrest --stanza=scrabble --type=full backup
docker exec scrabble-postgres pgbackrest --stanza=scrabble check
docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble stanza-create
docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check
docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble --type=full backup
docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble info # (info takes no --pg1-user)
```
4. Enable the daily timer: `ansible-playbook site.yml -e pitr_enabled=true` (installs + starts
`pgbackrest-backup.timer` on the main host).
The few `archive-push` failures logged between `archive_mode=on` and `stanza-create` are the
expected transient (WAL retained, not lost); clear the counter afterwards with
`docker exec -u postgres scrabble-postgres psql -U scrabble -d scrabble -c "SELECT pg_stat_reset_shared('archiver');"`
so it does not trip the failing-archive alert.
4. Enable the daily timer: `ansible-playbook site.yml --limit main -e pitr_enabled=true`
(installs + starts `pgbackrest-backup.timer` on the main host).
5. Confirm in Grafana that the two archiving alerts are green (`last_archive_age` now tracks a
real number, `failed_count` flat).
@@ -292,17 +302,26 @@ an **isolated, one-shot** target (a throwaway VM or a container with no ingress)
the matching Postgres major, an empty `PGDATA`, and the same `PGBACKREST_*` environment:
```sh
pgbackrest --stanza=scrabble --type=time "--target=YYYY-MM-DD HH:MM:SS+00" --delta restore
# start Postgres; it replays WAL to the target; then verify a known row / the ledger tail
# Throwaway container from the DB image (carries pgBackRest), the live PGBACKREST_* env, an
# empty PGDATA and no app network; restore, recover, verify, then wipe (-v drops the data).
IMG=$(docker inspect -f '{{.Config.Image}}' scrabble-postgres)
docker exec scrabble-postgres env | grep '^PGBACKREST_' > /tmp/drill.env; echo POSTGRES_PASSWORD=drill >> /tmp/drill.env
docker run -d --name pitr-drill --env-file /tmp/drill.env --entrypoint sleep "$IMG" infinity
docker exec pitr-drill sh -c 'rm -rf /var/lib/postgresql/data/*; chown -R postgres:postgres /var/lib/postgresql/data; chmod 700 /var/lib/postgresql/data'
# --type=immediate = to the base backup's consistency point; --type=time "--target=<ts>" for PITR
docker exec -u postgres pitr-drill pgbackrest --stanza=scrabble --pg1-path=/var/lib/postgresql/data --type=immediate restore
docker exec -u postgres pitr-drill pg_ctl -D /var/lib/postgresql/data -w start
docker exec -u postgres pitr-drill psql -U scrabble -d scrabble -c "SELECT count(*) FROM backend.accounts;" # verify data intact
docker rm -fv pitr-drill; rm -f /tmp/drill.env # wipe the restored real data
```
The target holds **real money + personal data** while it exists — keep it network-isolated and
**destroy it (wipe `PGDATA` + the instance) afterwards**. Record each drill (date, target
timestamp, outcome) here:
| Date | Target timestamp | Result |
| Date | Target | Result |
| --- | --- | --- |
| _pending first arming_ | — | — |
| 2026-07-09 | latest (`--type=immediate`) | PASS — v1.13.0 arming: 31.9 MB cluster restored from the encrypted S3 repo (3.7 MB compressed), recovered to consistency, `backend.accounts` intact; drill instance wiped. |
**bot-link cert rotation:** regenerate (`deploy/gen-certs.sh /tmp/c --force`), reset the
five `PROD_BOTLINK_*` secrets from `/tmp/c`, and re-run the workflow — both hosts redeploy