fix(deploy): run the base-backup timer's pgBackRest as the postgres role
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m9s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m46s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m9s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m46s
docker exec defaults to root, so the systemd base-backup timer connected to the database as role "root" (then "postgres") — neither exists; the superuser role is POSTGRES_USER (scrabble). Run the timer's pgBackRest as the postgres OS user (-u postgres, for lock-dir/PGDATA consistency with archive-push) and connect with --pg1-user=scrabble. archive_command (run by the postgres server process) was already correct. Also record the point-in-time-recovery arming + restore drill in deploy/README.md (correct the runbook commands to the same invocation, fill the drill log) and mark the durability work done in PLAN.md.
This commit is contained in:
+31
-12
@@ -236,6 +236,7 @@ The main host archives Postgres continuously with **pgBackRest** to **Selectel S
|
||||
(encrypted at rest, path-style addressing) so the database can be restored to any moment —
|
||||
protecting the money ledger and the game state against corruption or host loss. It is the
|
||||
primary recovery path; the migration-window `pg_dump` above is the secondary net.
|
||||
**Live on the prod main host since v1.13.0** (armed + restore-drilled 2026-07-09).
|
||||
|
||||
**Shape.** A daily full **base backup** (a systemd timer on the main host, `04:00`) plus
|
||||
**continuous WAL** archived by Postgres `archive_command` (a segment is forced at least every
|
||||
@@ -258,8 +259,9 @@ absent/NaN-safe, so they stay quiet until archiving is armed.
|
||||
`pg_stat_wal`: WAL is generated at **~0.77 MB/day** and the database is **~9.6 MB**. At
|
||||
30-day retention on Selectel S3 (~2 ₽/GB·month) the archive is **under ~0.3 GB → well under
|
||||
1 ₽/month** (compression halves it again); request volume is trivial. Performance impact is
|
||||
**negligible**: archive-push moves tiny compressed segments, and the daily full base backup
|
||||
is a ~10 MB, sub-second job on the 2 vCPU host. Revisit both if traffic grows ~100× (watch
|
||||
**negligible**: archive-push moves tiny compressed segments, and the daily full base backup is
|
||||
small (the whole cluster is ~32 MB → ~3.7 MB compressed in the repo) and checkpoint-bound
|
||||
(~1.5 min wall, minimal CPU/I-O) on the 2 vCPU host. Revisit if traffic grows ~100× (watch
|
||||
`node_exporter` during a base backup).
|
||||
|
||||
**Arming (owner-coordinated, once, before real payments).** Ships disarmed; to turn it on:
|
||||
@@ -276,14 +278,22 @@ is a ~10 MB, sub-second job on the 2 vCPU host. Revisit both if traffic grows ~1
|
||||
2. Promote `development → master`, tag, and run **prod-deploy**. The roll recreates postgres
|
||||
with `archive_mode=on` behind the maintenance page. (Archive pushes fail harmlessly for the
|
||||
minute until step 3 creates the repository — the WAL is retained, not lost.)
|
||||
3. On the main host, create the repository, take the first base backup, and verify:
|
||||
3. On the main host, create the repository, verify archiving, take the first base backup. Run
|
||||
pgBackRest **as the `postgres` OS user** (`-u postgres` — lock-dir/PGDATA consistency with
|
||||
`archive-push`) and connect to the DB **as the superuser role** (`--pg1-user=scrabble`, the
|
||||
`POSTGRES_USER`, not `postgres`); it inherits the container's `PGBACKREST_*` env:
|
||||
```sh
|
||||
docker exec scrabble-postgres pgbackrest --stanza=scrabble stanza-create
|
||||
docker exec scrabble-postgres pgbackrest --stanza=scrabble --type=full backup
|
||||
docker exec scrabble-postgres pgbackrest --stanza=scrabble check
|
||||
docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble stanza-create
|
||||
docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check
|
||||
docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble --type=full backup
|
||||
docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble info # (info takes no --pg1-user)
|
||||
```
|
||||
4. Enable the daily timer: `ansible-playbook site.yml -e pitr_enabled=true` (installs + starts
|
||||
`pgbackrest-backup.timer` on the main host).
|
||||
The few `archive-push` failures logged between `archive_mode=on` and `stanza-create` are the
|
||||
expected transient (WAL retained, not lost); clear the counter afterwards with
|
||||
`docker exec -u postgres scrabble-postgres psql -U scrabble -d scrabble -c "SELECT pg_stat_reset_shared('archiver');"`
|
||||
so it does not trip the failing-archive alert.
|
||||
4. Enable the daily timer: `ansible-playbook site.yml --limit main -e pitr_enabled=true`
|
||||
(installs + starts `pgbackrest-backup.timer` on the main host).
|
||||
5. Confirm in Grafana that the two archiving alerts are green (`last_archive_age` now tracks a
|
||||
real number, `failed_count` flat).
|
||||
|
||||
@@ -292,17 +302,26 @@ an **isolated, one-shot** target (a throwaway VM or a container with no ingress)
|
||||
the matching Postgres major, an empty `PGDATA`, and the same `PGBACKREST_*` environment:
|
||||
|
||||
```sh
|
||||
pgbackrest --stanza=scrabble --type=time "--target=YYYY-MM-DD HH:MM:SS+00" --delta restore
|
||||
# start Postgres; it replays WAL to the target; then verify a known row / the ledger tail
|
||||
# Throwaway container from the DB image (carries pgBackRest), the live PGBACKREST_* env, an
|
||||
# empty PGDATA and no app network; restore, recover, verify, then wipe (-v drops the data).
|
||||
IMG=$(docker inspect -f '{{.Config.Image}}' scrabble-postgres)
|
||||
docker exec scrabble-postgres env | grep '^PGBACKREST_' > /tmp/drill.env; echo POSTGRES_PASSWORD=drill >> /tmp/drill.env
|
||||
docker run -d --name pitr-drill --env-file /tmp/drill.env --entrypoint sleep "$IMG" infinity
|
||||
docker exec pitr-drill sh -c 'rm -rf /var/lib/postgresql/data/*; chown -R postgres:postgres /var/lib/postgresql/data; chmod 700 /var/lib/postgresql/data'
|
||||
# --type=immediate = to the base backup's consistency point; --type=time "--target=<ts>" for PITR
|
||||
docker exec -u postgres pitr-drill pgbackrest --stanza=scrabble --pg1-path=/var/lib/postgresql/data --type=immediate restore
|
||||
docker exec -u postgres pitr-drill pg_ctl -D /var/lib/postgresql/data -w start
|
||||
docker exec -u postgres pitr-drill psql -U scrabble -d scrabble -c "SELECT count(*) FROM backend.accounts;" # verify data intact
|
||||
docker rm -fv pitr-drill; rm -f /tmp/drill.env # wipe the restored real data
|
||||
```
|
||||
|
||||
The target holds **real money + personal data** while it exists — keep it network-isolated and
|
||||
**destroy it (wipe `PGDATA` + the instance) afterwards**. Record each drill (date, target
|
||||
timestamp, outcome) here:
|
||||
|
||||
| Date | Target timestamp | Result |
|
||||
| Date | Target | Result |
|
||||
| --- | --- | --- |
|
||||
| _pending first arming_ | — | — |
|
||||
| 2026-07-09 | latest (`--type=immediate`) | PASS — v1.13.0 arming: 31.9 MB cluster restored from the encrypted S3 repo (3.7 MB compressed), recovered to consistency, `backend.accounts` intact; drill instance wiped. |
|
||||
|
||||
**bot-link cert rotation:** regenerate (`deploy/gen-certs.sh /tmp/c --force`), reset the
|
||||
five `PROD_BOTLINK_*` secrets from `/tmp/c`, and re-run the workflow — both hosts redeploy
|
||||
|
||||
Reference in New Issue
Block a user