diff --git a/PLAN.md b/PLAN.md index 0e22e08..f182eb9 100644 --- a/PLAN.md +++ b/PLAN.md @@ -33,7 +33,7 @@ status — without re-deriving decisions. | E1 | Trusted platform signal | 1 | DONE | | E2 | Currency + benefit core | 1 | DONE | | E3 | Wallet UI | 1 | DONE | -| E4 | Durability (PITR) | 2 | WIP | +| E4 | Durability (PITR) | 2 | DONE | | E5 | Payment intake | 2 | TODO | | E6 | Ads | 2 | TODO | | E7 | Admin & reports | 2 | TODO | @@ -446,8 +446,8 @@ noun agreement). Svelte whitespace/`$state` naming gotchas apply. ## E4 — Durability (PITR) -**Status:** WIP — repo artifacts landed; **prod arming pending** (owner-coordinated, before -E5). · **Release 2** · depends on: E0 (schema exists) · mechanics: PAYMENTS §14 (D4). +**Status:** DONE — armed + restore-drilled on prod (v1.13.0, 2026-07-09). · **Release 2** · +depends on: E0 (schema exists) · mechanics: PAYMENTS §14 (D4). **Goal.** Continuous WAL archiving with point-in-time recovery, armed **before the first real money** is accepted (E5 prod). Protects both money and game data. @@ -479,24 +479,26 @@ real money** is accepted (E5 prod). Protects both money and game data. `-n backend`, silently excluding `payments`); manual-restore runbook updated. - Full PITR runbook + arming sequence + recorded assessment in `deploy/README.md`. -**Prod arming (SEPARATE — owner-coordinated, before E5; NOT the artifacts PR).** Owner creates -the Selectel S3 bucket + the `PROD_PGBACKREST_*` secrets/variables (incl. `ARCHIVE_MODE=on`); -then promote `development → master` → `prod-deploy` (archive_mode on behind the maintenance -window), `pgbackrest stanza-create` + first base backup + `check`, re-run Ansible with -`-e pitr_enabled=true`, and run the restore drill on an isolated one-shot target. Exact steps: -`deploy/README.md` (point-in-time recovery — arming). +**Prod arming (completed 2026-07-09 with the v1.13.0 release).** Owner created the Selectel S3 +bucket (`erudite`, ru-6) + the `PROD_PGBACKREST_*` secrets/variables (incl. `ARCHIVE_MODE=on`); +promoted `development → master` → `prod-deploy` (archive_mode on behind the maintenance window), +then `stanza-create` + first base backup (31.9 MB cluster → 3.7 MB in the repo) + `check`, the +Ansible `-e pitr_enabled=true` timer, and a restore drill on an isolated one-shot target (data +intact, then wiped). Exact steps: `deploy/README.md` (point-in-time recovery — arming). **Tests.** Restore drill on an isolated one-shot instance (base + WAL → target timestamp), recorded in `deploy/README.md`. No app-level tests. Local verification: `docker compose config` valid + the custom PG image builds (archiving inert on the contour). -**Done-criteria.** Artifacts merged with archiving inert. **Fully done** at arming: WAL -archiving live on prod PG; a timed restore verified; cost + perf assessment reviewed; runbook -current in `deploy/README.md`. +**Done-criteria (met).** WAL archiving live on prod PG (v1.13.0); a restore verified on an +isolated one-shot target; cost + perf assessment reviewed; runbook current in +`deploy/README.md`. **Notes/risks.** The repository **cipher passphrase is unrecoverable if lost** — stored apart -from the S3 keys. Enabling `archive_mode` restarts postgres (rides the prod-deploy maintenance -window). Migrations stay expand-contract so image rollback remains DB-safe alongside PITR. +from the S3 keys. Manual/timer pgBackRest runs use `docker exec -u postgres … --pg1-user=scrabble` +(docker exec is root; the DB superuser role is `scrabble`, not `postgres`) — the systemd timer ++ the runbook carry this. Enabling `archive_mode` restarts postgres (rode the prod-deploy +maintenance window). Migrations stay expand-contract so image rollback remains DB-safe with PITR. --- diff --git a/deploy/README.md b/deploy/README.md index f1dc886..c9d7170 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -236,6 +236,7 @@ The main host archives Postgres continuously with **pgBackRest** to **Selectel S (encrypted at rest, path-style addressing) so the database can be restored to any moment — protecting the money ledger and the game state against corruption or host loss. It is the primary recovery path; the migration-window `pg_dump` above is the secondary net. +**Live on the prod main host since v1.13.0** (armed + restore-drilled 2026-07-09). **Shape.** A daily full **base backup** (a systemd timer on the main host, `04:00`) plus **continuous WAL** archived by Postgres `archive_command` (a segment is forced at least every @@ -258,8 +259,9 @@ absent/NaN-safe, so they stay quiet until archiving is armed. `pg_stat_wal`: WAL is generated at **~0.77 MB/day** and the database is **~9.6 MB**. At 30-day retention on Selectel S3 (~2 ₽/GB·month) the archive is **under ~0.3 GB → well under 1 ₽/month** (compression halves it again); request volume is trivial. Performance impact is -**negligible**: archive-push moves tiny compressed segments, and the daily full base backup -is a ~10 MB, sub-second job on the 2 vCPU host. Revisit both if traffic grows ~100× (watch +**negligible**: archive-push moves tiny compressed segments, and the daily full base backup is +small (the whole cluster is ~32 MB → ~3.7 MB compressed in the repo) and checkpoint-bound +(~1.5 min wall, minimal CPU/I-O) on the 2 vCPU host. Revisit if traffic grows ~100× (watch `node_exporter` during a base backup). **Arming (owner-coordinated, once, before real payments).** Ships disarmed; to turn it on: @@ -276,14 +278,22 @@ is a ~10 MB, sub-second job on the 2 vCPU host. Revisit both if traffic grows ~1 2. Promote `development → master`, tag, and run **prod-deploy**. The roll recreates postgres with `archive_mode=on` behind the maintenance page. (Archive pushes fail harmlessly for the minute until step 3 creates the repository — the WAL is retained, not lost.) -3. On the main host, create the repository, take the first base backup, and verify: +3. On the main host, create the repository, verify archiving, take the first base backup. Run + pgBackRest **as the `postgres` OS user** (`-u postgres` — lock-dir/PGDATA consistency with + `archive-push`) and connect to the DB **as the superuser role** (`--pg1-user=scrabble`, the + `POSTGRES_USER`, not `postgres`); it inherits the container's `PGBACKREST_*` env: ```sh - docker exec scrabble-postgres pgbackrest --stanza=scrabble stanza-create - docker exec scrabble-postgres pgbackrest --stanza=scrabble --type=full backup - docker exec scrabble-postgres pgbackrest --stanza=scrabble check + docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble stanza-create + docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check + docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble --type=full backup + docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble info # (info takes no --pg1-user) ``` -4. Enable the daily timer: `ansible-playbook site.yml -e pitr_enabled=true` (installs + starts - `pgbackrest-backup.timer` on the main host). + The few `archive-push` failures logged between `archive_mode=on` and `stanza-create` are the + expected transient (WAL retained, not lost); clear the counter afterwards with + `docker exec -u postgres scrabble-postgres psql -U scrabble -d scrabble -c "SELECT pg_stat_reset_shared('archiver');"` + so it does not trip the failing-archive alert. +4. Enable the daily timer: `ansible-playbook site.yml --limit main -e pitr_enabled=true` + (installs + starts `pgbackrest-backup.timer` on the main host). 5. Confirm in Grafana that the two archiving alerts are green (`last_archive_age` now tracks a real number, `failed_count` flat). @@ -292,17 +302,26 @@ an **isolated, one-shot** target (a throwaway VM or a container with no ingress) the matching Postgres major, an empty `PGDATA`, and the same `PGBACKREST_*` environment: ```sh -pgbackrest --stanza=scrabble --type=time "--target=YYYY-MM-DD HH:MM:SS+00" --delta restore -# start Postgres; it replays WAL to the target; then verify a known row / the ledger tail +# Throwaway container from the DB image (carries pgBackRest), the live PGBACKREST_* env, an +# empty PGDATA and no app network; restore, recover, verify, then wipe (-v drops the data). +IMG=$(docker inspect -f '{{.Config.Image}}' scrabble-postgres) +docker exec scrabble-postgres env | grep '^PGBACKREST_' > /tmp/drill.env; echo POSTGRES_PASSWORD=drill >> /tmp/drill.env +docker run -d --name pitr-drill --env-file /tmp/drill.env --entrypoint sleep "$IMG" infinity +docker exec pitr-drill sh -c 'rm -rf /var/lib/postgresql/data/*; chown -R postgres:postgres /var/lib/postgresql/data; chmod 700 /var/lib/postgresql/data' +# --type=immediate = to the base backup's consistency point; --type=time "--target=" for PITR +docker exec -u postgres pitr-drill pgbackrest --stanza=scrabble --pg1-path=/var/lib/postgresql/data --type=immediate restore +docker exec -u postgres pitr-drill pg_ctl -D /var/lib/postgresql/data -w start +docker exec -u postgres pitr-drill psql -U scrabble -d scrabble -c "SELECT count(*) FROM backend.accounts;" # verify data intact +docker rm -fv pitr-drill; rm -f /tmp/drill.env # wipe the restored real data ``` The target holds **real money + personal data** while it exists — keep it network-isolated and **destroy it (wipe `PGDATA` + the instance) afterwards**. Record each drill (date, target timestamp, outcome) here: -| Date | Target timestamp | Result | +| Date | Target | Result | | --- | --- | --- | -| _pending first arming_ | — | — | +| 2026-07-09 | latest (`--type=immediate`) | PASS — v1.13.0 arming: 31.9 MB cluster restored from the encrypted S3 repo (3.7 MB compressed), recovered to consistency, `backend.accounts` intact; drill instance wiped. | **bot-link cert rotation:** regenerate (`deploy/gen-certs.sh /tmp/c --force`), reset the five `PROD_BOTLINK_*` secrets from `/tmp/c`, and re-run the workflow — both hosts redeploy diff --git a/deploy/ansible/roles/main/templates/pgbackrest-backup.service.j2 b/deploy/ansible/roles/main/templates/pgbackrest-backup.service.j2 index 7565a09..389d893 100644 --- a/deploy/ansible/roles/main/templates/pgbackrest-backup.service.j2 +++ b/deploy/ansible/roles/main/templates/pgbackrest-backup.service.j2 @@ -10,6 +10,8 @@ Requires=docker.service [Service] Type=oneshot -# docker exec runs as the image's postgres user and inherits the container's PGBACKREST_* -# environment (the S3 repository + cipher), so no repository config is needed on the host. -ExecStart=/usr/bin/docker exec scrabble-postgres pgbackrest --stanza=scrabble --type=full backup +# docker exec defaults to root, so run as the postgres OS user (-u postgres) for lock-dir and +# PGDATA consistency with archive-push, and connect to the database as the superuser role via +# --pg1-user: that role is the POSTGRES_USER (scrabble), not "postgres". It inherits the +# container's PGBACKREST_* environment (the S3 repository + cipher), so no host-side config. +ExecStart=/usr/bin/docker exec -u postgres scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble --type=full backup