feat(deploy): continuous WAL archiving to S3 for point-in-time recovery
CI / changes (pull_request) Successful in 4s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 24s
CI / ui (pull_request) Successful in 1m9s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m41s

Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.

- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
  in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
  repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
  write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
  disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
  test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
  silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
  deploy/README.md; record the measured cost/perf assessment.
This commit is contained in:
Ilia Denisov
2026-07-09 00:58:04 +02:00
parent 04976a64e8
commit e922f5f3b9
16 changed files with 359 additions and 36 deletions
+15
View File
@@ -13,6 +13,21 @@ POSTGRES_DB=scrabble
POSTGRES_USER=scrabble
POSTGRES_PASSWORD=change-me # required
# --- Point-in-time recovery (pgBackRest -> S3; PROD main host only) ----------
# Continuous WAL archiving for PITR. PROD-ONLY: the test contour never archives (these
# stay unset there and archive_mode stays off). The artifact ships DISARMED —
# PGBACKREST_ARCHIVE_MODE defaults off; arm it only after setting up the S3 repository +
# secrets and creating the stanza, then flip it on (deploy/README.md, point-in-time
# recovery). Gitea: PROD_PGBACKREST_* — endpoint/bucket/region/archive-mode are variables,
# the two S3 keys + the cipher passphrase are secrets.
PGBACKREST_ARCHIVE_MODE=off # on = archiving active (arm only after the stanza exists)
PGBACKREST_S3_ENDPOINT= # e.g. s3.ru-1.storage.selcloud.ru (Selectel S3, path-style addressing)
PGBACKREST_S3_BUCKET= # bucket name; lowercase, no dots/underscores
PGBACKREST_S3_REGION= # e.g. ru-1
PGBACKREST_S3_KEY= # secret: S3 access key id
PGBACKREST_S3_KEY_SECRET= # secret: S3 secret access key
PGBACKREST_CIPHER_PASS= # secret: repo encryption passphrase — KEEP SAFE, stored apart from the S3 keys (losing it makes the archive unrecoverable)
# --- Dictionary -------------------------------------------------------------
# scrabble-dictionary release tag baked into the image as the SEED dictionary for a
# FRESH volume (image build-arg; also labels the resident seed version). After first