feat(deploy): continuous WAL archiving to S3 for point-in-time recovery
CI / changes (pull_request) Successful in 4s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 24s
CI / ui (pull_request) Successful in 1m9s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m41s

Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.

- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
  in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
  repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
  write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
  disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
  test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
  silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
  deploy/README.md; record the measured cost/perf assessment.
This commit is contained in:
Ilia Denisov
2026-07-09 00:58:04 +02:00
parent 04976a64e8
commit e922f5f3b9
16 changed files with 359 additions and 36 deletions
@@ -212,3 +212,62 @@ groups:
- evaluator: { type: gt, params: [0.8] }
labels: { severity: warning }
annotations: { summary: 'Postgres using over 80% of max_connections.' }
# Continuous WAL archiving health (pgBackRest -> S3, point-in-time recovery). Both rules
# read the postgres_exporter's built-in pg_stat_archiver metrics and are absent/NaN-safe:
# on the test contour (and any host before archiving is armed) archive_mode is off, so
# failed_count stays 0 and last_archive_age is NaN — neither threshold trips.
- orgId: 1
name: scrabble-backup
folder: Alerts
interval: 1m
rules:
- uid: pg_archive_failing
title: WAL archiving failing
condition: C
for: 15m
noDataState: OK
execErrState: OK
data:
- refId: A
relativeTimeRange: { from: 900, to: 0 }
datasourceUid: prometheus
model:
refId: A
expr: increase(pg_stat_archiver_failed_count[15m])
instant: true
- refId: C
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: A
conditions:
- evaluator: { type: gt, params: [0] }
labels: { severity: critical }
annotations: { summary: 'pgBackRest archive_command is failing — PITR is degrading and pg_wal can fill the disk. Run `docker exec scrabble-postgres pgbackrest --stanza=scrabble check`.' }
- uid: pg_archive_stalled
title: WAL archiving stalled
condition: C
for: 15m
noDataState: OK
execErrState: OK
data:
- refId: A
relativeTimeRange: { from: 300, to: 0 }
datasourceUid: prometheus
model:
refId: A
expr: pg_stat_archiver_last_archive_age
instant: true
- refId: C
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: A
conditions:
- evaluator: { type: gt, params: [1800] }
labels: { severity: critical }
annotations: { summary: 'No WAL segment archived for over 30 minutes (archive_timeout is 5m) — archiving is stalled; the PITR recovery point is frozen and pg_wal may grow.' }