feat(deploy): continuous WAL archiving to S3 for point-in-time recovery

Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.

- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
  in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
  repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
  write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
  disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
  test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
  silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
  deploy/README.md; record the measured cost/perf assessment.

deploy/ansible/README.md

@@ -45,5 +45,7 @@ safe (idempotent) and survives a host resize.
   10m×3 log rotation), `deploy` user (docker group, no sudo), key-only sshd,
   `ufw` default-deny incoming + allow SSH, fail2ban sshd jail, unattended
   upgrades, chrony, `/opt/scrabble/{config,certs,dumps,images}`.
-- **main**: `ufw` opens 80/443/9443; the external `edge` docker network.
+- **main**: `ufw` opens 80/443/9443; the external `edge` docker network; the pgBackRest
+  daily base-backup systemd timer — installed only when `pitr_enabled=true` (off by default;
+  turned on as part of arming point-in-time recovery, see [`../README.md`](../README.md)).
 - **tg**: verifies direct `api.telegram.org` egress (the no-VPN assumption).

deploy/ansible/group_vars/all.yml

@@ -27,3 +27,12 @@ docker_log_max_file: "3"
 # path (a slow service beats a killed one). Set swap_size to "0" to skip provisioning.
 swap_size: "1G"
 swap_swappiness: 10
+
+# Point-in-time recovery (pgBackRest). The base-backup systemd timer on the main host is
+# gated by pitr_enabled so provisioning stays inert until archiving is armed — the timer
+# would fail nightly if it ran before the S3 repository + stanza exist. Arm it as part of
+# the PITR rollout (deploy/README.md): flip it on with `-e pitr_enabled=true` (or a main
+# host_var) once the stanza is created. Only the `main` role reads these.
+pitr_enabled: false
+# systemd OnCalendar for the daily full base backup — a low-traffic hour on the main host.
+pitr_backup_oncalendar: "*-*-* 04:00:00"

deploy/ansible/roles/main/tasks/main.yml

@@ -16,3 +16,30 @@
   community.docker.docker_network:
     name: edge
     state: present
+
+# --- pgBackRest base-backup schedule (point-in-time recovery) ------------------
+# A daily full base backup via a systemd timer that runs pgBackRest inside the postgres
+# container; archive_command handles the continuous WAL between backups. Gated by
+# pitr_enabled so the schedule stays inert until archiving is armed (the S3 repository +
+# stanza must exist first, or the run fails nightly). Arm per deploy/README.md.
+- name: Install the pgBackRest base-backup service
+  ansible.builtin.template:
+    src: pgbackrest-backup.service.j2
+    dest: /etc/systemd/system/pgbackrest-backup.service
+    mode: "0644"
+  when: pitr_enabled | default(false) | bool
+
+- name: Install the pgBackRest base-backup timer
+  ansible.builtin.template:
+    src: pgbackrest-backup.timer.j2
+    dest: /etc/systemd/system/pgbackrest-backup.timer
+    mode: "0644"
+  when: pitr_enabled | default(false) | bool
+
+- name: Enable and start the pgBackRest base-backup timer
+  ansible.builtin.systemd:
+    name: pgbackrest-backup.timer
+    enabled: true
+    state: started
+    daemon_reload: true
+  when: pitr_enabled | default(false) | bool

deploy/ansible/roles/main/templates/pgbackrest-backup.service.j2

@@ -0,0 +1,15 @@
+# Managed by Ansible (deploy/ansible).
+# Daily full pgBackRest base backup for the payments/game database. Runs pgBackRest
+# inside the postgres container (where the repository configuration and PGDATA live);
+# the continuous WAL between base backups is handled by the container's archive_command.
+# Installed only when pitr_enabled is set (see deploy/ansible/group_vars/all.yml).
+[Unit]
+Description=pgBackRest full base backup (scrabble)
+After=docker.service
+Requires=docker.service
+
+[Service]
+Type=oneshot
+# docker exec runs as the image's postgres user and inherits the container's PGBACKREST_*
+# environment (the S3 repository + cipher), so no repository config is needed on the host.
+ExecStart=/usr/bin/docker exec scrabble-postgres pgbackrest --stanza=scrabble --type=full backup

deploy/ansible/roles/main/templates/pgbackrest-backup.timer.j2

@@ -0,0 +1,12 @@
+# Managed by Ansible (deploy/ansible).
+# Fires the daily full base backup. Persistent=true catches up a run missed while the
+# host was down. Installed only when pitr_enabled is set.
+[Unit]
+Description=Daily pgBackRest full base backup (scrabble)
+
+[Timer]
+OnCalendar={{ pitr_backup_oncalendar }}
+Persistent=true
+
+[Install]
+WantedBy=timers.target