feat(deploy): continuous WAL archiving to S3 for point-in-time recovery

Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.

- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
  in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
  repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
  write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
  disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
  test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
  silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
  deploy/README.md; record the measured cost/perf assessment.

deploy/ansible/roles/main/tasks/main.yml

@@ -16,3 +16,30 @@
   community.docker.docker_network:
     name: edge
     state: present
+
+# --- pgBackRest base-backup schedule (point-in-time recovery) ------------------
+# A daily full base backup via a systemd timer that runs pgBackRest inside the postgres
+# container; archive_command handles the continuous WAL between backups. Gated by
+# pitr_enabled so the schedule stays inert until archiving is armed (the S3 repository +
+# stanza must exist first, or the run fails nightly). Arm per deploy/README.md.
+- name: Install the pgBackRest base-backup service
+  ansible.builtin.template:
+    src: pgbackrest-backup.service.j2
+    dest: /etc/systemd/system/pgbackrest-backup.service
+    mode: "0644"
+  when: pitr_enabled | default(false) | bool
+
+- name: Install the pgBackRest base-backup timer
+  ansible.builtin.template:
+    src: pgbackrest-backup.timer.j2
+    dest: /etc/systemd/system/pgbackrest-backup.timer
+    mode: "0644"
+  when: pitr_enabled | default(false) | bool
+
+- name: Enable and start the pgBackRest base-backup timer
+  ansible.builtin.systemd:
+    name: pgbackrest-backup.timer
+    enabled: true
+    state: started
+    daemon_reload: true
+  when: pitr_enabled | default(false) | bool

deploy/ansible/roles/main/templates/pgbackrest-backup.service.j2

@@ -0,0 +1,15 @@
+# Managed by Ansible (deploy/ansible).
+# Daily full pgBackRest base backup for the payments/game database. Runs pgBackRest
+# inside the postgres container (where the repository configuration and PGDATA live);
+# the continuous WAL between base backups is handled by the container's archive_command.
+# Installed only when pitr_enabled is set (see deploy/ansible/group_vars/all.yml).
+[Unit]
+Description=pgBackRest full base backup (scrabble)
+After=docker.service
+Requires=docker.service
+
+[Service]
+Type=oneshot
+# docker exec runs as the image's postgres user and inherits the container's PGBACKREST_*
+# environment (the S3 repository + cipher), so no repository config is needed on the host.
+ExecStart=/usr/bin/docker exec scrabble-postgres pgbackrest --stanza=scrabble --type=full backup

deploy/ansible/roles/main/templates/pgbackrest-backup.timer.j2

@@ -0,0 +1,12 @@
+# Managed by Ansible (deploy/ansible).
+# Fires the daily full base backup. Persistent=true catches up a run missed while the
+# host was down. Installed only when pitr_enabled is set.
+[Unit]
+Description=Daily pgBackRest full base backup (scrabble)
+
+[Timer]
+OnCalendar={{ pitr_backup_oncalendar }}
+Persistent=true
+
+[Install]
+WantedBy=timers.target