docker exec defaults to root, so the systemd base-backup timer connected to the
database as role "root" (then "postgres") — neither exists; the superuser role is
POSTGRES_USER (scrabble). Run the timer's pgBackRest as the postgres OS user (-u postgres,
for lock-dir/PGDATA consistency with archive-push) and connect with --pg1-user=scrabble.
archive_command (run by the postgres server process) was already correct.
Also record the point-in-time-recovery arming + restore drill in deploy/README.md
(correct the runbook commands to the same invocation, fill the drill log) and mark the
durability work done in PLAN.md.
Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.
- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
deploy/README.md; record the measured cost/perf assessment.
Two prod-deploy hardening measures (owner-requested):
- Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container
memory caps enforce that one service can't eat all RAM, but they overcommit the
1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel
OOM-killer (and it might pick postgres). A small swap absorbs the overshoot.
Idempotent, builtin-only (no ansible.posix).
- Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap /
migration window that the caddy edge serves a static 503 "технические работы" page
from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm
(Grafana) stays reachable. Cleared on any exit (success, health failure + rollback,
or error) by a shell trap so it can never stick on. The 503 carries Retry-After +
an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned
window apart from a transient error (the static page only catches a fresh load; an
in-session user needs the app-side overlay). Not zero-downtime — the single
stateful backend still blips — but the window is graceful instead of raw 502s.
Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC
edge and serves the page + markers; /_gm bypasses; toggling needs no reload
(per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
Idempotent playbooks under deploy/ansible/ prepare both production hosts:
docker-ce + compose plugin, a non-sudo deploy service account holding the CI
deploy key, key-only sshd, default-deny ufw, fail2ban, unattended upgrades and
chrony. The main host also opens 80/443/9443 and creates the external edge
network; the tg host verifies direct Bot API egress (the no-VPN assumption).
The application is deployed separately by the prod-deploy workflow (later
phase), running as the deploy account this playbook provisions.
Ilia Denisov