Ilia Denisov
e922f5f3b9
Add pgBackRest-based PITR for the production database, shipped disarmed (archive_mode and the base-backup timer gated off) so it stays inert until the operator arms it before real payments — an un-armed deploy cannot pile WAL onto the disk. - Postgres now builds from a thin image carrying pgBackRest (archive_command runs in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3 repository with 30-day retention and a daily full base-backup systemd timer. - Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot disarm archiving. - Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the test contour, which never archives. - Fix the pre-migration pg_dump to snapshot the whole database (was backend-only, silently excluding payments). - Document the PITR runbook, the arming sequence and the restore drill in deploy/README.md; record the measured cost/perf assessment.
Prod host provisioning
Idempotent Ansible that prepares the two production hosts. It installs Docker, a
non-sudo deploy service account, SSH hardening, a default-deny firewall,
fail2ban, unattended security upgrades and time sync. It does not deploy the
application — that is .gitea/workflows/prod-deploy.yaml's job, running as the
deploy account this playbook creates.
Hosts are referenced by ~/.ssh/config aliases (scrabble-main-ops,
scrabble-tg-ops), so no IPs or key paths live in the repo.
Prerequisites (controller)
ansiblewith the bundled collections (community.general,community.docker,ansible.posix).- The two hosts reachable as root via the ssh-config aliases, host keys already
accepted into
known_hosts(host_key_checking = True).
One-time: the CI deploy key
The CI prod-deploy workflow logs into the hosts as deploy using a dedicated
key. Generate it once on the controller, authorize its public half via the
playbook, and store its private half only in the Gitea PROD_SSH_KEY secret:
ssh-keygen -t ed25519 -N '' -C scrabble-ci-deploy \
-f ~/.ssh/scrabble_ci_deploy_ed25519
# private half -> Gitea secret PROD_SSH_KEY (set via API); never commit it
Run
cd deploy/ansible
ansible-playbook site.yml
The playbook reads the public key from ~/.ssh/scrabble_ci_deploy_ed25519.pub by
default; override with -e deploy_ci_pubkey_path=/path/to/key.pub. Re-running is
safe (idempotent) and survives a host resize.
What each host gets
- both (
common): docker-ce + compose plugin,daemon.json(live-restore, 10m×3 log rotation),deployuser (docker group, no sudo), key-only sshd,ufwdefault-deny incoming + allow SSH, fail2ban sshd jail, unattended upgrades, chrony,/opt/scrabble/{config,certs,dumps,images}. - main:
ufwopens 80/443/9443; the externaledgedocker network; the pgBackRest daily base-backup systemd timer — installed only whenpitr_enabled=true(off by default; turned on as part of arming point-in-time recovery, see../README.md). - tg: verifies direct
api.telegram.orgegress (the no-VPN assumption).