Ilia Denisov
e922f5f3b9
CI / changes (pull_request) Successful in 4s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 24s
CI / ui (pull_request) Successful in 1m9s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m41s
Add pgBackRest-based PITR for the production database, shipped disarmed (archive_mode and the base-backup timer gated off) so it stays inert until the operator arms it before real payments — an un-armed deploy cannot pile WAL onto the disk. - Postgres now builds from a thin image carrying pgBackRest (archive_command runs in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3 repository with 30-day retention and a daily full base-backup systemd timer. - Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot disarm archiving. - Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the test contour, which never archives. - Fix the pre-migration pg_dump to snapshot the whole database (was backend-only, silently excluding payments). - Document the PITR runbook, the arming sequence and the restore drill in deploy/README.md; record the measured cost/perf assessment.
52 lines
2.1 KiB
Markdown
52 lines
2.1 KiB
Markdown
# Prod host provisioning
|
||
|
||
Idempotent Ansible that prepares the two production hosts. It installs Docker, a
|
||
non-sudo `deploy` service account, SSH hardening, a default-deny firewall,
|
||
fail2ban, unattended security upgrades and time sync. It does **not** deploy the
|
||
application — that is `.gitea/workflows/prod-deploy.yaml`'s job, running as the
|
||
`deploy` account this playbook creates.
|
||
|
||
Hosts are referenced by `~/.ssh/config` aliases (`scrabble-main-ops`,
|
||
`scrabble-tg-ops`), so no IPs or key paths live in the repo.
|
||
|
||
## Prerequisites (controller)
|
||
|
||
- `ansible` with the bundled collections (`community.general`, `community.docker`,
|
||
`ansible.posix`).
|
||
- The two hosts reachable as root via the ssh-config aliases, host keys already
|
||
accepted into `known_hosts` (`host_key_checking = True`).
|
||
|
||
## One-time: the CI deploy key
|
||
|
||
The CI prod-deploy workflow logs into the hosts as `deploy` using a dedicated
|
||
key. Generate it once on the controller, authorize its public half via the
|
||
playbook, and store its private half **only** in the Gitea `PROD_SSH_KEY` secret:
|
||
|
||
```sh
|
||
ssh-keygen -t ed25519 -N '' -C scrabble-ci-deploy \
|
||
-f ~/.ssh/scrabble_ci_deploy_ed25519
|
||
# private half -> Gitea secret PROD_SSH_KEY (set via API); never commit it
|
||
```
|
||
|
||
## Run
|
||
|
||
```sh
|
||
cd deploy/ansible
|
||
ansible-playbook site.yml
|
||
```
|
||
|
||
The playbook reads the public key from `~/.ssh/scrabble_ci_deploy_ed25519.pub` by
|
||
default; override with `-e deploy_ci_pubkey_path=/path/to/key.pub`. Re-running is
|
||
safe (idempotent) and survives a host resize.
|
||
|
||
## What each host gets
|
||
|
||
- **both** (`common`): docker-ce + compose plugin, `daemon.json` (live-restore,
|
||
10m×3 log rotation), `deploy` user (docker group, no sudo), key-only sshd,
|
||
`ufw` default-deny incoming + allow SSH, fail2ban sshd jail, unattended
|
||
upgrades, chrony, `/opt/scrabble/{config,certs,dumps,images}`.
|
||
- **main**: `ufw` opens 80/443/9443; the external `edge` docker network; the pgBackRest
|
||
daily base-backup systemd timer — installed only when `pitr_enabled=true` (off by default;
|
||
turned on as part of arming point-in-time recovery, see [`../README.md`](../README.md)).
|
||
- **tg**: verifies direct `api.telegram.org` egress (the no-VPN assumption).
|