- ARCHITECTURE §13 prod bullet -> the realized mechanism: registry transport,
two-host, rolling + auto-rollback, migration maintenance window, node_exporter,
the undersized launch; the contour paragraph notes node_exporter + the
telegram-local profile.
- deploy/README gains a prod rollout runbook (how to run, migrations/restore, cert
rotation, sizing/monitoring, the full PROD_ set) + node_exporter row, the
telegram-local profile note, and the soft AWG_CONF note.
- PLAN Stage 18 records the resolved open details and the remaining live cutover
(pending erudit-game.ru DNS); the tracker reads 'machinery built; cutover pending DNS'.
- PRERELEASE TX/AG note the prod wiring is built.
A workflow_dispatch-only rollout from master (confirm=deploy):
- .gitea/workflows/prod-deploy.yaml builds + pushes the images to the registry,
ships the compose/config/certs/env over SSH, deploys the main host via
prod-deploy.sh, then the bot host, then verifies the public site.
- deploy/prod-deploy.sh rolls the main stack one service at a time in dependency
order (postgres->backend->gateway->landing->validator->caddy), health-checking
after each; any failure rolls the whole stack back to the previous tag. A schema
migration adds a maintenance window: the backend (sole writer) is stopped for a
consistent pg_dump before migrating; image rollback stays DB-safe (expand-contract),
the dump is kept for a manual restore.
- prod overlay: pull the four main images from the registry by tag.
- Runtime secrets reach the host via a sourced env.sh (single-quoted values keep the
bcrypt hash's literal $ intact, unlike a --env-file).
Split the contour across the two prod hosts and retune for the small main host:
- Gate vpn+bot to the telegram-local profile. The CI test deploy now passes
--profile telegram-local so the test contour still brings them; the prod main
host omits both, and the prod bot runs standalone from docker-compose.bot.yml.
- docker-compose.prod.yml (main-host overlay): publish caddy 80/443 (no host
caddy in prod; caddy owns ACME) and gateway 9443 (the remote bot dials in over
mTLS); GOMAXPROCS=2, smaller memory caps and 7d Prometheus retention for the
2 vCPU / 1.9 GiB host. It launches deliberately undersized; resize reactively.
- docker-compose.bot.yml: standalone bot for the tg host (no VPN, OTLP off since
otelcol is unreachable from there, dials the main host's bot-link).
- Add node_exporter + a Prometheus scrape so host memory pressure (the OOM
signal on the tight main host), not just per-container docker_stats, is visible.
- Soften AWG_CONF to a default: only the profiled vpn sidecar consumes it, and
compose interpolates profiled-out services too, so prod must not require it.
Idempotent playbooks under deploy/ansible/ prepare both production hosts:
docker-ce + compose plugin, a non-sudo deploy service account holding the CI
deploy key, key-only sshd, default-deny ufw, fail2ban, unattended upgrades and
chrony. The main host also opens 80/443/9443 and creates the external edge
network; the tg host verifies direct Bot API egress (the no-VPN assumption).
The application is deployed separately by the prod-deploy workflow (later
phase), running as the deploy account this playbook provisions.
Ilia Denisov