Stage 18 — prod contour deploy (two-host registry rollout, rolling + auto-rollback) #103

Merged

developer merged 5 commits from feature/prod-contour-deploy into development 2026-06-22 04:59:46 +00:00

Conversation 1 Commits 5 Files Changed 23

+1021 -33

developer commented 2026-06-21 22:30:59 +00:00

Owner

Copy Link

Copy Source

Stage 18 — prod contour deploy (machinery)

Two-host production deploy: main (erudit-game.ru, full stack + ACME) and tg (bot only, native Bot API, no VPN). The first live cutover is pending the erudit-game.ru DNS delegation (ACME needs it); everything else is built and verified.

What's here

• Provisioning — deploy/ansible/ (idempotent): docker, a non-sudo deploy user with the CI key, key-only sshd, default-deny ufw, fail2ban. Already run + verified on both hosts.
• Compose split — vpn+bot gated to a telegram-local profile (the test deploy now passes --profile telegram-local); docker-compose.prod.yml (main overlay: publish 80/443/9443, GOMAXPROCS=2, trimmed limits, 7d retention for the 2 vCPU / 1.9 GiB host); docker-compose.bot.yml (standalone tg bot); node_exporter + a Prometheus scrape for host-memory monitoring.
• Pipeline — .gitea/workflows/prod-deploy.yaml (manual workflow_dispatch from master, confirm=deploy) builds + pushes to the registry, ships over SSH, then deploy/prod-deploy.sh rolls the main stack one service at a time in dependency order, health-gated, auto-rollback to the previous tag; a schema migration adds a maintenance window + consistent pg_dump.
• Secrets — the full PROD_ set is in Gitea.
• Docs — ARCHITECTURE §13, deploy/README.md (prod runbook), PLAN Stage 18, PRERELEASE TX/AG.

Notes

• Test contour stays green: the deploy keeps bringing the bot+vpn (via the profile) and gains node_exporter.
• No Go/UI/schema change — deploy-only.
• Stage 18 stays open until the first live cutover (DNS).

## Stage 18 — prod contour deploy (machinery) Two-host production deploy: **main** (`erudit-game.ru`, full stack + ACME) and **tg** (bot only, native Bot API, no VPN). The first live cutover is **pending the `erudit-game.ru` DNS delegation** (ACME needs it); everything else is built and verified. ### What's here - **Provisioning** — `deploy/ansible/` (idempotent): docker, a non-sudo `deploy` user with the CI key, key-only sshd, default-deny ufw, fail2ban. **Already run + verified on both hosts.** - **Compose split** — `vpn`+`bot` gated to a `telegram-local` profile (the test deploy now passes `--profile telegram-local`); `docker-compose.prod.yml` (main overlay: publish 80/443/9443, `GOMAXPROCS=2`, trimmed limits, 7d retention for the 2 vCPU / 1.9 GiB host); `docker-compose.bot.yml` (standalone tg bot); **`node_exporter`** + a Prometheus scrape for host-memory monitoring. - **Pipeline** — `.gitea/workflows/prod-deploy.yaml` (manual `workflow_dispatch` from `master`, `confirm=deploy`) builds + pushes to the registry, ships over SSH, then `deploy/prod-deploy.sh` rolls the main stack one service at a time in dependency order, health-gated, **auto-rollback to the previous tag**; a schema migration adds a maintenance window + consistent `pg_dump`. - **Secrets** — the full `PROD_` set is in Gitea. - **Docs** — ARCHITECTURE §13, `deploy/README.md` (prod runbook), PLAN Stage 18, PRERELEASE TX/AG. ### Notes - Test contour stays green: the deploy keeps bringing the bot+vpn (via the profile) and gains `node_exporter`. - No Go/UI/schema change — deploy-only. - Stage 18 stays open until the first live cutover (DNS).

developer added 4 commits 2026-06-21 22:30:59 +00:00

feat(deploy): Ansible provisioning for prod hosts (Stage 18) ... f5f45e7afb

Idempotent playbooks under deploy/ansible/ prepare both production hosts:
docker-ce + compose plugin, a non-sudo deploy service account holding the CI
deploy key, key-only sshd, default-deny ufw, fail2ban, unattended upgrades and
chrony. The main host also opens 80/443/9443 and creates the external edge
network; the tg host verifies direct Bot API egress (the no-VPN assumption).

The application is deployed separately by the prod-deploy workflow (later
phase), running as the deploy account this playbook provisions.

feat(deploy): prod compose split + host-memory monitoring (Stage 18) ... 2b399d0838

Split the contour across the two prod hosts and retune for the small main host:

- Gate vpn+bot to the telegram-local profile. The CI test deploy now passes
  --profile telegram-local so the test contour still brings them; the prod main
  host omits both, and the prod bot runs standalone from docker-compose.bot.yml.
- docker-compose.prod.yml (main-host overlay): publish caddy 80/443 (no host
  caddy in prod; caddy owns ACME) and gateway 9443 (the remote bot dials in over
  mTLS); GOMAXPROCS=2, smaller memory caps and 7d Prometheus retention for the
  2 vCPU / 1.9 GiB host. It launches deliberately undersized; resize reactively.
- docker-compose.bot.yml: standalone bot for the tg host (no VPN, OTLP off since
  otelcol is unreachable from there, dials the main host's bot-link).
- Add node_exporter + a Prometheus scrape so host memory pressure (the OOM
  signal on the tight main host), not just per-container docker_stats, is visible.
- Soften AWG_CONF to a default: only the profiled vpn sidecar consumes it, and
  compose interpolates profiled-out services too, so prod must not require it.

feat(deploy): manual prod-deploy pipeline with rolling rollback (Stage 18) ... 171b71b7e0

A workflow_dispatch-only rollout from master (confirm=deploy):

- .gitea/workflows/prod-deploy.yaml builds + pushes the images to the registry,
  ships the compose/config/certs/env over SSH, deploys the main host via
  prod-deploy.sh, then the bot host, then verifies the public site.
- deploy/prod-deploy.sh rolls the main stack one service at a time in dependency
  order (postgres->backend->gateway->landing->validator->caddy), health-checking
  after each; any failure rolls the whole stack back to the previous tag. A schema
  migration adds a maintenance window: the backend (sole writer) is stopped for a
  consistent pg_dump before migrating; image rollback stays DB-safe (expand-contract),
  the dump is kept for a manual restore.
- prod overlay: pull the four main images from the registry by tag.
- Runtime secrets reach the host via a sourced env.sh (single-quoted values keep the
  bcrypt hash's literal $ intact, unlike a --env-file).

docs(deploy): bake Stage 18 prod-deploy decisions into the live docs ... 3d9ba3ac3d

- ARCHITECTURE §13 prod bullet -> the realized mechanism: registry transport,
  two-host, rolling + auto-rollback, migration maintenance window, node_exporter,
  the undersized launch; the contour paragraph notes node_exporter + the
  telegram-local profile.
- deploy/README gains a prod rollout runbook (how to run, migrations/restore, cert
  rotation, sizing/monitoring, the full PROD_ set) + node_exporter row, the
  telegram-local profile note, and the soft AWG_CONF note.
- PLAN Stage 18 records the resolved open details and the remaining live cutover
  (pending erudit-game.ru DNS); the tracker reads 'machinery built; cutover pending DNS'.
- PRERELEASE TX/AG note the prod wiring is built.

developer added 1 commit 2026-06-21 22:35:21 +00:00

fix(deploy): guard cd and split DOCKER_GID assignment (shellcheck) ... 9f970495ee

cd $COMPOSE_DIR now aborts on failure instead of deploying from the wrong dir;
DOCKER_GID is declared then exported so the subshell exit isn't masked.

owner approved these changes 2026-06-22 04:59:19 +00:00

developer merged commit 520a9092fe into development 2026-06-22 04:59:46 +00:00

developer deleted branch feature/prod-contour-deploy 2026-06-22 04:59:46 +00:00

developer referenced this pull request 2026-06-22 05:37:34 +00:00

Deploy v2 — release versioning + visible deploy jobs + manual rollback #105

Sign in to join this conversation.

Reviewers

No Reviewers

owner

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

owner (Ilia) developer

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: developer/scrabble-game#103