docs(deploy): bake Stage 18 prod-deploy decisions into the live docs
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 16s
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m8s

- ARCHITECTURE §13 prod bullet -> the realized mechanism: registry transport,
  two-host, rolling + auto-rollback, migration maintenance window, node_exporter,
  the undersized launch; the contour paragraph notes node_exporter + the
  telegram-local profile.
- deploy/README gains a prod rollout runbook (how to run, migrations/restore, cert
  rotation, sizing/monitoring, the full PROD_ set) + node_exporter row, the
  telegram-local profile note, and the soft AWG_CONF note.
- PLAN Stage 18 records the resolved open details and the remaining live cutover
  (pending erudit-game.ru DNS); the tracker reads 'machinery built; cutover pending DNS'.
- PRERELEASE TX/AG note the prod wiring is built.
This commit is contained in:
Ilia Denisov
2026-06-22 00:30:30 +02:00
parent 171b71b7e0
commit 3d9ba3ac3d
4 changed files with 104 additions and 30 deletions
+28 -12
View File
@@ -1056,8 +1056,10 @@ plaintext relay (`GATEWAY_BOTLINK_RELAY_ADDR`) the backend admin console calls.
The full contour (`deploy/docker-compose.yml`) runs one `gateway`, one `backend`,
one Postgres, the static `landing`, the Telegram `validator` and `bot` (+ the bot's VPN
sidecar) and the **observability stack**
OTel Collector (OTLP/gRPC ingest → Prometheus metrics + Tempo traces) and Grafana
sidecar — the `bot`+`vpn` pair is gated to a `telegram-local` compose profile so the prod
main host can omit them) and the **observability stack**
OTel Collector (OTLP/gRPC ingest → Prometheus metrics + Tempo traces), a `node_exporter`
for host CPU/memory (the prod main host's OOM signal), and Grafana
with provisioned datasources and dashboards. All services export OTLP to the
collector; the bot shares the VPN sidecar's netns, so its `AWG_CONF` must not
carry a `DNS=` directive (that would hijack resolv.conf and stop it resolving
@@ -1081,16 +1083,30 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`):
generated by `deploy/gen-certs.sh` before `compose up`; the bot keeps its VPN sidecar
for Telegram egress and dials the gateway by its internal name, so the bot-link stays
on the internal network.
- **Prod**: a manual SSH deploy after `development → master`. There is no
host caddy, so the contour ships its own caddy terminating TLS — set
`CADDY_SITE_ADDRESS` to the domain and the caddy does its own ACME. The **bot runs
on a separate host** with native Telegram access (no VPN), deployed by SSH alongside
the main app (rolled together so the bot-link protocol versions never skew); the
gateway **publishes** the bot-link port and the certificates come from `PROD_`
secrets — a long-lived CA with leaves rotated by a scheduled job. The bot dials the
gateway's public bot-link endpoint and holds no inbound port; login is unaffected if
that host or the link is down. *(This prod wiring is the deferred final stage; the
code and the unified test contour land first — see `PRERELEASE.md`.)*
- **Prod**: a **manual** rollout — `.gitea/workflows/prod-deploy.yaml`, `workflow_dispatch`
only (from `master`, `confirm=deploy`), run after `development → master` is merged green.
It builds and pushes the images to the registry (`docker.iliadenisov.ru`), then deploys
over SSH onto **two hosts** provisioned by `deploy/ansible/` (docker, a non-sudo `deploy`
service account holding a dedicated CI key, key-only sshd, default-deny ufw, fail2ban):
the **main host** runs the full stack (`docker-compose.yml` + `docker-compose.prod.yml`),
the **bot host** runs only the bot (`docker-compose.bot.yml`, no VPN — native Bot API
egress, telemetry off). There is no host caddy, so the contour caddy terminates TLS —
`CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. The gateway **publishes**
the bot-link `:9443`; the remote bot dials it over mTLS (certs from `PROD_BOTLINK_*`,
ServerName `gateway`, so TLS validation is independent of the public dial address), holds
no inbound port, and login is unaffected if that host or the link is down.
`deploy/prod-deploy.sh` rolls the main stack **one service at a time in dependency order**
(postgres → backend → gateway → landing → validator → caddy), health-checking after each;
any failure **rolls the whole stack back to the previous image tag**. A **schema migration**
adds a maintenance window: the backend (the sole writer) is stopped for a consistent
`pg_dump` before the new backend migrates forward — image rollback stays DB-safe under the
expand-contract migration rule, and the dump is kept for a manual restore. The main host is
intentionally **launch-sized** (2 vCPU / 1.9 GiB): the prod overlay trims the R7 limits
(`GOMAXPROCS=2`, smaller caps, 7d Prometheus retention) and a **node_exporter** feeds
host-memory metrics to Grafana so it can be resized reactively as players arrive.
`GATEWAY_ABUSE_BAN_ENABLED=true` in prod (the per-IP ban is meaningful only with real
client IPs). The `vpn`+`bot` pair is gated to a `telegram-local` compose profile the test
contour activates; the prod main host omits it.
## 14. CI & branches