docs(deploy): bake Stage 18 prod-deploy decisions into the live docs
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 16s
CI / ui (pull_request) Successful in 57s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m8s

- ARCHITECTURE §13 prod bullet -> the realized mechanism: registry transport,
  two-host, rolling + auto-rollback, migration maintenance window, node_exporter,
  the undersized launch; the contour paragraph notes node_exporter + the
  telegram-local profile.
- deploy/README gains a prod rollout runbook (how to run, migrations/restore, cert
  rotation, sizing/monitoring, the full PROD_ set) + node_exporter row, the
  telegram-local profile note, and the soft AWG_CONF note.
- PLAN Stage 18 records the resolved open details and the remaining live cutover
  (pending erudit-game.ru DNS); the tracker reads 'machinery built; cutover pending DNS'.
- PRERELEASE TX/AG note the prod wiring is built.
This commit is contained in:
Ilia Denisov
2026-06-22 00:30:30 +02:00
parent 171b71b7e0
commit 3d9ba3ac3d
4 changed files with 104 additions and 30 deletions
+23 -13
View File
@@ -51,7 +51,7 @@ independent (see ARCHITECTURE §9.1).
| 15 | Dual Telegram bots & language-gated variants | **done** | | 15 | Dual Telegram bots & language-gated variants | **done** |
| 16 | Deploy infra & test contour (Dockerfiles, gateway static UI, compose, observability) | **done** | | 16 | Deploy infra & test contour (Dockerfiles, gateway static UI, compose, observability) | **done** |
| 17 | Test-contour verification & defect fixes | **done** | | 17 | Test-contour verification & defect fixes | **done** |
| 18 | Prod contour deploy (SSH export/import, manual after merge) | todo | | 18 | Prod contour deploy (registry, two-host, rolling + auto-rollback; manual after merge) | machinery built; first cutover pending DNS |
| 19 | User feedback (in-app submit + attachment, admin review/reply, account roles) | **done** | | 19 | User feedback (in-app submit + attachment, admin review/reply, account roles) | **done** |
Scaffolding is incremental: `go.work` lists only existing modules; each stage Scaffolding is incremental: `go.work` lists only existing modules; each stage
@@ -413,18 +413,28 @@ raw list is kept here as the record of what the first contour run surfaced.
"что-то пошло не так". при этом "new -> эрудит" работает. Попробуй посмотреть в логах сейчас, может что-то есть. Или как-то иначе проанализируй, или давай вместе будем смотреть, если не получится. "что-то пошло не так". при этом "new -> эрудит" работает. Попробуй посмотреть в логах сейчас, может что-то есть. Или как-то иначе проанализируй, или давай вместе будем смотреть, если не получится.
### Stage 18 — Prod contour deploy ### Stage 18 — Prod contour deploy
Scope: the **production contour** on a remote host over SSH. Deploy by **container export/import** Scope: the **production contour** on **two remote hosts** over SSH — main (full stack, `erudit-game.ru`)
(`docker save``scp`/ssh → `docker load``docker compose up` on the remote), the SSH key + host IP and tg (the bot only). Resolved open details (re-interviewed):
in Gitea secrets; **strictly manual** (`workflow_dispatch`) after `development` is merged to `master` - **Transport: a registry** (not export/import) — build + push to `docker.iliadenisov.ru`, the hosts pull by tag.
(the Stage 16 branch model: `feature/* → development → master`, merge gated green). Two-contour config - **Cert: ACME** at the contour caddy (`CADDY_SITE_ADDRESS=erudit-game.ru www.erudit-game.ru`, no host caddy).
uses **`TEST_`/`PROD_` secret/variable prefixes** — Gitea 1.26 has no deployment environments (verified: - **No prod VPN** — the bot host has native Bot API egress (verified `api.telegram.org` → 200).
the `environments` API 404s), so a flat prefixed namespace is the convention. - **Rollback** — rolling per-service deploy (least → most dependent), health-gated, auto-rollback to the
Reuses the Stage 16 `deploy/docker-compose.yml` as-is, mapping the **`PROD_`** set onto the same previous image tag; a maintenance window + consistent `pg_dump` only on a schema migration
unprefixed compose vars. **No host caddy on prod**, so the contour's own caddy terminates TLS — set (expand-contract keeps the auto-rollback image-only; the dump is a manual safety net).
`CADDY_SITE_ADDRESS` to the prod domain so caddy does its own ACME (the Caddyfile is already
parameterised for this; the test contour leaves it `:80` behind the host caddy). **Strictly manual** (`workflow_dispatch` from `master`, `confirm=deploy`) after `development → master`
Open details (re-interview): export/import vs a registry trade-off; prod domain/cert source (ACME vs a is merged green. `TEST_`/`PROD_` prefixed Gitea secrets/variables (Gitea 1.26 has no deployment
provided cert) at the contour caddy; prod VPN; rollback. environments — the `environments` API 404s). Hosts are provisioned by **`deploy/ansible/`** (docker, a
non-sudo `deploy` user with the CI key, key-only sshd, ufw, fail2ban). The main host is **launch-sized**
(2 vCPU / 1.9 GiB): `docker-compose.prod.yml` trims the R7 limits (`GOMAXPROCS=2`, smaller caps, 7d
Prometheus retention) and adds `node_exporter` for host-memory monitoring (launch undersized, resize at
Selectel reactively). `vpn`+`bot` are gated to a `telegram-local` compose profile (test only); the prod
bot runs standalone from `docker-compose.bot.yml`. `GATEWAY_ABUSE_BAN_ENABLED=true`.
**Built:** `deploy/ansible/` (both hosts provisioned + verified), the compose split + `node_exporter`,
`.gitea/workflows/prod-deploy.yaml` + `deploy/prod-deploy.sh`, the full `PROD_` secret/variable set.
**Remaining (acceptance):** the **first live cutover** — waits on the `erudit-game.ru` DNS delegation
(`A`/`www` → the main host) that ACME requires; then run the workflow and verify the public site end-to-end.
### Stage 19 — User feedback *(done)* ### Stage 19 — User feedback *(done)*
A user→operator feedback channel, sequenced after the numbered stages but shipped **before** the Stage 18 A user→operator feedback channel, sequenced after the numbered stages but shipped **before** the Stage 18
+2 -2
View File
@@ -39,8 +39,8 @@ the edge before prod. Each phase maps back to the owner's raw pre-release TODO l
| FM | First-move tile draw (official rules): each seated player draws a tile, the one closest to "A" leads (a blank beats every letter), ties re-drawing until a single leader; **honest per-draw `crypto/rand` entropy**, not the bag seed, so the **record** (`game_setup_draws`, migration `00013`) — not a seed — is the only account of the outcome, kept for future **tournaments** (designed as a discrete per-tile "player N draws" step). Friend/AI draws at create; **auto-match draws at *open*** against a synthetic `uuid.Nil` opponent whose draw rows are back-filled on join, so the opener's seat is fixed up front and the existing open-game pre-move is preserved (no reseating, no play-gating). Admin `/_gm/games/:id` gains the recorded draw list + a simple **step-by-step board replay** (`ReplayTimeline`). | owner ad-hoc | **done** | | FM | First-move tile draw (official rules): each seated player draws a tile, the one closest to "A" leads (a blank beats every letter), ties re-drawing until a single leader; **honest per-draw `crypto/rand` entropy**, not the bag seed, so the **record** (`game_setup_draws`, migration `00013`) — not a seed — is the only account of the outcome, kept for future **tournaments** (designed as a discrete per-tile "player N draws" step). Friend/AI draws at create; **auto-match draws at *open*** against a synthetic `uuid.Nil` opponent whose draw rows are back-filled on join, so the opener's seat is fixed up front and the existing open-game pre-move is preserved (no reseating, no play-gating). Admin `/_gm/games/:id` gains the recorded draw list + a simple **step-by-step board replay** (`ReplayTimeline`). | owner ad-hoc | **done** |
| SB | Single Telegram bot + per-user variant preferences: the two per-language bots collapse into **one** (drop `accounts.service_language`, `supported_languages`, the `*_EN`/`*_RU` env vars and game-language push routing — the single bot renders in the recipient's `preferred_language`); New Game variant gating moves to a profile **`variant_preferences`** set (default Erudit only, Erudit-first, server-enforced on the caller's auto-match/vs-AI/invitation-create paths, an invited friend may accept any variant); env vars collapse to unsuffixed `TELEGRAM_BOT_TOKEN`/`TELEGRAM_GAME_CHANNEL_ID`/`VITE_TELEGRAM_LINK`/`VITE_TELEGRAM_GAME_CHANNEL_NAME` and `GATEWAY_DEFAULT_SUPPORTED_LANGUAGES` is removed; wire drops `service_language`/`supported_languages` (Session, ValidateInitDataResponse) + the push `language` routing field and adds `variant_preferences` to Profile/UpdateProfile. | owner ad-hoc | **done** | | SB | Single Telegram bot + per-user variant preferences: the two per-language bots collapse into **one** (drop `accounts.service_language`, `supported_languages`, the `*_EN`/`*_RU` env vars and game-language push routing — the single bot renders in the recipient's `preferred_language`); New Game variant gating moves to a profile **`variant_preferences`** set (default Erudit only, Erudit-first, server-enforced on the caller's auto-match/vs-AI/invitation-create paths, an invited friend may accept any variant); env vars collapse to unsuffixed `TELEGRAM_BOT_TOKEN`/`TELEGRAM_GAME_CHANNEL_ID`/`VITE_TELEGRAM_LINK`/`VITE_TELEGRAM_GAME_CHANNEL_NAME` and `GATEWAY_DEFAULT_SUPPORTED_LANGUAGES` is removed; wire drops `service_language`/`supported_languages` (Session, ValidateInitDataResponse) + the push `language` routing field and adds `variant_preferences` to Profile/UpdateProfile. | owner ad-hoc | **done** |
| DV | Dictionary version hygiene: CI + image/compose seed track the current release (`v1.2.1`); a **seed-drift guard** records the flat dir's seed in an authoritative `.seed_version` marker so a bumped build seed on a live volume is ignored (it can't relabel live bytes — which would mis-serve the dictionary + void games pinned to the prior label); `DICT_VERSION` is the fresh-volume seed only, a live contour migrates through the admin console | owner ad-hoc | **done** | | DV | Dictionary version hygiene: CI + image/compose seed track the current release (`v1.2.1`); a **seed-drift guard** records the flat dir's seed in an authoritative `.seed_version` marker so a bumped build seed on a live volume is ignored (it can't relabel live bytes — which would mis-serve the dictionary + void games pinned to the prior label); `DICT_VERSION` is the fresh-volume seed only, a live contour migrates through the admin console | owner ad-hoc | **done** |
| TX | Telegram egress off the main host: split the connector into a home **validator** (Mini App / Login-Widget HMAC, no VPN, no Bot API — so game login no longer depends on Telegram being reachable) and a remote **bot** (Bot API long-poll + `sendMessage`) that holds **no inbound port** and dials the gateway over a reverse **mTLS bot-link** (`pkg/proto/botlink/v1`); the gateway funnels out-of-app push (fire-and-forget, at-most-once) and the backend admin broadcasts (a relay that awaits the bot's ack) down the link. The bot is Telegram-rate-limited; **one bot now**, with seams (a bot registry + `owns_updates` + command ids) for N later; **no webhook** (rejected: one URL per token, adds inbound + a static address). The **unified test contour** runs the split (the bot keeps its VPN sidecar and dials the gateway by its internal name; certs from `deploy/gen-certs.sh`). The **prod** wiring — the bot on a separate host (no VPN), the gateway bot-link port published, `PROD_` certs with scheduled rotation, an SSH deploy of both hosts together — is the **deferred final stage** (Stage 18). | owner ad-hoc | **done** (code + test contour; prod wiring Stage 18) | | TX | Telegram egress off the main host: split the connector into a home **validator** (Mini App / Login-Widget HMAC, no VPN, no Bot API — so game login no longer depends on Telegram being reachable) and a remote **bot** (Bot API long-poll + `sendMessage`) that holds **no inbound port** and dials the gateway over a reverse **mTLS bot-link** (`pkg/proto/botlink/v1`); the gateway funnels out-of-app push (fire-and-forget, at-most-once) and the backend admin broadcasts (a relay that awaits the bot's ack) down the link. The bot is Telegram-rate-limited; **one bot now**, with seams (a bot registry + `owns_updates` + command ids) for N later; **no webhook** (rejected: one URL per token, adds inbound + a static address). The **unified test contour** runs the split (the bot keeps its VPN sidecar and dials the gateway by its internal name; certs from `deploy/gen-certs.sh`). The **prod** wiring — the bot on a separate host (no VPN), the gateway bot-link port published, `PROD_` certs, an SSH deploy of both hosts together — is **built in Stage 18** (the two-host registry rollout; first cutover pending the `erudit-game.ru` DNS). | owner ad-hoc | **done** (code + test contour; prod wiring built — Stage 18) |
| AG | Anti-abuse IP ban + honeypot/honeytoken (prod-only): a fail2ban-style in-memory `ratelimit.Banlist` keyed by client IP, fed by sustained rate-limiter rejections (the IP-keyed public/email/admin classes — the user class stays the soft-flag's concern), a **honeypot** decoy path (the contour caddy tags `/.env`, `/.git`, `/wp-*`, … with `X-Scrabble-Honeypot` and routes them to the gateway), and a **honeytoken** (`GATEWAY_HONEYTOKEN`, a planted bearer). The `abuseGuard` edge middleware refuses a banned IP with **429** before any work — closing the R3 gap that the static SPA/landing was outside the token bucket. Off by default — it keys by the real client IP the shared-NAT test contour does not expose (detection still logs there); enabled in prod via `GATEWAY_ABUSE_BAN_ENABLED`. Operators see + lift bans on the console **Throttled** page; the gateway syncs its active set to the backend (`/api/v1/internal/bans/sync`, `internal/banview`) every 30 s and applies operator unbans. | owner ad-hoc | **done** (code + test contour; ban enabled in prod Stage 18) | | AG | Anti-abuse IP ban + honeypot/honeytoken (prod-only): a fail2ban-style in-memory `ratelimit.Banlist` keyed by client IP, fed by sustained rate-limiter rejections (the IP-keyed public/email/admin classes — the user class stays the soft-flag's concern), a **honeypot** decoy path (the contour caddy tags `/.env`, `/.git`, `/wp-*`, … with `X-Scrabble-Honeypot` and routes them to the gateway), and a **honeytoken** (`GATEWAY_HONEYTOKEN`, a planted bearer). The `abuseGuard` edge middleware refuses a banned IP with **429** before any work — closing the R3 gap that the static SPA/landing was outside the token bucket. Off by default — it keys by the real client IP the shared-NAT test contour does not expose (detection still logs there); enabled in prod via `GATEWAY_ABUSE_BAN_ENABLED`. Operators see + lift bans on the console **Throttled** page; the gateway syncs its active set to the backend (`/api/v1/internal/bans/sync`, `internal/banview`) every 30 s and applies operator unbans. | owner ad-hoc | **done** (code + test contour; ban on in prod via Stage 18 — machinery built, cutover pending DNS) |
| CM | Channel-chat moderation + promo bot: a second standalone bot in the bot container answers `/start` with a localized message + a **URL** button into the **main** bot's Mini App (`?startapp`; a `web_app` button would sign initData with the promo token, which the main validator rejects). The **main** bot gates write access in a channel's linked discussion chat. The chat **allows sending by default** and the bot only restricts (Telegram intersects the chat default with the per-user permission, so a per-user grant cannot exceed a deny-by-default group): it **mutes** a member who is not registered or is admin-suspended or holding a new **`chat_muted`** role, and **un-mutes** an eligible one it had muted, for a member currently in the chat (a `getChatMember` guard, since bots cannot list members). Eligibility = `registered AND NOT suspended AND NOT chat_muted` (the game suspension dominates), resolved once in the backend and reached two ways: the bot's `ResolveChatEligibility` on a `chat_member` event over the existing mTLS bot-link, and a backend `chat_access_changed` event → gateway → `ChatGate` command (emitted on block/unblock, a `chat_muted` change, a first registration, or a temporary-block expiry via a sweeper; idempotent). No schema change — `chat_muted` reuses `account_roles`. | owner ad-hoc | **done** | | CM | Channel-chat moderation + promo bot: a second standalone bot in the bot container answers `/start` with a localized message + a **URL** button into the **main** bot's Mini App (`?startapp`; a `web_app` button would sign initData with the promo token, which the main validator rejects). The **main** bot gates write access in a channel's linked discussion chat. The chat **allows sending by default** and the bot only restricts (Telegram intersects the chat default with the per-user permission, so a per-user grant cannot exceed a deny-by-default group): it **mutes** a member who is not registered or is admin-suspended or holding a new **`chat_muted`** role, and **un-mutes** an eligible one it had muted, for a member currently in the chat (a `getChatMember` guard, since bots cannot list members). Eligibility = `registered AND NOT suspended AND NOT chat_muted` (the game suspension dominates), resolved once in the backend and reached two ways: the bot's `ResolveChatEligibility` on a `chat_member` event over the existing mTLS bot-link, and a backend `chat_access_changed` event → gateway → `ChatGate` command (emitted on block/unblock, a `chat_muted` change, a first registration, or a temporary-block expiry via a sweeper; idempotent). No schema change — `chat_muted` reuses `account_roles`. | owner ad-hoc | **done** |
| → | Stage 18 — prod contour deploy | — | see [`PLAN.md`](PLAN.md) | | → | Stage 18 — prod contour deploy | — | see [`PLAN.md`](PLAN.md) |
+51 -3
View File
@@ -17,11 +17,12 @@ operational reference for **every environment variable**.
| `backend` | built (`backend/Dockerfile`) | Domain service; bakes in the DAWG dictionaries; runs migrations at boot. | | `backend` | built (`backend/Dockerfile`) | Domain service; bakes in the DAWG dictionaries; runs migrations at boot. |
| `postgres` | `postgres:17-alpine` | Database (named volume, `pg_isready` healthcheck). | | `postgres` | `postgres:17-alpine` | Database (named volume, `pg_isready` healthcheck). |
| `validator` | built (`platform/telegram/Dockerfile`, target `validator`) | Telegram HMAC validator (no VPN, no Bot API); internal gRPC at `validator:9091`. Game login depends only on this. | | `validator` | built (`platform/telegram/Dockerfile`, target `validator`) | Telegram HMAC validator (no VPN, no Bot API); internal gRPC at `validator:9091`. Game login depends only on this. |
| `vpn` + `bot` | sidecar + built (`platform/telegram/Dockerfile`, target `bot`) | Telegram bot; egresses through the AmneziaWG sidecar; holds no inbound port — dials the gateway bot-link (mTLS) at `gateway:9443`. | | `vpn` + `bot` | sidecar + built (`platform/telegram/Dockerfile`, target `bot`) | Telegram bot, gated to the **`telegram-local`** profile; egresses through the AmneziaWG sidecar and dials the gateway bot-link (mTLS) at `gateway:9443`. The test contour activates the profile; the prod **main** host omits it and runs the bot standalone on its **own host** (`docker-compose.bot.yml`, no VPN — native Bot API egress). |
| `otelcol` | `otel/opentelemetry-collector-contrib` | OTLP/gRPC `:4317` → Prometheus scrape (`:9464`) + Tempo. | | `otelcol` | `otel/opentelemetry-collector-contrib` | OTLP/gRPC `:4317` → Prometheus scrape (`:9464`) + Tempo. |
| `prometheus` | `prom/prometheus` | Metrics, 15d retention. | | `prometheus` | `prom/prometheus` | Metrics, 15d retention (7d in prod). |
| `tempo` | `grafana/tempo` | Traces, 72h retention. | | `tempo` | `grafana/tempo` | Traces, 72h retention. |
| `grafana` | `grafana/grafana` | Dashboards (provisioned), anonymous-admin behind caddy's `/_gm/grafana`. | | `grafana` | `grafana/grafana` | Dashboards (provisioned), anonymous-admin behind caddy's `/_gm/grafana`. |
| `node_exporter` | `quay.io/prometheus/node-exporter` | Host CPU/memory/disk metrics (Prometheus job `node`); the OOM signal on the tight prod main host (2 vCPU / 1.9 GiB). |
Networking: inter-service traffic is on the private `internal` network Networking: inter-service traffic is on the private `internal` network
(project-scoped DNS); only `caddy` joins the shared external `edge` network so the (project-scoped DNS); only `caddy` joins the shared external `edge` network so the
@@ -59,7 +60,6 @@ compose binds from this directory.
| Variable | Gitea kind | Purpose | | Variable | Gitea kind | Purpose |
| --- | --- | --- | | --- | --- | --- |
| `POSTGRES_PASSWORD` | secret | Postgres password (also embedded in `BACKEND_POSTGRES_DSN`). | | `POSTGRES_PASSWORD` | secret | Postgres password (also embedded in `BACKEND_POSTGRES_DSN`). |
| `AWG_CONF` | secret | AmneziaWG config for the VPN sidecar (the bot's only Telegram egress in the test contour). **Must not contain a `DNS=` line** — it hijacks the shared netns's resolv.conf and breaks the bot resolving `otelcol` / `gateway`. Without it, Docker's resolver handles `otelcol`, `gateway` and `api.telegram.org`. |
| `GM_BASICAUTH_HASH` | secret | bcrypt hash gating `/_gm` (admin console + Grafana). Generate with `docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'`. | | `GM_BASICAUTH_HASH` | secret | bcrypt hash gating `/_gm` (admin console + Grafana). Generate with `docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'`. |
| `TELEGRAM_MINIAPP_URL` | variable | The Mini App URL the bot hands out in deep links / buttons. | | `TELEGRAM_MINIAPP_URL` | variable | The Mini App URL the bot hands out in deep links / buttons. |
@@ -67,6 +67,13 @@ compose binds from this directory.
secret) and the bot (Bot API). It defaults to empty in compose, but both **fail at secret) and the bot (Bot API). It defaults to empty in compose, but both **fail at
boot** when it is empty. boot** when it is empty.
**Conditionally — `AWG_CONF`** (secret): the AmneziaWG config for the VPN sidecar, needed
only when the `telegram-local` profile runs (the test contour and local runs with the
bot). It is **not** `:?`-guarded — compose interpolates profiled-out services too, so the
prod main host (no VPN) must not require it. It **must not contain a `DNS=` line** — that
hijacks the shared netns's resolv.conf and breaks the bot resolving `otelcol` / `gateway`;
without it Docker's resolver handles `otelcol`, `gateway` and `api.telegram.org`.
## Optional variables (with defaults) ## Optional variables (with defaults)
| Variable | Gitea kind | Default | Purpose | | Variable | Gitea kind | Default | Purpose |
@@ -110,6 +117,47 @@ collector's / gateway's internal IP is fine (connected route), but its `AWG_CONF
which resolves `otelcol`, `gateway` and `api.telegram.org`. `GATEWAY_ADMIN_*` is which resolves `otelcol`, `gateway` and `api.telegram.org`. `GATEWAY_ADMIN_*` is
intentionally **unset** — caddy owns `/_gm` in the contour. intentionally **unset** — caddy owns `/_gm` in the contour.
## Production rollout
Prod runs on **two hosts** (main = full stack + ACME on the domain; tg = the bot only,
native Bot API, no VPN), one-time provisioned by **[`ansible/`](ansible/)** (docker, a
non-sudo `deploy` user holding the CI key, key-only sshd, default-deny ufw, fail2ban).
Re-run `ansible/` after a host resize — it is idempotent.
**To roll out:** merge `development → master` (CI green), then run the **`prod-deploy`**
workflow manually (Gitea → Actions → prod-deploy → run from `master`, input
`confirm=deploy`). It builds + pushes the images to the registry, ships the
compose/config/certs/env over SSH, deploys the main host with `prod-deploy.sh` (rolling,
health-gated, **auto-rollback to the previous tag**), then the bot host, then probes the
public site. After `master` is green this workflow is the **only** thing that touches
prod — nothing auto-deploys there.
**Migrations** must be **expand-contract** (backward-compatible; goose is forward-only):
the automatic rollback is image-only and never restores the DB. A deploy that changes
`backend/internal/postgres/migrations/` opens a maintenance window — the backend (sole
writer) is stopped for a consistent `pg_dump` into `/opt/scrabble/dumps` before the new
backend migrates. **Manual DB restore** (only if a migration was destructive):
`docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA backend CASCADE'`,
then pipe the dump into the same `psql`, and redeploy the matching old tag.
**bot-link cert rotation:** regenerate (`deploy/gen-certs.sh /tmp/c --force`), reset the
five `PROD_BOTLINK_*` secrets from `/tmp/c`, and re-run the workflow — both hosts redeploy
together with the fresh CA.
**Sizing / monitoring:** the main host launches undersized (2 vCPU / 1.9 GiB); the prod
overlay trims limits + `GOMAXPROCS=2` + 7d Prometheus retention, and `node_exporter` feeds
host memory to Grafana (`/_gm/grafana/`). Watch host memory and resize at Selectel when
players arrive.
**`PROD_` Gitea set** (mirrors `TEST_`, mapped onto the unprefixed names above) — secrets:
`PROD_{POSTGRES_PASSWORD, GM_BASICAUTH_HASH, GRAFANA_ADMIN_PASSWORD, TELEGRAM_BOT_TOKEN,
TELEGRAM_PROMO_BOT_TOKEN, REGISTRY_PASSWORD, SSH_KEY, SSH_KNOWN_HOSTS, BOTLINK_CA,
BOTLINK_GATEWAY_CERT, BOTLINK_GATEWAY_KEY, BOTLINK_BOT_CERT, BOTLINK_BOT_KEY}`; variables:
`PROD_{REGISTRY_USER, MAIN_HOST, TG_HOST, CADDY_SITE_ADDRESS, GM_BASICAUTH_USER,
GRAFANA_ROOT_URL, LOG_LEVEL, DICT_VERSION, TELEGRAM_MINIAPP_URL, TELEGRAM_GAME_CHANNEL_ID,
TELEGRAM_CHAT_ID, TELEGRAM_BOT_USERNAME, VITE_TELEGRAM_BOT_ID, VITE_TELEGRAM_LINK,
VITE_TELEGRAM_GAME_CHANNEL_NAME}`.
## Host-side setup (outside this repo) ## Host-side setup (outside this repo)
- **`edge` network** must exist on the host (`docker network create edge`). - **`edge` network** must exist on the host (`docker network create edge`).
+28 -12
View File
@@ -1056,8 +1056,10 @@ plaintext relay (`GATEWAY_BOTLINK_RELAY_ADDR`) the backend admin console calls.
The full contour (`deploy/docker-compose.yml`) runs one `gateway`, one `backend`, The full contour (`deploy/docker-compose.yml`) runs one `gateway`, one `backend`,
one Postgres, the static `landing`, the Telegram `validator` and `bot` (+ the bot's VPN one Postgres, the static `landing`, the Telegram `validator` and `bot` (+ the bot's VPN
sidecar) and the **observability stack** sidecar — the `bot`+`vpn` pair is gated to a `telegram-local` compose profile so the prod
OTel Collector (OTLP/gRPC ingest → Prometheus metrics + Tempo traces) and Grafana main host can omit them) and the **observability stack**
OTel Collector (OTLP/gRPC ingest → Prometheus metrics + Tempo traces), a `node_exporter`
for host CPU/memory (the prod main host's OOM signal), and Grafana
with provisioned datasources and dashboards. All services export OTLP to the with provisioned datasources and dashboards. All services export OTLP to the
collector; the bot shares the VPN sidecar's netns, so its `AWG_CONF` must not collector; the bot shares the VPN sidecar's netns, so its `AWG_CONF` must not
carry a `DNS=` directive (that would hijack resolv.conf and stop it resolving carry a `DNS=` directive (that would hijack resolv.conf and stop it resolving
@@ -1081,16 +1083,30 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`):
generated by `deploy/gen-certs.sh` before `compose up`; the bot keeps its VPN sidecar generated by `deploy/gen-certs.sh` before `compose up`; the bot keeps its VPN sidecar
for Telegram egress and dials the gateway by its internal name, so the bot-link stays for Telegram egress and dials the gateway by its internal name, so the bot-link stays
on the internal network. on the internal network.
- **Prod**: a manual SSH deploy after `development → master`. There is no - **Prod**: a **manual** rollout — `.gitea/workflows/prod-deploy.yaml`, `workflow_dispatch`
host caddy, so the contour ships its own caddy terminating TLS — set only (from `master`, `confirm=deploy`), run after `development → master` is merged green.
`CADDY_SITE_ADDRESS` to the domain and the caddy does its own ACME. The **bot runs It builds and pushes the images to the registry (`docker.iliadenisov.ru`), then deploys
on a separate host** with native Telegram access (no VPN), deployed by SSH alongside over SSH onto **two hosts** provisioned by `deploy/ansible/` (docker, a non-sudo `deploy`
the main app (rolled together so the bot-link protocol versions never skew); the service account holding a dedicated CI key, key-only sshd, default-deny ufw, fail2ban):
gateway **publishes** the bot-link port and the certificates come from `PROD_` the **main host** runs the full stack (`docker-compose.yml` + `docker-compose.prod.yml`),
secrets — a long-lived CA with leaves rotated by a scheduled job. The bot dials the the **bot host** runs only the bot (`docker-compose.bot.yml`, no VPN — native Bot API
gateway's public bot-link endpoint and holds no inbound port; login is unaffected if egress, telemetry off). There is no host caddy, so the contour caddy terminates TLS —
that host or the link is down. *(This prod wiring is the deferred final stage; the `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. The gateway **publishes**
code and the unified test contour land first — see `PRERELEASE.md`.)* the bot-link `:9443`; the remote bot dials it over mTLS (certs from `PROD_BOTLINK_*`,
ServerName `gateway`, so TLS validation is independent of the public dial address), holds
no inbound port, and login is unaffected if that host or the link is down.
`deploy/prod-deploy.sh` rolls the main stack **one service at a time in dependency order**
(postgres → backend → gateway → landing → validator → caddy), health-checking after each;
any failure **rolls the whole stack back to the previous image tag**. A **schema migration**
adds a maintenance window: the backend (the sole writer) is stopped for a consistent
`pg_dump` before the new backend migrates forward — image rollback stays DB-safe under the
expand-contract migration rule, and the dump is kept for a manual restore. The main host is
intentionally **launch-sized** (2 vCPU / 1.9 GiB): the prod overlay trims the R7 limits
(`GOMAXPROCS=2`, smaller caps, 7d Prometheus retention) and a **node_exporter** feeds
host-memory metrics to Grafana so it can be resized reactively as players arrive.
`GATEWAY_ABUSE_BAN_ENABLED=true` in prod (the per-IP ban is meaningful only with real
client IPs). The `vpn`+`bot` pair is gated to a `telegram-local` compose profile the test
contour activates; the prod main host omits it.
## 14. CI & branches ## 14. CI & branches