Ilia Denisov
a58f2ce0e4
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 19s
CI / ui (pull_request) Successful in 1m8s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 1s
CI / deploy (pull_request) Successful in 1m43s
The S3 endpoint is a host only; some providers publish a separate, non-443 port. Add an optional PGBACKREST_S3_PORT (default 443) alongside the endpoint host, wired through the prod overlay, write-prod-env.sh and both prod workflows, and clarify in the docs that the endpoint variable takes the host alone.
354 lines
27 KiB
Markdown
354 lines
27 KiB
Markdown
# deploy
|
||
|
||
The full Scrabble contour: `backend` + `gateway` + the static `landing` + Postgres +
|
||
the Telegram `validator` + `bot` (the bot with a VPN sidecar) + the observability stack
|
||
(OTel Collector → Prometheus + Tempo → Grafana), fronted by a **caddy** that owns a single
|
||
`/_gm` Basic-Auth (the admin console + Grafana). Topology and the decision record are in
|
||
[`../docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md) §13; this file is the
|
||
operational reference for **every environment variable**.
|
||
|
||
## Services
|
||
|
||
| Service | Image | Role |
|
||
| --- | --- | --- |
|
||
| `caddy` | `caddy:2-alpine` | Edge proxy (alias `scrabble` on `edge`): single `/_gm` Basic-Auth → admin console + Grafana; `/app/`, `/telegram/` + the Connect path → gateway; the catch-all (incl. `/`) → landing. TLS per `CADDY_SITE_ADDRESS`. |
|
||
| `gateway` | built (`gateway/Dockerfile`, target `gateway`) | Public edge; serves the embedded game SPA at `/app/` + `/telegram/`; Connect-RPC edge. `/` redirects to `/app/`. |
|
||
| `landing` | built (`gateway/Dockerfile`, target `landing`) | Static landing page at `/` (caddy:2-alpine + the shared Vite build, `deploy/landing/Caddyfile`); absorbs stray public paths. |
|
||
| `backend` | built (`backend/Dockerfile`) | Domain service; bakes in the DAWG dictionaries; runs migrations at boot. |
|
||
| `postgres` | `postgres:17-alpine` | Database (named volume, `pg_isready` healthcheck). |
|
||
| `renderer` | built (`renderer/Dockerfile`) | Finished-game image-render sidecar (Node + skia-canvas running the shared `ui/src/lib/gameimage.ts`); internal-only HTTP at `renderer:8090`, called by the backend for the PNG export artifact. |
|
||
| `validator` | built (`platform/telegram/Dockerfile`, target `validator`) | Telegram HMAC validator (no VPN, no Bot API); internal gRPC at `validator:9091`. Game login depends only on this. |
|
||
| `vpn` + `bot` | sidecar + built (`platform/telegram/Dockerfile`, target `bot`) | Telegram bot, gated to the **`telegram-local`** profile; egresses through the AmneziaWG sidecar and dials the gateway bot-link (mTLS) at `gateway:9443`. The test contour activates the profile; the prod **main** host omits it and runs the bot standalone on its **own host** (`docker-compose.bot.yml`, no VPN — native Bot API egress). |
|
||
| `otelcol` | `otel/opentelemetry-collector-contrib` | OTLP/gRPC `:4317` → Prometheus scrape (`:9464`) + Tempo. |
|
||
| `prometheus` | `prom/prometheus` | Metrics, 15d retention (7d in prod). |
|
||
| `tempo` | `grafana/tempo` | Traces, 72h retention. |
|
||
| `grafana` | `grafana/grafana` | Dashboards (provisioned), anonymous-admin behind caddy's `/_gm/grafana`. |
|
||
| `node_exporter` | `quay.io/prometheus/node-exporter` | Host CPU/memory/disk metrics (Prometheus job `node`); the OOM signal on the tight prod main host (2 vCPU / 1.9 GiB). |
|
||
|
||
Networking: inter-service traffic is on the private `internal` network
|
||
(project-scoped DNS); only `caddy` joins the shared external `edge` network so the
|
||
host caddy can reach it at `scrabble:80`. `edge` must already exist on the host
|
||
(`docker network create edge`).
|
||
|
||
## Run it
|
||
|
||
**Locally** — copy the template, fill the required values, bring it up:
|
||
|
||
```sh
|
||
cp deploy/.env.example deploy/.env # then edit deploy/.env
|
||
docker network create edge # once, if it does not exist
|
||
cd deploy && docker compose up -d --build
|
||
```
|
||
|
||
**In CI** (the test contour) — `.gitea/workflows/ci.yaml`'s `deploy` job maps the
|
||
Gitea **`TEST_`-prefixed** secrets/variables onto the unprefixed names below and
|
||
runs `docker compose up -d --build` on the runner host. The prod deploy maps the
|
||
**`PROD_`** set the same way. So a Gitea secret named `TEST_POSTGRES_PASSWORD`
|
||
feeds the compose's `POSTGRES_PASSWORD`, etc.
|
||
|
||
Three naming classes in Gitea:
|
||
- **Per-contour** (`TEST_`/`PROD_<NAME>`) — values that differ between the contours
|
||
(bots, hosts, public origin, log level, email senders): the common case below.
|
||
- **Shared** (one unprefixed `<NAME>`, no prefix) — values identical on every contour,
|
||
stored once: `DICT_VERSION`, `SMTP_RELAY_HOST`/`PORT`/`TLS`/`USER`/`PASS`,
|
||
`GRAFANA_SMTP_PORT`, `VITE_VK_APP_LINK`, `VITE_VK_APP_ID`, `GATEWAY_VK_APP_SECRET`,
|
||
`GATEWAY_VK_ID_CLIENT_SECRET` (one Selectel relay + one pair of VK apps for all contours).
|
||
- **Derived** — not stored at all; the deploy computes them from `PUBLIC_BASE_URL`
|
||
(`deploy/write-prod-env.sh`, and the `ci.yaml` deploy step): `TELEGRAM_MINIAPP_URL`
|
||
(`+ /telegram/`), `GRAFANA_ROOT_URL` (`+ /_gm/grafana/`), `VITE_VK_ID_REDIRECT_URL`
|
||
(`+ /app/`). Set them directly only for a local `.env` run.
|
||
|
||
The deploy job also **seeds the config files** (`caddy`, `otelcol`, `prometheus`,
|
||
`tempo`, `grafana`) to a stable host path (`$HOME/.scrabble-deploy`) and sets
|
||
`SCRABBLE_CONFIG_DIR` to it before `up`. The runner's checkout is an ephemeral act
|
||
workspace that is removed after the job — binding config straight from it would
|
||
dangle the mounts in the long-lived containers (Grafana would log
|
||
`no such file or directory`). Locally `SCRABBLE_CONFIG_DIR` defaults to `.`, so the
|
||
compose binds from this directory.
|
||
|
||
## Required variables
|
||
|
||
`docker compose` aborts immediately if any of these is unset (they use `:?`):
|
||
|
||
| Variable | Gitea kind | Purpose |
|
||
| --- | --- | --- |
|
||
| `POSTGRES_PASSWORD` | secret | Postgres password (also embedded in `BACKEND_POSTGRES_DSN`). |
|
||
| `GM_BASICAUTH_HASH` | secret | bcrypt hash gating `/_gm` (admin console + Grafana). Generate with `docker run --rm caddy:2-alpine caddy hash-password --plaintext '<pw>'`. |
|
||
| `TELEGRAM_MINIAPP_URL` | derived | The Mini App URL the bot hands out in deep links / buttons. The deploy derives `PUBLIC_BASE_URL + /telegram/`; set it directly only for a local run (compose still `:?`-requires it). |
|
||
| `EXPORT_SIGN_KEY` | secret | HMAC key signing the public finished-game export download URLs (`/dl/*`). Generate with `openssl rand -base64 32`. |
|
||
|
||
**Plus the bot token** — `TELEGRAM_BOT_TOKEN` (secret), shared by the validator (HMAC
|
||
secret) and the bot (Bot API). It defaults to empty in compose, but both **fail at
|
||
boot** when it is empty.
|
||
|
||
**Conditionally — `AWG_CONF`** (secret): the AmneziaWG config for the VPN sidecar, needed
|
||
only when the `telegram-local` profile runs (the test contour and local runs with the
|
||
bot). It is **not** `:?`-guarded — compose interpolates profiled-out services too, so the
|
||
prod main host (no VPN) must not require it. It **must not contain a `DNS=` line** — that
|
||
hijacks the shared netns's resolv.conf and breaks the bot resolving `otelcol` / `gateway`;
|
||
without it Docker's resolver handles `otelcol`, `gateway` and `api.telegram.org`.
|
||
|
||
## Optional variables (with defaults)
|
||
|
||
| Variable | Gitea kind | Default | Purpose |
|
||
| --- | --- | --- | --- |
|
||
| `POSTGRES_DB` | variable | `scrabble` | Database name. |
|
||
| `POSTGRES_USER` | variable | `scrabble` | Database user. |
|
||
| `DICT_VERSION` | variable (shared) | `v1.3.1` | `scrabble-dictionary` release tag baked into the backend image as the **seed for a fresh volume** (build-arg). A live contour changes dictionary through the admin console, not this; on a seeded volume a changed value is ignored (the recorded `.seed_version` marker wins — the seed-drift guard, ARCHITECTURE.md §5). One shared unprefixed Gitea variable seeds both contours and pins the CI test suite. |
|
||
| `LOG_LEVEL` | variable | `info` | Shared log level for backend / gateway / validator / bot (`debug\|info\|warn\|error`). |
|
||
| `CADDY_SITE_ADDRESS` | variable | `:80` | Caddy site address. Test: `:80` (host caddy terminates TLS). Prod: a domain, so caddy does its own ACME. |
|
||
| `GM_BASICAUTH_USER` | variable | `gm` | Username for the `/_gm` Basic-Auth. |
|
||
| `GRAFANA_ROOT_URL` | derived | `/_gm/grafana/` | Grafana root URL (sub-path serving). The deploy derives `PUBLIC_BASE_URL + /_gm/grafana/`; set the full `https://<domain>/_gm/grafana/` only for a local run. |
|
||
| `GRAFANA_ADMIN_PASSWORD` | secret | `admin` | Grafana admin password. Low impact (the login form is disabled, access is anonymous-admin behind caddy) but set it anyway. |
|
||
| `TELEGRAM_GAME_CHANNEL_ID` | variable | _(empty)_ | The bot's game-channel id; empty/`0` disables channel posts. |
|
||
| `TELEGRAM_TEST_ENV` | _pinned_ | `false` | `true` routes the bot through Telegram's test environment (`.../bot<token>/test/METHOD`). **The CI test contour pins this to `true` in `ci.yaml`** (the contour is the test environment) — it is not a Gitea variable. Set it in `.env` for a local run; prod leaves it `false`. |
|
||
| `TELEGRAM_API_BASE_URL` | variable | _(empty)_ | Override the Bot API host (a mock/self-hosted server); empty = `https://api.telegram.org`. |
|
||
| `VITE_TELEGRAM_BOT_ID` | variable | _(empty)_ | UI build-arg: numeric bot id for the web Login Widget. |
|
||
| `VITE_TELEGRAM_LINK` | variable | _(empty)_ | UI build-arg: friend-invite Mini App link (full URL, `https://t.me/<bot>/<app>` — `<app>` is the Mini App short name from BotFather). |
|
||
| `VITE_TELEGRAM_GAME_CHANNEL_NAME` | variable | _(empty)_ | UI build-arg: the landing "Play in Telegram" link, the bot's game channel (e.g. `https://t.me/Erudit_Game`). |
|
||
| `VITE_VK_APP_LINK` | variable (shared) | _(empty)_ | UI build-arg: the landing "Play on VK" link, the VK Mini App (full URL, `https://vk.com/app<id>`). One VK Mini App for all contours. |
|
||
| `VITE_VK_APP_ID` | variable (shared) | _(empty)_ | VK ID "Web" app id (`client_id`) for VK web-login linking. Baked into the SPA authorize URL and reused as the gateway's `GATEWAY_VK_ID_APP_ID`. Empty disables the `link.vk.*` ops. One VK ID "Web" app for all contours. |
|
||
| `VITE_VK_ID_REDIRECT_URL` | derived | _(empty)_ | VK ID trusted redirect URL — must match the app's registered redirect exactly. The deploy derives `PUBLIC_BASE_URL + /app/` (used by the SPA and as the gateway's exchange `redirect_uri`); set it only for a local run. |
|
||
| `GATEWAY_VK_APP_SECRET` | secret (shared) | _(empty)_ | The VK Mini App's protected key (`client_secret`): the gateway verifies the launch-parameter signature in-process under it (offline HMAC, no VK API call). Empty disables the VK auth path (`auth.vk`). One VK Mini App for all contours. |
|
||
| `GATEWAY_VK_ID_CLIENT_SECRET` | secret (shared) | _(empty)_ | The VK ID "Web" app's protected key (`client_secret`) for the gateway's server-side confidential code exchange. A SEPARATE VK app from `GATEWAY_VK_APP_SECRET` (the Mini App). One VK ID "Web" app for all contours. |
|
||
| `GATEWAY_HONEYTOKEN` | secret | _(empty)_ | Planted honeytoken bearer value: presenting it earns a 24h IP ban + a high-severity alarm where the IP ban is on (prod), or logs + a `gateway_abuse_banned_total{reason="honeytoken"}` metric (test, ban off). Plant the value somewhere an attacker would find it; empty disables the trap. Per-contour `TEST_`/`PROD_GATEWAY_HONEYTOKEN`. |
|
||
| `VITE_GATEWAY_URL` | variable | _(empty)_ | UI build-arg: gateway origin; empty = same-origin (the usual single-origin deploy). |
|
||
| `SMTP_RELAY_HOST` | variable (shared) | _(empty)_ | Selectel SMTP relay host for confirm-code email. Empty leaves the backend on the log mailer (email disabled) — the contour still boots. One relay for every contour (limit 100 msgs / 5 min). |
|
||
| `SMTP_RELAY_PORT` | variable (shared) | `465` | Relay port. No client certificate is needed (the server cert is validated against the system roots). |
|
||
| `SMTP_RELAY_TLS` | variable (shared) | _(empty)_ | Transport security: `ssl` (implicit TLS) or `starttls`. Empty derives it from the port (implicit on `465`, STARTTLS otherwise); required for a relay on a non-standard port (e.g. Selectel's `1127` = SSL, `1126` = STARTTLS). |
|
||
| `SMTP_RELAY_USER` | secret (shared) | _(empty)_ | Relay SMTP AUTH username. |
|
||
| `SMTP_RELAY_PASS` | secret (shared) | _(empty)_ | Relay SMTP AUTH password. |
|
||
| `SMTP_RELAY_FROM` | variable | `no-reply@localhost` | Sender address. **Must use the prod domain** (`no-reply@erudit-game.ru`) — Selectel only accepts the verified sender domain — so it is the same on every contour. |
|
||
| `PUBLIC_BASE_URL` | variable | _(empty)_ | Canonical public https origin for links in the email (the contour's own URL, e.g. `https://erudit-game.ru`). **Required by the backend whenever `SMTP_RELAY_HOST` is set** — it is never derived from a request Host header (anti-injection). |
|
||
| `SMTP_RELAY_ADMIN_FROM` | variable | _(empty)_ | Backend operator-alert sender (new feedback / word complaints), distinct from the confirm-code From. Empty (with `ADMIN_EMAIL`) disables the alert worker. |
|
||
| `ADMIN_EMAIL` | variable | _(empty)_ | Backend operator-alert recipient(s); several comma-separated addresses allowed. |
|
||
| `SMTP_RELAY_SERVICE_FROM` | variable | _(empty)_ | Grafana infra-alert sender address. |
|
||
| `SERVICE_EMAIL` | variable | _(empty)_ | Grafana infra-alert recipient(s); comma-separated allowed (read by the provisioned contact point via `$__env{SERVICE_EMAIL}`). |
|
||
| `GRAFANA_SMTP_PORT` | variable (shared) | _(empty)_ | STARTTLS port Grafana dials on `SMTP_RELAY_HOST` (its client can't do the backend's implicit TLS), e.g. Selectel's `1126`. Reuses `SMTP_RELAY_HOST`/`USER`/`PASS`. |
|
||
| `GF_SMTP_ENABLED` | variable | `false` | Set `true` to enable Grafana alert emails. |
|
||
|
||
The six `VITE_*` are **build-args** baked into the gateway and landing images at
|
||
build time (both targets share one UI build stage — keep the args identical so it is
|
||
built once), so changing them requires a rebuild (`--build`), not just a restart.
|
||
|
||
## Fixed internal wiring (not operator-set)
|
||
|
||
These are hard-wired in `docker-compose.yml` (no `${...}`), pointing the services
|
||
at each other on the `internal` network — listed here so they are not mistaken for
|
||
missing config: `BACKEND_POSTGRES_DSN` (→ `postgres`, `search_path=backend`),
|
||
`GATEWAY_BACKEND_HTTP_URL`/`_GRPC_ADDR` (→ `backend`),
|
||
`GATEWAY_VALIDATOR_ADDR` (→ `validator:9091`), `BACKEND_CONNECTOR_ADDR` (→ the gateway
|
||
bot-link relay `gateway:9092`), `BACKEND_RENDERER_URL` (→ `renderer:8090`, the image-render
|
||
sidecar), the bot's `TELEGRAM_GATEWAY_ADDR` (→ `gateway:9443`,
|
||
mTLS) with the `GATEWAY_BOTLINK_*` / `TELEGRAM_BOTLINK_*` cert paths under `/certs` (the
|
||
mTLS material is generated by `deploy/gen-certs.sh`, gitignored, regenerated each
|
||
deploy), and all services' `*_OTEL_*_EXPORTER=otlp` →
|
||
`OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317`
|
||
(`_INSECURE=true`). The bot shares the VPN sidecar's netns: routing to the
|
||
collector's / gateway's internal IP is fine (connected route), but its `AWG_CONF` must
|
||
**not** set a `DNS=` directive — that hijacks resolv.conf and breaks resolving `otelcol`
|
||
/ `gateway` ("produced zero addresses"); without it the netns uses Docker's resolver,
|
||
which resolves `otelcol`, `gateway` and `api.telegram.org`. `GATEWAY_ADMIN_*` is
|
||
intentionally **unset** — caddy owns `/_gm` in the contour.
|
||
|
||
## Bumping the dictionary version
|
||
|
||
The dictionary ships as a versioned **release artifact** (`scrabble-dawg-vX.Y.Z.tar.gz`) from
|
||
[`scrabble-dictionary`](https://gitea.iliadenisov.ru/developer/scrabble-dictionary). The tag is
|
||
a build-time input with **no default** in the images. It is a single Gitea repo variable —
|
||
**`DICT_VERSION`** (unprefixed, shared across contours) — so a release bump is **one edit**:
|
||
|
||
- the CI test jobs read it via `.gitea/workflows/ci.yaml`'s top-level
|
||
`env.DICT_VERSION: ${{ vars.DICT_VERSION }}` (the unit/integration jobs download that dawg), and
|
||
- both deploy jobs feed the same variable to `compose` as the `DICT_VERSION` build-arg that
|
||
bakes a **fresh** volume's seed.
|
||
|
||
For local builds set `DICT_VERSION` in `deploy/.env` (template: `.env.example`); a bare
|
||
`docker build` needs `--build-arg DICT_VERSION=vX.Y.Z`. The Dockerfiles and `compose` carry no
|
||
default — a missing value fails loudly instead of baking a stale tag.
|
||
|
||
Bumping the seed is a **no-op on a live volume** (the `.seed_version` marker wins — the
|
||
seed-drift guard). A running contour/prod moves to a new release **through the admin console**
|
||
`/_gm/dictionary` (upload the tarball, preview the per-variant diff, confirm); in-flight games
|
||
keep their pinned version, new games use the new one (ARCHITECTURE.md §5).
|
||
|
||
## Production rollout
|
||
|
||
Prod runs on **two hosts** (main = full stack + ACME on the domain; tg = the bot only,
|
||
native Bot API, no VPN), one-time provisioned by **[`ansible/`](ansible/)** (docker, a
|
||
non-sudo `deploy` user holding the CI key, key-only sshd, default-deny ufw, fail2ban).
|
||
Re-run `ansible/` after a host resize — it is idempotent.
|
||
|
||
**To roll out:** merge `development → master` (CI green), then run the **`prod-deploy`**
|
||
workflow manually (Gitea → Actions → prod-deploy → run from `master`, input
|
||
`confirm=deploy`). It builds + pushes the images to the registry, ships the
|
||
compose/config/certs/env over SSH, deploys the main host with `prod-deploy.sh` (rolling,
|
||
health-gated, **auto-rollback to the previous tag**; caddy is force-recreated on its roll so
|
||
a bind-mounted `Caddyfile` change applies — its image is pinned and admin is off, so neither a
|
||
new tag nor a hot reload would pick it up), then the bot host, then probes the
|
||
public site. After `master` is green this workflow is the **only** thing that touches
|
||
prod — nothing auto-deploys there. It runs four visible jobs: **build → deploy-main →
|
||
deploy-bot → verify** (the per-service rolling shows in the deploy-main log).
|
||
|
||
**Maintenance page.** During the roll (and any migration window) `prod-deploy.sh` raises a
|
||
flag the edge caddy serves a static 503 "технические работы" page from
|
||
(`deploy/caddy/maintenance.html`), for the user-facing routes only — `/_gm` (Grafana) stays
|
||
reachable. The flag is cleared on any exit (success, a health failure + rollback, or an
|
||
error) by a shell trap, so it can never stick on. It arms from this feature's own deploy
|
||
onward (the caddy carrying the gate must be live first). This is **not** a zero-downtime
|
||
deploy — the backend is a single stateful instance (in-memory game state + push hub, no
|
||
Redis), so its swap still blips (clients auto-reconnect the live stream); the page just
|
||
makes the window graceful instead of raw 502s.
|
||
|
||
**Versioning.** Each release is a git tag `vX.Y.Z` on `master`; the deploy stamps
|
||
`git describe --tags` into every image tag, every binary (`-ldflags` → `pkg/version` →
|
||
the `service.version` telemetry attribute) and the SPA About screen. Tag the release
|
||
before running the deploy:
|
||
|
||
```sh
|
||
git tag -a v1.0.0 -m v1.0.0 && git push origin v1.0.0
|
||
```
|
||
|
||
**Manual rollback** (any time after a successful deploy). Run the **`prod-rollback`**
|
||
workflow (Gitea → Actions → prod-rollback, `confirm=rollback`). Leave `target_version`
|
||
blank to roll back to the previously deployed version (read from the host's
|
||
`PREVIOUS_TAG`), or set it to a release tag from the **Releases** page. It re-deploys
|
||
that already-published image rolling + health-gated — no rebuild, no DB migration
|
||
(image rollback is DB-safe under the expand-contract rule). The registry keeps every
|
||
release tag, so any prior release is reachable.
|
||
|
||
**Migrations** must be **expand-contract** (backward-compatible; goose is forward-only):
|
||
the automatic rollback is image-only and never restores the DB. A deploy that changes
|
||
`backend/internal/postgres/migrations/` opens a maintenance window — the backend (sole
|
||
writer) is stopped for a consistent **whole-database** `pg_dump` into `/opt/scrabble/dumps`
|
||
(it covers `backend` **and** `payments`) before the new backend migrates. **Manual DB
|
||
restore** (only if a migration was destructive): drop the app schemas in the same instance
|
||
and pipe the dump back —
|
||
`docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA IF EXISTS backend CASCADE; DROP SCHEMA IF EXISTS payments CASCADE;'`,
|
||
then `docker exec -i scrabble-postgres psql -U scrabble -d scrabble < the-dump.sql`, and
|
||
redeploy the matching old tag. This dump is a belt-and-braces net for a bad migration;
|
||
**point-in-time recovery** (below) is the primary recovery path once armed, and the only one
|
||
that survives losing the host.
|
||
|
||
## Point-in-time recovery (PITR)
|
||
|
||
The main host archives Postgres continuously with **pgBackRest** to **Selectel S3**
|
||
(encrypted at rest, path-style addressing) so the database can be restored to any moment —
|
||
protecting the money ledger and the game state against corruption or host loss. It is the
|
||
primary recovery path; the migration-window `pg_dump` above is the secondary net.
|
||
|
||
**Shape.** A daily full **base backup** (a systemd timer on the main host, `04:00`) plus
|
||
**continuous WAL** archived by Postgres `archive_command` (a segment is forced at least every
|
||
5 minutes, so the recovery point is never more than a few minutes behind). Retention is **30
|
||
days** (`repo1-retention-full=30`); the repository is AES-256-CBC encrypted. This is main-host
|
||
only — the test contour never archives.
|
||
|
||
**Wiring.** pgBackRest ships inside the DB image (`deploy/postgres/Dockerfile`); the
|
||
repository + S3 credentials + cipher are the `PGBACKREST_*` environment on the postgres
|
||
service in `docker-compose.prod.yml`, rendered from the `PROD_` Gitea set by
|
||
`write-prod-env.sh`. Archiving is gated by **`PGBACKREST_ARCHIVE_MODE` (default `off`)**, so
|
||
shipping or redeploying this stack does **not** start archiving — the artifact is inert until
|
||
armed, which is why an un-armed prod deploy can never pile WAL onto the disk. The base-backup
|
||
timer is provisioned by the Ansible `main` role behind `pitr_enabled` (also default off). Two
|
||
Grafana alerts watch health: `WAL archiving failing` (`pg_stat_archiver_failed_count` rising)
|
||
and `WAL archiving stalled` (`pg_stat_archiver_last_archive_age` over 30 min) — both
|
||
absent/NaN-safe, so they stay quiet until archiving is armed.
|
||
|
||
**Assessment (owner-reviewed; the gate before the first real money).** Measured on prod
|
||
`pg_stat_wal`: WAL is generated at **~0.77 MB/day** and the database is **~9.6 MB**. At
|
||
30-day retention on Selectel S3 (~2 ₽/GB·month) the archive is **under ~0.3 GB → well under
|
||
1 ₽/month** (compression halves it again); request volume is trivial. Performance impact is
|
||
**negligible**: archive-push moves tiny compressed segments, and the daily full base backup
|
||
is a ~10 MB, sub-second job on the 2 vCPU host. Revisit both if traffic grows ~100× (watch
|
||
`node_exporter` during a base backup).
|
||
|
||
**Arming (owner-coordinated, once, before real payments).** Ships disarmed; to turn it on:
|
||
|
||
1. **Owner (Selectel + Gitea):** create an S3 bucket (name **lowercase, no dots/underscores**)
|
||
and an S3 access key/secret; pick a repository **cipher passphrase** and store it **apart
|
||
from the S3 keys** (losing it makes the archive unrecoverable). Set the Gitea `PROD_` set —
|
||
variables `PROD_PGBACKREST_S3_ENDPOINT` (the S3 **host only** — no `https://`, no port, no
|
||
bucket) / `_S3_BUCKET` / `_S3_REGION`, an optional `_S3_PORT` (default 443, set only for a
|
||
non-standard provider port), and `PROD_PGBACKREST_ARCHIVE_MODE=on`; secrets
|
||
`PROD_PGBACKREST_S3_KEY` (the S3 **access key**) / `_S3_KEY_SECRET` (its paired **secret
|
||
key**, shown once at creation) / `_CIPHER_PASS` (a fresh repository passphrase, e.g.
|
||
`openssl rand -base64 48`).
|
||
2. Promote `development → master`, tag, and run **prod-deploy**. The roll recreates postgres
|
||
with `archive_mode=on` behind the maintenance page. (Archive pushes fail harmlessly for the
|
||
minute until step 3 creates the repository — the WAL is retained, not lost.)
|
||
3. On the main host, create the repository, take the first base backup, and verify:
|
||
```sh
|
||
docker exec scrabble-postgres pgbackrest --stanza=scrabble stanza-create
|
||
docker exec scrabble-postgres pgbackrest --stanza=scrabble --type=full backup
|
||
docker exec scrabble-postgres pgbackrest --stanza=scrabble check
|
||
```
|
||
4. Enable the daily timer: `ansible-playbook site.yml -e pitr_enabled=true` (installs + starts
|
||
`pgbackrest-backup.timer` on the main host).
|
||
5. Confirm in Grafana that the two archiving alerts are green (`last_archive_age` now tracks a
|
||
real number, `failed_count` flat).
|
||
|
||
**Restore drill (proves recoverability; re-run after arming and after any major change).** On
|
||
an **isolated, one-shot** target (a throwaway VM or a container with no ingress), pgBackRest,
|
||
the matching Postgres major, an empty `PGDATA`, and the same `PGBACKREST_*` environment:
|
||
|
||
```sh
|
||
pgbackrest --stanza=scrabble --type=time "--target=YYYY-MM-DD HH:MM:SS+00" --delta restore
|
||
# start Postgres; it replays WAL to the target; then verify a known row / the ledger tail
|
||
```
|
||
|
||
The target holds **real money + personal data** while it exists — keep it network-isolated and
|
||
**destroy it (wipe `PGDATA` + the instance) afterwards**. Record each drill (date, target
|
||
timestamp, outcome) here:
|
||
|
||
| Date | Target timestamp | Result |
|
||
| --- | --- | --- |
|
||
| _pending first arming_ | — | — |
|
||
|
||
**bot-link cert rotation:** regenerate (`deploy/gen-certs.sh /tmp/c --force`), reset the
|
||
five `PROD_BOTLINK_*` secrets from `/tmp/c`, and re-run the workflow — both hosts redeploy
|
||
together with the fresh CA.
|
||
|
||
**Sizing / monitoring:** the main host launches undersized (2 vCPU / 1.9 GiB); the prod
|
||
overlay trims limits + `GOMAXPROCS=2` + 7d Prometheus retention, and `node_exporter` feeds
|
||
host memory to Grafana (`/_gm/grafana/`). Watch host memory and resize at Selectel when
|
||
players arrive. The per-container memory caps are enforced (Compose v2 → cgroup), so no one
|
||
service can eat all RAM, but they **overcommit** the host (~2.8 GiB of caps vs 1.9 GiB) — a
|
||
**1 GiB swap file** (Ansible `common` role, `swap_size`, `vm.swappiness=10`) cushions a
|
||
simultaneous spike into swap instead of the kernel OOM-killer. The `host_mem_low` Grafana
|
||
alert fires under 10% available.
|
||
|
||
**Shared Gitea set** (one unprefixed entry each, used by every contour) — secrets:
|
||
`GATEWAY_VK_APP_SECRET, GATEWAY_VK_ID_CLIENT_SECRET, SMTP_RELAY_USER, SMTP_RELAY_PASS`;
|
||
variables: `DICT_VERSION, SMTP_RELAY_HOST, SMTP_RELAY_PORT, SMTP_RELAY_TLS, GRAFANA_SMTP_PORT,
|
||
VITE_VK_APP_LINK, VITE_VK_APP_ID`. **Derived from `PUBLIC_BASE_URL` at deploy** (not stored):
|
||
`TELEGRAM_MINIAPP_URL, GRAFANA_ROOT_URL, VITE_VK_ID_REDIRECT_URL`.
|
||
|
||
**`PROD_` Gitea set** (per-contour, mirrors `TEST_`, mapped onto the unprefixed names above)
|
||
— secrets:
|
||
`PROD_{POSTGRES_PASSWORD, GM_BASICAUTH_HASH, GRAFANA_ADMIN_PASSWORD, TELEGRAM_BOT_TOKEN,
|
||
TELEGRAM_PROMO_BOT_TOKEN, EXPORT_SIGN_KEY, GATEWAY_HONEYTOKEN, REGISTRY_PASSWORD, SSH_KEY,
|
||
SSH_KNOWN_HOSTS, BOTLINK_CA, BOTLINK_GATEWAY_CERT, BOTLINK_GATEWAY_KEY, BOTLINK_BOT_CERT,
|
||
BOTLINK_BOT_KEY, PGBACKREST_S3_KEY, PGBACKREST_S3_KEY_SECRET, PGBACKREST_CIPHER_PASS}`;
|
||
variables:
|
||
`PROD_{REGISTRY_USER, MAIN_HOST, TG_HOST, CADDY_SITE_ADDRESS, GM_BASICAUTH_USER, LOG_LEVEL,
|
||
TELEGRAM_GAME_CHANNEL_ID, TELEGRAM_CHAT_ID, TELEGRAM_SUPPORT_CHAT_ID, TELEGRAM_BOT_USERNAME,
|
||
VITE_TELEGRAM_BOT_ID, VITE_TELEGRAM_LINK, VITE_TELEGRAM_GAME_CHANNEL_NAME, SMTP_RELAY_FROM,
|
||
PUBLIC_BASE_URL, SMTP_RELAY_ADMIN_FROM, ADMIN_EMAIL, SMTP_RELAY_SERVICE_FROM, SERVICE_EMAIL,
|
||
GF_SMTP_ENABLED, PGBACKREST_S3_ENDPOINT, PGBACKREST_S3_PORT, PGBACKREST_S3_BUCKET,
|
||
PGBACKREST_S3_REGION, PGBACKREST_ARCHIVE_MODE}`. The test contour uses the same names under `TEST_`, minus the
|
||
prod-only infra (`MAIN_HOST`/`TG_HOST`/`REGISTRY_*`/`SSH_*`/`BOTLINK_*` and the `PGBACKREST_*`
|
||
PITR set, which is prod-only — the test contour never archives) and plus `TEST_AWG_CONF` (the
|
||
bot's VPN egress).
|
||
|
||
## Host-side setup (outside this repo)
|
||
|
||
- **`edge` network** must exist on the host (`docker network create edge`).
|
||
- **Host caddy** route `<domain> → scrabble:80` (the in-compose caddy serves HTTP
|
||
in the test contour; the host caddy terminates TLS). Not needed on prod, where the
|
||
contour caddy owns TLS (set `CADDY_SITE_ADDRESS` to the domain).
|
||
- **Branch protection** requires the single status check `CI / gate`.
|
||
The `unit` / `integration` / `ui` jobs are path-conditional (they skip when their
|
||
code did not change), and the always-running `gate` job aggregates them (passing
|
||
when each succeeded or was skipped), so a skipped job never blocks a merge. See
|
||
[`../CLAUDE.md`](../CLAUDE.md) "Branching & CI".
|