feat(deploy): prod OOM swap cushion + edge maintenance page
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s

Two prod-deploy hardening measures (owner-requested):

- Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container
  memory caps enforce that one service can't eat all RAM, but they overcommit the
  1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel
  OOM-killer (and it might pick postgres). A small swap absorbs the overshoot.
  Idempotent, builtin-only (no ansible.posix).

- Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap /
  migration window that the caddy edge serves a static 503 "технические работы" page
  from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm
  (Grafana) stays reachable. Cleared on any exit (success, health failure + rollback,
  or error) by a shell trap so it can never stick on. The 503 carries Retry-After +
  an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned
  window apart from a transient error (the static page only catches a fresh load; an
  in-session user needs the app-side overlay). Not zero-downtime — the single
  stateful backend still blips — but the window is graceful instead of raw 502s.

Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC
edge and serves the page + markers; /_gm bypasses; toggling needs no reload
(per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
This commit is contained in:
Ilia Denisov
2026-07-03 22:09:37 +02:00
parent 4b4dcab9b6
commit c8601c0115
7 changed files with 172 additions and 1 deletions
+15 -1
View File
@@ -189,6 +189,16 @@ public site. After `master` is green this workflow is the **only** thing that to
prod — nothing auto-deploys there. It runs four visible jobs: **build → deploy-main →
deploy-bot → verify** (the per-service rolling shows in the deploy-main log).
**Maintenance page.** During the roll (and any migration window) `prod-deploy.sh` raises a
flag the edge caddy serves a static 503 "технические работы" page from
(`deploy/caddy/maintenance.html`), for the user-facing routes only — `/_gm` (Grafana) stays
reachable. The flag is cleared on any exit (success, a health failure + rollback, or an
error) by a shell trap, so it can never stick on. It arms from this feature's own deploy
onward (the caddy carrying the gate must be live first). This is **not** a zero-downtime
deploy — the backend is a single stateful instance (in-memory game state + push hub, no
Redis), so its swap still blips (clients auto-reconnect the live stream); the page just
makes the window graceful instead of raw 502s.
**Versioning.** Each release is a git tag `vX.Y.Z` on `master`; the deploy stamps
`git describe --tags` into every image tag, every binary (`-ldflags``pkg/version`
the `service.version` telemetry attribute) and the SPA About screen. Tag the release
@@ -221,7 +231,11 @@ together with the fresh CA.
**Sizing / monitoring:** the main host launches undersized (2 vCPU / 1.9 GiB); the prod
overlay trims limits + `GOMAXPROCS=2` + 7d Prometheus retention, and `node_exporter` feeds
host memory to Grafana (`/_gm/grafana/`). Watch host memory and resize at Selectel when
players arrive.
players arrive. The per-container memory caps are enforced (Compose v2 → cgroup), so no one
service can eat all RAM, but they **overcommit** the host (~2.8 GiB of caps vs 1.9 GiB) — a
**1 GiB swap file** (Ansible `common` role, `swap_size`, `vm.swappiness=10`) cushions a
simultaneous spike into swap instead of the kernel OOM-killer. The `host_mem_low` Grafana
alert fires under 10% available.
**Shared Gitea set** (one unprefixed entry each, used by every contour) — secrets:
`GATEWAY_VK_APP_SECRET, GATEWAY_VK_ID_CLIENT_SECRET, SMTP_RELAY_USER, SMTP_RELAY_PASS`;