feat(deploy): prod OOM swap cushion + edge maintenance page
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s
Two prod-deploy hardening measures (owner-requested): - Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container memory caps enforce that one service can't eat all RAM, but they overcommit the 1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel OOM-killer (and it might pick postgres). A small swap absorbs the overshoot. Idempotent, builtin-only (no ansible.posix). - Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap / migration window that the caddy edge serves a static 503 "технические работы" page from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm (Grafana) stays reachable. Cleared on any exit (success, health failure + rollback, or error) by a shell trap so it can never stick on. The 503 carries Retry-After + an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned window apart from a transient error (the static page only catches a fresh load; an in-session user needs the app-side overlay). Not zero-downtime — the single stateful backend still blips — but the window is graceful instead of raw 502s. Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC edge and serves the page + markers; /_gm bypasses; toggling needs no reload (per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
This commit is contained in:
+15
-1
@@ -189,6 +189,16 @@ public site. After `master` is green this workflow is the **only** thing that to
|
||||
prod — nothing auto-deploys there. It runs four visible jobs: **build → deploy-main →
|
||||
deploy-bot → verify** (the per-service rolling shows in the deploy-main log).
|
||||
|
||||
**Maintenance page.** During the roll (and any migration window) `prod-deploy.sh` raises a
|
||||
flag the edge caddy serves a static 503 "технические работы" page from
|
||||
(`deploy/caddy/maintenance.html`), for the user-facing routes only — `/_gm` (Grafana) stays
|
||||
reachable. The flag is cleared on any exit (success, a health failure + rollback, or an
|
||||
error) by a shell trap, so it can never stick on. It arms from this feature's own deploy
|
||||
onward (the caddy carrying the gate must be live first). This is **not** a zero-downtime
|
||||
deploy — the backend is a single stateful instance (in-memory game state + push hub, no
|
||||
Redis), so its swap still blips (clients auto-reconnect the live stream); the page just
|
||||
makes the window graceful instead of raw 502s.
|
||||
|
||||
**Versioning.** Each release is a git tag `vX.Y.Z` on `master`; the deploy stamps
|
||||
`git describe --tags` into every image tag, every binary (`-ldflags` → `pkg/version` →
|
||||
the `service.version` telemetry attribute) and the SPA About screen. Tag the release
|
||||
@@ -221,7 +231,11 @@ together with the fresh CA.
|
||||
**Sizing / monitoring:** the main host launches undersized (2 vCPU / 1.9 GiB); the prod
|
||||
overlay trims limits + `GOMAXPROCS=2` + 7d Prometheus retention, and `node_exporter` feeds
|
||||
host memory to Grafana (`/_gm/grafana/`). Watch host memory and resize at Selectel when
|
||||
players arrive.
|
||||
players arrive. The per-container memory caps are enforced (Compose v2 → cgroup), so no one
|
||||
service can eat all RAM, but they **overcommit** the host (~2.8 GiB of caps vs 1.9 GiB) — a
|
||||
**1 GiB swap file** (Ansible `common` role, `swap_size`, `vm.swappiness=10`) cushions a
|
||||
simultaneous spike into swap instead of the kernel OOM-killer. The `host_mem_low` Grafana
|
||||
alert fires under 10% available.
|
||||
|
||||
**Shared Gitea set** (one unprefixed entry each, used by every contour) — secrets:
|
||||
`GATEWAY_VK_APP_SECRET, GATEWAY_VK_ID_CLIENT_SECRET, SMTP_RELAY_USER, SMTP_RELAY_PASS`;
|
||||
|
||||
Reference in New Issue
Block a user