feat(deploy): prod OOM swap cushion + edge maintenance page
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s

Two prod-deploy hardening measures (owner-requested):

- Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container
  memory caps enforce that one service can't eat all RAM, but they overcommit the
  1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel
  OOM-killer (and it might pick postgres). A small swap absorbs the overshoot.
  Idempotent, builtin-only (no ansible.posix).

- Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap /
  migration window that the caddy edge serves a static 503 "технические работы" page
  from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm
  (Grafana) stays reachable. Cleared on any exit (success, health failure + rollback,
  or error) by a shell trap so it can never stick on. The 503 carries Retry-After +
  an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned
  window apart from a transient error (the static page only catches a fresh load; an
  in-session user needs the app-side overlay). Not zero-downtime — the single
  stateful backend still blips — but the window is graceful instead of raw 502s.

Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC
edge and serves the page + markers; /_gm bypasses; toggling needs no reload
(per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
This commit is contained in:
Ilia Denisov
2026-07-03 22:09:37 +02:00
parent 4b4dcab9b6
commit c8601c0115
7 changed files with 172 additions and 1 deletions
+16
View File
@@ -42,6 +42,15 @@ PREV_STATE_FILE="${PREV_STATE_FILE:-/opt/scrabble/PREVIOUS_TAG}"
PG_USER="${POSTGRES_USER:-scrabble}"
PG_DB="${POSTGRES_DB:-scrabble}"
# Edge maintenance page. caddy serves a static 503 "works in progress" page for the
# user-facing routes while this flag file exists (deploy/caddy/Caddyfile). We hold it
# across the roll / migration window and clear it on ANY exit — success, a health
# failure + rollback, or an unexpected error — via the trap, so users get a graceful
# page instead of raw mid-swap 502s and the flag can never get stuck on.
MAINT_FLAG="${MAINT_FLAG:-${SCRABBLE_CONFIG_DIR:-/opt/scrabble}/caddy/on}"
maint_off() { rm -f "$MAINT_FLAG" 2>/dev/null || true; }
trap maint_off EXIT
cd "$COMPOSE_DIR" || { echo "compose dir $COMPOSE_DIR missing"; exit 1; }
export REGISTRY
# otelcol joins the host docker group to read the socket; the GID varies per host.
@@ -119,6 +128,13 @@ if [ -z "$(docker ps -aq -f name=scrabble-backend)" ]; then
exit 0
fi
# Existing stack: raise the maintenance page for the whole roll (and any migration
# window). It shows once caddy carries the gate (from this feature's own deploy
# onward); the EXIT trap lowers it however this run ends.
mkdir -p "$(dirname "$MAINT_FLAG")"
: > "$MAINT_FLAG"
echo "maintenance page raised ($MAINT_FLAG)"
# Migration deploy: freeze writes and snapshot a consistent dump before migrating.
if [ "$MIGRATION" = 1 ]; then
echo "migration deploy: opening maintenance window (stopping the backend = the only writer)"