feat(deploy): prod OOM swap cushion + edge maintenance page
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s
Two prod-deploy hardening measures (owner-requested): - Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container memory caps enforce that one service can't eat all RAM, but they overcommit the 1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel OOM-killer (and it might pick postgres). A small swap absorbs the overshoot. Idempotent, builtin-only (no ansible.posix). - Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap / migration window that the caddy edge serves a static 503 "технические работы" page from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm (Grafana) stays reachable. Cleared on any exit (success, health failure + rollback, or error) by a shell trap so it can never stick on. The 503 carries Retry-After + an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned window apart from a transient error (the static page only catches a fresh load; an in-session user needs the app-side overlay). Not zero-downtime — the single stateful backend still blips — but the window is graceful instead of raw 502s. Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC edge and serves the page + markers; /_gm bypasses; toggling needs no reload (per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
This commit is contained in:
@@ -42,6 +42,15 @@ PREV_STATE_FILE="${PREV_STATE_FILE:-/opt/scrabble/PREVIOUS_TAG}"
|
||||
PG_USER="${POSTGRES_USER:-scrabble}"
|
||||
PG_DB="${POSTGRES_DB:-scrabble}"
|
||||
|
||||
# Edge maintenance page. caddy serves a static 503 "works in progress" page for the
|
||||
# user-facing routes while this flag file exists (deploy/caddy/Caddyfile). We hold it
|
||||
# across the roll / migration window and clear it on ANY exit — success, a health
|
||||
# failure + rollback, or an unexpected error — via the trap, so users get a graceful
|
||||
# page instead of raw mid-swap 502s and the flag can never get stuck on.
|
||||
MAINT_FLAG="${MAINT_FLAG:-${SCRABBLE_CONFIG_DIR:-/opt/scrabble}/caddy/on}"
|
||||
maint_off() { rm -f "$MAINT_FLAG" 2>/dev/null || true; }
|
||||
trap maint_off EXIT
|
||||
|
||||
cd "$COMPOSE_DIR" || { echo "compose dir $COMPOSE_DIR missing"; exit 1; }
|
||||
export REGISTRY
|
||||
# otelcol joins the host docker group to read the socket; the GID varies per host.
|
||||
@@ -119,6 +128,13 @@ if [ -z "$(docker ps -aq -f name=scrabble-backend)" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Existing stack: raise the maintenance page for the whole roll (and any migration
|
||||
# window). It shows once caddy carries the gate (from this feature's own deploy
|
||||
# onward); the EXIT trap lowers it however this run ends.
|
||||
mkdir -p "$(dirname "$MAINT_FLAG")"
|
||||
: > "$MAINT_FLAG"
|
||||
echo "maintenance page raised ($MAINT_FLAG)"
|
||||
|
||||
# Migration deploy: freeze writes and snapshot a consistent dump before migrating.
|
||||
if [ "$MIGRATION" = 1 ]; then
|
||||
echo "migration deploy: opening maintenance window (stopping the backend = the only writer)"
|
||||
|
||||
Reference in New Issue
Block a user