feat(deploy): prod OOM swap cushion + edge maintenance page
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s
Two prod-deploy hardening measures (owner-requested): - Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container memory caps enforce that one service can't eat all RAM, but they overcommit the 1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel OOM-killer (and it might pick postgres). A small swap absorbs the overshoot. Idempotent, builtin-only (no ansible.posix). - Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap / migration window that the caddy edge serves a static 503 "технические работы" page from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm (Grafana) stays reachable. Cleared on any exit (success, health failure + rollback, or error) by a shell trap so it can never stick on. The 503 carries Retry-After + an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned window apart from a transient error (the static page only catches a fresh load; an in-session user needs the app-side overlay). Not zero-downtime — the single stateful backend still blips — but the window is graceful instead of raw 502s. Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC edge and serves the page + markers; /_gm bypasses; toggling needs no reload (per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
This commit is contained in:
@@ -33,6 +33,39 @@
|
||||
# is the prod fix. Background + alternatives (incl. serving h3 for real): docs/EDGE_HTTP3.md.
|
||||
header Alt-Svc clear
|
||||
|
||||
# Maintenance gate. While the deploy holds the flag file /srv/maint/on (touched by
|
||||
# prod-deploy.sh around a rolling swap, removed on the script's exit), serve a static
|
||||
# 503 "works in progress" page instead of proxying to a mid-recreate upstream —
|
||||
# a graceful signal rather than raw 502s. Operator surfaces (/_gm) are excluded so
|
||||
# Grafana stays reachable during the window. Caddy stat()s the flag per request, so
|
||||
# toggling it needs no reload; the page + flag live in the caddy config dir bind-mounted
|
||||
# read-only at /srv/maint (docker-compose.yml). The test contour never sets the flag
|
||||
# (only prod-deploy.sh does), so this is inert there.
|
||||
@maintenance {
|
||||
not path /_gm /_gm/*
|
||||
file {
|
||||
root /srv/maint
|
||||
try_files on
|
||||
}
|
||||
}
|
||||
handle @maintenance {
|
||||
error 503
|
||||
}
|
||||
handle_errors {
|
||||
@maint503 expression {err.status_code} == 503
|
||||
handle @maint503 {
|
||||
root * /srv/maint
|
||||
rewrite * /maintenance.html
|
||||
header Retry-After 120
|
||||
# Distinctive maintenance marker so the SPA can tell a planned window apart from
|
||||
# a transient upstream 503 and show its own dimmed overlay (an in-session user
|
||||
# never sees this static page — it only renders on a fresh load). The header
|
||||
# rides every gated response, incl. the Connect/gRPC edge the app polls.
|
||||
header X-Scrabble-Maintenance 1
|
||||
file_server
|
||||
}
|
||||
}
|
||||
|
||||
# Operator surfaces under /_gm: a single shared Basic-Auth, then route.
|
||||
@gm path /_gm /_gm/*
|
||||
handle @gm {
|
||||
|
||||
Reference in New Issue
Block a user