feat(deploy): prod OOM swap cushion + edge maintenance page
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s

Two prod-deploy hardening measures (owner-requested):

- Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container
  memory caps enforce that one service can't eat all RAM, but they overcommit the
  1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel
  OOM-killer (and it might pick postgres). A small swap absorbs the overshoot.
  Idempotent, builtin-only (no ansible.posix).

- Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap /
  migration window that the caddy edge serves a static 503 "технические работы" page
  from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm
  (Grafana) stays reachable. Cleared on any exit (success, health failure + rollback,
  or error) by a shell trap so it can never stick on. The 503 carries Retry-After +
  an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned
  window apart from a transient error (the static page only catches a fresh load; an
  in-session user needs the app-side overlay). Not zero-downtime — the single
  stateful backend still blips — but the window is graceful instead of raw 502s.

Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC
edge and serves the page + markers; /_gm bypasses; toggling needs no reload
(per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
This commit is contained in:
Ilia Denisov
2026-07-03 22:09:37 +02:00
parent 4b4dcab9b6
commit c8601c0115
7 changed files with 172 additions and 1 deletions
+8
View File
@@ -19,3 +19,11 @@ scrabble_base_dir: /opt/scrabble
# the host's own containers (and any ad-hoc runs) rotate identically.
docker_log_max_size: "10m"
docker_log_max_file: "3"
# Swap file as an OOM cushion. The per-container memory caps (docker-compose.prod.yml)
# sum to more than the tight main host's RAM (2 vCPU / 1.9 GiB), so a simultaneous
# spike could otherwise hit the kernel OOM-killer and it might pick postgres. A small
# swap absorbs the overshoot; swappiness stays low so swap is a cushion, not a hot
# path (a slow service beats a killed one). Set swap_size to "0" to skip provisioning.
swap_size: "1G"
swap_swappiness: 10
@@ -150,6 +150,57 @@
enabled: true
state: started
# --- Swap file (OOM cushion) ---------------------------------------------------
# The prod main host is tight (1.9 GiB) and the per-container memory caps
# (docker-compose.prod.yml) sum to more than RAM, so a simultaneous spike could hit
# the kernel OOM-killer (which might pick postgres). A small swap absorbs the
# overshoot. Idempotent; skipped when swap_size == "0". Builtin-only (no ansible.posix).
- name: Check whether the swap file is already active
ansible.builtin.command: swapon --show=NAME --noheadings
register: swap_active
changed_when: false
when: swap_size != "0"
- name: Allocate the swap file
ansible.builtin.command:
cmd: "fallocate -l {{ swap_size }} /swapfile"
creates: /swapfile
when: swap_size != "0" and '/swapfile' not in swap_active.stdout
- name: Secure the swap file (0600)
ansible.builtin.file:
path: /swapfile
owner: root
group: root
mode: "0600"
when: swap_size != "0"
- name: Format and enable the swap file
ansible.builtin.shell: "mkswap /swapfile && swapon /swapfile"
when: swap_size != "0" and '/swapfile' not in swap_active.stdout
- name: Persist the swap file in /etc/fstab
ansible.builtin.lineinfile:
path: /etc/fstab
line: "/swapfile none swap sw 0 0"
regexp: '^/swapfile\s'
state: present
when: swap_size != "0"
- name: Keep swap a cushion, not a hot path (low swappiness)
ansible.builtin.copy:
dest: /etc/sysctl.d/60-scrabble-swap.conf
mode: "0644"
content: "vm.swappiness = {{ swap_swappiness }}\n"
register: swappiness_conf
when: swap_size != "0"
- name: Apply swappiness now
ansible.builtin.command: sysctl --system
changed_when: false
when: swap_size != "0" and swappiness_conf is changed
# --- Deploy directories --------------------------------------------------------
- name: Create the scrabble base directories