feat(deploy): prod OOM swap cushion + edge maintenance page
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m39s
Two prod-deploy hardening measures (owner-requested): - Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container memory caps enforce that one service can't eat all RAM, but they overcommit the 1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel OOM-killer (and it might pick postgres). A small swap absorbs the overshoot. Idempotent, builtin-only (no ansible.posix). - Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap / migration window that the caddy edge serves a static 503 "технические работы" page from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm (Grafana) stays reachable. Cleared on any exit (success, health failure + rollback, or error) by a shell trap so it can never stick on. The 503 carries Retry-After + an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned window apart from a transient error (the static page only catches a fresh load; an in-session user needs the app-side overlay). Not zero-downtime — the single stateful backend still blips — but the window is graceful instead of raw 502s. Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC edge and serves the page + markers; /_gm bypasses; toggling needs no reload (per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
This commit is contained in:
@@ -19,3 +19,11 @@ scrabble_base_dir: /opt/scrabble
|
||||
# the host's own containers (and any ad-hoc runs) rotate identically.
|
||||
docker_log_max_size: "10m"
|
||||
docker_log_max_file: "3"
|
||||
|
||||
# Swap file as an OOM cushion. The per-container memory caps (docker-compose.prod.yml)
|
||||
# sum to more than the tight main host's RAM (2 vCPU / 1.9 GiB), so a simultaneous
|
||||
# spike could otherwise hit the kernel OOM-killer and it might pick postgres. A small
|
||||
# swap absorbs the overshoot; swappiness stays low so swap is a cushion, not a hot
|
||||
# path (a slow service beats a killed one). Set swap_size to "0" to skip provisioning.
|
||||
swap_size: "1G"
|
||||
swap_swappiness: 10
|
||||
|
||||
@@ -150,6 +150,57 @@
|
||||
enabled: true
|
||||
state: started
|
||||
|
||||
# --- Swap file (OOM cushion) ---------------------------------------------------
|
||||
# The prod main host is tight (1.9 GiB) and the per-container memory caps
|
||||
# (docker-compose.prod.yml) sum to more than RAM, so a simultaneous spike could hit
|
||||
# the kernel OOM-killer (which might pick postgres). A small swap absorbs the
|
||||
# overshoot. Idempotent; skipped when swap_size == "0". Builtin-only (no ansible.posix).
|
||||
|
||||
- name: Check whether the swap file is already active
|
||||
ansible.builtin.command: swapon --show=NAME --noheadings
|
||||
register: swap_active
|
||||
changed_when: false
|
||||
when: swap_size != "0"
|
||||
|
||||
- name: Allocate the swap file
|
||||
ansible.builtin.command:
|
||||
cmd: "fallocate -l {{ swap_size }} /swapfile"
|
||||
creates: /swapfile
|
||||
when: swap_size != "0" and '/swapfile' not in swap_active.stdout
|
||||
|
||||
- name: Secure the swap file (0600)
|
||||
ansible.builtin.file:
|
||||
path: /swapfile
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0600"
|
||||
when: swap_size != "0"
|
||||
|
||||
- name: Format and enable the swap file
|
||||
ansible.builtin.shell: "mkswap /swapfile && swapon /swapfile"
|
||||
when: swap_size != "0" and '/swapfile' not in swap_active.stdout
|
||||
|
||||
- name: Persist the swap file in /etc/fstab
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/fstab
|
||||
line: "/swapfile none swap sw 0 0"
|
||||
regexp: '^/swapfile\s'
|
||||
state: present
|
||||
when: swap_size != "0"
|
||||
|
||||
- name: Keep swap a cushion, not a hot path (low swappiness)
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/sysctl.d/60-scrabble-swap.conf
|
||||
mode: "0644"
|
||||
content: "vm.swappiness = {{ swap_swappiness }}\n"
|
||||
register: swappiness_conf
|
||||
when: swap_size != "0"
|
||||
|
||||
- name: Apply swappiness now
|
||||
ansible.builtin.command: sysctl --system
|
||||
changed_when: false
|
||||
when: swap_size != "0" and swappiness_conf is changed
|
||||
|
||||
# --- Deploy directories --------------------------------------------------------
|
||||
|
||||
- name: Create the scrabble base directories
|
||||
|
||||
Reference in New Issue
Block a user