The Ansible base-directory loop created /opt/scrabble/certs at mode 0750
(deploy:deploy), like config/dumps/images. The gateway (and backend) run as
the distroless nonroot UID 65532 — not the deploy user — so the container
cannot traverse a 0750 certs dir and fails at startup with
"mtls: load server keypair: ... permission denied", crash-looping.
This is latent: a long-running container holds the keypair in memory and
never re-reads the file, so the misconfig only bites when a container
restarts (a host reboot / redeploy). A hoster maintenance reboot exposed it
on prod — the gateway came back crash-looping while the deploy could not SSH
in mid-reboot.
Split certs out of the 0750 loop and create it 0755 (traversable). The keys
stay 0644 by design (the gateway compose relies on it); the host is
single-tenant + SSH-access-controlled, so a traversable certs dir adds no
meaningful exposure. The live prod host was already chmod-fixed by hand; this
keeps the next provisioning run from re-tightening it.
Two prod-deploy hardening measures (owner-requested):
- Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container
memory caps enforce that one service can't eat all RAM, but they overcommit the
1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel
OOM-killer (and it might pick postgres). A small swap absorbs the overshoot.
Idempotent, builtin-only (no ansible.posix).
- Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap /
migration window that the caddy edge serves a static 503 "технические работы" page
from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm
(Grafana) stays reachable. Cleared on any exit (success, health failure + rollback,
or error) by a shell trap so it can never stick on. The 503 carries Retry-After +
an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned
window apart from a transient error (the static page only catches a fresh load; an
in-session user needs the app-side overlay). Not zero-downtime — the single
stateful backend still blips — but the window is graceful instead of raw 502s.
Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC
edge and serves the page + markers; /_gm bypasses; toggling needs no reload
(per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
Idempotent playbooks under deploy/ansible/ prepare both production hosts:
docker-ce + compose plugin, a non-sudo deploy service account holding the CI
deploy key, key-only sshd, default-deny ufw, fail2ban, unattended upgrades and
chrony. The main host also opens 80/443/9443 and creates the external edge
network; the tg host verifies direct Bot API egress (the no-VPN assumption).
The application is deployed separately by the prod-deploy workflow (later
phase), running as the deploy account this playbook provisions.
Ilia Denisov