The tg (bot) VPS came up on Europe/Moscow while the main host is UTC, so host-level
timestamps (journald, file mtimes, cron) sat 3h apart across the fleet. Add a
community.general.timezone task to the common role so every provisioned host is UTC
(idempotent — a no-op on the already-UTC main host). Applied live to the tg host in
the same change; the containerised services run in UTC regardless, so no restart.
The Ansible base-directory loop created /opt/scrabble/certs at mode 0750
(deploy:deploy), like config/dumps/images. The gateway (and backend) run as
the distroless nonroot UID 65532 — not the deploy user — so the container
cannot traverse a 0750 certs dir and fails at startup with
"mtls: load server keypair: ... permission denied", crash-looping.
This is latent: a long-running container holds the keypair in memory and
never re-reads the file, so the misconfig only bites when a container
restarts (a host reboot / redeploy). A hoster maintenance reboot exposed it
on prod — the gateway came back crash-looping while the deploy could not SSH
in mid-reboot.
Split certs out of the 0750 loop and create it 0755 (traversable). The keys
stay 0644 by design (the gateway compose relies on it); the host is
single-tenant + SSH-access-controlled, so a traversable certs dir adds no
meaningful exposure. The live prod host was already chmod-fixed by hand; this
keeps the next provisioning run from re-tightening it.
docker exec defaults to root, so the systemd base-backup timer connected to the
database as role "root" (then "postgres") — neither exists; the superuser role is
POSTGRES_USER (scrabble). Run the timer's pgBackRest as the postgres OS user (-u postgres,
for lock-dir/PGDATA consistency with archive-push) and connect with --pg1-user=scrabble.
archive_command (run by the postgres server process) was already correct.
Also record the point-in-time-recovery arming + restore drill in deploy/README.md
(correct the runbook commands to the same invocation, fill the drill log) and mark the
durability work done in PLAN.md.
Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.
- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
deploy/README.md; record the measured cost/perf assessment.
Two prod-deploy hardening measures (owner-requested):
- Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container
memory caps enforce that one service can't eat all RAM, but they overcommit the
1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel
OOM-killer (and it might pick postgres). A small swap absorbs the overshoot.
Idempotent, builtin-only (no ansible.posix).
- Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap /
migration window that the caddy edge serves a static 503 "технические работы" page
from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm
(Grafana) stays reachable. Cleared on any exit (success, health failure + rollback,
or error) by a shell trap so it can never stick on. The 503 carries Retry-After +
an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned
window apart from a transient error (the static page only catches a fresh load; an
in-session user needs the app-side overlay). Not zero-downtime — the single
stateful backend still blips — but the window is graceful instead of raw 502s.
Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC
edge and serves the page + markers; /_gm bypasses; toggling needs no reload
(per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
The project is live in production, so the staged-development scaffolding is removed.
- Delete the staged trackers PLAN.md and PRERELEASE.md.
- Rewrite CLAUDE.md: drop the per-stage workflow; codify the ongoing development
principles (How we work) and the production model (Branching, CI & production):
manual prod-deploy / prod-rollback, semver release tags, Ansible provisioning,
expand-contract migrations.
- De-stage the living docs (README, ARCHITECTURE, TESTING, deploy/ansible, loadtest,
platform/telegram READMEs) and the docker-compose tuning comments: drop the
Stage N / R1-R7 / pre-release labels, keep every number and rationale, and fix the
now-dangling PLAN.md / PRERELEASE.md references to describe the current state.
- Reword stale 'later stage' Go doc comments for subsystems that have shipped.
Idempotent playbooks under deploy/ansible/ prepare both production hosts:
docker-ce + compose plugin, a non-sudo deploy service account holding the CI
deploy key, key-only sshd, default-deny ufw, fail2ban, unattended upgrades and
chrony. The main host also opens 80/443/9443 and creates the external edge
network; the tg host verifies direct Bot API egress (the no-VPN assumption).
The application is deployed separately by the prod-deploy workflow (later
phase), running as the deploy account this playbook provisions.
Ilia Denisov