Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.
- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
deploy/README.md; record the measured cost/perf assessment.
Two prod-deploy hardening measures (owner-requested):
- Swap file (Ansible common role, swap_size=1G, vm.swappiness=10). The per-container
memory caps enforce that one service can't eat all RAM, but they overcommit the
1.9 GiB main host (~2.8 GiB of caps), so a simultaneous spike could hit the kernel
OOM-killer (and it might pick postgres). A small swap absorbs the overshoot.
Idempotent, builtin-only (no ansible.posix).
- Edge maintenance page. prod-deploy.sh raises a flag around the rolling swap /
migration window that the caddy edge serves a static 503 "технические работы" page
from (deploy/caddy/maintenance.html), for the user-facing routes only — /_gm
(Grafana) stays reachable. Cleared on any exit (success, health failure + rollback,
or error) by a shell trap so it can never stick on. The 503 carries Retry-After +
an X-Scrabble-Maintenance marker so a follow-up SPA overlay can tell a planned
window apart from a transient error (the static page only catches a fresh load; an
in-session user needs the app-side overlay). Not zero-downtime — the single
stateful backend still blips — but the window is graceful instead of raw 502s.
Verified: caddy validate; the gate 503s every non-/_gm path incl. the Connect/gRPC
edge and serves the page + markers; /_gm bypasses; toggling needs no reload
(per-request stat). Ansible --syntax-check + compose config (base+prod) pass.
The finished-game export (GCG + a new PNG of the final position) is one
signed, short-lived relative URL (game.export_url; HMAC-SHA256, 10-min
TTL, BACKEND_EXPORT_SIGN_KEY) resolved against the client's own origin
and delivered by the best affordance each platform has (five on-device
review rounds):
- TG Android/desktop: native showPopup chooser -> native downloadFile
dialog (bridge-only chain, activation-safe).
- TG iOS: app-modal chooser -> OS share sheet with the fetched file
(a popup callback cannot supply the activation the sheet needs).
- VK iOS: VKWebAppDownloadFile for both formats.
- VK Android: the PNG opens in VK's native image viewer, the GCG copies
to the clipboard (the VK Android downloader hangs on any download,
Content-Length/Range notwithstanding).
- VK desktop iframe / desktop browsers: plain anchor downloads.
- Mobile browsers: the OS share sheet (fetch-then-share).
- Legacy TG (< Bot API 8.0): app modal + GCG clipboard, no image option.
The PNG is rasterized on demand by the new internal `renderer` sidecar
(node:22-slim + skia-canvas + baked Liberation/Noto Color Emoji fonts)
executing the SAME ui/src/lib/gameimage.ts the ui project unit-tests;
the backend rebuilds the render payload from the journal +
engine.AlphabetTable, and the device date locale, IANA time zone and
localized non-play labels ride the signed URL. Nothing is stored — the
artifact re-derives from the immutable journal on each GET. The gateway
forwards /dl/* (caddy @gateway matcher extended) behind the per-IP
public rate limiter and serves bytes via http.ServeContent.
Deploy: renderer service in compose + prod overlay + rolling order +
prod push list; TEST_/PROD_EXPORT_SIGN_KEY secrets; the sidecar smoke
runs in the ui CI job. Docs: ARCHITECTURE, FUNCTIONAL(+_ru), UI_DESIGN,
TESTING, deploy/README, renderer/README.
The prod rolling deploy rolls each service with `compose up -d --no-deps <svc>`.
For caddy that is a no-op on a config-only release: its image is pinned
(caddy:2-alpine, no $TAG), so the compose definition is unchanged between
releases, compose treats the container as current and does not recreate it, and
admin is off so there is no hot reload. The new bind-mounted Caddyfile is seeded
to the host but never loaded -- the v1.2.2 `Alt-Svc: clear` edge fix deployed
green yet did not take effect until caddy was restarted by hand.
Force a recreate for caddy on its roll (every other service already recreates on
its new $TAG image), so a bind-mounted Caddyfile change always applies. Costs a
~1-2s caddy blip per deploy, acceptable for the infrequent manual prod rollout.
- prod-deploy.yaml is now four visible sequential jobs (build -> deploy-main ->
deploy-bot -> verify) so the rollout stages show in the Actions UI; the
per-service rolling stays in the deploy-main log.
- prod-rollback.yaml: a separate manual workflow_dispatch. Leave target_version
blank to roll back to the previous deployed version (the host now tracks
DEPLOYED_TAG + PREVIOUS_TAG), or pick a release tag. Re-deploys an already
published image rolling + health-gated, image-only (no rebuild, no DB migration).
- prod-deploy.sh tracks the previous tag (commit_tag) for the blank-input rollback.
- Docs: ARCHITECTURE §13 + deploy/README runbook cover versioning + rollback.
A workflow_dispatch-only rollout from master (confirm=deploy):
- .gitea/workflows/prod-deploy.yaml builds + pushes the images to the registry,
ships the compose/config/certs/env over SSH, deploys the main host via
prod-deploy.sh, then the bot host, then verifies the public site.
- deploy/prod-deploy.sh rolls the main stack one service at a time in dependency
order (postgres->backend->gateway->landing->validator->caddy), health-checking
after each; any failure rolls the whole stack back to the previous tag. A schema
migration adds a maintenance window: the backend (sole writer) is stopped for a
consistent pg_dump before migrating; image rollback stays DB-safe (expand-contract),
the dump is kept for a manual restore.
- prod overlay: pull the four main images from the registry by tag.
- Runtime secrets reach the host via a sourced env.sh (single-quoted values keep the
bcrypt hash's literal $ intact, unlike a --env-file).
Ilia Denisov
developer