Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.
- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
deploy/README.md; record the measured cost/perf assessment.
The finished-game export (GCG + a new PNG of the final position) is one
signed, short-lived relative URL (game.export_url; HMAC-SHA256, 10-min
TTL, BACKEND_EXPORT_SIGN_KEY) resolved against the client's own origin
and delivered by the best affordance each platform has (five on-device
review rounds):
- TG Android/desktop: native showPopup chooser -> native downloadFile
dialog (bridge-only chain, activation-safe).
- TG iOS: app-modal chooser -> OS share sheet with the fetched file
(a popup callback cannot supply the activation the sheet needs).
- VK iOS: VKWebAppDownloadFile for both formats.
- VK Android: the PNG opens in VK's native image viewer, the GCG copies
to the clipboard (the VK Android downloader hangs on any download,
Content-Length/Range notwithstanding).
- VK desktop iframe / desktop browsers: plain anchor downloads.
- Mobile browsers: the OS share sheet (fetch-then-share).
- Legacy TG (< Bot API 8.0): app modal + GCG clipboard, no image option.
The PNG is rasterized on demand by the new internal `renderer` sidecar
(node:22-slim + skia-canvas + baked Liberation/Noto Color Emoji fonts)
executing the SAME ui/src/lib/gameimage.ts the ui project unit-tests;
the backend rebuilds the render payload from the journal +
engine.AlphabetTable, and the device date locale, IANA time zone and
localized non-play labels ride the signed URL. Nothing is stored — the
artifact re-derives from the immutable journal on each GET. The gateway
forwards /dl/* (caddy @gateway matcher extended) behind the per-IP
public rate limiter and serves bytes via http.ServeContent.
Deploy: renderer service in compose + prod overlay + rolling order +
prod push list; TEST_/PROD_EXPORT_SIGN_KEY secrets; the sidecar smoke
runs in the ui CI job. Docs: ARCHITECTURE, FUNCTIONAL(+_ru), UI_DESIGN,
TESTING, deploy/README, renderer/README.
The project is live in production, so the staged-development scaffolding is removed.
- Delete the staged trackers PLAN.md and PRERELEASE.md.
- Rewrite CLAUDE.md: drop the per-stage workflow; codify the ongoing development
principles (How we work) and the production model (Branching, CI & production):
manual prod-deploy / prod-rollback, semver release tags, Ansible provisioning,
expand-contract migrations.
- De-stage the living docs (README, ARCHITECTURE, TESTING, deploy/ansible, loadtest,
platform/telegram READMEs) and the docker-compose tuning comments: drop the
Stage N / R1-R7 / pre-release labels, keep every number and rationale, and fix the
now-dangling PLAN.md / PRERELEASE.md references to describe the current state.
- Reword stale 'later stage' Go doc comments for subsystems that have shipped.
A workflow_dispatch-only rollout from master (confirm=deploy):
- .gitea/workflows/prod-deploy.yaml builds + pushes the images to the registry,
ships the compose/config/certs/env over SSH, deploys the main host via
prod-deploy.sh, then the bot host, then verifies the public site.
- deploy/prod-deploy.sh rolls the main stack one service at a time in dependency
order (postgres->backend->gateway->landing->validator->caddy), health-checking
after each; any failure rolls the whole stack back to the previous tag. A schema
migration adds a maintenance window: the backend (sole writer) is stopped for a
consistent pg_dump before migrating; image rollback stays DB-safe (expand-contract),
the dump is kept for a manual restore.
- prod overlay: pull the four main images from the registry by tag.
- Runtime secrets reach the host via a sourced env.sh (single-quoted values keep the
bcrypt hash's literal $ intact, unlike a --env-file).
Split the contour across the two prod hosts and retune for the small main host:
- Gate vpn+bot to the telegram-local profile. The CI test deploy now passes
--profile telegram-local so the test contour still brings them; the prod main
host omits both, and the prod bot runs standalone from docker-compose.bot.yml.
- docker-compose.prod.yml (main-host overlay): publish caddy 80/443 (no host
caddy in prod; caddy owns ACME) and gateway 9443 (the remote bot dials in over
mTLS); GOMAXPROCS=2, smaller memory caps and 7d Prometheus retention for the
2 vCPU / 1.9 GiB host. It launches deliberately undersized; resize reactively.
- docker-compose.bot.yml: standalone bot for the tg host (no VPN, OTLP off since
otelcol is unreachable from there, dials the main host's bot-link).
- Add node_exporter + a Prometheus scrape so host memory pressure (the OOM
signal on the tight main host), not just per-container docker_stats, is visible.
- Soften AWG_CONF to a default: only the profiled vpn sidecar consumes it, and
compose interpolates profiled-out services too, so prod must not require it.
Ilia Denisov
developer