Files
scrabble-game/deploy/docker-compose.prod.yml
T
Ilia Denisov e922f5f3b9
CI / changes (pull_request) Successful in 4s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 24s
CI / ui (pull_request) Successful in 1m9s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m41s
feat(deploy): continuous WAL archiving to S3 for point-in-time recovery
Add pgBackRest-based PITR for the production database, shipped disarmed
(archive_mode and the base-backup timer gated off) so it stays inert until the
operator arms it before real payments — an un-armed deploy cannot pile WAL onto
the disk.

- Postgres now builds from a thin image carrying pgBackRest (archive_command runs
  in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3
  repository with 30-day retention and a daily full base-backup systemd timer.
- Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and
  write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot
  disarm archiving.
- Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the
  test contour, which never archives.
- Fix the pre-migration pg_dump to snapshot the whole database (was backend-only,
  silently excluding payments).
- Document the PITR runbook, the arming sequence and the restore drill in
  deploy/README.md; record the measured cost/perf assessment.
2026-07-09 00:58:04 +02:00

142 lines
4.5 KiB
YAML

# Production main-host overlay, applied on top of docker-compose.yml on the main host:
# docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
#
# It (1) publishes caddy 80/443 — there is no host caddy in prod, so the contour caddy
# owns the edge and does its own ACME on CADDY_SITE_ADDRESS — and the gateway bot-link
# :9443 the remote bot dials in over mTLS; and (2) retunes the baseline limits down for the
# 2 vCPU / 1.9 GiB host (GOMAXPROCS=2, smaller memory caps, shorter Prometheus
# retention). The contour launches deliberately undersized at zero players; the added
# node_exporter + Grafana watch host memory so it can be resized at Selectel when
# traffic arrives.
#
# The bot + its VPN sidecar are absent here (the telegram-local profile is not
# activated); the prod bot runs on its own host from docker-compose.bot.yml.
services:
caddy:
ports:
- "80:80"
- "443:443"
deploy:
resources:
limits:
memory: 96M
gateway:
# Prod pulls the pushed image by tag instead of building locally; the base
# build: section stays dormant because the deploy always pulls first.
image: ${REGISTRY:?set REGISTRY}/scrabble-gateway:${TAG:?set TAG}
ports:
- "9443:9443"
environment:
# 2 vCPU host: align the Go scheduler with the cgroup quota (the baseline's 3-core gateway needs 3 cores).
GOMAXPROCS: "2"
deploy:
resources:
limits:
cpus: "2.0"
memory: 384M
backend:
image: ${REGISTRY:?set REGISTRY}/scrabble-backend:${TAG:?set TAG}
deploy:
resources:
limits:
memory: 384M
postgres:
image: ${REGISTRY:?set REGISTRY}/scrabble-postgres:${TAG:?set TAG}
# Continuous WAL archiving to S3 via pgBackRest (point-in-time recovery), on the prod
# main host only. archive_mode is gated by PGBACKREST_ARCHIVE_MODE (default "off"), so
# merging/redeploying this stack does NOT start archiving until the operator arms it:
# archiving stays inert — and cannot pile WAL onto the disk — until the S3 repository is
# set up, the stanza is created and the switch is flipped on. See deploy/README.md
# (point-in-time recovery — arming). Non-secret settings are inline; the endpoint,
# bucket, region, S3 keys and repository cipher passphrase come from the prod env.sh
# (deploy/write-prod-env.sh, PROD_ secrets/variables).
environment:
PGBACKREST_STANZA: scrabble
PGBACKREST_PG1_PATH: /var/lib/postgresql/data
PGBACKREST_REPO1_TYPE: s3
PGBACKREST_REPO1_PATH: /pgbackrest
PGBACKREST_REPO1_S3_URI_STYLE: path
PGBACKREST_REPO1_S3_ENDPOINT: ${PGBACKREST_S3_ENDPOINT:-}
PGBACKREST_REPO1_S3_BUCKET: ${PGBACKREST_S3_BUCKET:-}
PGBACKREST_REPO1_S3_REGION: ${PGBACKREST_S3_REGION:-}
PGBACKREST_REPO1_S3_KEY: ${PGBACKREST_S3_KEY:-}
PGBACKREST_REPO1_S3_KEY_SECRET: ${PGBACKREST_S3_KEY_SECRET:-}
PGBACKREST_REPO1_CIPHER_TYPE: aes-256-cbc
PGBACKREST_REPO1_CIPHER_PASS: ${PGBACKREST_CIPHER_PASS:-}
PGBACKREST_REPO1_RETENTION_FULL: "30"
PGBACKREST_COMPRESS_TYPE: zst
PGBACKREST_LOG_LEVEL_CONSOLE: info
PGBACKREST_LOG_LEVEL_FILE: "off"
# archive_command is inert while archive_mode is off, so it is always present and only
# the mode switches. A forced 5-minute segment switch bounds the recovery point.
command:
- postgres
- -c
- archive_mode=${PGBACKREST_ARCHIVE_MODE:-off}
- -c
- "archive_command=pgbackrest --stanza=scrabble archive-push %p"
- -c
- archive_timeout=300
deploy:
resources:
limits:
memory: 384M
renderer:
image: ${REGISTRY:?set REGISTRY}/scrabble-renderer:${TAG:?set TAG}
deploy:
resources:
limits:
memory: 160M
validator:
image: ${REGISTRY:?set REGISTRY}/scrabble-telegram-validator:${TAG:?set TAG}
deploy:
resources:
limits:
memory: 96M
landing:
image: ${REGISTRY:?set REGISTRY}/scrabble-landing:${TAG:?set TAG}
deploy:
resources:
limits:
memory: 64M
otelcol:
deploy:
resources:
limits:
memory: 256M
prometheus:
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.retention.time=7d
deploy:
resources:
limits:
memory: 256M
tempo:
deploy:
resources:
limits:
memory: 384M
grafana:
deploy:
resources:
limits:
memory: 256M
postgres_exporter:
deploy:
resources:
limits:
memory: 64M