e922f5f3b9
CI / changes (pull_request) Successful in 4s
CI / unit (pull_request) Successful in 12s
CI / integration (pull_request) Successful in 24s
CI / ui (pull_request) Successful in 1m9s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m41s
Add pgBackRest-based PITR for the production database, shipped disarmed (archive_mode and the base-backup timer gated off) so it stays inert until the operator arms it before real payments — an un-armed deploy cannot pile WAL onto the disk. - Postgres now builds from a thin image carrying pgBackRest (archive_command runs in-process); the prod overlay wires an encrypted (AES-256-CBC), path-style S3 repository with 30-day retention and a daily full base-backup systemd timer. - Repository config/secrets flow through the PROD_PGBACKREST_* Gitea set and write-prod-env.sh; prod-rollback re-renders the same env so a rollback cannot disarm archiving. - Grafana alerts watch pg_stat_archiver (failing / stalled), absent-safe on the test contour, which never archives. - Fix the pre-migration pg_dump to snapshot the whole database (was backend-only, silently excluding payments). - Document the PITR runbook, the arming sequence and the restore drill in deploy/README.md; record the measured cost/perf assessment.
142 lines
4.5 KiB
YAML
142 lines
4.5 KiB
YAML
# Production main-host overlay, applied on top of docker-compose.yml on the main host:
|
|
# docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
|
|
#
|
|
# It (1) publishes caddy 80/443 — there is no host caddy in prod, so the contour caddy
|
|
# owns the edge and does its own ACME on CADDY_SITE_ADDRESS — and the gateway bot-link
|
|
# :9443 the remote bot dials in over mTLS; and (2) retunes the baseline limits down for the
|
|
# 2 vCPU / 1.9 GiB host (GOMAXPROCS=2, smaller memory caps, shorter Prometheus
|
|
# retention). The contour launches deliberately undersized at zero players; the added
|
|
# node_exporter + Grafana watch host memory so it can be resized at Selectel when
|
|
# traffic arrives.
|
|
#
|
|
# The bot + its VPN sidecar are absent here (the telegram-local profile is not
|
|
# activated); the prod bot runs on its own host from docker-compose.bot.yml.
|
|
|
|
services:
|
|
caddy:
|
|
ports:
|
|
- "80:80"
|
|
- "443:443"
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 96M
|
|
|
|
gateway:
|
|
# Prod pulls the pushed image by tag instead of building locally; the base
|
|
# build: section stays dormant because the deploy always pulls first.
|
|
image: ${REGISTRY:?set REGISTRY}/scrabble-gateway:${TAG:?set TAG}
|
|
ports:
|
|
- "9443:9443"
|
|
environment:
|
|
# 2 vCPU host: align the Go scheduler with the cgroup quota (the baseline's 3-core gateway needs 3 cores).
|
|
GOMAXPROCS: "2"
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
cpus: "2.0"
|
|
memory: 384M
|
|
|
|
backend:
|
|
image: ${REGISTRY:?set REGISTRY}/scrabble-backend:${TAG:?set TAG}
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 384M
|
|
|
|
postgres:
|
|
image: ${REGISTRY:?set REGISTRY}/scrabble-postgres:${TAG:?set TAG}
|
|
# Continuous WAL archiving to S3 via pgBackRest (point-in-time recovery), on the prod
|
|
# main host only. archive_mode is gated by PGBACKREST_ARCHIVE_MODE (default "off"), so
|
|
# merging/redeploying this stack does NOT start archiving until the operator arms it:
|
|
# archiving stays inert — and cannot pile WAL onto the disk — until the S3 repository is
|
|
# set up, the stanza is created and the switch is flipped on. See deploy/README.md
|
|
# (point-in-time recovery — arming). Non-secret settings are inline; the endpoint,
|
|
# bucket, region, S3 keys and repository cipher passphrase come from the prod env.sh
|
|
# (deploy/write-prod-env.sh, PROD_ secrets/variables).
|
|
environment:
|
|
PGBACKREST_STANZA: scrabble
|
|
PGBACKREST_PG1_PATH: /var/lib/postgresql/data
|
|
PGBACKREST_REPO1_TYPE: s3
|
|
PGBACKREST_REPO1_PATH: /pgbackrest
|
|
PGBACKREST_REPO1_S3_URI_STYLE: path
|
|
PGBACKREST_REPO1_S3_ENDPOINT: ${PGBACKREST_S3_ENDPOINT:-}
|
|
PGBACKREST_REPO1_S3_BUCKET: ${PGBACKREST_S3_BUCKET:-}
|
|
PGBACKREST_REPO1_S3_REGION: ${PGBACKREST_S3_REGION:-}
|
|
PGBACKREST_REPO1_S3_KEY: ${PGBACKREST_S3_KEY:-}
|
|
PGBACKREST_REPO1_S3_KEY_SECRET: ${PGBACKREST_S3_KEY_SECRET:-}
|
|
PGBACKREST_REPO1_CIPHER_TYPE: aes-256-cbc
|
|
PGBACKREST_REPO1_CIPHER_PASS: ${PGBACKREST_CIPHER_PASS:-}
|
|
PGBACKREST_REPO1_RETENTION_FULL: "30"
|
|
PGBACKREST_COMPRESS_TYPE: zst
|
|
PGBACKREST_LOG_LEVEL_CONSOLE: info
|
|
PGBACKREST_LOG_LEVEL_FILE: "off"
|
|
# archive_command is inert while archive_mode is off, so it is always present and only
|
|
# the mode switches. A forced 5-minute segment switch bounds the recovery point.
|
|
command:
|
|
- postgres
|
|
- -c
|
|
- archive_mode=${PGBACKREST_ARCHIVE_MODE:-off}
|
|
- -c
|
|
- "archive_command=pgbackrest --stanza=scrabble archive-push %p"
|
|
- -c
|
|
- archive_timeout=300
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 384M
|
|
|
|
renderer:
|
|
image: ${REGISTRY:?set REGISTRY}/scrabble-renderer:${TAG:?set TAG}
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 160M
|
|
|
|
validator:
|
|
image: ${REGISTRY:?set REGISTRY}/scrabble-telegram-validator:${TAG:?set TAG}
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 96M
|
|
|
|
landing:
|
|
image: ${REGISTRY:?set REGISTRY}/scrabble-landing:${TAG:?set TAG}
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 64M
|
|
|
|
otelcol:
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 256M
|
|
|
|
prometheus:
|
|
command:
|
|
- --config.file=/etc/prometheus/prometheus.yml
|
|
- --storage.tsdb.retention.time=7d
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 256M
|
|
|
|
tempo:
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 384M
|
|
|
|
grafana:
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 256M
|
|
|
|
postgres_exporter:
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 64M
|