From f5f45e7afb52f9e5d943ce8200694bd59d86b194 Mon Sep 17 00:00:00 2001 From: Ilia Denisov Date: Sun, 21 Jun 2026 23:54:57 +0200 Subject: [PATCH 1/5] feat(deploy): Ansible provisioning for prod hosts (Stage 18) Idempotent playbooks under deploy/ansible/ prepare both production hosts: docker-ce + compose plugin, a non-sudo deploy service account holding the CI deploy key, key-only sshd, default-deny ufw, fail2ban, unattended upgrades and chrony. The main host also opens 80/443/9443 and creates the external edge network; the tg host verifies direct Bot API egress (the no-VPN assumption). The application is deployed separately by the prod-deploy workflow (later phase), running as the deploy account this playbook provisions. --- deploy/ansible/README.md | 49 +++++ deploy/ansible/ansible.cfg | 11 ++ deploy/ansible/group_vars/all.yml | 21 +++ deploy/ansible/inventory.ini | 19 ++ deploy/ansible/roles/common/handlers/main.yml | 15 ++ deploy/ansible/roles/common/tasks/main.yml | 167 ++++++++++++++++++ .../roles/common/templates/daemon.json.j2 | 8 + .../roles/common/templates/jail.local.j2 | 9 + .../common/templates/sshd-hardening.conf.j2 | 6 + deploy/ansible/roles/main/tasks/main.yml | 18 ++ deploy/ansible/roles/tg/tasks/main.yml | 19 ++ deploy/ansible/site.yml | 31 ++++ 12 files changed, 373 insertions(+) create mode 100644 deploy/ansible/README.md create mode 100644 deploy/ansible/ansible.cfg create mode 100644 deploy/ansible/group_vars/all.yml create mode 100644 deploy/ansible/inventory.ini create mode 100644 deploy/ansible/roles/common/handlers/main.yml create mode 100644 deploy/ansible/roles/common/tasks/main.yml create mode 100644 deploy/ansible/roles/common/templates/daemon.json.j2 create mode 100644 deploy/ansible/roles/common/templates/jail.local.j2 create mode 100644 deploy/ansible/roles/common/templates/sshd-hardening.conf.j2 create mode 100644 deploy/ansible/roles/main/tasks/main.yml create mode 100644 deploy/ansible/roles/tg/tasks/main.yml create mode 100644 deploy/ansible/site.yml diff --git a/deploy/ansible/README.md b/deploy/ansible/README.md new file mode 100644 index 0000000..36fd44e --- /dev/null +++ b/deploy/ansible/README.md @@ -0,0 +1,49 @@ +# Prod host provisioning (Stage 18) + +Idempotent Ansible that prepares the two production hosts. It installs Docker, a +non-sudo `deploy` service account, SSH hardening, a default-deny firewall, +fail2ban, unattended security upgrades and time sync. It does **not** deploy the +application — that is `.gitea/workflows/prod-deploy.yaml`'s job, running as the +`deploy` account this playbook creates. + +Hosts are referenced by `~/.ssh/config` aliases (`scrabble-main-ops`, +`scrabble-tg-ops`), so no IPs or key paths live in the repo. + +## Prerequisites (controller) + +- `ansible` with the bundled collections (`community.general`, `community.docker`, + `ansible.posix`). +- The two hosts reachable as root via the ssh-config aliases, host keys already + accepted into `known_hosts` (`host_key_checking = True`). + +## One-time: the CI deploy key + +The CI prod-deploy workflow logs into the hosts as `deploy` using a dedicated +key. Generate it once on the controller, authorize its public half via the +playbook, and store its private half **only** in the Gitea `PROD_SSH_KEY` secret: + +```sh +ssh-keygen -t ed25519 -N '' -C scrabble-ci-deploy \ + -f ~/.ssh/scrabble_ci_deploy_ed25519 +# private half -> Gitea secret PROD_SSH_KEY (set via API); never commit it +``` + +## Run + +```sh +cd deploy/ansible +ansible-playbook site.yml +``` + +The playbook reads the public key from `~/.ssh/scrabble_ci_deploy_ed25519.pub` by +default; override with `-e deploy_ci_pubkey_path=/path/to/key.pub`. Re-running is +safe (idempotent) and survives a host resize. + +## What each host gets + +- **both** (`common`): docker-ce + compose plugin, `daemon.json` (live-restore, + 10m×3 log rotation), `deploy` user (docker group, no sudo), key-only sshd, + `ufw` default-deny incoming + allow SSH, fail2ban sshd jail, unattended + upgrades, chrony, `/opt/scrabble/{config,certs,dumps,images}`. +- **main**: `ufw` opens 80/443/9443; the external `edge` docker network. +- **tg**: verifies direct `api.telegram.org` egress (the no-VPN assumption). diff --git a/deploy/ansible/ansible.cfg b/deploy/ansible/ansible.cfg new file mode 100644 index 0000000..f846548 --- /dev/null +++ b/deploy/ansible/ansible.cfg @@ -0,0 +1,11 @@ +[defaults] +inventory = inventory.ini +roles_path = roles +interpreter_python = /usr/bin/python3 +host_key_checking = True +stdout_callback = yaml +deprecation_warnings = False +retry_files_enabled = False + +[ssh_connection] +pipelining = True diff --git a/deploy/ansible/group_vars/all.yml b/deploy/ansible/group_vars/all.yml new file mode 100644 index 0000000..bb1f07a --- /dev/null +++ b/deploy/ansible/group_vars/all.yml @@ -0,0 +1,21 @@ +--- +# Service account the CI prod-deploy workflow uses to drive docker on the hosts. +# Membership in the docker group is root-equivalent (docker socket access), which +# is all the deploy workflow needs; the account is deliberately not given sudo. +deploy_user: deploy + +# Public half of the dedicated CI deploy SSH key, read from the controller at run +# time. The private half is generated on the controller during provisioning and +# stored ONLY in the Gitea PROD_SSH_KEY secret; it is never committed. Override the +# path with -e deploy_ci_pubkey_path=/path/to/key.pub if the key lives elsewhere. +deploy_ci_pubkey_path: "{{ lookup('env', 'HOME') }}/.ssh/scrabble_ci_deploy_ed25519.pub" +deploy_ci_pubkey: "{{ lookup('file', deploy_ci_pubkey_path) }}" + +# Base directory the deploy workflow rsyncs compose files, config, certs and dumps +# into. Owned by deploy_user so the workflow needs no elevation. +scrabble_base_dir: /opt/scrabble + +# Docker daemon json-file log rotation, mirroring the compose x-logging anchor so +# the host's own containers (and any ad-hoc runs) rotate identically. +docker_log_max_size: "10m" +docker_log_max_file: "3" diff --git a/deploy/ansible/inventory.ini b/deploy/ansible/inventory.ini new file mode 100644 index 0000000..25b80df --- /dev/null +++ b/deploy/ansible/inventory.ini @@ -0,0 +1,19 @@ +# Production inventory for Stage 18. +# +# Hosts resolve through the operator's ~/.ssh/config aliases, so HostName (public +# IP), User and IdentityFile live there — no IPs or key paths are committed here. +# scrabble-main-ops -> main stack host (public IP, domain erudit-game.ru) +# scrabble-tg-ops -> Telegram bot host (direct Bot API egress, no VPN) + +[main] +scrabble-main-ops + +[tg] +scrabble-tg-ops + +[prod:children] +main +tg + +[prod:vars] +ansible_user=root diff --git a/deploy/ansible/roles/common/handlers/main.yml b/deploy/ansible/roles/common/handlers/main.yml new file mode 100644 index 0000000..017e54b --- /dev/null +++ b/deploy/ansible/roles/common/handlers/main.yml @@ -0,0 +1,15 @@ +--- +- name: restart docker + ansible.builtin.service: + name: docker + state: restarted + +- name: reload sshd + ansible.builtin.service: + name: ssh + state: reloaded + +- name: restart fail2ban + ansible.builtin.service: + name: fail2ban + state: restarted diff --git a/deploy/ansible/roles/common/tasks/main.yml b/deploy/ansible/roles/common/tasks/main.yml new file mode 100644 index 0000000..5c0a7a3 --- /dev/null +++ b/deploy/ansible/roles/common/tasks/main.yml @@ -0,0 +1,167 @@ +--- +# Common baseline applied to both prod hosts: Docker engine, a non-sudo deploy +# service account, SSH hardening, a default-deny firewall, fail2ban, unattended +# security upgrades and time sync. Every task is idempotent. + +- name: Install base packages + ansible.builtin.apt: + name: + - ca-certificates + - curl + - gnupg + - ufw + - fail2ban + - unattended-upgrades + - chrony + state: present + update_cache: true + cache_valid_time: 3600 + +# --- Docker engine (official repo; trixie is published upstream) --------------- + +- name: Create apt keyring directory + ansible.builtin.file: + path: /etc/apt/keyrings + state: directory + mode: "0755" + +- name: Install Docker apt GPG key + ansible.builtin.get_url: + url: https://download.docker.com/linux/debian/gpg + dest: /etc/apt/keyrings/docker.asc + mode: "0644" + +- name: Add Docker apt repository + ansible.builtin.apt_repository: + repo: >- + deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] + https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable + filename: docker + state: present + +- name: Install Docker engine and the compose plugin + ansible.builtin.apt: + name: + - docker-ce + - docker-ce-cli + - containerd.io + - docker-buildx-plugin + - docker-compose-plugin + state: present + update_cache: true + +- name: Configure the Docker daemon (live-restore + log rotation) + ansible.builtin.template: + src: daemon.json.j2 + dest: /etc/docker/daemon.json + mode: "0644" + notify: restart docker + +- name: Enable and start Docker + ansible.builtin.service: + name: docker + enabled: true + state: started + +# --- Deploy service account ---------------------------------------------------- + +- name: Create the deploy service account + ansible.builtin.user: + name: "{{ deploy_user }}" + groups: docker + append: true + shell: /bin/bash + create_home: true + +- name: Ensure the deploy .ssh directory + ansible.builtin.file: + path: "/home/{{ deploy_user }}/.ssh" + state: directory + owner: "{{ deploy_user }}" + group: "{{ deploy_user }}" + mode: "0700" + +- name: Authorize the CI deploy SSH key (exclusive) + ansible.builtin.copy: + dest: "/home/{{ deploy_user }}/.ssh/authorized_keys" + content: "{{ deploy_ci_pubkey }}\n" + owner: "{{ deploy_user }}" + group: "{{ deploy_user }}" + mode: "0600" + +# --- SSH hardening ------------------------------------------------------------- + +- name: Harden sshd (key-only auth) + ansible.builtin.template: + src: sshd-hardening.conf.j2 + dest: /etc/ssh/sshd_config.d/10-scrabble-hardening.conf + mode: "0644" + validate: sshd -t -f %s + notify: reload sshd + +# --- Firewall (default deny incoming) ------------------------------------------ +# SSH is allowed before the policy flips so enabling ufw never locks us out. + +- name: Allow SSH through the firewall + community.general.ufw: + rule: allow + name: OpenSSH + +- name: Default-deny incoming, allow outgoing + community.general.ufw: + direction: "{{ item.direction }}" + policy: "{{ item.policy }}" + loop: + - { direction: incoming, policy: deny } + - { direction: outgoing, policy: allow } + +- name: Enable the firewall + community.general.ufw: + state: enabled + +# --- fail2ban ------------------------------------------------------------------ + +- name: Configure the fail2ban sshd jail + ansible.builtin.template: + src: jail.local.j2 + dest: /etc/fail2ban/jail.local + mode: "0644" + notify: restart fail2ban + +- name: Enable and start fail2ban + ansible.builtin.service: + name: fail2ban + enabled: true + state: started + +# --- Unattended security upgrades + time sync ---------------------------------- + +- name: Enable unattended upgrades + ansible.builtin.copy: + dest: /etc/apt/apt.conf.d/20auto-upgrades + mode: "0644" + content: | + APT::Periodic::Update-Package-Lists "1"; + APT::Periodic::Unattended-Upgrade "1"; + +- name: Enable and start chrony + ansible.builtin.service: + name: chrony + enabled: true + state: started + +# --- Deploy directories -------------------------------------------------------- + +- name: Create the scrabble base directories + ansible.builtin.file: + path: "{{ scrabble_base_dir }}/{{ item }}" + state: directory + owner: "{{ deploy_user }}" + group: "{{ deploy_user }}" + mode: "0750" + loop: + - "" + - config + - certs + - dumps + - images diff --git a/deploy/ansible/roles/common/templates/daemon.json.j2 b/deploy/ansible/roles/common/templates/daemon.json.j2 new file mode 100644 index 0000000..50f7040 --- /dev/null +++ b/deploy/ansible/roles/common/templates/daemon.json.j2 @@ -0,0 +1,8 @@ +{ + "live-restore": true, + "log-driver": "json-file", + "log-opts": { + "max-size": "{{ docker_log_max_size }}", + "max-file": "{{ docker_log_max_file }}" + } +} diff --git a/deploy/ansible/roles/common/templates/jail.local.j2 b/deploy/ansible/roles/common/templates/jail.local.j2 new file mode 100644 index 0000000..e037958 --- /dev/null +++ b/deploy/ansible/roles/common/templates/jail.local.j2 @@ -0,0 +1,9 @@ +# Managed by Ansible (deploy/ansible). +[DEFAULT] +bantime = 1h +findtime = 10m +maxretry = 5 +backend = systemd + +[sshd] +enabled = true diff --git a/deploy/ansible/roles/common/templates/sshd-hardening.conf.j2 b/deploy/ansible/roles/common/templates/sshd-hardening.conf.j2 new file mode 100644 index 0000000..7293298 --- /dev/null +++ b/deploy/ansible/roles/common/templates/sshd-hardening.conf.j2 @@ -0,0 +1,6 @@ +# Managed by Ansible (deploy/ansible). Key-only authentication. +# root stays reachable by key (prohibit-password) for provisioning re-runs. +PasswordAuthentication no +PermitRootLogin prohibit-password +PubkeyAuthentication yes +KbdInteractiveAuthentication no diff --git a/deploy/ansible/roles/main/tasks/main.yml b/deploy/ansible/roles/main/tasks/main.yml new file mode 100644 index 0000000..3cb7cd7 --- /dev/null +++ b/deploy/ansible/roles/main/tasks/main.yml @@ -0,0 +1,18 @@ +--- +# Main stack host: public web + bot-link ports and the external 'edge' network +# the compose stack attaches caddy to. + +- name: Open public web and bot-link ports + community.general.ufw: + rule: allow + port: "{{ item }}" + proto: tcp + loop: + - "80" # HTTP (ACME challenge + redirect to HTTPS) + - "443" # HTTPS (caddy edge) + - "9443" # bot-link mTLS (remote bot dials in; mutual TLS gates access) + +- name: Ensure the external 'edge' docker network exists + community.docker.docker_network: + name: edge + state: present diff --git a/deploy/ansible/roles/tg/tasks/main.yml b/deploy/ansible/roles/tg/tasks/main.yml new file mode 100644 index 0000000..ad62307 --- /dev/null +++ b/deploy/ansible/roles/tg/tasks/main.yml @@ -0,0 +1,19 @@ +--- +# Telegram bot host: holds no inbound port beyond SSH (the bot dials out to the +# Bot API and into the main host's bot-link). We only verify direct Bot API +# egress here, since the "no VPN" decision depends on it. + +- name: Verify direct Telegram Bot API egress (no VPN on this host) + ansible.builtin.uri: + url: https://api.telegram.org/ + method: GET + status_code: [200, 301, 302, 401, 404] # any HTTP reply proves reachability + timeout: 10 + register: tg_egress + failed_when: false + +- name: Report Telegram reachability + ansible.builtin.debug: + msg: >- + api.telegram.org reachable: + {{ (tg_egress.status | default(0) | int) > 0 }} (status {{ tg_egress.status | default('none') }}) diff --git a/deploy/ansible/site.yml b/deploy/ansible/site.yml new file mode 100644 index 0000000..a977ee8 --- /dev/null +++ b/deploy/ansible/site.yml @@ -0,0 +1,31 @@ +--- +# Stage 18 host provisioning. Idempotent: safe to re-run after a host resize. +# Prepares hosts only (docker, hardening, service account, firewall); the +# application is deployed separately by .gitea/workflows/prod-deploy.yaml. + +- name: Common baseline (both hosts) + hosts: prod + become: true + pre_tasks: + - name: Require a well-formed CI deploy public key + ansible.builtin.assert: + that: + - deploy_ci_pubkey | length > 0 + - deploy_ci_pubkey is search('^(ssh|ecdsa)-') + fail_msg: >- + deploy_ci_pubkey is empty or malformed. Generate the key first + (see deploy/ansible/README.md) or override deploy_ci_pubkey_path. + roles: + - common + +- name: Main stack host + hosts: main + become: true + roles: + - main + +- name: Telegram bot host + hosts: tg + become: true + roles: + - tg -- 2.52.0 From 2b399d0838f24d5da826235ad876f6036f84b858 Mon Sep 17 00:00:00 2001 From: Ilia Denisov Date: Mon, 22 Jun 2026 00:12:43 +0200 Subject: [PATCH 2/5] feat(deploy): prod compose split + host-memory monitoring (Stage 18) Split the contour across the two prod hosts and retune for the small main host: - Gate vpn+bot to the telegram-local profile. The CI test deploy now passes --profile telegram-local so the test contour still brings them; the prod main host omits both, and the prod bot runs standalone from docker-compose.bot.yml. - docker-compose.prod.yml (main-host overlay): publish caddy 80/443 (no host caddy in prod; caddy owns ACME) and gateway 9443 (the remote bot dials in over mTLS); GOMAXPROCS=2, smaller memory caps and 7d Prometheus retention for the 2 vCPU / 1.9 GiB host. It launches deliberately undersized; resize reactively. - docker-compose.bot.yml: standalone bot for the tg host (no VPN, OTLP off since otelcol is unreachable from there, dials the main host's bot-link). - Add node_exporter + a Prometheus scrape so host memory pressure (the OOM signal on the tight main host), not just per-container docker_stats, is visible. - Soften AWG_CONF to a default: only the profiled vpn sidecar consumes it, and compose interpolates profiled-out services too, so prod must not require it. --- .gitea/workflows/ci.yaml | 7 ++- deploy/docker-compose.bot.yml | 53 ++++++++++++++++++ deploy/docker-compose.prod.yml | 92 ++++++++++++++++++++++++++++++++ deploy/docker-compose.yml | 31 ++++++++++- deploy/prometheus/prometheus.yml | 5 ++ 5 files changed, 185 insertions(+), 3 deletions(-) create mode 100644 deploy/docker-compose.bot.yml create mode 100644 deploy/docker-compose.prod.yml diff --git a/.gitea/workflows/ci.yaml b/.gitea/workflows/ci.yaml index 1299e62..fe5a438 100644 --- a/.gitea/workflows/ci.yaml +++ b/.gitea/workflows/ci.yaml @@ -301,8 +301,11 @@ jobs: # App version for the About screen: the git tag if present, else the short SHA # (the test checkout is shallow/untagged, so this is the SHA here — fine). export APP_VERSION="$(git -C "$GITHUB_WORKSPACE" describe --tags --always 2>/dev/null || echo dev)" - docker compose --ansi never build --progress plain - docker compose --ansi never up -d --remove-orphans + # The telegram-local profile brings the bot + its VPN sidecar; prod runs the + # bot on its own host instead (deploy/docker-compose.bot.yml), and the prod + # main host omits both. Without the profile they would not start here. + docker compose --ansi never --profile telegram-local build --progress plain + docker compose --ansi never --profile telegram-local up -d --remove-orphans # The config-only services bind-mount the reseeded config dir. A plain `up -d` # leaves them on the previous bind mount (the dir was rm'd + recreated), so a # changed Caddyfile or Grafana dashboard is ignored — force-recreate them to diff --git a/deploy/docker-compose.bot.yml b/deploy/docker-compose.bot.yml new file mode 100644 index 0000000..b4c0975 --- /dev/null +++ b/deploy/docker-compose.bot.yml @@ -0,0 +1,53 @@ +# Production Telegram bot host descriptor (standalone — NOT an overlay). Run only on +# the bot host: +# docker compose -f docker-compose.bot.yml up -d +# +# The bot egresses to the Bot API directly (no VPN sidecar) and dials the main host's +# published bot-link :9443 over mTLS. It exports no telemetry — otelcol lives on the +# main host and is unreachable from here — so observe it via `docker logs` on this host. +# Values come from the prod-deploy workflow (PROD_ secrets/variables); BOT_IMAGE is the +# pushed registry tag and BOTLINK_GATEWAY_ADDR is the main host's :9443. +name: scrabble-bot + +services: + bot: + container_name: scrabble-telegram-bot + image: ${BOT_IMAGE:?set BOT_IMAGE to the registry tag} + restart: unless-stopped + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + environment: + TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:?set TELEGRAM_BOT_TOKEN} + TELEGRAM_GAME_CHANNEL_ID: ${TELEGRAM_GAME_CHANNEL_ID:-} + TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID:-} + TELEGRAM_PROMO_BOT_TOKEN: ${TELEGRAM_PROMO_BOT_TOKEN:-} + TELEGRAM_BOT_USERNAME: ${TELEGRAM_BOT_USERNAME:-} + TELEGRAM_BOT_LINK: ${TELEGRAM_BOT_LINK:-} + TELEGRAM_MINIAPP_URL: ${TELEGRAM_MINIAPP_URL:?set TELEGRAM_MINIAPP_URL} + # Real Bot API in prod (the test contour pins TELEGRAM_TEST_ENV=true instead). + TELEGRAM_TEST_ENV: "false" + TELEGRAM_API_BASE_URL: ${TELEGRAM_API_BASE_URL:-} + TELEGRAM_OWNS_UPDATES: "true" + # Dials the main host's published bot-link. ServerName stays `gateway` (the cert + # SAN), so TLS validation is independent of the dial address. + TELEGRAM_GATEWAY_ADDR: ${BOTLINK_GATEWAY_ADDR:?set BOTLINK_GATEWAY_ADDR (main:9443)} + TELEGRAM_BOTLINK_SERVER_NAME: gateway + TELEGRAM_BOTLINK_TLS_CERT: /certs/bot.crt + TELEGRAM_BOTLINK_TLS_KEY: /certs/bot.key + TELEGRAM_BOTLINK_TLS_CA: /certs/ca.crt + TELEGRAM_LOG_LEVEL: ${LOG_LEVEL:-info} + TELEGRAM_SERVICE_NAME: scrabble-telegram-bot + # No telemetry export: otelcol is on the main host, unreachable from here. + TELEGRAM_OTEL_TRACES_EXPORTER: none + TELEGRAM_OTEL_METRICS_EXPORTER: none + GOMAXPROCS: "1" + volumes: + - ${SCRABBLE_CONFIG_DIR:-.}/certs:/certs:ro + deploy: + resources: + limits: + cpus: "1.0" + memory: 256M diff --git a/deploy/docker-compose.prod.yml b/deploy/docker-compose.prod.yml new file mode 100644 index 0000000..6b45826 --- /dev/null +++ b/deploy/docker-compose.prod.yml @@ -0,0 +1,92 @@ +# Production main-host overlay, applied on top of docker-compose.yml on the main host: +# docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d +# +# It (1) publishes caddy 80/443 — there is no host caddy in prod, so the contour caddy +# owns the edge and does its own ACME on CADDY_SITE_ADDRESS — and the gateway bot-link +# :9443 the remote bot dials in over mTLS; and (2) retunes the R7 limits down for the +# 2 vCPU / 1.9 GiB host (GOMAXPROCS=2, smaller memory caps, shorter Prometheus +# retention). The contour launches deliberately undersized at zero players; the added +# node_exporter + Grafana watch host memory so it can be resized at Selectel when +# traffic arrives. +# +# The bot + its VPN sidecar are absent here (the telegram-local profile is not +# activated); the prod bot runs on its own host from docker-compose.bot.yml. + +services: + caddy: + ports: + - "80:80" + - "443:443" + deploy: + resources: + limits: + memory: 96M + + gateway: + ports: + - "9443:9443" + environment: + # 2 vCPU host: align the Go scheduler with the cgroup quota (R7's 3 needs 3 cores). + GOMAXPROCS: "2" + deploy: + resources: + limits: + cpus: "2.0" + memory: 384M + + backend: + deploy: + resources: + limits: + memory: 384M + + postgres: + deploy: + resources: + limits: + memory: 384M + + validator: + deploy: + resources: + limits: + memory: 96M + + landing: + deploy: + resources: + limits: + memory: 64M + + otelcol: + deploy: + resources: + limits: + memory: 256M + + prometheus: + command: + - --config.file=/etc/prometheus/prometheus.yml + - --storage.tsdb.retention.time=7d + deploy: + resources: + limits: + memory: 256M + + tempo: + deploy: + resources: + limits: + memory: 384M + + grafana: + deploy: + resources: + limits: + memory: 256M + + postgres_exporter: + deploy: + resources: + limits: + memory: 64M diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index cd17de5..fce7650 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -240,14 +240,22 @@ services: networks: [internal] # --- Telegram bot (egress via the VPN sidecar in test; dials the gateway) --- + # vpn + bot are gated to the `telegram-local` profile: the test contour runs them + # locally (CI passes --profile telegram-local), the prod main host omits them, and + # the prod bot runs on its own host from deploy/docker-compose.bot.yml. vpn: container_name: scrabble-telegram-vpn image: docker.iliadenisov.ru/developer/amneziawg-sidecar:latest + profiles: ["telegram-local"] restart: unless-stopped logging: *default-logging privileged: true environment: - AWG_CONF: ${AWG_CONF:?set AWG_CONF} + # Required by the vpn sidecar, which is gated to the telegram-local profile. + # Compose can't scope a `:?` guard to a profile (interpolation runs for + # profiled-out services too) and the prod main host has no VPN, so this is a soft + # default; the test contour always supplies TEST_AWG_CONF and the sidecar validates it. + AWG_CONF: ${AWG_CONF:-} networks: internal: aliases: [telegram] @@ -255,6 +263,7 @@ services: bot: container_name: scrabble-telegram-bot image: scrabble-telegram-bot:latest + profiles: ["telegram-local"] build: context: .. dockerfile: platform/telegram/Dockerfile @@ -444,6 +453,26 @@ services: memory: 128M networks: [internal] + # node_exporter exports host CPU/memory/disk metrics. The prod main host runs a tight + # 1.9 GiB budget, so host memory pressure — not just per-container docker_stats — is + # what warns before an OOM. Prometheus scrapes it at :9100 (see prometheus.yml). + node_exporter: + container_name: scrabble-node-exporter + image: quay.io/prometheus/node-exporter:v1.8.2 + restart: unless-stopped + logging: *default-logging + command: + - --path.rootfs=/host + - --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host)($|/) + pid: host + volumes: + - /:/host:ro,rslave + deploy: + resources: + limits: + memory: 64M + networks: [internal] + networks: internal: name: scrabble-internal diff --git a/deploy/prometheus/prometheus.yml b/deploy/prometheus/prometheus.yml index 71d6cd8..6dc3d43 100644 --- a/deploy/prometheus/prometheus.yml +++ b/deploy/prometheus/prometheus.yml @@ -18,3 +18,8 @@ scrape_configs: - job_name: postgres_exporter static_configs: - targets: ["postgres_exporter:9187"] + # Host-level metrics (memory/CPU/disk). Matters most on the prod main host's tight + # 1.9 GiB budget, where total host memory is the OOM-proximity signal. + - job_name: node + static_configs: + - targets: ["node_exporter:9100"] -- 2.52.0 From 171b71b7e0e23016bf5359690acdc7b6c8bafb9d Mon Sep 17 00:00:00 2001 From: Ilia Denisov Date: Mon, 22 Jun 2026 00:25:09 +0200 Subject: [PATCH 3/5] feat(deploy): manual prod-deploy pipeline with rolling rollback (Stage 18) A workflow_dispatch-only rollout from master (confirm=deploy): - .gitea/workflows/prod-deploy.yaml builds + pushes the images to the registry, ships the compose/config/certs/env over SSH, deploys the main host via prod-deploy.sh, then the bot host, then verifies the public site. - deploy/prod-deploy.sh rolls the main stack one service at a time in dependency order (postgres->backend->gateway->landing->validator->caddy), health-checking after each; any failure rolls the whole stack back to the previous tag. A schema migration adds a maintenance window: the backend (sole writer) is stopped for a consistent pg_dump before migrating; image rollback stays DB-safe (expand-contract), the dump is kept for a manual restore. - prod overlay: pull the four main images from the registry by tag. - Runtime secrets reach the host via a sourced env.sh (single-quoted values keep the bcrypt hash's literal $ intact, unlike a --env-file). --- .gitea/workflows/prod-deploy.yaml | 219 ++++++++++++++++++++++++++++++ deploy/docker-compose.prod.yml | 6 + deploy/prod-deploy.sh | 133 ++++++++++++++++++ 3 files changed, 358 insertions(+) create mode 100644 .gitea/workflows/prod-deploy.yaml create mode 100755 deploy/prod-deploy.sh diff --git a/.gitea/workflows/prod-deploy.yaml b/.gitea/workflows/prod-deploy.yaml new file mode 100644 index 0000000..8ebd502 --- /dev/null +++ b/.gitea/workflows/prod-deploy.yaml @@ -0,0 +1,219 @@ +# Manual production rollout. Runs ONLY from master, ONLY on an explicit +# workflow_dispatch with confirm=deploy (development->master is merged + green first; +# this is the separate, deliberate prod step). It builds and pushes the images to the +# registry, then deploys over SSH: the main host via deploy/prod-deploy.sh (rolling, +# health-gated, auto-rollback; a maintenance window + consistent dump on a migration), +# then the bot host. See deploy/README.md (prod runbook) for operator steps. +name: prod-deploy +run-name: "prod deploy ${{ github.sha }}" + +on: + workflow_dispatch: + inputs: + confirm: + description: 'Type "deploy" to confirm a production rollout from master.' + required: true + default: "" + +permissions: + contents: read + +jobs: + deploy: + if: ${{ github.ref == 'refs/heads/master' && inputs.confirm == 'deploy' }} + runs-on: ubuntu-latest + defaults: + run: + shell: bash + env: + NO_COLOR: "1" + DOCKER_CLI_HINTS: "false" + REGISTRY: docker.iliadenisov.ru/developer + # SSH + registry + PROD_REGISTRY_USER: ${{ vars.PROD_REGISTRY_USER }} + PROD_REGISTRY_PASSWORD: ${{ secrets.PROD_REGISTRY_PASSWORD }} + PROD_SSH_KEY: ${{ secrets.PROD_SSH_KEY }} + PROD_SSH_KNOWN_HOSTS: ${{ secrets.PROD_SSH_KNOWN_HOSTS }} + MAIN_HOST: ${{ vars.PROD_MAIN_HOST }} + TG_HOST: ${{ vars.PROD_TG_HOST }} + # SPA build args (baked into the gateway/landing images) + VITE_TELEGRAM_BOT_ID: ${{ vars.PROD_VITE_TELEGRAM_BOT_ID }} + VITE_TELEGRAM_LINK: ${{ vars.PROD_VITE_TELEGRAM_LINK }} + VITE_TELEGRAM_GAME_CHANNEL_NAME: ${{ vars.PROD_VITE_TELEGRAM_GAME_CHANNEL_NAME }} + VITE_GATEWAY_URL: ${{ vars.PROD_VITE_GATEWAY_URL }} + # Runtime secrets + POSTGRES_PASSWORD: ${{ secrets.PROD_POSTGRES_PASSWORD }} + GM_BASICAUTH_HASH: ${{ secrets.PROD_GM_BASICAUTH_HASH }} + GRAFANA_ADMIN_PASSWORD: ${{ secrets.PROD_GRAFANA_ADMIN_PASSWORD }} + TELEGRAM_BOT_TOKEN: ${{ secrets.PROD_TELEGRAM_BOT_TOKEN }} + TELEGRAM_PROMO_BOT_TOKEN: ${{ secrets.PROD_TELEGRAM_PROMO_BOT_TOKEN }} + PROD_BOTLINK_CA: ${{ secrets.PROD_BOTLINK_CA }} + PROD_BOTLINK_GATEWAY_CERT: ${{ secrets.PROD_BOTLINK_GATEWAY_CERT }} + PROD_BOTLINK_GATEWAY_KEY: ${{ secrets.PROD_BOTLINK_GATEWAY_KEY }} + PROD_BOTLINK_BOT_CERT: ${{ secrets.PROD_BOTLINK_BOT_CERT }} + PROD_BOTLINK_BOT_KEY: ${{ secrets.PROD_BOTLINK_BOT_KEY }} + # Runtime variables + GM_BASICAUTH_USER: ${{ vars.PROD_GM_BASICAUTH_USER }} + GRAFANA_ROOT_URL: ${{ vars.PROD_GRAFANA_ROOT_URL }} + CADDY_SITE_ADDRESS: ${{ vars.PROD_CADDY_SITE_ADDRESS }} + LOG_LEVEL: ${{ vars.PROD_LOG_LEVEL }} + DICT_VERSION: ${{ vars.PROD_DICT_VERSION }} + POSTGRES_DB: ${{ vars.PROD_POSTGRES_DB }} + POSTGRES_USER: ${{ vars.PROD_POSTGRES_USER }} + TELEGRAM_MINIAPP_URL: ${{ vars.PROD_TELEGRAM_MINIAPP_URL }} + TELEGRAM_GAME_CHANNEL_ID: ${{ vars.PROD_TELEGRAM_GAME_CHANNEL_ID }} + TELEGRAM_CHAT_ID: ${{ vars.PROD_TELEGRAM_CHAT_ID }} + TELEGRAM_BOT_USERNAME: ${{ vars.PROD_TELEGRAM_BOT_USERNAME }} + TELEGRAM_BOT_LINK: ${{ vars.PROD_VITE_TELEGRAM_LINK }} + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Compute image tag and app version + run: | + echo "TAG=$(git rev-parse --short=12 HEAD)" >> "$GITHUB_ENV" + echo "APP_VERSION=$(git describe --tags --always)" >> "$GITHUB_ENV" + + - name: Registry login + run: echo "$PROD_REGISTRY_PASSWORD" | docker login "${REGISTRY%%/*}" -u "$PROD_REGISTRY_USER" --password-stdin + + - name: Build and push images + working-directory: deploy + run: | + export TAG SCRABBLE_CONFIG_DIR=. + # The four main-stack images via compose (reuses the compose build args); the + # bot separately, since it is profiled out of the prod compose. + docker compose -f docker-compose.yml -f docker-compose.prod.yml build + docker compose -f docker-compose.yml -f docker-compose.prod.yml push backend gateway landing validator + docker build -f ../platform/telegram/Dockerfile --target bot -t "$REGISTRY/scrabble-telegram-bot:$TAG" .. + docker push "$REGISTRY/scrabble-telegram-bot:$TAG" + + - name: Set up SSH + run: | + mkdir -p ~/.ssh && chmod 700 ~/.ssh + printf '%s\n' "$PROD_SSH_KEY" > ~/.ssh/id_deploy + chmod 600 ~/.ssh/id_deploy + printf '%s\n' "$PROD_SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts + + - name: Determine previous tag and migration + run: | + ssh_main() { ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$MAIN_HOST" "$@"; } + PREV_TAG="$(ssh_main 'cat /opt/scrabble/DEPLOYED_TAG 2>/dev/null || echo none')" + MIGRATION=0 + if [ "$PREV_TAG" != none ]; then + if ! git cat-file -e "$PREV_TAG^{commit}" 2>/dev/null; then + MIGRATION=1 # unknown previous tag: take the safe window + dump + elif git diff --name-only "$PREV_TAG..$TAG" -- backend/internal/postgres/migrations/ | grep -q .; then + MIGRATION=1 + fi + fi + echo "PREV_TAG=$PREV_TAG" >> "$GITHUB_ENV" + echo "MIGRATION=$MIGRATION" >> "$GITHUB_ENV" + echo "prev=$PREV_TAG migration=$MIGRATION" + + - name: Render env files and certs + run: | + umask 077 + mkdir -p stage/certs-main stage/certs-bot + # Main-host runtime env (single-quoted so the literal '$' in the bcrypt hash + # survives; prod-deploy.sh sources this, compose reads it from the environment). + cat > stage/env.sh < stage/env.bot.sh < stage/certs-main/ca.crt + printf '%s\n' "$PROD_BOTLINK_GATEWAY_CERT" > stage/certs-main/gateway.crt + printf '%s\n' "$PROD_BOTLINK_GATEWAY_KEY" > stage/certs-main/gateway.key + printf '%s\n' "$PROD_BOTLINK_CA" > stage/certs-bot/ca.crt + printf '%s\n' "$PROD_BOTLINK_BOT_CERT" > stage/certs-bot/bot.crt + printf '%s\n' "$PROD_BOTLINK_BOT_KEY" > stage/certs-bot/bot.key + chmod 644 stage/certs-main/* stage/certs-bot/* + + - name: Deploy the main host + run: | + ssh_main() { ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$MAIN_HOST" "$@"; } + ssh_main 'mkdir -p /opt/scrabble/compose' + # Compose files + config + script. + tar -C deploy -czf - docker-compose.yml docker-compose.prod.yml prod-deploy.sh \ + | ssh_main 'tar -C /opt/scrabble/compose -xzf -' + tar -C deploy -czf - caddy otelcol prometheus tempo grafana \ + | ssh_main 'tar -C /opt/scrabble -xzf -' + tar -C stage -czf - certs-main \ + | ssh_main 'rm -rf /opt/scrabble/certs && mkdir -p /opt/scrabble/certs && tar -C /opt/scrabble/certs --strip-components=1 -xzf -' + scp -i ~/.ssh/id_deploy -o BatchMode=yes stage/env.sh "deploy@$MAIN_HOST:/opt/scrabble/env.sh" + # Registry login on the host so compose can pull the private images. + echo "$PROD_REGISTRY_PASSWORD" | ssh_main "docker login ${REGISTRY%%/*} -u $PROD_REGISTRY_USER --password-stdin" + ssh_main "TAG='$TAG' PREV_TAG='$PREV_TAG' MIGRATION='$MIGRATION' bash /opt/scrabble/compose/prod-deploy.sh" + + - name: Deploy the bot host + run: | + ssh_tg() { ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$TG_HOST" "$@"; } + ssh_tg 'mkdir -p /opt/scrabble/compose' + tar -C deploy -czf - docker-compose.bot.yml | ssh_tg 'tar -C /opt/scrabble/compose -xzf -' + tar -C stage -czf - certs-bot \ + | ssh_tg 'rm -rf /opt/scrabble/certs && mkdir -p /opt/scrabble/certs && tar -C /opt/scrabble/certs --strip-components=1 -xzf -' + scp -i ~/.ssh/id_deploy -o BatchMode=yes stage/env.bot.sh "deploy@$TG_HOST:/opt/scrabble/env.bot.sh" + echo "$PROD_REGISTRY_PASSWORD" | ssh_tg "docker login ${REGISTRY%%/*} -u $PROD_REGISTRY_USER --password-stdin" + ssh_tg 'set -a; . /opt/scrabble/env.bot.sh; set +a; cd /opt/scrabble/compose; + docker compose -f docker-compose.bot.yml pull; + docker compose -f docker-compose.bot.yml up -d' + # Bot liveness: running, not restarting, stable restart count. + ssh_tg 'for i in $(seq 1 20); do + s=$(docker inspect -f "{{.State.Status}}" scrabble-telegram-bot 2>/dev/null || echo missing) + r=$(docker inspect -f "{{.State.Restarting}}" scrabble-telegram-bot 2>/dev/null || echo true) + if [ "$s" = running ] && [ "$r" = false ]; then + c1=$(docker inspect -f "{{.RestartCount}}" scrabble-telegram-bot); sleep 5 + c2=$(docker inspect -f "{{.RestartCount}}" scrabble-telegram-bot) + [ "$c1" = "$c2" ] && { echo "bot healthy"; exit 0; } + fi + sleep 3 + done + echo "bot not healthy:"; docker logs --tail 80 scrabble-telegram-bot; exit 1' + + - name: Verify the public site + run: | + ssh_main() { ssh -i ~/.ssh/id_deploy -o BatchMode=yes "deploy@$MAIN_HOST" "$@"; } + domain="${CADDY_SITE_ADDRESS%% *}" # first domain of "erudit-game.ru www..." + # Probe through caddy with the right vhost (force-resolved to the host); -k + # tolerates an ACME-pending cert. Also check the backend is actually ready. + ssh_main "for i in \$(seq 1 20); do + if curl -fsS -k --resolve $domain:443:127.0.0.1 https://$domain/ -o /dev/null && + curl -fsS -k --resolve $domain:443:127.0.0.1 https://$domain/app/ -o /dev/null && + docker run --rm --network scrabble-internal alpine:3.20 wget -q -T 5 -O /dev/null http://backend:8080/readyz; then + echo 'public site + /app/ + backend healthy'; exit 0 + fi + sleep 5 + done + echo 'public verify failed; recent caddy + gateway + backend logs:' + docker logs --tail 40 scrabble-caddy; docker logs --tail 40 scrabble-gateway; docker logs --tail 40 scrabble-backend + exit 1" diff --git a/deploy/docker-compose.prod.yml b/deploy/docker-compose.prod.yml index 6b45826..03e7be7 100644 --- a/deploy/docker-compose.prod.yml +++ b/deploy/docker-compose.prod.yml @@ -23,6 +23,9 @@ services: memory: 96M gateway: + # Prod pulls the pushed image by tag instead of building locally; the base + # build: section stays dormant because the deploy always pulls first. + image: ${REGISTRY:?set REGISTRY}/scrabble-gateway:${TAG:?set TAG} ports: - "9443:9443" environment: @@ -35,6 +38,7 @@ services: memory: 384M backend: + image: ${REGISTRY:?set REGISTRY}/scrabble-backend:${TAG:?set TAG} deploy: resources: limits: @@ -47,12 +51,14 @@ services: memory: 384M validator: + image: ${REGISTRY:?set REGISTRY}/scrabble-telegram-validator:${TAG:?set TAG} deploy: resources: limits: memory: 96M landing: + image: ${REGISTRY:?set REGISTRY}/scrabble-landing:${TAG:?set TAG} deploy: resources: limits: diff --git a/deploy/prod-deploy.sh b/deploy/prod-deploy.sh new file mode 100755 index 0000000..aec789b --- /dev/null +++ b/deploy/prod-deploy.sh @@ -0,0 +1,133 @@ +#!/usr/bin/env bash +# Production main-host deploy driver. Runs ON the main host, invoked over SSH by +# .gitea/workflows/prod-deploy.yaml as the deploy user (which must already be +# `docker login`ed to the registry). It pulls the images at the new tag and rolls +# the stack ONE service at a time in dependency order (least -> most dependent), +# health-checking after each; any failure rolls the whole stack back to the +# previously deployed tag. +# +# A schema migration adds a maintenance window: the backend (the only writer) is +# stopped so a consistent pg_dump is taken before the new backend migrates forward. +# Image rollback alone is safe under the expand-contract migration rule, so the +# automatic rollback never touches the database; the dump is kept for a MANUAL +# restore if a migration turned out to be destructive (see deploy/prod/README.md). +# +# Required env (exported by the workflow over SSH): +# REGISTRY registry namespace, e.g. docker.iliadenisov.ru/developer +# TAG new image tag (the deployed git SHA) +# PREV_TAG previously deployed tag, or "none" on the first deploy +# MIGRATION "1" when the deploy carries a schema migration, else "0" +# Optional: COMPOSE_DIR ENV_FILE DUMP_DIR STATE_FILE POSTGRES_USER POSTGRES_DB +set -uo pipefail + +# Runtime compose vars (POSTGRES_*, GM_*, GRAFANA_*, CADDY_*, TELEGRAM_*, REGISTRY, +# SCRABBLE_CONFIG_DIR, ...) come from a shell-sourceable env file the workflow writes +# with single-quoted values. Exporting them into the process environment lets compose +# interpolate ${...} without re-parsing the value — a plain --env-file would mangle the +# literal '$' in the bcrypt GM_BASICAUTH_HASH. +ENV_FILE="${ENV_FILE:-/opt/scrabble/env.sh}" +# shellcheck disable=SC1090 +[ -f "$ENV_FILE" ] && . "$ENV_FILE" + +REGISTRY="${REGISTRY:?REGISTRY required (env.sh)}" +TAG="${TAG:?TAG required}" +PREV_TAG="${PREV_TAG:-none}" +MIGRATION="${MIGRATION:-0}" +COMPOSE_DIR="${COMPOSE_DIR:-/opt/scrabble/compose}" +DUMP_DIR="${DUMP_DIR:-/opt/scrabble/dumps}" +STATE_FILE="${STATE_FILE:-/opt/scrabble/DEPLOYED_TAG}" +PG_USER="${POSTGRES_USER:-scrabble}" +PG_DB="${POSTGRES_DB:-scrabble}" + +cd "$COMPOSE_DIR" +export REGISTRY +# otelcol joins the host docker group to read the socket; the GID varies per host. +export DOCKER_GID="$(getent group docker | cut -d: -f3)" + +dc() { docker compose -f docker-compose.yml -f docker-compose.prod.yml "$@"; } +use_tag() { export TAG="$1"; } + +# --- health probes (one-off containers on the contour networks, like CI) -------- +_probe() { docker run --rm --network "$1" alpine:3.20 wget -q -T 5 -O /dev/null "$2"; } +health_backend() { for _ in $(seq 1 20); do _probe scrabble-internal http://backend:8080/readyz && return 0; sleep 3; done; return 1; } +health_landing() { for _ in $(seq 1 20); do _probe scrabble-internal http://landing:80/ && return 0; sleep 3; done; return 1; } +health_postgres() { for _ in $(seq 1 30); do [ "$(docker inspect -f '{{.State.Health.Status}}' scrabble-postgres 2>/dev/null)" = healthy ] && return 0; sleep 2; done; return 1; } +health_running() { # health_running : running, not restarting, stable restart count + local n="$1" s r c1 c2 + for _ in $(seq 1 20); do + s="$(docker inspect -f '{{.State.Status}}' "$n" 2>/dev/null || echo missing)" + r="$(docker inspect -f '{{.State.Restarting}}' "$n" 2>/dev/null || echo true)" + if [ "$s" = running ] && [ "$r" = false ]; then + c1="$(docker inspect -f '{{.RestartCount}}' "$n")"; sleep 5 + c2="$(docker inspect -f '{{.RestartCount}}' "$n")" + [ "$c1" = "$c2" ] && return 0 + fi + sleep 3 + done + return 1 +} + +roll() { # roll + local svc="$1"; shift + echo ">>> rolling $svc -> $TAG" + dc up -d --no-build --no-deps "$svc" || return 1 + "$@" || { echo "!!! $svc failed health check"; return 1; } + echo "<<< $svc healthy" +} + +rollback() { + echo "########## ROLLBACK -> $PREV_TAG ##########" + if [ "$PREV_TAG" = none ]; then + echo "no previous tag (first deploy): cannot roll back; leaving the stack up for inspection." + return + fi + use_tag "$PREV_TAG" + dc up -d --no-build --remove-orphans + echo "rolled back to $PREV_TAG." + [ "$MIGRATION" = 1 ] && echo "NOTE: the DB is forward-migrated; a pre-deploy dump is in $DUMP_DIR — restore manually ONLY if the migration was destructive (see deploy/README.md, prod runbook)." +} + +mkdir -p "$DUMP_DIR" +echo "=== prod deploy: tag=$TAG prev=$PREV_TAG migration=$MIGRATION ===" +use_tag "$TAG" +dc pull + +# First deploy: nothing to roll from; bring the whole stack up and gate on health. +if [ -z "$(docker ps -aq -f name=scrabble-backend)" ]; then + echo "first deploy: bringing the whole stack up" + dc up -d --no-build --remove-orphans || { echo "compose up failed"; exit 1; } + health_backend || { echo "backend not ready"; exit 1; } + health_landing || { echo "landing not ready"; exit 1; } + echo "$TAG" > "$STATE_FILE" + echo "first deploy healthy ($TAG)." + exit 0 +fi + +# Migration deploy: freeze writes and snapshot a consistent dump before migrating. +if [ "$MIGRATION" = 1 ]; then + echo "migration deploy: opening maintenance window (stopping the backend = the only writer)" + dc stop backend + dump="$DUMP_DIR/pre-$TAG-$(date +%Y%m%d-%H%M%S).sql" + if ! docker exec scrabble-postgres pg_dump -U "$PG_USER" -d "$PG_DB" -n backend > "$dump"; then + echo "pg_dump failed; restarting the old backend and aborting" + dc start backend + exit 1 + fi + echo "consistent dump: $dump" +fi + +# Roll one service at a time, least -> most dependent; any failure rolls everything back. +roll postgres health_postgres || { rollback; exit 1; } +roll backend health_backend || { rollback; exit 1; } +roll gateway health_running scrabble-gateway || { rollback; exit 1; } +roll landing health_landing || { rollback; exit 1; } +roll validator health_running scrabble-telegram-validator || { rollback; exit 1; } +roll caddy health_running scrabble-caddy || { rollback; exit 1; } + +# Observability + node_exporter: bring up the remainder and pick up any config changes. +dc up -d --no-build --remove-orphans || { rollback; exit 1; } + +# Final internal sanity before committing the new tag. +health_backend || { rollback; exit 1; } +echo "$TAG" > "$STATE_FILE" +echo "=== deploy healthy ($TAG) ===" -- 2.52.0 From 3d9ba3ac3d5d588fdb1445b26c7a3239bed5ccb6 Mon Sep 17 00:00:00 2001 From: Ilia Denisov Date: Mon, 22 Jun 2026 00:30:30 +0200 Subject: [PATCH 4/5] docs(deploy): bake Stage 18 prod-deploy decisions into the live docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - ARCHITECTURE §13 prod bullet -> the realized mechanism: registry transport, two-host, rolling + auto-rollback, migration maintenance window, node_exporter, the undersized launch; the contour paragraph notes node_exporter + the telegram-local profile. - deploy/README gains a prod rollout runbook (how to run, migrations/restore, cert rotation, sizing/monitoring, the full PROD_ set) + node_exporter row, the telegram-local profile note, and the soft AWG_CONF note. - PLAN Stage 18 records the resolved open details and the remaining live cutover (pending erudit-game.ru DNS); the tracker reads 'machinery built; cutover pending DNS'. - PRERELEASE TX/AG note the prod wiring is built. --- PLAN.md | 36 ++++++++++++++++++----------- PRERELEASE.md | 4 ++-- deploy/README.md | 54 +++++++++++++++++++++++++++++++++++++++++--- docs/ARCHITECTURE.md | 40 ++++++++++++++++++++++---------- 4 files changed, 104 insertions(+), 30 deletions(-) diff --git a/PLAN.md b/PLAN.md index 10836dd..81955d1 100644 --- a/PLAN.md +++ b/PLAN.md @@ -51,7 +51,7 @@ independent (see ARCHITECTURE §9.1). | 15 | Dual Telegram bots & language-gated variants | **done** | | 16 | Deploy infra & test contour (Dockerfiles, gateway static UI, compose, observability) | **done** | | 17 | Test-contour verification & defect fixes | **done** | -| 18 | Prod contour deploy (SSH export/import, manual after merge) | todo | +| 18 | Prod contour deploy (registry, two-host, rolling + auto-rollback; manual after merge) | machinery built; first cutover pending DNS | | 19 | User feedback (in-app submit + attachment, admin review/reply, account roles) | **done** | Scaffolding is incremental: `go.work` lists only existing modules; each stage @@ -413,18 +413,28 @@ raw list is kept here as the record of what the first contour run surfaced. "что-то пошло не так". при этом "new -> эрудит" работает. Попробуй посмотреть в логах сейчас, может что-то есть. Или как-то иначе проанализируй, или давай вместе будем смотреть, если не получится. ### Stage 18 — Prod contour deploy -Scope: the **production contour** on a remote host over SSH. Deploy by **container export/import** -(`docker save` → `scp`/ssh → `docker load` → `docker compose up` on the remote), the SSH key + host IP -in Gitea secrets; **strictly manual** (`workflow_dispatch`) after `development` is merged to `master` -(the Stage 16 branch model: `feature/* → development → master`, merge gated green). Two-contour config -uses **`TEST_`/`PROD_` secret/variable prefixes** — Gitea 1.26 has no deployment environments (verified: -the `environments` API 404s), so a flat prefixed namespace is the convention. -Reuses the Stage 16 `deploy/docker-compose.yml` as-is, mapping the **`PROD_`** set onto the same -unprefixed compose vars. **No host caddy on prod**, so the contour's own caddy terminates TLS — set -`CADDY_SITE_ADDRESS` to the prod domain so caddy does its own ACME (the Caddyfile is already -parameterised for this; the test contour leaves it `:80` behind the host caddy). -Open details (re-interview): export/import vs a registry trade-off; prod domain/cert source (ACME vs a -provided cert) at the contour caddy; prod VPN; rollback. +Scope: the **production contour** on **two remote hosts** over SSH — main (full stack, `erudit-game.ru`) +and tg (the bot only). Resolved open details (re-interviewed): +- **Transport: a registry** (not export/import) — build + push to `docker.iliadenisov.ru`, the hosts pull by tag. +- **Cert: ACME** at the contour caddy (`CADDY_SITE_ADDRESS=erudit-game.ru www.erudit-game.ru`, no host caddy). +- **No prod VPN** — the bot host has native Bot API egress (verified `api.telegram.org` → 200). +- **Rollback** — rolling per-service deploy (least → most dependent), health-gated, auto-rollback to the + previous image tag; a maintenance window + consistent `pg_dump` only on a schema migration + (expand-contract keeps the auto-rollback image-only; the dump is a manual safety net). + +**Strictly manual** (`workflow_dispatch` from `master`, `confirm=deploy`) after `development → master` +is merged green. `TEST_`/`PROD_` prefixed Gitea secrets/variables (Gitea 1.26 has no deployment +environments — the `environments` API 404s). Hosts are provisioned by **`deploy/ansible/`** (docker, a +non-sudo `deploy` user with the CI key, key-only sshd, ufw, fail2ban). The main host is **launch-sized** +(2 vCPU / 1.9 GiB): `docker-compose.prod.yml` trims the R7 limits (`GOMAXPROCS=2`, smaller caps, 7d +Prometheus retention) and adds `node_exporter` for host-memory monitoring (launch undersized, resize at +Selectel reactively). `vpn`+`bot` are gated to a `telegram-local` compose profile (test only); the prod +bot runs standalone from `docker-compose.bot.yml`. `GATEWAY_ABUSE_BAN_ENABLED=true`. + +**Built:** `deploy/ansible/` (both hosts provisioned + verified), the compose split + `node_exporter`, +`.gitea/workflows/prod-deploy.yaml` + `deploy/prod-deploy.sh`, the full `PROD_` secret/variable set. +**Remaining (acceptance):** the **first live cutover** — waits on the `erudit-game.ru` DNS delegation +(`A`/`www` → the main host) that ACME requires; then run the workflow and verify the public site end-to-end. ### Stage 19 — User feedback *(done)* A user→operator feedback channel, sequenced after the numbered stages but shipped **before** the Stage 18 diff --git a/PRERELEASE.md b/PRERELEASE.md index 6c1d91b..32417ee 100644 --- a/PRERELEASE.md +++ b/PRERELEASE.md @@ -39,8 +39,8 @@ the edge before prod. Each phase maps back to the owner's raw pre-release TODO l | FM | First-move tile draw (official rules): each seated player draws a tile, the one closest to "A" leads (a blank beats every letter), ties re-drawing until a single leader; **honest per-draw `crypto/rand` entropy**, not the bag seed, so the **record** (`game_setup_draws`, migration `00013`) — not a seed — is the only account of the outcome, kept for future **tournaments** (designed as a discrete per-tile "player N draws" step). Friend/AI draws at create; **auto-match draws at *open*** against a synthetic `uuid.Nil` opponent whose draw rows are back-filled on join, so the opener's seat is fixed up front and the existing open-game pre-move is preserved (no reseating, no play-gating). Admin `/_gm/games/:id` gains the recorded draw list + a simple **step-by-step board replay** (`ReplayTimeline`). | owner ad-hoc | **done** | | SB | Single Telegram bot + per-user variant preferences: the two per-language bots collapse into **one** (drop `accounts.service_language`, `supported_languages`, the `*_EN`/`*_RU` env vars and game-language push routing — the single bot renders in the recipient's `preferred_language`); New Game variant gating moves to a profile **`variant_preferences`** set (default Erudit only, Erudit-first, server-enforced on the caller's auto-match/vs-AI/invitation-create paths, an invited friend may accept any variant); env vars collapse to unsuffixed `TELEGRAM_BOT_TOKEN`/`TELEGRAM_GAME_CHANNEL_ID`/`VITE_TELEGRAM_LINK`/`VITE_TELEGRAM_GAME_CHANNEL_NAME` and `GATEWAY_DEFAULT_SUPPORTED_LANGUAGES` is removed; wire drops `service_language`/`supported_languages` (Session, ValidateInitDataResponse) + the push `language` routing field and adds `variant_preferences` to Profile/UpdateProfile. | owner ad-hoc | **done** | | DV | Dictionary version hygiene: CI + image/compose seed track the current release (`v1.2.1`); a **seed-drift guard** records the flat dir's seed in an authoritative `.seed_version` marker so a bumped build seed on a live volume is ignored (it can't relabel live bytes — which would mis-serve the dictionary + void games pinned to the prior label); `DICT_VERSION` is the fresh-volume seed only, a live contour migrates through the admin console | owner ad-hoc | **done** | -| TX | Telegram egress off the main host: split the connector into a home **validator** (Mini App / Login-Widget HMAC, no VPN, no Bot API — so game login no longer depends on Telegram being reachable) and a remote **bot** (Bot API long-poll + `sendMessage`) that holds **no inbound port** and dials the gateway over a reverse **mTLS bot-link** (`pkg/proto/botlink/v1`); the gateway funnels out-of-app push (fire-and-forget, at-most-once) and the backend admin broadcasts (a relay that awaits the bot's ack) down the link. The bot is Telegram-rate-limited; **one bot now**, with seams (a bot registry + `owns_updates` + command ids) for N later; **no webhook** (rejected: one URL per token, adds inbound + a static address). The **unified test contour** runs the split (the bot keeps its VPN sidecar and dials the gateway by its internal name; certs from `deploy/gen-certs.sh`). The **prod** wiring — the bot on a separate host (no VPN), the gateway bot-link port published, `PROD_` certs with scheduled rotation, an SSH deploy of both hosts together — is the **deferred final stage** (Stage 18). | owner ad-hoc | **done** (code + test contour; prod wiring → Stage 18) | -| AG | Anti-abuse IP ban + honeypot/honeytoken (prod-only): a fail2ban-style in-memory `ratelimit.Banlist` keyed by client IP, fed by sustained rate-limiter rejections (the IP-keyed public/email/admin classes — the user class stays the soft-flag's concern), a **honeypot** decoy path (the contour caddy tags `/.env`, `/.git`, `/wp-*`, … with `X-Scrabble-Honeypot` and routes them to the gateway), and a **honeytoken** (`GATEWAY_HONEYTOKEN`, a planted bearer). The `abuseGuard` edge middleware refuses a banned IP with **429** before any work — closing the R3 gap that the static SPA/landing was outside the token bucket. Off by default — it keys by the real client IP the shared-NAT test contour does not expose (detection still logs there); enabled in prod via `GATEWAY_ABUSE_BAN_ENABLED`. Operators see + lift bans on the console **Throttled** page; the gateway syncs its active set to the backend (`/api/v1/internal/bans/sync`, `internal/banview`) every 30 s and applies operator unbans. | owner ad-hoc | **done** (code + test contour; ban enabled in prod → Stage 18) | +| TX | Telegram egress off the main host: split the connector into a home **validator** (Mini App / Login-Widget HMAC, no VPN, no Bot API — so game login no longer depends on Telegram being reachable) and a remote **bot** (Bot API long-poll + `sendMessage`) that holds **no inbound port** and dials the gateway over a reverse **mTLS bot-link** (`pkg/proto/botlink/v1`); the gateway funnels out-of-app push (fire-and-forget, at-most-once) and the backend admin broadcasts (a relay that awaits the bot's ack) down the link. The bot is Telegram-rate-limited; **one bot now**, with seams (a bot registry + `owns_updates` + command ids) for N later; **no webhook** (rejected: one URL per token, adds inbound + a static address). The **unified test contour** runs the split (the bot keeps its VPN sidecar and dials the gateway by its internal name; certs from `deploy/gen-certs.sh`). The **prod** wiring — the bot on a separate host (no VPN), the gateway bot-link port published, `PROD_` certs, an SSH deploy of both hosts together — is **built in Stage 18** (the two-host registry rollout; first cutover pending the `erudit-game.ru` DNS). | owner ad-hoc | **done** (code + test contour; prod wiring built — Stage 18) | +| AG | Anti-abuse IP ban + honeypot/honeytoken (prod-only): a fail2ban-style in-memory `ratelimit.Banlist` keyed by client IP, fed by sustained rate-limiter rejections (the IP-keyed public/email/admin classes — the user class stays the soft-flag's concern), a **honeypot** decoy path (the contour caddy tags `/.env`, `/.git`, `/wp-*`, … with `X-Scrabble-Honeypot` and routes them to the gateway), and a **honeytoken** (`GATEWAY_HONEYTOKEN`, a planted bearer). The `abuseGuard` edge middleware refuses a banned IP with **429** before any work — closing the R3 gap that the static SPA/landing was outside the token bucket. Off by default — it keys by the real client IP the shared-NAT test contour does not expose (detection still logs there); enabled in prod via `GATEWAY_ABUSE_BAN_ENABLED`. Operators see + lift bans on the console **Throttled** page; the gateway syncs its active set to the backend (`/api/v1/internal/bans/sync`, `internal/banview`) every 30 s and applies operator unbans. | owner ad-hoc | **done** (code + test contour; ban on in prod via Stage 18 — machinery built, cutover pending DNS) | | CM | Channel-chat moderation + promo bot: a second standalone bot in the bot container answers `/start` with a localized message + a **URL** button into the **main** bot's Mini App (`?startapp`; a `web_app` button would sign initData with the promo token, which the main validator rejects). The **main** bot gates write access in a channel's linked discussion chat. The chat **allows sending by default** and the bot only restricts (Telegram intersects the chat default with the per-user permission, so a per-user grant cannot exceed a deny-by-default group): it **mutes** a member who is not registered or is admin-suspended or holding a new **`chat_muted`** role, and **un-mutes** an eligible one it had muted, for a member currently in the chat (a `getChatMember` guard, since bots cannot list members). Eligibility = `registered AND NOT suspended AND NOT chat_muted` (the game suspension dominates), resolved once in the backend and reached two ways: the bot's `ResolveChatEligibility` on a `chat_member` event over the existing mTLS bot-link, and a backend `chat_access_changed` event → gateway → `ChatGate` command (emitted on block/unblock, a `chat_muted` change, a first registration, or a temporary-block expiry via a sweeper; idempotent). No schema change — `chat_muted` reuses `account_roles`. | owner ad-hoc | **done** | | → | Stage 18 — prod contour deploy | — | see [`PLAN.md`](PLAN.md) | diff --git a/deploy/README.md b/deploy/README.md index 2386f41..c2d8d94 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -17,11 +17,12 @@ operational reference for **every environment variable**. | `backend` | built (`backend/Dockerfile`) | Domain service; bakes in the DAWG dictionaries; runs migrations at boot. | | `postgres` | `postgres:17-alpine` | Database (named volume, `pg_isready` healthcheck). | | `validator` | built (`platform/telegram/Dockerfile`, target `validator`) | Telegram HMAC validator (no VPN, no Bot API); internal gRPC at `validator:9091`. Game login depends only on this. | -| `vpn` + `bot` | sidecar + built (`platform/telegram/Dockerfile`, target `bot`) | Telegram bot; egresses through the AmneziaWG sidecar; holds no inbound port — dials the gateway bot-link (mTLS) at `gateway:9443`. | +| `vpn` + `bot` | sidecar + built (`platform/telegram/Dockerfile`, target `bot`) | Telegram bot, gated to the **`telegram-local`** profile; egresses through the AmneziaWG sidecar and dials the gateway bot-link (mTLS) at `gateway:9443`. The test contour activates the profile; the prod **main** host omits it and runs the bot standalone on its **own host** (`docker-compose.bot.yml`, no VPN — native Bot API egress). | | `otelcol` | `otel/opentelemetry-collector-contrib` | OTLP/gRPC `:4317` → Prometheus scrape (`:9464`) + Tempo. | -| `prometheus` | `prom/prometheus` | Metrics, 15d retention. | +| `prometheus` | `prom/prometheus` | Metrics, 15d retention (7d in prod). | | `tempo` | `grafana/tempo` | Traces, 72h retention. | | `grafana` | `grafana/grafana` | Dashboards (provisioned), anonymous-admin behind caddy's `/_gm/grafana`. | +| `node_exporter` | `quay.io/prometheus/node-exporter` | Host CPU/memory/disk metrics (Prometheus job `node`); the OOM signal on the tight prod main host (2 vCPU / 1.9 GiB). | Networking: inter-service traffic is on the private `internal` network (project-scoped DNS); only `caddy` joins the shared external `edge` network so the @@ -59,7 +60,6 @@ compose binds from this directory. | Variable | Gitea kind | Purpose | | --- | --- | --- | | `POSTGRES_PASSWORD` | secret | Postgres password (also embedded in `BACKEND_POSTGRES_DSN`). | -| `AWG_CONF` | secret | AmneziaWG config for the VPN sidecar (the bot's only Telegram egress in the test contour). **Must not contain a `DNS=` line** — it hijacks the shared netns's resolv.conf and breaks the bot resolving `otelcol` / `gateway`. Without it, Docker's resolver handles `otelcol`, `gateway` and `api.telegram.org`. | | `GM_BASICAUTH_HASH` | secret | bcrypt hash gating `/_gm` (admin console + Grafana). Generate with `docker run --rm caddy:2-alpine caddy hash-password --plaintext ''`. | | `TELEGRAM_MINIAPP_URL` | variable | The Mini App URL the bot hands out in deep links / buttons. | @@ -67,6 +67,13 @@ compose binds from this directory. secret) and the bot (Bot API). It defaults to empty in compose, but both **fail at boot** when it is empty. +**Conditionally — `AWG_CONF`** (secret): the AmneziaWG config for the VPN sidecar, needed +only when the `telegram-local` profile runs (the test contour and local runs with the +bot). It is **not** `:?`-guarded — compose interpolates profiled-out services too, so the +prod main host (no VPN) must not require it. It **must not contain a `DNS=` line** — that +hijacks the shared netns's resolv.conf and breaks the bot resolving `otelcol` / `gateway`; +without it Docker's resolver handles `otelcol`, `gateway` and `api.telegram.org`. + ## Optional variables (with defaults) | Variable | Gitea kind | Default | Purpose | @@ -110,6 +117,47 @@ collector's / gateway's internal IP is fine (connected route), but its `AWG_CONF which resolves `otelcol`, `gateway` and `api.telegram.org`. `GATEWAY_ADMIN_*` is intentionally **unset** — caddy owns `/_gm` in the contour. +## Production rollout + +Prod runs on **two hosts** (main = full stack + ACME on the domain; tg = the bot only, +native Bot API, no VPN), one-time provisioned by **[`ansible/`](ansible/)** (docker, a +non-sudo `deploy` user holding the CI key, key-only sshd, default-deny ufw, fail2ban). +Re-run `ansible/` after a host resize — it is idempotent. + +**To roll out:** merge `development → master` (CI green), then run the **`prod-deploy`** +workflow manually (Gitea → Actions → prod-deploy → run from `master`, input +`confirm=deploy`). It builds + pushes the images to the registry, ships the +compose/config/certs/env over SSH, deploys the main host with `prod-deploy.sh` (rolling, +health-gated, **auto-rollback to the previous tag**), then the bot host, then probes the +public site. After `master` is green this workflow is the **only** thing that touches +prod — nothing auto-deploys there. + +**Migrations** must be **expand-contract** (backward-compatible; goose is forward-only): +the automatic rollback is image-only and never restores the DB. A deploy that changes +`backend/internal/postgres/migrations/` opens a maintenance window — the backend (sole +writer) is stopped for a consistent `pg_dump` into `/opt/scrabble/dumps` before the new +backend migrates. **Manual DB restore** (only if a migration was destructive): +`docker exec -i scrabble-postgres psql -U scrabble -d scrabble -c 'DROP SCHEMA backend CASCADE'`, +then pipe the dump into the same `psql`, and redeploy the matching old tag. + +**bot-link cert rotation:** regenerate (`deploy/gen-certs.sh /tmp/c --force`), reset the +five `PROD_BOTLINK_*` secrets from `/tmp/c`, and re-run the workflow — both hosts redeploy +together with the fresh CA. + +**Sizing / monitoring:** the main host launches undersized (2 vCPU / 1.9 GiB); the prod +overlay trims limits + `GOMAXPROCS=2` + 7d Prometheus retention, and `node_exporter` feeds +host memory to Grafana (`/_gm/grafana/`). Watch host memory and resize at Selectel when +players arrive. + +**`PROD_` Gitea set** (mirrors `TEST_`, mapped onto the unprefixed names above) — secrets: +`PROD_{POSTGRES_PASSWORD, GM_BASICAUTH_HASH, GRAFANA_ADMIN_PASSWORD, TELEGRAM_BOT_TOKEN, +TELEGRAM_PROMO_BOT_TOKEN, REGISTRY_PASSWORD, SSH_KEY, SSH_KNOWN_HOSTS, BOTLINK_CA, +BOTLINK_GATEWAY_CERT, BOTLINK_GATEWAY_KEY, BOTLINK_BOT_CERT, BOTLINK_BOT_KEY}`; variables: +`PROD_{REGISTRY_USER, MAIN_HOST, TG_HOST, CADDY_SITE_ADDRESS, GM_BASICAUTH_USER, +GRAFANA_ROOT_URL, LOG_LEVEL, DICT_VERSION, TELEGRAM_MINIAPP_URL, TELEGRAM_GAME_CHANNEL_ID, +TELEGRAM_CHAT_ID, TELEGRAM_BOT_USERNAME, VITE_TELEGRAM_BOT_ID, VITE_TELEGRAM_LINK, +VITE_TELEGRAM_GAME_CHANNEL_NAME}`. + ## Host-side setup (outside this repo) - **`edge` network** must exist on the host (`docker network create edge`). diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index a10d3f9..f95b818 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1056,8 +1056,10 @@ plaintext relay (`GATEWAY_BOTLINK_RELAY_ADDR`) the backend admin console calls. The full contour (`deploy/docker-compose.yml`) runs one `gateway`, one `backend`, one Postgres, the static `landing`, the Telegram `validator` and `bot` (+ the bot's VPN -sidecar) and the **observability stack** — -OTel Collector (OTLP/gRPC ingest → Prometheus metrics + Tempo traces) and Grafana +sidecar — the `bot`+`vpn` pair is gated to a `telegram-local` compose profile so the prod +main host can omit them) and the **observability stack** — +OTel Collector (OTLP/gRPC ingest → Prometheus metrics + Tempo traces), a `node_exporter` +for host CPU/memory (the prod main host's OOM signal), and Grafana with provisioned datasources and dashboards. All services export OTLP to the collector; the bot shares the VPN sidecar's netns, so its `AWG_CONF` must not carry a `DNS=` directive (that would hijack resolv.conf and stop it resolving @@ -1081,16 +1083,30 @@ Two contours, two secret/variable prefixes (`TEST_` / `PROD_`): generated by `deploy/gen-certs.sh` before `compose up`; the bot keeps its VPN sidecar for Telegram egress and dials the gateway by its internal name, so the bot-link stays on the internal network. -- **Prod**: a manual SSH deploy after `development → master`. There is no - host caddy, so the contour ships its own caddy terminating TLS — set - `CADDY_SITE_ADDRESS` to the domain and the caddy does its own ACME. The **bot runs - on a separate host** with native Telegram access (no VPN), deployed by SSH alongside - the main app (rolled together so the bot-link protocol versions never skew); the - gateway **publishes** the bot-link port and the certificates come from `PROD_` - secrets — a long-lived CA with leaves rotated by a scheduled job. The bot dials the - gateway's public bot-link endpoint and holds no inbound port; login is unaffected if - that host or the link is down. *(This prod wiring is the deferred final stage; the - code and the unified test contour land first — see `PRERELEASE.md`.)* +- **Prod**: a **manual** rollout — `.gitea/workflows/prod-deploy.yaml`, `workflow_dispatch` + only (from `master`, `confirm=deploy`), run after `development → master` is merged green. + It builds and pushes the images to the registry (`docker.iliadenisov.ru`), then deploys + over SSH onto **two hosts** provisioned by `deploy/ansible/` (docker, a non-sudo `deploy` + service account holding a dedicated CI key, key-only sshd, default-deny ufw, fail2ban): + the **main host** runs the full stack (`docker-compose.yml` + `docker-compose.prod.yml`), + the **bot host** runs only the bot (`docker-compose.bot.yml`, no VPN — native Bot API + egress, telemetry off). There is no host caddy, so the contour caddy terminates TLS — + `CADDY_SITE_ADDRESS` is the domain and caddy does its own ACME. The gateway **publishes** + the bot-link `:9443`; the remote bot dials it over mTLS (certs from `PROD_BOTLINK_*`, + ServerName `gateway`, so TLS validation is independent of the public dial address), holds + no inbound port, and login is unaffected if that host or the link is down. + `deploy/prod-deploy.sh` rolls the main stack **one service at a time in dependency order** + (postgres → backend → gateway → landing → validator → caddy), health-checking after each; + any failure **rolls the whole stack back to the previous image tag**. A **schema migration** + adds a maintenance window: the backend (the sole writer) is stopped for a consistent + `pg_dump` before the new backend migrates forward — image rollback stays DB-safe under the + expand-contract migration rule, and the dump is kept for a manual restore. The main host is + intentionally **launch-sized** (2 vCPU / 1.9 GiB): the prod overlay trims the R7 limits + (`GOMAXPROCS=2`, smaller caps, 7d Prometheus retention) and a **node_exporter** feeds + host-memory metrics to Grafana so it can be resized reactively as players arrive. + `GATEWAY_ABUSE_BAN_ENABLED=true` in prod (the per-IP ban is meaningful only with real + client IPs). The `vpn`+`bot` pair is gated to a `telegram-local` compose profile the test + contour activates; the prod main host omits it. ## 14. CI & branches -- 2.52.0 From 9f970495ee675ccbdb46bc322f14075f71e7e543 Mon Sep 17 00:00:00 2001 From: Ilia Denisov Date: Mon, 22 Jun 2026 00:35:20 +0200 Subject: [PATCH 5/5] fix(deploy): guard cd and split DOCKER_GID assignment (shellcheck) cd $COMPOSE_DIR now aborts on failure instead of deploying from the wrong dir; DOCKER_GID is declared then exported so the subshell exit isn't masked. --- deploy/prod-deploy.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/deploy/prod-deploy.sh b/deploy/prod-deploy.sh index aec789b..91a408e 100755 --- a/deploy/prod-deploy.sh +++ b/deploy/prod-deploy.sh @@ -39,10 +39,11 @@ STATE_FILE="${STATE_FILE:-/opt/scrabble/DEPLOYED_TAG}" PG_USER="${POSTGRES_USER:-scrabble}" PG_DB="${POSTGRES_DB:-scrabble}" -cd "$COMPOSE_DIR" +cd "$COMPOSE_DIR" || { echo "compose dir $COMPOSE_DIR missing"; exit 1; } export REGISTRY # otelcol joins the host docker group to read the socket; the GID varies per host. -export DOCKER_GID="$(getent group docker | cut -d: -f3)" +DOCKER_GID="$(getent group docker | cut -d: -f3)" +export DOCKER_GID dc() { docker compose -f docker-compose.yml -f docker-compose.prod.yml "$@"; } use_tag() { export TAG="$1"; } -- 2.52.0