Ilia Denisov
2d2dd2bc47
The Ansible base-directory loop created /opt/scrabble/certs at mode 0750 (deploy:deploy), like config/dumps/images. The gateway (and backend) run as the distroless nonroot UID 65532 — not the deploy user — so the container cannot traverse a 0750 certs dir and fails at startup with "mtls: load server keypair: ... permission denied", crash-looping. This is latent: a long-running container holds the keypair in memory and never re-reads the file, so the misconfig only bites when a container restarts (a host reboot / redeploy). A hoster maintenance reboot exposed it on prod — the gateway came back crash-looping while the deploy could not SSH in mid-reboot. Split certs out of the 0750 loop and create it 0755 (traversable). The keys stay 0644 by design (the gateway compose relies on it); the host is single-tenant + SSH-access-controlled, so a traversable certs dir adds no meaningful exposure. The live prod host was already chmod-fixed by hand; this keeps the next provisioning run from re-tightening it.
Prod host provisioning
Idempotent Ansible that prepares the two production hosts. It installs Docker, a
non-sudo deploy service account, SSH hardening, a default-deny firewall,
fail2ban, unattended security upgrades and time sync. It does not deploy the
application — that is .gitea/workflows/prod-deploy.yaml's job, running as the
deploy account this playbook creates.
Hosts are referenced by ~/.ssh/config aliases (scrabble-main-ops,
scrabble-tg-ops), so no IPs or key paths live in the repo.
Prerequisites (controller)
ansiblewith the bundled collections (community.general,community.docker,ansible.posix).- The two hosts reachable as root via the ssh-config aliases, host keys already
accepted into
known_hosts(host_key_checking = True).
One-time: the CI deploy key
The CI prod-deploy workflow logs into the hosts as deploy using a dedicated
key. Generate it once on the controller, authorize its public half via the
playbook, and store its private half only in the Gitea PROD_SSH_KEY secret:
ssh-keygen -t ed25519 -N '' -C scrabble-ci-deploy \
-f ~/.ssh/scrabble_ci_deploy_ed25519
# private half -> Gitea secret PROD_SSH_KEY (set via API); never commit it
Run
cd deploy/ansible
ansible-playbook site.yml
The playbook reads the public key from ~/.ssh/scrabble_ci_deploy_ed25519.pub by
default; override with -e deploy_ci_pubkey_path=/path/to/key.pub. Re-running is
safe (idempotent) and survives a host resize.
What each host gets
- both (
common): docker-ce + compose plugin,daemon.json(live-restore, 10m×3 log rotation),deployuser (docker group, no sudo), key-only sshd,ufwdefault-deny incoming + allow SSH, fail2ban sshd jail, unattended upgrades, chrony,/opt/scrabble/{config,certs,dumps,images}. - main:
ufwopens 80/443/9443; the externaledgedocker network; the pgBackRest daily base-backup systemd timer — installed only whenpitr_enabled=true(off by default; turned on as part of arming point-in-time recovery, see../README.md). - tg: verifies direct
api.telegram.orgegress (the no-VPN assumption).