feat(deploy): Grafana infra alerts + blackbox cert probe + admin-alert wiring

Grafana: GF_SMTP from the shared relay (STARTTLS host:port) + alerting provisioning
(contact point → SERVICE_EMAIL, route-all policy, and rules for scrape-target down,
gateway internal-error rate + p99 latency, host mem/disk/cpu, postgres connections, and
TLS cert < 20 days). All rules noDataState=OK so an absent metric never false-alerts.
blackbox_exporter probes the edge caddy's TLS (probe_ssl_earliest_cert_expiry) — effective
on prod (caddy terminates TLS; contour caddy is HTTP-only so the metric is absent).
Wires the new env through compose (backend admin From/To, Grafana SMTP), ci.yaml (TEST_),
prod-deploy (PROD_ + env.sh), .env.example and the README var table.
This commit is contained in:
Ilia Denisov
2026-07-03 15:08:09 +02:00
parent 8d2cd97e17
commit 55f6176538
10 changed files with 346 additions and 1 deletions
+34
View File
@@ -151,6 +151,10 @@ services:
BACKEND_SMTP_PASSWORD: ${SMTP_RELAY_PASS:-}
BACKEND_SMTP_FROM: ${SMTP_RELAY_FROM:-no-reply@localhost}
BACKEND_PUBLIC_BASE_URL: ${PUBLIC_BASE_URL:-}
# Operator alert emails (new feedback / word complaints): a distinct sender and the
# recipient(s) (comma-separated allowed). Both empty disables the alert worker.
BACKEND_SMTP_ADMIN_FROM: ${SMTP_RELAY_ADMIN_FROM:-}
BACKEND_ADMIN_EMAIL: ${ADMIN_EMAIL:-}
# The dictionary lives on a named volume seeded from the image on first boot
# (the image's /opt/dawg is owned by the nonroot UID, which the fresh volume
# inherits). The admin console writes new version subdirectories here, and the
@@ -497,6 +501,18 @@ services:
# caddy's Basic-Auth and re-prompts for the password on every dashboard; the
# dashboards poll and do not need Live.
GF_LIVE_MAX_CONNECTIONS: "0"
# SMTP for alert emails, reusing the shared relay credentials on its STARTTLS host:port
# (GRAFANA_SMTP_HOST — Grafana's client speaks STARTTLS, not the backend's implicit-TLS
# port), the SERVICE From, and the SERVICE recipient. Disabled unless GF_SMTP_ENABLED.
GF_SMTP_ENABLED: ${GF_SMTP_ENABLED:-false}
GF_SMTP_HOST: ${GRAFANA_SMTP_HOST:-}
GF_SMTP_USER: ${SMTP_RELAY_USER:-}
GF_SMTP_PASSWORD: ${SMTP_RELAY_PASS:-}
GF_SMTP_FROM_ADDRESS: ${SMTP_RELAY_SERVICE_FROM:-}
GF_SMTP_FROM_NAME: ${GRAFANA_SMTP_FROM_NAME:-Erudit Alerts}
GF_SMTP_STARTTLS_POLICY: MandatoryStartTLS
# The alert recipient(s), read by the provisioned contact point via $__env{SERVICE_EMAIL}.
SERVICE_EMAIL: ${SERVICE_EMAIL:-}
volumes:
- ${SCRABBLE_CONFIG_DIR:-.}/grafana/provisioning:/etc/grafana/provisioning:ro
# Dashboards live under /etc/grafana (NOT /var/lib/grafana, which the
@@ -547,6 +563,24 @@ services:
memory: 64M
networks: [internal]
# blackbox_exporter lets Prometheus alert on TLS certificate expiry (a Caddy ACME
# renewal failure) via probe_ssl_earliest_cert_expiry. It probes the edge caddy that
# terminates TLS (prod: the published scrabble-caddy; the test contour has no compose
# caddy, so the probe simply finds no target and the cert metric is absent — the rule is
# absent-safe). See prometheus.yml and grafana alerting rules.
blackbox_exporter:
container_name: scrabble-blackbox-exporter
image: prom/blackbox-exporter:v0.25.0
restart: unless-stopped
logging: *default-logging
volumes:
- ${SCRABBLE_CONFIG_DIR:-.}/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
deploy:
resources:
limits:
memory: 64M
networks: [internal, edge]
networks:
internal:
name: scrabble-internal