feat(deploy): Grafana infra alerts + blackbox cert probe + admin-alert wiring
Grafana: GF_SMTP from the shared relay (STARTTLS host:port) + alerting provisioning (contact point → SERVICE_EMAIL, route-all policy, and rules for scrape-target down, gateway internal-error rate + p99 latency, host mem/disk/cpu, postgres connections, and TLS cert < 20 days). All rules noDataState=OK so an absent metric never false-alerts. blackbox_exporter probes the edge caddy's TLS (probe_ssl_earliest_cert_expiry) — effective on prod (caddy terminates TLS; contour caddy is HTTP-only so the metric is absent). Wires the new env through compose (backend admin From/To, Grafana SMTP), ci.yaml (TEST_), prod-deploy (PROD_ + env.sh), .env.example and the README var table.
This commit is contained in:
@@ -151,6 +151,10 @@ services:
|
||||
BACKEND_SMTP_PASSWORD: ${SMTP_RELAY_PASS:-}
|
||||
BACKEND_SMTP_FROM: ${SMTP_RELAY_FROM:-no-reply@localhost}
|
||||
BACKEND_PUBLIC_BASE_URL: ${PUBLIC_BASE_URL:-}
|
||||
# Operator alert emails (new feedback / word complaints): a distinct sender and the
|
||||
# recipient(s) (comma-separated allowed). Both empty disables the alert worker.
|
||||
BACKEND_SMTP_ADMIN_FROM: ${SMTP_RELAY_ADMIN_FROM:-}
|
||||
BACKEND_ADMIN_EMAIL: ${ADMIN_EMAIL:-}
|
||||
# The dictionary lives on a named volume seeded from the image on first boot
|
||||
# (the image's /opt/dawg is owned by the nonroot UID, which the fresh volume
|
||||
# inherits). The admin console writes new version subdirectories here, and the
|
||||
@@ -497,6 +501,18 @@ services:
|
||||
# caddy's Basic-Auth and re-prompts for the password on every dashboard; the
|
||||
# dashboards poll and do not need Live.
|
||||
GF_LIVE_MAX_CONNECTIONS: "0"
|
||||
# SMTP for alert emails, reusing the shared relay credentials on its STARTTLS host:port
|
||||
# (GRAFANA_SMTP_HOST — Grafana's client speaks STARTTLS, not the backend's implicit-TLS
|
||||
# port), the SERVICE From, and the SERVICE recipient. Disabled unless GF_SMTP_ENABLED.
|
||||
GF_SMTP_ENABLED: ${GF_SMTP_ENABLED:-false}
|
||||
GF_SMTP_HOST: ${GRAFANA_SMTP_HOST:-}
|
||||
GF_SMTP_USER: ${SMTP_RELAY_USER:-}
|
||||
GF_SMTP_PASSWORD: ${SMTP_RELAY_PASS:-}
|
||||
GF_SMTP_FROM_ADDRESS: ${SMTP_RELAY_SERVICE_FROM:-}
|
||||
GF_SMTP_FROM_NAME: ${GRAFANA_SMTP_FROM_NAME:-Erudit Alerts}
|
||||
GF_SMTP_STARTTLS_POLICY: MandatoryStartTLS
|
||||
# The alert recipient(s), read by the provisioned contact point via $__env{SERVICE_EMAIL}.
|
||||
SERVICE_EMAIL: ${SERVICE_EMAIL:-}
|
||||
volumes:
|
||||
- ${SCRABBLE_CONFIG_DIR:-.}/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
# Dashboards live under /etc/grafana (NOT /var/lib/grafana, which the
|
||||
@@ -547,6 +563,24 @@ services:
|
||||
memory: 64M
|
||||
networks: [internal]
|
||||
|
||||
# blackbox_exporter lets Prometheus alert on TLS certificate expiry (a Caddy ACME
|
||||
# renewal failure) via probe_ssl_earliest_cert_expiry. It probes the edge caddy that
|
||||
# terminates TLS (prod: the published scrabble-caddy; the test contour has no compose
|
||||
# caddy, so the probe simply finds no target and the cert metric is absent — the rule is
|
||||
# absent-safe). See prometheus.yml and grafana alerting rules.
|
||||
blackbox_exporter:
|
||||
container_name: scrabble-blackbox-exporter
|
||||
image: prom/blackbox-exporter:v0.25.0
|
||||
restart: unless-stopped
|
||||
logging: *default-logging
|
||||
volumes:
|
||||
- ${SCRABBLE_CONFIG_DIR:-.}/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 64M
|
||||
networks: [internal, edge]
|
||||
|
||||
networks:
|
||||
internal:
|
||||
name: scrabble-internal
|
||||
|
||||
Reference in New Issue
Block a user