feat(observability): Grafana infra alerts + admin email notifications (PR4) #165

Merged
developer merged 5 commits from feature/email-relay-pr4 into development 2026-07-03 14:05:45 +00:00

5 Commits

Author SHA1 Message Date
Ilia Denisov 854c4b3005 fix(deploy): stage blackbox config + derive bare Grafana SMTP from-address
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m28s
Two contour-deploy failures from PR4: (1) the deploy did not stage deploy/blackbox/, so
blackbox_exporter crash-looped on a missing config mount — add blackbox to the ci.yaml cp
and the prod-deploy tar; (2) Grafana rejects the 'Name <addr>' From form the backend go-mail
accepts and validates it even when SMTP is disabled, crash-looping Grafana — the deploy now
splits SMTP_RELAY_SERVICE_FROM into a bare GF_SMTP_FROM_ADDRESS + GRAFANA_SMTP_FROM_NAME
(handles both display and bare forms). Verified the sed split + compose render.
2026-07-03 15:30:39 +02:00
Ilia Denisov ec1bdfca00 fix(deploy): Grafana SMTP reuses relay host + explicit GRAFANA_SMTP_PORT
CI / changes (pull_request) Successful in 1s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 17s
CI / ui (pull_request) Successful in 1m5s
CI / conformance (pull_request) Successful in 8s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Failing after 1s
Drop the separate GRAFANA_SMTP_HOST (it duplicated SMTP_RELAY_HOST): Grafana differs from
the backend only in needing the STARTTLS port, so GF_SMTP_HOST is composed from
SMTP_RELAY_HOST + GRAFANA_SMTP_PORT (an explicit per-contour var, no magic default), reusing
SMTP_RELAY_USER/PASS. Wired through ci/prod/.env.example/README.
2026-07-03 15:17:43 +02:00
Ilia Denisov 70f0f9e36a docs(architecture): observability alerting + admin-alert worker
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 18s
CI / ui (pull_request) Successful in 1m6s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Failing after 1m8s
§11 gains the alerting layer: Grafana infra rules (scrape-down, edge error-rate/p99, host
mem/disk/cpu, postgres connections, TLS cert < 20d via blackbox), noDataState=OK, and the
backend admin-alert worker (coalesced email on new feedback / complaints).
2026-07-03 15:08:57 +02:00
Ilia Denisov 55f6176538 feat(deploy): Grafana infra alerts + blackbox cert probe + admin-alert wiring
Grafana: GF_SMTP from the shared relay (STARTTLS host:port) + alerting provisioning
(contact point → SERVICE_EMAIL, route-all policy, and rules for scrape-target down,
gateway internal-error rate + p99 latency, host mem/disk/cpu, postgres connections, and
TLS cert < 20 days). All rules noDataState=OK so an absent metric never false-alerts.
blackbox_exporter probes the edge caddy's TLS (probe_ssl_earliest_cert_expiry) — effective
on prod (caddy terminates TLS; contour caddy is HTTP-only so the metric is absent).
Wires the new env through compose (backend admin From/To, Grafana SMTP), ci.yaml (TEST_),
prod-deploy (PROD_ + env.sh), .env.example and the README var table.
2026-07-03 15:08:09 +02:00
Ilia Denisov 8d2cd97e17 feat(adminalert): operator email on new feedback / word complaints
New adminalert worker polls for feedback + word complaints arriving since the last check
and coalesces a burst into one digest email per interval (5 min), inert unless a distinct
admin sender (BACKEND_SMTP_ADMIN_FROM) and recipient (BACKEND_ADMIN_EMAIL, comma-separated
allowed) are configured. The mailer gains a per-message From override and splits a
comma-separated To into separate recipients (go-mail needs them as a list). Feedback/game
stores gain CountSince/CountComplaintsSince. Unit tests cover the digest, the skip-when-
empty, and the recipient split.
2026-07-03 14:53:51 +02:00