Ilia Denisov
55f6176538
Grafana: GF_SMTP from the shared relay (STARTTLS host:port) + alerting provisioning (contact point → SERVICE_EMAIL, route-all policy, and rules for scrape-target down, gateway internal-error rate + p99 latency, host mem/disk/cpu, postgres connections, and TLS cert < 20 days). All rules noDataState=OK so an absent metric never false-alerts. blackbox_exporter probes the edge caddy's TLS (probe_ssl_earliest_cert_expiry) — effective on prod (caddy terminates TLS; contour caddy is HTTP-only so the metric is absent). Wires the new env through compose (backend admin From/To, Grafana SMTP), ci.yaml (TEST_), prod-deploy (PROD_ + env.sh), .env.example and the README var table.
44 lines
1.8 KiB
YAML
44 lines
1.8 KiB
YAML
# Prometheus scrape config for the Scrabble contour. The OTel Collector exposes
|
|
# every service's metrics on its prometheus exporter; Prometheus scrapes that one
|
|
# endpoint. Retention (15d) is set on the command line in docker-compose.yml.
|
|
global:
|
|
scrape_interval: 30s
|
|
evaluation_interval: 30s
|
|
|
|
scrape_configs:
|
|
# otelcol exposes both the services' OTLP metrics and the docker_stats receiver's
|
|
# per-container resource metrics (CPU/memory/network) on one endpoint.
|
|
- job_name: otelcol
|
|
static_configs:
|
|
- targets: ["otelcol:9464"]
|
|
- job_name: prometheus
|
|
static_configs:
|
|
- targets: ["localhost:9090"]
|
|
# Postgres server metrics (connections, cache hit ratio, transactions, db size).
|
|
- job_name: postgres_exporter
|
|
static_configs:
|
|
- targets: ["postgres_exporter:9187"]
|
|
# Host-level metrics (memory/CPU/disk). Matters most on the prod main host's tight
|
|
# 1.9 GiB budget, where total host memory is the OOM-proximity signal.
|
|
- job_name: node
|
|
static_configs:
|
|
- targets: ["node_exporter:9100"]
|
|
# TLS certificate expiry of the edge caddy (probe_ssl_earliest_cert_expiry), for the
|
|
# certificate-renewal-failure alert. Effective on prod, where caddy terminates TLS on
|
|
# :443; on the test contour caddy is HTTP-only, so the probe finds nothing and the metric
|
|
# is absent (the alert rule is absent-safe). The exporter probes the target passed as a
|
|
# scrape parameter and answers on its own :9115.
|
|
- job_name: blackbox_tls
|
|
metrics_path: /probe
|
|
params:
|
|
module: [tls_cert]
|
|
static_configs:
|
|
- targets: ["caddy:443"]
|
|
relabel_configs:
|
|
- source_labels: [__address__]
|
|
target_label: __param_target
|
|
- source_labels: [__param_target]
|
|
target_label: instance
|
|
- target_label: __address__
|
|
replacement: blackbox_exporter:9115
|