Ilia Denisov
4ba9da6721
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 24s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 10s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m49s
Refuse a client whose IP is in a curated CIDR feed with 403 in the same
abuseGuard, before the fail2ban ban. Prod-only (keys by real client IP), off by
default.
- ratelimit.Blocklist: a sorted-range IPv4 matcher (binary search) with an
allowlist checked first; ParseDROP reads the feed; ApplyRefresh keeps the
last-good feed on a transient fetch failure and drops it fail-open once stale
(better to under-block than block a legitimate client on a frozen feed). A
separate static CIDR set, not the per-IP fail2ban store.
- gateway: a refresher goroutine re-fetches every few hours (bounded fetch + size
cap); config GATEWAY_BLOCKLIST_{ENABLED,URL,ALLOW,REFRESH,MAX_STALENESS}.
IPv6 is not matched (a v6 client is still covered by fail2ban / the honeypot).
- observability: gateway_blocklist_blocked_total + entries/age gauges; a Grafana
alert warns before the feed is dropped; service-overview panels.
- deploy: compose env + write-prod-env.sh + prod-deploy/rollback wiring (opt-in
via PROD_ vars).
- docs (ARCHITECTURE, deploy/README); unit tests (match, allowlist, IPv6 skip,
parser, fault-tolerance) + a 403 abuse-guard integration test.
394 lines
15 KiB
YAML
394 lines
15 KiB
YAML
# Grafana provisioned alert rules for the Scrabble contour. Each rule is a Prometheus
|
|
# instant query (refId A) fed into a threshold expression (refId C). noDataState/execErrState
|
|
# are OK so an absent metric never raises a false alert — notably cert-expiry, whose blackbox
|
|
# probe has no target on the test contour (caddy is HTTP-only there). Metric names are the
|
|
# real ones Prometheus exposes (edge_request_* from the gateway via the collector,
|
|
# node_*/pg_*/probe_ssl_* from the exporters). All route to the ops-email contact point.
|
|
apiVersion: 1
|
|
groups:
|
|
- orgId: 1
|
|
name: scrabble-service
|
|
folder: Alerts
|
|
interval: 1m
|
|
rules:
|
|
- uid: svc_target_down
|
|
title: Scrape target down
|
|
condition: C
|
|
for: 3m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 300, to: 0 }
|
|
datasourceUid: prometheus
|
|
model: { refId: A, expr: up, instant: true }
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: lt, params: [1] }
|
|
labels: { severity: critical }
|
|
annotations: { summary: 'A Prometheus scrape target is down (up < 1).' }
|
|
|
|
- uid: edge_error_rate
|
|
title: Gateway internal-error rate high
|
|
condition: C
|
|
for: 10m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 600, to: 0 }
|
|
datasourceUid: prometheus
|
|
model:
|
|
refId: A
|
|
expr: sum by (service_name) (rate(edge_request_duration_count{result="internal"}[5m]))
|
|
instant: true
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: gt, params: [0.05] }
|
|
labels: { severity: warning }
|
|
annotations: { summary: 'Sustained internal (5xx-equivalent) errors at the edge.' }
|
|
|
|
- uid: edge_latency_p99
|
|
title: Gateway request latency p99 high
|
|
condition: C
|
|
for: 10m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 600, to: 0 }
|
|
datasourceUid: prometheus
|
|
model:
|
|
refId: A
|
|
expr: histogram_quantile(0.99, sum by (le, service_name) (rate(edge_request_duration_bucket[5m])))
|
|
instant: true
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: gt, params: [1] }
|
|
labels: { severity: warning }
|
|
annotations: { summary: 'Edge request p99 latency above 1s.' }
|
|
|
|
- uid: tls_cert_expiry
|
|
title: TLS certificate nearing expiry
|
|
condition: C
|
|
for: 15m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 300, to: 0 }
|
|
datasourceUid: prometheus
|
|
model:
|
|
refId: A
|
|
expr: (probe_ssl_earliest_cert_expiry - time()) / 86400
|
|
instant: true
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: lt, params: [20] }
|
|
labels: { severity: critical }
|
|
annotations: { summary: 'Edge TLS cert has under 20 days left — Caddy ACME renewal may have failed.' }
|
|
|
|
# The community IP blocklist feed has not refreshed in over a day (the gateway re-fetches every
|
|
# few hours). Absent/NaN-safe: the age gauge appears only once a feed has loaded, so a disabled
|
|
# or never-fetched blocklist stays quiet. The feed itself is dropped (fail-open) at its
|
|
# max-staleness window; this warns well before that so the operator can fix the fetch.
|
|
- uid: blocklist_stale
|
|
title: IP blocklist feed not refreshing
|
|
condition: C
|
|
for: 15m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 300, to: 0 }
|
|
datasourceUid: prometheus
|
|
model: { refId: A, expr: gateway_blocklist_age_seconds, instant: true }
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: gt, params: [86400] }
|
|
labels: { severity: warning }
|
|
annotations: { summary: 'The community IP blocklist feed has not refreshed in over 24h — the fetch is failing; it will be dropped (fail-open) once stale. Check the gateway blocklist refresher logs and the feed URL.' }
|
|
|
|
- orgId: 1
|
|
name: scrabble-host
|
|
folder: Alerts
|
|
interval: 1m
|
|
rules:
|
|
- uid: host_mem_low
|
|
title: Host memory low
|
|
condition: C
|
|
for: 5m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 300, to: 0 }
|
|
datasourceUid: prometheus
|
|
model:
|
|
refId: A
|
|
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
|
|
instant: true
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: lt, params: [0.1] }
|
|
labels: { severity: critical }
|
|
annotations: { summary: 'Under 10% host memory available.' }
|
|
|
|
- uid: host_disk_low
|
|
title: Host disk low
|
|
condition: C
|
|
for: 10m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 300, to: 0 }
|
|
datasourceUid: prometheus
|
|
model:
|
|
refId: A
|
|
expr: min(node_filesystem_avail_bytes / node_filesystem_size_bytes)
|
|
instant: true
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: lt, params: [0.1] }
|
|
labels: { severity: critical }
|
|
annotations: { summary: 'A host filesystem is under 10% free.' }
|
|
|
|
- uid: host_cpu_high
|
|
title: Host CPU saturated
|
|
condition: C
|
|
for: 10m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 600, to: 0 }
|
|
datasourceUid: prometheus
|
|
model:
|
|
refId: A
|
|
expr: 1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))
|
|
instant: true
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: gt, params: [0.9] }
|
|
labels: { severity: warning }
|
|
annotations: { summary: 'Host CPU above 90% for 10 minutes.' }
|
|
|
|
- uid: pg_connections_high
|
|
title: Postgres connections high
|
|
condition: C
|
|
for: 5m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 300, to: 0 }
|
|
datasourceUid: prometheus
|
|
model:
|
|
refId: A
|
|
expr: sum(pg_stat_activity_count) / max(pg_settings_max_connections)
|
|
instant: true
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: gt, params: [0.8] }
|
|
labels: { severity: warning }
|
|
annotations: { summary: 'Postgres using over 80% of max_connections.' }
|
|
|
|
# Continuous WAL archiving health (pgBackRest -> S3, point-in-time recovery). Both rules
|
|
# read the postgres_exporter's built-in pg_stat_archiver metrics and are absent/NaN-safe:
|
|
# on the test contour (and any host before archiving is armed) archive_mode is off, so
|
|
# failed_count stays 0 and last_archive_age is NaN — neither threshold trips.
|
|
- orgId: 1
|
|
name: scrabble-backup
|
|
folder: Alerts
|
|
interval: 1m
|
|
rules:
|
|
- uid: pg_archive_failing
|
|
title: WAL archiving failing
|
|
condition: C
|
|
for: 15m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 900, to: 0 }
|
|
datasourceUid: prometheus
|
|
model:
|
|
refId: A
|
|
expr: increase(pg_stat_archiver_failed_count[15m])
|
|
instant: true
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: gt, params: [0] }
|
|
labels: { severity: critical }
|
|
annotations: { summary: 'pgBackRest archive_command is failing — PITR is degrading and pg_wal can fill the disk. Run `docker exec scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check`.' }
|
|
|
|
# Fires only when the last archive is >30 min old AND pg_wal is actually growing (WAL is being
|
|
# produced but not archived). A genuinely idle database archives nothing — Postgres does not
|
|
# force-switch an empty segment on archive_timeout — so `last_archive_age` alone false-triggers
|
|
# during quiet hours; the pg_wal-growth guard removes that. `and on()` bridges the differing
|
|
# label sets (last_archive_age has a server label, the pg_wal gauge does not); delta() (not
|
|
# increase) suits the pg_wal_size_bytes gauge. Real archive failures are caught by pg_archive_failing.
|
|
- uid: pg_archive_stalled
|
|
title: WAL archiving stalled
|
|
condition: C
|
|
for: 15m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 2100, to: 0 }
|
|
datasourceUid: prometheus
|
|
model:
|
|
refId: A
|
|
expr: (pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16777216)
|
|
instant: true
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: gt, params: [1800] }
|
|
labels: { severity: critical }
|
|
annotations: { summary: 'No WAL archived for over 30 minutes while pg_wal keeps growing — archiving is genuinely stalled; the PITR recovery point is frozen and pg_wal will fill the disk. Run `docker exec scrabble-postgres pgbackrest --stanza=scrabble --pg1-user=scrabble check`.' }
|
|
|
|
# Remote Telegram bot health. The bot runs on its own host and exports no telemetry of its own; the
|
|
# gateway observes it through the bot-link (botlink_connected_bots) and the health it reports over
|
|
# that stream (bot_tg_*). All absent/NaN-safe: before a bot ever connects the metrics are absent, so
|
|
# noDataState OK keeps them quiet on a fresh contour. The alert email does NOT go through the bot, so
|
|
# a bot-down alert is deliverable.
|
|
- orgId: 1
|
|
name: scrabble-bot
|
|
folder: Alerts
|
|
interval: 1m
|
|
rules:
|
|
- uid: bot_disconnected
|
|
title: Telegram bot disconnected
|
|
condition: C
|
|
for: 5m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 300, to: 0 }
|
|
datasourceUid: prometheus
|
|
model: { refId: A, expr: botlink_connected_bots, instant: true }
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: lt, params: [1] }
|
|
labels: { severity: critical }
|
|
annotations: { summary: 'No Telegram bot is connected to the gateway bot-link — out-of-app push and admin sends are down. Check the bot host.' }
|
|
|
|
# Positive liveness: a bot is connected but its last successful Bot API call is over 5 minutes
|
|
# old — the getUpdates long-poll returns every ~minute even when idle, so a stale stamp means the
|
|
# bot is wedged (no errors, no traffic). Guarded by "and connected" so a mere disconnect (owned by
|
|
# bot_disconnected) does not double-fire.
|
|
- uid: bot_tg_stale
|
|
title: Telegram bot not reaching the Bot API
|
|
condition: C
|
|
for: 5m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 600, to: 0 }
|
|
datasourceUid: prometheus
|
|
model:
|
|
refId: A
|
|
expr: (time() - bot_tg_last_ok_unix) and on() (botlink_connected_bots >= 1)
|
|
instant: true
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: gt, params: [300] }
|
|
labels: { severity: critical }
|
|
annotations: { summary: 'A connected Telegram bot has not reached the Bot API in over 5 minutes — the update poll is wedged. Check the bot host and Telegram reachability.' }
|
|
|
|
# 429s should be ~never once the bot honours Retry-After; any sustained rate-limiting is a symptom
|
|
# (a send loop, a misbehaving path) worth investigating.
|
|
- uid: bot_tg_rate_limited
|
|
title: Telegram bot rate-limited (429)
|
|
condition: C
|
|
for: 5m
|
|
noDataState: OK
|
|
execErrState: OK
|
|
data:
|
|
- refId: A
|
|
relativeTimeRange: { from: 900, to: 0 }
|
|
datasourceUid: prometheus
|
|
model:
|
|
refId: A
|
|
expr: increase(bot_tg_errors_total{kind="rate_limited"}[15m])
|
|
instant: true
|
|
- refId: C
|
|
datasourceUid: __expr__
|
|
model:
|
|
refId: C
|
|
type: threshold
|
|
expression: A
|
|
conditions:
|
|
- evaluator: { type: gt, params: [0] }
|
|
labels: { severity: warning }
|
|
annotations: { summary: 'The Telegram bot is being rate-limited (HTTP 429). It should honour Retry-After, so sustained 429s point to a send loop or a hot path.' }
|