fix(monitoring): don't false-fire the WAL-stalled alert on an idle database #241

Merged
developer merged 1 commits from fix/pg-archive-stalled-idle-falsepositive into development 2026-07-10 14:31:57 +00:00

1 Commits

Author SHA1 Message Date
Ilia Denisov c66bf1eceb fix(monitoring): pg_archive_stalled must not false-fire on an idle database
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 11s
CI / integration (pull_request) Successful in 24s
CI / ui (pull_request) Successful in 1m11s
CI / conformance (pull_request) Successful in 9s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m55s
The "WAL archiving stalled" alert fired on prod during a quiet period. An idle Postgres
archives nothing — it never force-switches an empty WAL segment on archive_timeout — so
pg_stat_archiver_last_archive_age grows past 30 min even though archiving is perfectly
healthy (failed_count 0, no .ready segments, pg_wal flat, backups current). A false
positive: no data and no disk at risk (nothing was written, so the frozen recovery point
equals the live state).

Gate the age condition on pg_wal actually growing:

  (pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16MB)

so it fires only when WAL is being produced but not archived (the real "pg_wal fills the
disk" danger; genuine archive_command failures are already caught by pg_archive_failing).
`and on()` bridges the two metrics' differing label sets (last_archive_age carries a server
label, the pg_wal gauge does not); delta() (not increase) suits the gauge. pg_stat_wal is
not exported by this postgres_exporter, so pg_wal size growth is the available signal.
Validated on prod with promtool: the expression is empty while idle.

Also fix the runbook command in both archive alerts' annotations: `pgbackrest check` needs
`--pg1-user=scrabble`, or it connects as role "root" (which does not exist) and aborts with
"no database found".
2026-07-10 16:25:01 +02:00