fix(monitoring): don't false-fire the WAL-stalled alert on an idle database #241

Merged

developer merged 1 commits from fix/pg-archive-stalled-idle-falsepositive into development 2026-07-10 14:31:57 +00:00

Conversation 1 Commits 1 Files Changed 2

+14 -6

developer commented 2026-07-10 14:25:35 +00:00

Owner

Copy Link

Copy Source

Why

A WAL archiving stalled critical alert fired on prod during a quiet period. Investigated
on the host — it was a false positive:

• disk 25 %, DB healthy, pg_wal 64 MB and not growing;
• pg_stat_archiver.failed_count = 0, zero .ready segments — archiving is not failing;
• pg_current_wal_insert_lsn() frozen across samples, current segment essentially empty — the
  DB had simply had no writes for ~2.5 h;
• pgbackrest --stanza=scrabble --pg1-user=scrabble check succeeded end-to-end (segment archived
  to S3), and daily backups are current.

Root cause: an idle Postgres archives nothing — it never force-switches an empty WAL segment on
archive_timeout — so pg_stat_archiver_last_archive_age climbs past 30 min even though
archiving is perfectly healthy. The rule watched only that age, so it false-fires every quiet
hour. Nothing was at risk (no writes ⇒ the "frozen" recovery point equals the live state).

What

• pg_archive_stalled now fires only when the age is high and pg_wal is actually growing
  (WAL produced but not archived — the real "pg_wal fills the disk" danger):
  (pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16MB).
  Genuine archive_command failures stay covered by pg_archive_failing
  (increase(failed_count[15m]) > 0), untouched. and on() bridges the two metrics' differing
  label sets; delta() suits the pg_wal_size_bytes gauge (pg_stat_wal is not exported here).
  Validated on prod with promtool — the expression is empty while idle.
• Runbook fix in both alerts' annotations: pgbackrest check needs --pg1-user=scrabble,
  or it connects as role root (absent) and aborts with "no database found".

Deploy note

This is prod-only monitoring (PITR runs on prod, not the test contour). To take effect on prod
the Grafana provisioning must be re-read (recreate the scrabble-grafana container after the file
lands on the host). It rides a prod deploy, or I can apply it targeted (a --force-recreate grafana) once merged — your call. Not urgent: the alert already self-cleared (the pgbackrest check advanced the recovery point), and the false positive is benign.

No app code touched.

## Why A `WAL archiving stalled` critical alert fired on **prod** during a quiet period. Investigated on the host — it was a **false positive**: - disk 25 %, DB healthy, `pg_wal` 64 MB and not growing; - `pg_stat_archiver.failed_count = 0`, zero `.ready` segments — archiving is **not** failing; - `pg_current_wal_insert_lsn()` frozen across samples, current segment essentially empty — the DB had simply **had no writes** for ~2.5 h; - `pgbackrest --stanza=scrabble --pg1-user=scrabble check` succeeded end-to-end (segment archived to S3), and daily backups are current. Root cause: an idle Postgres archives nothing — it never force-switches an empty WAL segment on `archive_timeout` — so `pg_stat_archiver_last_archive_age` climbs past 30 min even though archiving is perfectly healthy. The rule watched **only** that age, so it false-fires every quiet hour. Nothing was at risk (no writes ⇒ the "frozen" recovery point equals the live state). ## What - **`pg_archive_stalled`** now fires only when the age is high **and pg_wal is actually growing** (WAL produced but not archived — the real "pg_wal fills the disk" danger): `(pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16MB)`. Genuine `archive_command` failures stay covered by `pg_archive_failing` (`increase(failed_count[15m]) > 0`), untouched. `and on()` bridges the two metrics' differing label sets; `delta()` suits the `pg_wal_size_bytes` gauge (`pg_stat_wal` is not exported here). Validated on prod with `promtool` — the expression is empty while idle. - **Runbook fix** in both alerts' annotations: `pgbackrest check` needs `--pg1-user=scrabble`, or it connects as role `root` (absent) and aborts with "no database found". ## Deploy note This is prod-only monitoring (PITR runs on prod, not the test contour). To take effect on prod the Grafana provisioning must be re-read (recreate the `scrabble-grafana` container after the file lands on the host). It rides a prod deploy, or I can apply it targeted (a `--force-recreate grafana`) once merged — your call. Not urgent: the alert already self-cleared (the `pgbackrest check` advanced the recovery point), and the false positive is benign. No app code touched.

developer added 1 commit 2026-07-10 14:25:35 +00:00

fix(monitoring): pg_archive_stalled must not false-fire on an idle database ... c66bf1eceb

The "WAL archiving stalled" alert fired on prod during a quiet period. An idle Postgres
archives nothing — it never force-switches an empty WAL segment on archive_timeout — so
pg_stat_archiver_last_archive_age grows past 30 min even though archiving is perfectly
healthy (failed_count 0, no .ready segments, pg_wal flat, backups current). A false
positive: no data and no disk at risk (nothing was written, so the frozen recovery point
equals the live state).

Gate the age condition on pg_wal actually growing:

  (pg_stat_archiver_last_archive_age > 1800) and on() (delta(pg_wal_size_bytes[35m]) > 16MB)

so it fires only when WAL is being produced but not archived (the real "pg_wal fills the
disk" danger; genuine archive_command failures are already caught by pg_archive_failing).
`and on()` bridges the two metrics' differing label sets (last_archive_age carries a server
label, the pg_wal gauge does not); delta() (not increase) suits the gauge. pg_stat_wal is
not exported by this postgres_exporter, so pg_wal size growth is the available signal.
Validated on prod with promtool: the expression is empty while idle.

Also fix the runbook command in both archive alerts' annotations: `pgbackrest check` needs
`--pg1-user=scrabble`, or it connects as role "root" (which does not exist) and aborts with
"no database found".

owner approved these changes 2026-07-10 14:28:08 +00:00

developer merged commit 3469278260 into development 2026-07-10 14:31:57 +00:00

developer deleted branch fix/pg-archive-stalled-idle-falsepositive 2026-07-10 14:31:57 +00:00

developer referenced this pull request 2026-07-10 16:12:00 +00:00

Release v1.15.0 — in-game UX + wallet redesign + WAL-alert fix #243

Sign in to join this conversation.

Reviewers

No Reviewers

owner

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

owner (Ilia) developer

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: developer/scrabble-game#241