fix(metrics): second-scale buckets for edge_request_duration #186

Merged

developer merged 1 commits from fix/edge-latency-histogram-buckets into development 2026-07-05 20:55:31 +00:00

Conversation 2 Commits 1 Files Changed 2

+22 -1

developer commented 2026-07-05 20:49:24 +00:00

Owner

Copy Link

Copy Source

Fixes the constantly-flapping "Gateway request latency p99 high" alert (fires then immediately resolves on nearly every app open, on both the test and prod contours).

Root cause

edge_request_duration records durations in seconds (WithUnit("s"), time.Since(start).Seconds()) but was created without explicit bucket boundaries, so it used the OTel SDK's default millisecond-calibrated boundaries (0, 5, 10, 25, …). Real request latencies are milliseconds (0.001–0.5 s), so every request under 5 s fell into the single first bucket le=5. histogram_quantile(0.99, …) then interpolates 0.99 × 5 = 4.95s — uniformly for every message type, including trivially-fast ones.

Confirmed on the contour's Prometheus: p99 = 4.950s for profile.get, friends.list, games.list, … — impossible as real latency.

That 4.95s > the 1s alert threshold, so the alert tripped on essentially every request and flapped fire/resolve as the sparse 5m rate window shifted.

Fix

Set explicit second-scale bucket boundaries on the histogram (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10) straddling the 1s SLO, so the p99 reflects the real (mostly sub-second) latency and the alert fires only on genuine slowness. No alert-rule change needed.

Tests

TestEdgeMetric extended to assert the histogram carries a sub-second boundary (guards against reverting to the ms-default that caused the flap). go build/vet/test ./gateway/... + gofmt green.

After deploy

Takes effect once the gateway is redeployed (buckets are set at instrument creation). New samples then bucket correctly; the p99 panel/alert become meaningful within ~5 min of traffic.

Fixes the constantly-flapping **"Gateway request latency p99 high"** alert (fires then immediately resolves on nearly every app open, on both the test and prod contours). ## Root cause `edge_request_duration` records durations in **seconds** (`WithUnit("s")`, `time.Since(start).Seconds()`) but was created without explicit bucket boundaries, so it used the OTel SDK's **default millisecond-calibrated boundaries** (`0, 5, 10, 25, …`). Real request latencies are milliseconds (0.001–0.5 s), so **every** request under 5 s fell into the single first bucket `le=5`. `histogram_quantile(0.99, …)` then interpolates `0.99 × 5 = 4.95s` — uniformly for **every** message type, including trivially-fast ones. Confirmed on the contour's Prometheus: p99 = **4.950s** for `profile.get`, `friends.list`, `games.list`, … — impossible as real latency. That 4.95s > the 1s alert threshold, so the alert tripped on essentially every request and flapped fire/resolve as the sparse 5m rate window shifted. ## Fix Set explicit **second-scale** bucket boundaries on the histogram (`0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10`) straddling the 1s SLO, so the p99 reflects the real (mostly sub-second) latency and the alert fires only on genuine slowness. No alert-rule change needed. ## Tests `TestEdgeMetric` extended to assert the histogram carries a **sub-second** boundary (guards against reverting to the ms-default that caused the flap). `go build/vet/test ./gateway/...` + `gofmt` green. ## After deploy Takes effect once the gateway is redeployed (buckets are set at instrument creation). New samples then bucket correctly; the p99 panel/alert become meaningful within ~5 min of traffic.

developer added 1 commit 2026-07-05 20:49:24 +00:00

fix(metrics): second-scale buckets for edge_request_duration ... 00bb66ba0f

The edge request-latency histogram records seconds but used the OTel SDK's
default millisecond-calibrated bucket boundaries (first boundary 5), so every
sub-5s request fell into one bucket and histogram_quantile(0.99) interpolated
to ~4.95s for every message type — regardless of the real (millisecond)
latency. That tripped the >1s "Gateway request latency p99 high" alert on
essentially every request, flapping fire/resolve on each app open (seen on the
test and prod contours).

Set explicit second-scale bucket boundaries (0.005 … 10s) straddling the 1s SLO
so the p99 reflects real latency and the alert fires only on genuine slowness.
Regression test asserts the histogram carries a sub-second boundary.

owner approved these changes 2026-07-05 20:51:31 +00:00

Dismissed

owner approved these changes 2026-07-05 20:55:08 +00:00

developer merged commit 3a85f64726 into development 2026-07-05 20:55:31 +00:00

developer deleted branch fix/edge-latency-histogram-buckets 2026-07-05 20:55:31 +00:00

developer referenced this pull request 2026-07-05 21:00:02 +00:00

Release v1.11.0: PWA install + code-only PWA login + email/metrics fixes #187

Sign in to join this conversation.

Reviewers

No Reviewers

owner

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

owner (Ilia) developer

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: developer/scrabble-game#186