fix(metrics): second-scale buckets for edge_request_duration
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 14s
CI / ui (pull_request) Has been skipped
CI / conformance (pull_request) Successful in 8s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m44s
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 14s
CI / ui (pull_request) Has been skipped
CI / conformance (pull_request) Successful in 8s
CI / gate (pull_request) Successful in 0s
CI / deploy (pull_request) Successful in 1m44s
The edge request-latency histogram records seconds but used the OTel SDK's default millisecond-calibrated bucket boundaries (first boundary 5), so every sub-5s request fell into one bucket and histogram_quantile(0.99) interpolated to ~4.95s for every message type — regardless of the real (millisecond) latency. That tripped the >1s "Gateway request latency p99 high" alert on essentially every request, flapping fire/resolve on each app open (seen on the test and prod contours). Set explicit second-scale bucket boundaries (0.005 … 10s) straddling the 1s SLO so the p99 reflects real latency and the alert fires only on genuine slowness. Regression test asserts the histogram carries a sub-second boundary.
This commit is contained in:
@@ -46,7 +46,13 @@ func newServerMetrics(meter metric.Meter) *serverMetrics {
|
||||
}
|
||||
h, err := meter.Float64Histogram("edge_request_duration",
|
||||
metric.WithUnit("s"),
|
||||
metric.WithDescription("Seconds to serve one Connect Execute call, by message type and result."))
|
||||
metric.WithDescription("Seconds to serve one Connect Execute call, by message type and result."),
|
||||
// Explicit second-scale buckets. The durations are recorded in seconds, so the SDK's
|
||||
// default millisecond-calibrated boundaries (first boundary 5) would bin every sub-5s
|
||||
// request into one bucket — making histogram_quantile(0.99) interpolate to ~4.95s
|
||||
// regardless of the real latency, which flapped the >1s edge-latency alert. These
|
||||
// boundaries straddle the 1s SLO so the p99 reflects real (mostly sub-second) latency.
|
||||
metric.WithExplicitBucketBoundaries(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10))
|
||||
if err != nil {
|
||||
h, _ = noop.NewMeterProvider().Meter(meterName).Float64Histogram("edge_request_duration")
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user