A percentile describes a population of requests
The 95th percentile is a latency below which approximately 95 percent of observations fall. A service percentile should represent the intended request population across replicas. Averaging the p95 already computed by each replica does not generally produce the service's p95, because the replicas can handle very different traffic volumes.
Histograms retain distribution information in buckets, allowing compatible observations to be combined before estimating a quantile. Summaries that expose precomputed quantiles have a different contract. First identify which instrument the application actually exports, rather than choosing a query from the metric name alone.
Setup: define the latency population
Document the metric unit, request boundaries, status filtering and bucket configuration. Decide whether the dashboard covers all requests, successful requests or a particular route family. Keep route templates instead of unbounded raw URLs where the instrumentation supports them.
Check whether the deployment emits classic or native histograms. Classic histograms expose bucket series with an le label. Native histograms carry distribution information in histogram samples and use a different aggregation expression. A mixed rollout needs an explicit compatibility plan and query verification.
Construct and inspect the query
- Inspect one replica's count and bucket series over a representative interval. Confirm that observations are arriving and that duration values are expressed in the documented unit.
- For a classic histogram, calculate bucket rates per original series, then sum compatible buckets while preserving
le. A scoped example ishistogram_quantile(0.95, sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))). - Keep dimensions that represent different user experiences, such as service or a deliberately selected operation. Removing a dimension can combine unrelated populations and hide a slow minority behind fast traffic.
- Examine bucket boundaries around the objective. A quantile is an estimate within the available distribution; wide buckets cannot magically provide fine precision around a narrow latency target.
Illustrative incident: the average of p95s hides a busy replica
In this illustrative scenario, a quiet replica handles a few fast health requests while another processes most customer work. A dashboard averages their reported percentiles and suggests the service is comfortably within its objective. Users continue reporting delays.
The team checks counts, confirms the unequal populations and replaces the expression with aggregation of compatible histogram data. It retains a per-operation view to investigate the busy route. This is a constructed teaching example; the article does not claim a customer benchmark or a particular performance improvement.
Diagnostic table
| Symptom | Evidence to collect | Next decision |
|---|---|---|
| Service p95 looks unexpectedly low | Counts and quantiles per replica | Was a percentile averaged? |
| Classic query returns no distribution | Bucket series and retained labels | Was le removed? |
| Latency changes after rollout | Histogram type, units and boundaries | Are the populations compatible? |
| Quantile appears unstable at night | Observation count in the window | Is traffic too sparse for interpretation? |
Verify the correction with known observations
Use a small test workload with deliberately varied durations and inspect counts, bucket distribution and the final quantile together. Compare the same operation and interval before drawing conclusions. A zero-traffic interval is not proof of perfect latency; document how empty populations appear in the dashboard and alerts.
For a threshold objective, also consider the proportion of requests within the target bucket when suitable boundaries exist. That answers compliance with a chosen threshold directly and complements percentile exploration. Keep query changes versioned with the instrumentation configuration.
Monitoring and SOP
Record instrument type, units, retained labels, bucket boundaries and the owner of the latency definition. Alert on missing collection separately from slow requests. Reassess queries after instrumentation migrations. The SOP should require a sample-volume view beside percentiles so an apparently precise chart cannot conceal a very small population.
See Clouvenix observability monitoring for implementation and operational handover.
Technical reference
Official reference 1; Official reference 2. Documentation checked 19 October 2026; confirm behavior for your deployed edition and version.