Why a restart changes the shape of a counter
A counter records a running total, such as requests handled by one application process. It usually grows until that process restarts. The restart creates a legitimate decrease, rather than negative traffic. A gauge, such as the number of active connections, can move in both directions and needs different analysis.
The operational question is whether a dashboard calculates a useful rate from each original counter before combining replicas. Prometheus can account for resets in individual series. Summing the raw counters first can hide the reset behind another replica's growth and create a misleading result.
Setup: identify the instrument and its labels
Record the metric's documented type, unit and label set. Check that a target or instance label distinguishes independent processes, and determine whether a scrape target serves one process or a changing aggregation of several workers. A counter exposed by a load-balanced endpoint can alternate between unrelated values even without a restart.
Choose a query range that contains enough samples for the configured scrape interval. Keep the actual interval and scrape failure history beside the query. A very short range during a missed scrape can produce missing results without demonstrating that the application received no requests.
Investigate the query in four steps
- Inspect the raw counter for one replica over the incident interval. Compare visible decreases with process start time and deployment events. Preserve the original label set while doing this.
- Calculate a per-replica rate and compare replicas. An illustrative service query is
sum by (service) (rate(http_requests_total[5m])); substitute metric and labels from your own instrumentation. - Compare that result with the dashboard's recording rules and transformations. Check whether an intermediate rule already aggregated the raw counter and therefore discarded evidence needed for reset detection.
- Inspect target health and label churn. A renamed instance is a new time series, while a missing scrape is an absence of evidence. Neither should automatically become a zero-valued business event.
Illustrative incident: a rollout produces a false traffic surge
In this illustrative example, a deployment replaces one API replica while another continues serving requests. A recording rule stores the sum of raw replica counters. A later dashboard calculates a rate from that sum and shows a sudden surge. Capacity changes are considered before anyone checks the query.
The operator compares the raw replicas, observes the restart and rewrites the recording rule to calculate rates before summing. Historical recording-rule samples do not repair themselves, so the team marks the query-change boundary and compares fresh intervals. The example represents query diagnosis, not a measured Clouvenix customer result.
Diagnostic evidence and correction
| Observation | Inspect | Interpretation |
|---|---|---|
| Spike begins at a rollout | Raw per-instance counters | Reset or series replacement |
| One target alternates totals | Scrape endpoint ownership | Different processes may share one target |
| Rate disappears briefly | Range and successful scrapes | Too few usable samples |
| Aggregate differs from replicas | Recording-rule expression | Aggregation may precede reset handling |
Use rate for appropriate counters rather than applying delta simply because the graph looks smoother. Keep units visible: a requests-per-second rate and an estimated count over an interval answer different questions. Reset handling does not correct a wrongly instrumented counter or recover events that were never observed.
Verification, alerting and SOP
Exercise one approved test-process restart and verify that remaining replicas continue contributing normally. Compare request logs or another independently defined count without expecting exact equality across different sampling windows. Record the query and recording-rule revisions.
The SOP should assign ownership of metric names, labels, scrape intervals and rule tests. Monitor collection health separately from traffic. A useful internal SLO can describe telemetry freshness; it does not automatically change the application's contractual availability SLA.
See Clouvenix observability monitoring for implementation and operational handover.
Technical reference
Official reference 1; Official reference 2. Documentation checked 19 October 2026; confirm behavior for your deployed edition and version.