Context
A dashboard based on a recording rule develops gaps while the underlying exporter appears available. Filling missing values with zero makes the graph look clean, but it can turn absent evaluation data into a false claim about service health. Recording rules have their own execution schedule and failure modes. Investigate that layer before changing the visualisation.
Understand the evaluation boundary
Prometheus documents rule groups and their evaluation intervals. If a group has not finished before its next scheduled evaluation, subsequent evaluations can be skipped until the original completes or times out, creating gaps. The documentation identifies a missed-iteration metric for this condition. Review the current rule configuration and Prometheus version when diagnosing the behaviour.
A missing recorded sample can have several causes: absent source data, an expression producing no series, an evaluation error, a skipped evaluation, or changed labels. These cases need different interpretations. A single no-data panel cannot distinguish them without further evidence.
An illustrative expensive rule group
Imagine a group containing a costly high-cardinality aggregation followed by several service-level rules. Evaluation duration increases beyond the configured interval, and recorded metrics develop gaps. Engineers increase scrape frequency, adding more work without addressing the rule bottleneck.
Compare rule-group evaluation duration, missed iterations, query errors, and source-series availability over the same period. Identify the expensive expression and the data growth that preceded the incident. A recent label change can increase work dramatically even when request volume remains steady.
Check raw and recorded data separately
Run the source expression at representative times using the intended query context. Confirm that the required labels and series exist. Then inspect the recorded metric and its label set. A rule rename or label change can make a dashboard query look empty while a different series is being produced correctly.
Use a small known population to verify the expression's meaning. Do not optimise a query before confirming it calculates the intended quantity. A fast rule that aggregates away the tenant or service boundary required by the alert is still incorrect.
Review group structure
Rules within a group have documented evaluation ordering and timing behaviour. Keep dependent calculations arranged according to supported semantics, and avoid assuming separate groups evaluate in a reliable cross-group sequence.
Consider splitting unrelated expensive work when appropriate, but review the resulting dependencies and resource use. Moving a rule does not remove its query cost. Measure the effect and confirm downstream rules do not begin reading an unintended prior or missing value.
Control cardinality at the source
Identify labels that create large or rapidly changing series populations. Request identifiers, unrestricted URLs, or user-specific attributes can make a recording expression expensive and the result difficult to operate. Review whether those dimensions belong in metrics.
Change instrumentation carefully. Removing labels can improve capacity but also change the meaning of existing dashboards and alerts. Document the migration and test queries against both old and new data where the transition requires it.
Treat missing data explicitly
Decide what a gap means for each alert and business report. No observed errors is different from no observations. Avoid unconditional zero-filling unless zero is a justified representation for the specific expression and absence condition.
Add an independent signal for recording-rule health where useful. This lets operators distinguish an application alert from a telemetry calculation failure. Keep the alert dependency understandable so the monitoring system does not suppress its own failure evidence accidentally.
Test rule changes before rollout
Use the available rule validation and testing tools to check syntax and expected expression outcomes. Include counter resets, missing series, label changes, and representative aggregation cases where relevant. Then load the rules in a controlled environment and observe actual evaluation duration.
A unit-level expression test does not measure production query cost. Use representative cardinality and retention when evaluating performance. Record the dataset assumptions and compare the proposed rule against the current configuration.
Quick diagnostic map
| Evidence | Interpretation | Next check |
|---|---|---|
| Recorded series has gaps | Evaluation may be skipped | Inspect missed iterations |
| Raw query also lacks data | Source boundary missing | Check source availability |
| Dashboard shows false zero | Absence was relabelled | Review display semantics |
A focused operator exercise
Build a small synthetic source population and list the expected recording-rule series and labels. Evaluate the source and recorded queries at matching timestamps. Include missing input, a label change, and a counter reset where relevant. This separates expression semantics from a rule group that failed to produce a sample at all.
Measure group duration and missed iterations under representative cardinality, then introduce one expensive test expression. Confirm the resulting gap and health signal are visible. Test the proposed correction without changing dashboard zero-filling, so the graph cannot conceal whether evaluation actually recovered. Review dependencies if rules move between groups.
After deployment, verify fresh samples, expected labels, and downstream alert behaviour. Preserve the historical incident interpretation and document any supported backfill separately. The acceptance receipt should explain the source data, expression result, evaluation schedule, and presentation choice. A visually continuous panel is accepted only when its values represent observations or a justified transformation rather than hidden telemetry failure.
Verify recovery without hiding history
After correcting the cause, confirm evaluations complete within the intended schedule and new samples appear as expected. Preserve the historical gap in incident interpretation unless a documented backfill process establishes valid replacement data.
Clouvenix can help connect Prometheus rule health with dashboard and alert semantics. The outcome is a monitoring calculation that remains observable and whose missing data is interpreted honestly.
Frequently asked questions
Should a gap always become zero in Grafana?
No. Zero can mean a measured absence of events, while a gap can mean the calculation never produced a sample. Choose the representation from the metric's semantics and keep evaluation failures visible.
Will a longer evaluation interval solve the problem?
It may reduce missed scheduling, but it also changes freshness and may leave the expensive query unexplained. Measure query cost, cardinality, and downstream expectations before selecting a new interval.
Explore Clouvenix observability monitoring for implementation and operational handover.