A missing measurement is a separate outcome
Grafana alert evaluation can return a threshold result, no usable data, or an execution error. No Data means the query did not provide the expected values. Error means evaluation failed, for example because a data source timed out. Neither outcome automatically establishes that the monitored service is healthy or unavailable.
The rule needs an explicit policy for these states. Setting missing data to Normal can hide a collection failure; treating every brief interruption as an immediate page can create noise. The useful choice depends on signal frequency, service impact and independent visibility into collection health.
Setup: document the evaluation contract
Record the query, relative time range, evaluation interval, pending period, expected sample cadence and notification owner. Decide whether the metric is continuous or legitimately sparse, such as a counter produced only when a scheduled job completes.
Review the installed Grafana version's behavior. Current documentation describes No Data and Error handling that can create independent DatasourceNoData and DatasourceError alert instances. Those instances include data-source and rule identifiers and can have labels different from the original service alert. Existing notification policies and silences may therefore match differently.
Investigate the evaluation path
- Run the exact rule query against the same data source, time range and identity. A dashboard viewed under a different user or a longer time window is not an equivalent test.
- Distinguish an empty result from an execution error. Inspect query responses, evaluation history, data-source connectivity and credentials without copying secret values into the incident record.
- Compare the chosen No Data and Error behaviors with the pending period. Determine whether an interruption is waiting, firing as a separate alert, held in its previous state or deliberately mapped to Normal.
- Inspect the resulting alert instance labels and match them against the notification policy. Confirm the route for collection failures as well as the route for an application threshold breach.
Illustrative incident: a silence misses the data-source alert
In this illustrative scenario, an engineer silences the service's latency alert during a planned data-source upgrade. Evaluation begins failing, and Grafana creates a separate data-source error instance. Its labels do not match the service silence, so another responder receives notifications that the maintenance owner did not anticipate.
The review documents the independent alert family and gives it a deliberate infrastructure route. It also defines maintenance suppression for the data-source operation separately from application failures. This is a teaching scenario and contains no claim about actual alert volume or customer availability.
Diagnostic table
| Observation | Evidence | Interpretation |
|---|---|---|
| Dashboard has data but rule is empty | Exact query identity and time range | Different evaluation context |
| Alert name becomes DatasourceError | Evaluation logs and generated labels | Query failure instance |
| Missing samples appear healthy | Configured No Data handling | Policy may map absence to Normal |
| State never changes during an outage | Keep Last State setting and duration | Prior state is retained |
Correct and verify the policy
Repair the evidenced query, identity, timing or transport failure. Adjust the state policy only after its intended meaning is clear. Expanding a query window can accommodate sparse samples, but an excessive range may increase execution cost and still leave collection failures unowned.
Use a test data source or controlled query to exercise normal data, an empty result and a deliberate error. Observe pending behavior, generated labels, routing and recovery. Verify the actual receiver with an approved test integration rather than assuming a green configuration screen proves delivery.
SOP and monitoring
Record the data-source owner, expected cadence, missing-data meaning and escalation path. If Keep Last State is used to tolerate brief disruptions, add an independent way to detect a prolonged loss of monitoring. Review these policies when changing evaluation intervals, query backends or rule provisioning.
See Clouvenix observability monitoring for implementation and operational handover.
Technical reference
Official reference 1. Documentation checked 19 October 2026; confirm behavior for your deployed edition and version.