A successful interactive query does not reproduce the alert automatically
An operator can find an event in Logs while a log search alert does not fire. The interactive search may use a different scope, time range or aggregation from the saved rule. A useful investigation reconstructs the rule's actual evaluation instead of comparing two superficially similar screens.
Log search alerts evaluate query results on a configured schedule and apply a condition. Alert state and notification routing are later concerns. This guide diagnoses evaluation and state behavior; if an alert instance already exists, investigate its action path separately rather than repeatedly changing the query.
Setup: preserve the complete rule contract
Record the rule resource, query, evaluation frequency, lookback, aggregation, dimensions and threshold. Identify the evaluated resource scope and any required access identity. Confirm whether the chosen alert behavior is stateful or stateless and how the operator expects repeated failures to appear.
A reproducibility worksheet is:
Exact evaluated resource scope:
Saved query and aggregation:
Event timestamp versus arrival timestamp:
Evaluation frequency and lookback:
Condition, dimensions and alert-state behavior:
Run the equivalent query against the same scope and period. A query over the last day is not evidence about a short evaluation window several minutes earlier. Preserve the time at which a relevant record became queryable.
Illustrative incident: late-arriving records miss a narrow assumption
In this fictional scenario, application events arrive later than the alert designer expected. An investigator searches a broad period and sees failures, while the saved evaluation's narrower condition did not include the same usable result at its evaluation time.
The team compares event time, arrival behavior and the rule window before changing thresholds. It adjusts the monitoring design through a controlled test, accounting for cost and repeat-alert semantics. It does not claim that every missed log alert is an ingestion-delay problem.
Troubleshooting: reproduce the saved evaluation
| Observation | Evidence | Question |
|---|---|---|
| Interactive query finds data but rule does not | Exact scope, time range and aggregation | Are both evaluations actually equivalent? |
| Rule is unhealthy or degraded | Rule health and query validation | Can the service execute the saved query? |
| First incident alerts but continuing events do not create expected repeats | Stateful/stateless configuration and instance state | Is the behavior intentionally stateful? |
| Alert exists but nobody receives it | Alert instance and action-group evidence | Has evaluation already succeeded? |
Resource dimensions can split one query result into separate evaluated series. Confirm the intended grouping and identify missing or changed dimension values. A syntactically valid query can still ask the wrong operational question, especially if the underlying log schema changed.
Correct one part of the alert contract
Repair the evidenced scope, query, access or window issue. Validate the query against the restrictions of the chosen evaluation frequency instead of assuming every interactive query is eligible unchanged. Treat stateful behavior as a design choice with a recovery condition, not an accidental notification failure.
Keep a controlled test case with known input, expected evaluation and expected state transition. Do not widen windows or lower thresholds indefinitely until an alert happens. That can create duplicate or misleading incidents without explaining why the original design failed.
Verification, monitoring and SOP
Verify a failing condition and its recovery in an approved test scope. Record input timing, evaluated result, alert instance and notification as separate acceptance points. Check that an empty result has a deliberate meaning; absence of a record is not automatically proof of health.
An internal SLO can track timely, correct alert evaluation for supported data paths; an external response SLA needs separate terms. The SOP should record rule ownership, query schema, time assumptions, state behavior, action routing, cost limits, regression cases and escalation when the collection path itself is unavailable.
See Clouvenix observability monitoring for a reviewed implementation and operational handover.
Technical reference
Microsoft log-alert troubleshooting, Microsoft log search rule creation. Reviewed 19 October 2026; use documentation for the installed product and release.