A healthy metrics endpoint still needs discovery and a collector
An application may expose a Prometheus-compatible endpoint without being discovered by the managed collection configuration. Even after discovery, a scrape can fail because of routing, authentication or parsing. The final workspace query is another stage with its own scope and label expectations.
Azure's managed collection uses different components for different target classes. This guide identifies the collector responsible for the affected target before inspecting logs. It does not assume that every scrape job runs in the same Pod or that an unrelated healthy collector proves the whole path works.
Setup: map the endpoint to its discovery rule
Record the cluster, namespace, target service or Pod, metrics port and path. Identify whether the target is enrolled through supported custom configuration or monitor custom resources. Confirm label selectors, endpoint names and the collection feature's supported version and authentication requirements.
A read-oriented starting inventory in an authorized cluster is:
kubectl get pods -n kube-system
kubectl get services -n example --show-labels
Use the installed monitoring procedure to identify the relevant collector and target diagnostics. Restrict any port-forward or diagnostic view to an authorized local session; do not expose a monitoring interface publicly to simplify an investigation.
Illustrative incident: the monitor selects yesterday's labels
In this illustrative scenario, an application deployment changes service labels. Its metrics endpoint remains available, but the reviewed ServiceMonitor or equivalent selection no longer matches. The managed workspace has no fresh series for the expected service.
The responder compares the actual service labels and monitor selection before changing workspace permissions. It fixes the enrollment contract and verifies target discovery, successful scraping and fresh ingestion. The fictional example does not imply that changing a selector repairs every missing metric.
Troubleshooting: follow discovery through ingestion
| Observation | Evidence | Boundary |
|---|---|---|
| Target never appears in discovery | Monitor selectors, namespace and service labels | Enrollment configuration |
| Target appears but scrape fails | Collector-owned target error and endpoint access | Route, authentication or endpoint format |
| Scrape succeeds but series is absent | Relabeling, ingestion errors and metric selection | Post-scrape filtering or delivery |
| Series exists but dashboard is empty | Workspace, labels and query window | Consumer selection |
Use the collector that owns the target type. The managed replica and node-level components do not scrape exactly the same classes of endpoints. Review any version-specific secret access rules when the selected monitor requires credentials, instead of widening Kubernetes permissions broadly.
Correct discovery with a bounded configuration change
Repair the evidenced selector, endpoint, scrape configuration or authorized credential reference. Test configuration parsing and rollout status before expecting data. Keep relabeling rules visible in review because a successfully scraped metric can still be intentionally dropped before ingestion.
Choose stable labels and control cardinality. A per-request or per-user label can make collection expensive and difficult to operate. Do not enable every optional metric as a debugging shortcut without reviewing the effect on ingestion limits and ownership of the shared workspace.
Verification, monitoring and SOP
Confirm the intended target is discovered, the responsible collector reports a successful scrape and a fresh expected series is queryable. Verify the alert expression and dashboard selectors against the resulting labels. Preserve a minimal safe test target for future collection changes.
An internal SLO can define scraping and ingestion freshness for enrolled targets; a customer-facing application SLA remains separate. The SOP should record collector ownership, discovery rules, endpoint access, version-sensitive authentication, relabeling, cardinality limits, rollback and escalation when the application endpoint is healthy but monitoring is blind.
See Clouvenix containers kubernetes for a reviewed implementation and operational handover.
Technical reference
Microsoft Prometheus collection troubleshooting, Microsoft monitor custom resources. Reviewed 19 October 2026; use documentation for the installed product and release.