Context
A VM Scale Set rolling upgrade replaces or updates instances in batches and uses health information to decide whether it should continue. A paused batch can therefore be a protective response to an application that has not become healthy. It can also reveal a health endpoint that no longer matches the deployed image.
Before investigation, identify the scale set's orchestration mode, upgrade operation and health monitoring source. Obtain access to instance views, extension status and application logs. Preserve the current batch state so that subsequent retries do not obscure which image and configuration failed first.
1. Locate the failing health decision
Read the rolling upgrade status and compare unhealthy instances before and after the latest batch. These are different policy checks: an unhealthy population that already existed can block progress, while failures introduced by the upgraded batch indicate a release problem. Record instance identifiers and image versions rather than assuming every instance runs the latest model.
Read the configured batch size and unhealthy thresholds. Do not raise those limits until you understand the affected application capacity. A threshold change can permit more unhealthy instances without repairing the reason they are unhealthy. Include the scale set's ordinary traffic and repair behavior in that assessment.
2. Inspect the local health contract
For the Application Health extension, check its protocol, port, path and status on one affected instance. It probes from inside the VM. A successful request through the public load balancer is useful application evidence, but it does not prove that the local endpoint configured in the extension is reachable.
Confirm that the endpoint is listening after startup and returns the expected status and, for rich health states, the required body. Check initialization grace against actual application startup work. Unknown is meaningful evidence: an unreachable endpoint or invalid response can produce it, and rolling upgrades treat it as unhealthy. Review whether an external health probe is also configured; orchestration should use a single supported health source.
3. Correct the cause at the smallest scope
Compare the affected image with a known healthy instance. Look for renamed routes, changed bind addresses, certificate behavior, missing configuration and unavailable startup dependencies. A health endpoint should represent readiness for the instance's required role without depending on an optional service that would unnecessarily stop the whole fleet's upgrade.
Test a corrected image or extension configuration in the release environment before applying it to production. If the release itself is broken, use the established rollback process. If initialization is legitimately longer, adjust the reviewed grace and batch plan based on observed startup rather than disabling health reporting.
Diagnostic checkpoints
| Observation | Likely cause | Evidence to collect |
|---|---|---|
| Unknown on new instances | Missing or invalid probe response | Local endpoint and extension logs |
| Old instances also unhealthy | Existing fleet issue | Pre-upgrade baseline |
| Healthy locally, wrong application output | Weak health contract | Business readiness test |
| One batch never settles | Startup or dependency failure | Boot and application timeline |
Illustrative scenario
A hypothetical reporting service changes its local status route during an image release. User traffic reaches a healthy process, but the extension continues probing the removed route. The first batch becomes Unknown and the rollout pauses. Comparing the route contract with the extension settings identifies the mismatch. This illustrates a diagnosis, not a claim about a real deployment.
Verify and maintain the upgrade SOP
Acceptance means the affected instance becomes healthy through the configured orchestration signal, serves its representative business request, and remains healthy through the observation period. Resume through the documented upgrade workflow and inspect the next batch before declaring the release complete.
Keep health endpoint ownership, expected startup duration and rollback criteria in the upgrade SOP. Alert on persistent Unknown or Unhealthy states independently of deployment failures. Review the contract whenever startup, ports or dependencies change so the next image release does not discover an obsolete health check.
Explore Clouvenix managed infrastructure for implementation and operational handover.