Context
A Kubernetes cluster can have free CPU and memory while a new Pod remains Pending. The scheduler is trying to satisfy more than capacity. It may also need to preserve a distribution across zones or nodes, respect affinity, tolerate taints, and meet storage placement requirements. Free resources in the wrong location are not necessarily usable resources.
Consider an illustrative three-zone application during a rolling update. Two zones have spare nodes, while the third is full. The new replica cannot be placed without violating the configured spread rule. Removing the rule may complete the rollout, but it also changes the availability design. First establish which constraint is actually blocking placement.
Read the spread contract precisely
Topology spread constraints describe how matching Pods should be distributed across topology domains. Fields such as topologyKey, labelSelector, maxSkew, and whenUnsatisfiable influence scheduling. A hard DoNotSchedule constraint can prevent placement, while ScheduleAnyway expresses a softer preference. Eligible-domain calculations and optional fields depend on the configuration and supported Kubernetes version. 1
Translate the YAML into a small table: domain, matching Pod count, available capacity, and eligibility. This makes the decision understandable without guessing from the total replica count. Include node labels and the incoming Pod's resource requests in the table.
Inspect the actual Pending Pod
Read scheduling events for the exact Pod and preserve the latest relevant failure message. Inspect the admitted Pod specification, since controllers or admission policies may have changed what was submitted. Compare node affinity, tolerations, resource requests, PVC topology, and every spread constraint together.
Do not assume the first mentioned constraint is the only one. A node can fail several checks simultaneously. If capacity is added to a zone but the Pod still cannot run, examine the next reported constraint rather than declaring the scheduler inconsistent. Each candidate node must satisfy the required combination.
Verify which Pods the selector counts
A selector that is too broad can count unrelated replicas, while a selector that misses the application's Pods can produce a misleading distribution. Inspect live labels on old and new revisions. Make the intended grouping explicit: all replicas of the service, one revision, or another supported boundary.
Label changes during releases deserve review. A chart update may alter selectors or add revision-specific matching behaviour. Test the resulting spread across a full rollout, not just the first new Pod. The desired steady-state distribution and the temporary distribution during replacement can differ substantially.
Include rollout surge in the capacity plan
A Deployment may need extra replicas while old ones remain available. That temporary capacity must exist in eligible domains, not merely somewhere in the cluster. Compare maxSurge, maxUnavailable, readiness time, and termination duration with the node capacity reserved for the rollout.
For an illustrative six-replica application, the team might have two replicas per zone at steady state. A rollout that needs an additional replica may still stall if the only eligible placement for that replica is full. The exact outcome depends on counts and constraints, so simulate the actual configuration rather than relying on a generic three-zone picture.
Distinguish capacity repair from policy relaxation
If the policy expresses a required failure-domain objective, add or free suitable capacity and verify the node labels. Check whether the cluster autoscaler's node-group design can supply the domain that is missing. Scaling a node group in another zone will not necessarily help a hard placement requirement.
If the business permits a temporary availability tradeoff, document that decision and change the specific constraint deliberately. Record what failure exposure increases, how long the exception lasts, and who restores the intended policy. Avoid turning an emergency softening into a permanent configuration drift that nobody remembers.
Test a zone loss and a normal rollout separately
A configuration that spreads well during normal operation may behave differently when a domain is unavailable. Test the intended failure case in a safe environment and observe replacement Pods, capacity, and customer-like requests. Verify whether the design prefers waiting for the missing domain or continuing in the remaining domains.
Do not infer application availability solely from placement. Database quorum, storage access, and external dependencies may determine the real outcome. The scheduling exercise should be one part of an application recovery test with measurable request success and recovery timing.
Keep the operational view domain-aware
Monitor Pending duration, scheduling failure reasons, available capacity by domain, and replica readiness. A cluster-wide utilization graph can hide the imbalance that matters. Keep a view of required node labels so a mislabeled replacement node is visible before the next rollout.
Record the diagnosed constraint, the domain-count table, the selected fix, and the successful rollout result. That evidence is more reusable than a note saying “added nodes.” It explains why those nodes needed to exist in that location and which availability objective they support.
A focused operator exercise: A constrained-zone rollout exercise
In staging, use a replica count and node layout representative of production, then remove spare capacity from one eligible test zone without disrupting unrelated workloads. Start the intended rolling update and capture the Pending Pod's events, matching replica counts by zone, and effective resource requests. Verify the diagnosis identifies the constrained domain.
Restore suitable capacity in that domain and observe rollout progress without changing the availability rule. If the business instead chooses a policy relaxation, repeat the exercise and record the resulting placement and failure exposure. The comparison makes the capacity-versus-availability decision concrete rather than treating every successful schedule as an equivalent outcome.
Quick diagnostic map
| Evidence | Review | Acceptance boundary |
|---|---|---|
| Eligible domains | Inspect node labels and scheduling constraints | Count usable topology domains |
| Replica match | Check label selectors and current placement | Include terminating and pending workload context |
| Rollout capacity | Model surge and unavailable replicas | Preserve capacity for a replacement |
Frequently asked questions
Should we switch every hard constraint to ScheduleAnyway? Only when the resulting availability tradeoff matches the application's requirements.
Clouvenix can review placement policy and release capacity together, helping teams keep deployments moving while preserving an explicit, tested failure-domain design.
For a scoped review of this workflow, discuss the relevant Clouvenix engineering service.