Context
A dashboard can show high CPU while the Horizontal Pod Autoscaler does not add replicas. That is not necessarily a contradiction. The dashboard and the HPA may use different data sources, time windows, containers, or denominators. Before lowering the target percentage, establish which metric the controller is actually using and whether it can calculate a valid recommendation.
Imagine an illustrative API with a CPU utilization target. A new sidecar is added without the expected resource configuration, and the HPA begins reporting an unknown metric. The application still consumes CPU, but the controller's input contract changed. More aggressive targets cannot repair an unavailable input.
Understand the utilization denominator
For a resource utilization target, HPA compares observed usage with the relevant resource requests. The controller also considers missing metrics and Pod readiness conservatively. Multiple metrics, stabilization, scaling policies, and replica bounds can affect the final decision. Raw usage targets and utilization targets are different configurations. 1
Write down the metric type, target, selected Pods, and resource requests. Confirm the target resource and selector match the intended workload. A generic CPU graph covering an entire namespace may include work the HPA does not scale.
Read status conditions and events first
Inspect the HPA's current and desired replicas, metric values, conditions, and recent events. Look for evidence of failed metric retrieval, a limited recommendation, or a deliberate stabilization decision. Preserve the condition reason and timestamp rather than relying on a screenshot of the headline replica count.
Then inspect the target workload and actual Pods. Compare the current template with running revisions and admission changes. If a sidecar or policy adds resource settings, use the effective values. The controller operates on the live objects, not the manifest an engineer remembers submitting last week.
Trace the metrics pipeline end to end
For resource metrics, verify that the relevant metrics API can supply recent values for the selected Pods. For custom or external metrics, inspect the adapter, query, labels, permissions, and data freshness. A healthy monitoring backend does not automatically imply a healthy Kubernetes metrics adapter.
Check the scope of missing data. One newly started Pod is different from every Pod losing metrics. Compare timestamps with a deployment, metrics component change, or network policy update. Avoid treating missing data as zero usage; that can lead to an unsafe manual reduction in replicas during an observability outage.
Reconcile CPU math with a small example
Suppose a container requests 500 millicores and uses 250 millicores. Its utilization relative to that request is 50 percent. That illustrative calculation is different from 250 millicores as a fraction of a node's total CPU. A dashboard using the latter denominator can show a different percentage without either calculation being wrong.
Use the actual configured metric and container scope to perform the comparison. If requests are unrealistic, changing them affects both scheduling and utilization math. Review capacity and resource policy together rather than adjusting requests only to make the autoscaler produce a desired number.
Inspect readiness and the path to useful capacity
New replicas are not immediately useful if they spend several minutes loading data or repeatedly fail readiness. Observe the interval from a scale recommendation to a Ready Pod handling requests. Include scheduling delay, image pull, startup, and application warm-up.
If the workload is limited by a database or external API, more replicas may increase pressure without increasing throughput. Measure completed requests, pool wait time, and downstream errors alongside CPU. Autoscaling should support a useful business capacity model, not merely chase one resource graph.
Review deliberate scaling limits
Check minimum and maximum replicas, scale-up policies, scale-down stabilization, and any other controller or deployment process writing replica counts. A GitOps configuration that repeatedly restores a fixed replica number can interfere with the intended operating model. Establish which component owns scale in steady state.
Distinguish “controller wants more replicas” from “cluster can schedule more replicas.” If desired replicas rise but Pods remain Pending, move the investigation to placement and node capacity. Raising the HPA maximum again will not create nodes in an unavailable topology domain or resolve an incompatible storage constraint.
Validate with a controlled load pattern
Use a gradual load increase that resembles real requests and stays within approved limits. Record metric values, recommendations, actual replicas, Ready replicas, latency, and successful throughput on one timeline. Hold the load long enough to observe the configured control behaviour rather than expecting instantaneous response.
Then reduce load and observe stabilization. A test that proves only scale-up can miss rapid oscillation or a scale-down policy that removes useful warm capacity too soon. Keep the test input reproducible so the team can compare the next release with the same baseline.
A focused operator exercise: A metrics-loss acceptance exercise
Run a bounded load that normally causes a scale recommendation, then make the relevant metric unavailable in a controlled test environment. Observe HPA conditions, reported values, recommendations, and the application's customer-like latency. Restore the metric and verify the controller returns to the documented operating behaviour without an unrelated configuration change.
Compare this with a separate test where metrics remain available but new Pods cannot schedule. The first case is an input problem; the second is a capacity-delivery problem. Recording both timelines helps responders avoid repeatedly lowering targets when the actual failure is missing data or unavailable placement, and gives the runbook a useful branch point.
Quick diagnostic map
| Evidence | Review | Acceptance boundary |
|---|---|---|
| Metric path | Trace collection to the autoscaler | Diagnose absent or stale samples |
| Resource requests | Inspect the admitted Pod requests | Use the intended utilization denominator |
| Ready capacity | Observe scheduling and startup delay | Separate desired replicas from serving replicas |
Frequently asked questions
Does high CPU always mean the HPA is broken? No. Verify metric scope, resource request math, readiness, and configured limits first.
Clouvenix can review the autoscaling input and the application's actual capacity path, producing a clear explanation of whether the next fix belongs in metrics, resource configuration, scheduling, or the application itself.
For a scoped review of this workflow, discuss the relevant Clouvenix engineering service.