Context
A health check is also an instruction about what to do when it fails. In Kubernetes, a failed liveness check can restart a container, while a failed readiness check changes whether it should receive normal service traffic. Using one deep dependency check for both can turn a temporary database slowdown into a fleet-wide restart storm.
Imagine an illustrative Java service that takes several minutes to warm its caches. A short liveness deadline kills it before initialization finishes, so it starts again and repeats the same work. The container is making progress, but the health policy interprets slow startup as a dead process. The fix begins by separating three different questions.
Give each probe a distinct purpose
Kubernetes provides startup, readiness, and liveness probes. A startup probe can protect slow initialization by delaying the other probes until startup succeeds. Readiness determines whether a container is ready to serve, while repeated liveness failure can trigger restart according to the configured behaviour. Probe type, thresholds, timing, and application semantics all matter. 1
Define the action first. Ask whether a failure should stop new traffic, restart the process, or simply produce an alert. A restart should have a plausible path to improve the condition. If the only failed component is a shared database, restarting every application replica may make recovery harder.
Design a meaningful startup boundary
Measure startup under realistic cold conditions, including image changes, empty caches, and dependency latency. Determine which initialization must finish before the application can serve safely. Set a startup budget with a deliberate upper bound rather than an arbitrary large delay that hides a permanently broken release.
Expose progress in logs or metrics so operators can distinguish slow progress from a stuck initializer. A startup probe that eventually fails should leave enough evidence to identify the blocked phase. Avoid placing real customer work inside a probe just to make it representative; health checks must remain safe to repeat frequently.
Keep liveness focused on restart-recoverable failure
A useful liveness check can identify a process that is alive at the operating-system level but unable to make required internal progress. Its implementation should be bounded and should not itself consume the same exhausted resource it is trying to observe without careful design.
Do not make liveness depend on every external service by default. A recommendation API outage does not necessarily mean the checkout process should restart. Decide which failure conditions indicate a local process defect and which should instead affect readiness or feature-level degradation. Document the reasoning so future engineers do not broaden the check casually.
Make readiness match the traffic contract
Readiness should answer whether this instance can accept the work it will receive. It may include critical initialization and a carefully chosen dependency condition. A deep check that waits on many services can add load and create synchronized readiness changes, so measure its cost and failure behaviour.
Consider partial capability. If the application can serve read-only traffic during a write dependency outage, one global boolean may be too coarse for the product architecture. The routing and application design must express that distinction deliberately. A readiness endpoint alone cannot create a safe degraded mode that the application does not implement.
Diagnose the current restart loop with evidence
Inspect Pod events, previous container logs, termination reasons, probe failures, and application startup timestamps. Separate probe-triggered restarts from out-of-memory kills, process crashes, and node problems. These may look similar in a restart counter but need different remedies.
Compare one failed instance with a healthy instance on the same release. Check resource limits, node pressure, initialization data, and configuration. If only cold instances fail, examine startup timing. If established instances fail during a dependency incident, review whether the probe is converting a shared failure into unnecessary local restarts.
Test failure modes one at a time
In a controlled environment, test slow startup, a local deadlock or equivalent safe failure simulation, an unavailable critical dependency, and an unavailable optional dependency. Record expected probe state, restart behaviour, traffic admission, and customer-like request outcomes for each case.
The acceptance matrix should show that a slow but progressing startup is allowed within its budget, a genuinely unrecoverable local process is restarted, and a shared outage does not create uncontrolled churn. Keep the application owner's approval for what each simulated failure means; the platform team cannot infer business readiness from a port opening alone.
Coordinate probes with deployment and shutdown
Probe settings affect rollout timing and available capacity. A new instance should not receive traffic before it can serve, and a terminating instance needs a separate graceful shutdown contract. Review readiness transitions, request draining, and termination deadlines together without assuming one probe solves the entire lifecycle.
An illustrative rollout may need extra capacity while new instances warm. If the cluster cannot supply that capacity, lengthening the startup allowance may expose an existing release-capacity gap. Record the dependency rather than declaring the probe change complete simply because the container stopped restarting.
A focused operator exercise: A probe-action acceptance matrix
Use a test application with controllable startup delay and a harmless simulated dependency failure. For slow startup, verify the process receives its intended initialization window. For a local unrecoverable condition, verify the liveness path produces the expected recovery. For an unavailable optional dependency, verify the application follows its documented degraded mode without unnecessary restarts.
Record probe state, container restarts, Ready capacity, and customer-like responses for each case. Repeat during a rolling update so the team can see the interaction with temporary capacity. The matrix should make the reason for each probe explicit and identify any case where the configured action makes the original condition worse.
Quick diagnostic map
| Evidence | Review | Acceptance boundary |
|---|---|---|
| Startup gate | Measure representative initialization time | Delay liveness until startup can finish |
| Readiness signal | Exercise serving dependencies | Remove traffic without needless restarts |
| Liveness action | Test a recoverable process failure | Avoid restarting on every downstream outage |
Frequently asked questions
Can the same endpoint serve every probe? Sometimes, but only when its meaning and failure action are appropriate for each use. Separate endpoints often make the contract clearer.
Clouvenix can review probe semantics alongside application startup and deployment behaviour, helping teams replace restart loops with measurable, intentional recovery actions.
For a scoped review of this workflow, discuss the relevant Clouvenix engineering service.