← All insights
ENGINEERING JOURNAL / Kubernetes

Kubernetes Probes: Separate Readiness from Recovery

Design Kubernetes startup, readiness and liveness probes around different failure modes. Test traffic removal and recovery before a rollout.

KUBERNETES / ORCHESTRATION
Kubernetes Probes: Separate Readiness from Recovery — Clouvenix engineering illustration
CHAPTER

Context

A slow-starting application and a deadlocked application need different responses. Restarting both immediately can turn a manageable startup delay into a continuous restart loop. Probe design should begin with an explanation of the application states that operators need Kubernetes to distinguish.

CHAPTER

Describe what each probe means

Use readiness to express whether a container should receive service traffic. Use liveness for a condition that restarting the container can reasonably repair. A startup probe can protect initialization before the other probes take over. These are separate decisions. Avoid defining every endpoint as a broad check of every downstream system. A temporary shared database outage should not automatically cause every otherwise healthy application process to restart together.

CHAPTER

Give initialization a measured budget

Measure application startup with cold caches, realistic configuration and the expected data size. Include the longest legitimate initialization path rather than copying timing values from another service. Select probe intervals, timeouts and thresholds deliberately, then record the assumptions in the deployment review. If initialization time changes after a release, investigate the change before repeatedly increasing thresholds. A larger budget can conceal an initialization regression as easily as it can accommodate legitimate work.

CHAPTER

Exercise traffic and failure behavior

In a test namespace, simulate a condition that should fail readiness while leaving the process alive. Confirm that requests are routed away as intended and that the container can become ready again. Separately simulate a recoverable stuck process and check the liveness response. Observe events, restart counts and request errors together. A green endpoint alone is not enough evidence if users still receive failures during the transition.

CHAPTER

Review the rollout as a user journey

Run a small synthetic transaction through the same ingress and service path users rely on. Check what happens while one replica starts, one becomes unready and an old replica terminates. Document dependency assumptions and who changes probe settings during an incident. Keep health responses small and avoid exposing configuration secrets. The final handover should explain each probe's purpose, its timing budget, expected recovery behavior and the evidence from the failure exercises. Revisit that contract whenever startup tasks or dependency handling changes.

Clouvenix can review the scope through Kubernetes engineering, with acceptance checks agreed before implementation.

Technical reference

Vendor documentation. Confirm behavior against the version and services used in your environment.