Context
A Pod starts terminating while its process is still handling requests. One dashboard shows it as not ready, another shows an endpoint, and a client receives a connection reset. These observations describe different parts of the shutdown lifecycle. EndpointSlice conditions help consumers understand whether an endpoint is ready, serving, or terminating, but applications and routing components must still coordinate the actual drain.
Read the conditions as distinct signals
Kubernetes documents three EndpointSlice conditions: ready, serving, and terminating. Serving describes whether the endpoint can currently serve responses, while terminating marks a shutdown lifecycle. Ready is generally a shortcut for serving without termination, with documented exceptions such as publishNotReadyAddresses. Consumers may handle terminating endpoints differently depending on their implementation and available alternatives.
Do not reduce the state to “listed means receiving traffic.” The Service, proxy, ingress, service mesh, and external load balancer can each have their own observation and update timing. Identify which component chooses the destination for the failing client before interpreting EndpointSlice data as a routing guarantee.
An illustrative rolling-update reset
Imagine an API whose process exits immediately on termination. Kubernetes updates endpoint information, but a routing component has not yet stopped all traffic to the old Pod. A few requests reach a listener that is already gone. Adding replicas increases spare capacity but does not coordinate the retirement sequence.
Build a timeline from Pod deletion, EndpointSlice updates, application termination signals, active requests, and proxy errors. Keep Pod UIDs and endpoint addresses so the old and replacement instances are not confused. This timeline reveals whether the application exited too early or a different routing layer retained stale information.
Map the complete traffic path
List the client, external load balancer, ingress or gateway, Service handling, sidecars, and application listener. Record which components watch EndpointSlices directly and which receive translated configuration from another controller. This determines where to collect evidence.
Test the actual production-like path. A request sent directly to a Pod IP bypasses several routing layers and cannot prove that a browser request through the gateway drains correctly. Direct tests are still useful when labelled as lower-layer checks rather than end-to-end acceptance.
Align application shutdown
The application should handle termination through its runtime's supported graceful shutdown mechanism. Stop accepting new work where appropriate, allow active bounded operations to finish, and exit before the platform's final deadline. Review sidecar shutdown behaviour if the Pod contains a proxy or other cooperating process.
Avoid relying on an arbitrary delay as the complete solution. A short wait may reduce a race but cannot prove active requests are complete. Track in-flight work and define a bounded fallback for requests that cannot finish. Long-running tasks may need durable job ownership rather than residence in a retiring web process.
Understand readiness during retirement
Readiness expresses whether the application should receive traffic through the selected mechanism, but it is not a kill switch with instantaneous global effect. Review how your routing stack interprets serving and terminating conditions, including fallback behaviour when alternatives are unavailable.
Do not make liveness checks fail simply to force a shutdown. Liveness and readiness have different purposes, and an artificial restart can complicate a controlled rollout. Keep planned retirement visible as such in logs and metrics so it is not confused with an unhealthy process crash.
Test persistent connections separately
Existing TCP, streaming, or WebSocket connections may behave differently from newly established requests. Determine which component owns them and what happens when its selected endpoint retires. A successful short-request test does not establish session continuity.
Give clients a defined reconnect and resynchronisation path where connections are expected to end. For commands with side effects, reconcile uncertain outcomes with stable operation identifiers. A disconnected client should not assume every unacknowledged operation failed and replay it blindly.
Rehearse under realistic timing
Create labelled traffic containing short requests, one slow request, and a persistent connection. Perform a rolling update and record the endpoint selected for each operation. Confirm new work moves to replacements and existing work follows the intended finish or recovery contract.
Repeat with reduced spare capacity and a slower routing update in a controlled environment. Observe whether the deployment stops safely or produces widespread errors. Keep the test's concurrency, application release, routing versions, and shutdown settings in the record so results remain interpretable after upgrades.
Quick diagnostic map
| Evidence | Interpretation | Next check |
|---|---|---|
| New requests hit retiring Pod | Routing transition incomplete | Correlate endpoint updates |
| Process exits with active work | Shutdown contract incomplete | Inspect termination timeline |
| Persistent clients lose progress | Recovery state missing | Test resynchronisation |
A focused operator exercise
Capture the selected Pod UID, endpoint address, readiness state, and EndpointSlice conditions while generating labelled requests through the actual ingress path. Start a controlled rollout and record deletion, routing updates, application signal handling, and exit. Keep the old and new endpoint identities distinct throughout the record so a replacement cannot be mistaken for continuing service.
Include a bounded slow request and a persistent connection. Verify the former follows the intended completion contract and the latter reconnects or recovers through its documented protocol. Compare the behaviour with a direct Pod request only as a lower-layer diagnostic; it bypasses the traffic consumers whose update timing is part of the test.
Repeat with reduced spare capacity in a representative environment and confirm the rollout's stop condition protects the service. The receipt should identify forced exits, resets, accepted operations, and recovery outcomes. This demonstrates how the installed routing stack interprets endpoint lifecycle information instead of assuming the presence of a condition produces instantaneous global traffic movement.
Monitor the release boundary
Track resets, gateway errors, active requests at exit, forced termination, and reconnect success around deployments. Correlate them with Pod and EndpointSlice lifecycle events. A small recurring error spike at every release deserves investigation even when daily availability looks acceptable.
Clouvenix can help review EndpointSlice interpretation alongside the full routing stack and application shutdown. The deliverable is a tested retirement sequence whose behaviour is understood from client request to process exit.
Frequently asked questions
Does ready=false mean the process has stopped?
No. The process may still be serving existing work while it retires, or it may be alive but unsuitable for new traffic. Inspect lifecycle and serving state separately, then verify how the actual routing component interprets those signals.
Can a preStop delay solve every deployment reset?
A delay can address part of a timing gap, but it does not replace graceful application shutdown, sufficient capacity, or routing verification. Test active requests and persistent connections across a rollout instead of choosing a delay without evidence.
Explore Clouvenix containers kubernetes for implementation and operational handover.