Context
Spot Instances provide interruptible compute capacity. A durable workload must survive losing an instance, with interruption notices helping it reduce unfinished work. Design recovery before implementing a notice handler. A shutdown script that works only when it receives ample warning leaves the application vulnerable to sudden failure or a signal it never processes.
Define what must survive the instance
Record the workload, purchase mix, interruption behavior, job source, checkpoint store and replacement mechanism. Identify the smallest unit of work that can be retried safely. Keep important state outside instance-local storage and describe how another worker discovers ownership and progress.
Use a representative test workload with synthetic data. Set an acceptance criterion such as completing every accepted job once at the business-effect boundary, while allowing repeated attempts internally. Assign an owner for jobs that cannot be reconciled automatically.
1. Make progress durable during normal execution
Write checkpoints at meaningful boundaries throughout processing instead of saving everything only during shutdown. Record enough state to resume safely, including input identity and the last durable output. Coordinate the checkpoint with business effects so a replacement can distinguish committed work from an uncertain attempt.
Test restart behavior before adding interruption logic. Terminating a test worker unexpectedly should cause recoverable work to become available to another worker under the documented ownership rules. A notice handler cannot repair an ambiguous checkpoint format.
2. Handle signals as bounded opportunities
AWS exposes Spot interruption warnings through instance metadata and EventBridge. The usual stop or terminate warning is about two minutes; hibernation starts immediately without that normal interval. Signals are best effort. Rebalance recommendations can arrive earlier, but not necessarily before the interruption warning.
On a relevant signal, stop accepting new work, preserve completed progress and release or record outstanding ownership according to the queue or scheduler contract. Make the handler idempotent because local and central signal paths can refer to the same instance. Keep its actions bounded by the available time.
3. Measure draining and replacement evidence
| Evidence | Risk | Corrective direction |
|---|---|---|
| Checkpoints exist only at shutdown | Missing notice loses progress | Persist during normal processing |
| Replacement starts from the beginning | Resume state is incomplete | Define checkpoint contract |
| Two workers commit the same effect | Ownership or idempotency gap | Add durable operation identity |
| Rebalance replacement is unavailable | Capacity diversity insufficient | Review instance types and zones |
An explicitly illustrative scenario: a video worker processes long inputs on Spot. It stores completed segment identifiers remotely as it runs. On a notice, it stops taking new videos and records the current segment's status. A replacement resumes only uncommitted segments. A separate abrupt-stop test verifies the same recovery path without relying on the warning handler to finish.
Correct and verify under interruption
Exercise both a controlled notice path and unexpected worker loss in a test environment. Confirm that accepted jobs remain discoverable, checkpoints are readable and downstream outputs are reconciled. Measure replacement time separately from drain time; a good shutdown cannot create unavailable capacity.
For Auto Scaling fleets, review Capacity Rebalancing, instance-type diversity and Availability Zones. Additional replacement capacity still needs application readiness checks. Avoid assuming a lifecycle hook can extend a Spot interruption deadline.
The SOP should record signal ownership, checkpoint frequency, retry rules, capacity alternatives and reconciliation steps. Monitor unfinished jobs and checkpoint age alongside fleet capacity. Repeat recovery tests when processing stages or external effects change so Spot remains an intentional execution choice with demonstrated recovery behavior.
Explore Clouvenix cloud architecture for implementation and operational handover.
Technical reference
Interruption warnings, rebalance recommendations, and Capacity Rebalancing explain available signals and fleet behavior.