Context
A Jenkins agent disappears during deployment. The pipeline retries its stage on another agent and reports success, but the remote server received the deployment twice. Retrying infrastructure failures can be useful, yet the retry boundary may include actions that already completed outside Jenkins. Recovery needs a durable record of the candidate and external operation, not just a fresh workspace.
Understand what can resume
Jenkins documents retry conditions for agent-related failures and nonresumable steps. Some pipeline steps can survive controller restart, while others cannot resume and need an appropriate retry strategy. Review the installed plugin behaviour and current step reference. Do not assume every failed step is safe to replay simply because the infrastructure caused the interruption.
A durable pipeline and a durable business operation are different guarantees. Jenkins may retain the execution graph while a remote command has an uncertain outcome. Conversely, an agent workspace can disappear while the artifact exists safely in a registry. Separate these states when designing recovery.
An illustrative lost acknowledgement
Imagine a stage invoking a remote deployment script. The script installs the release and restarts the service, but the agent disconnects before Jenkins receives the exit status. A broad retry repeats the whole stage, including a data migration that was not designed to run twice.
Before retrying, query the deployment target using a stable operation identifier. Determine whether the release is absent, complete, failed, or still in progress. The target should retain enough state to answer after the original SSH session and agent workspace are gone.
Define stage boundaries around evidence
Separate artifact preparation, validation, publication, deployment, and acceptance. Persist the candidate digest and required reports outside a disposable workspace through supported artifact storage. A replacement agent should consume the same candidate where the release contract requires it.
Avoid rebuilding a different artifact under the same release identity during recovery. If a rebuild is necessary, record it as a new candidate with its own validation. The pipeline's original source revision is useful context but does not freeze every dependency or base image.
Use conditional retries deliberately
Apply retry conditions to failures they are intended to handle. An application test failure, permission denial, and lost agent are different categories. Repeating a deterministic test failure several times can waste capacity and hide the first useful evidence.
Keep the retried block as small as practical while retaining a complete safe unit of work. A checkout or temporary download may be straightforward to retry; an external migration or release publication needs explicit idempotency or reconciliation. Review every command in the block rather than judging safety from the stage name.
Make external operations identifiable
Pass a stable release or operation identifier to the remote action and store its progress durably. The action should recognise an already completed operation and verify the intended state instead of repeating blindly.
For uncertain partial work, define a recovery action. A migration may need to inspect schema state, while a file deployment may verify checksums and active symlinks. Do not use one generic “already ran” file to skip every future attempt without checking whether the desired state is actually present.
Reconstruct the agent environment
A new agent needs the required toolchain, credentials, network access, and workspace inputs. Record the execution image or configuration so recovery does not depend on an undocumented long-lived worker.
Verify secret handling and logs during retries. Repeated debug output can expose values that were previously masked only in one shell context. Keep commands structured and minimise sensitive output. The recovery path deserves the same credential review as the normal successful run.
Test controller and agent failures separately
In a controlled environment, interrupt the agent during a local build, artifact publication, remote deployment, and acceptance check. Then test a controller restart during supported resumable and nonresumable steps. Record which operations resume and which are re-executed.
Inspect the target after each exercise. The useful test result includes artifact identity, number of external effects, final release, and diagnostic evidence. A pipeline returning to green is insufficient if it silently repeated a non-idempotent action.
Quick diagnostic map
| Evidence | Interpretation | Next check |
|---|---|---|
| Agent lost after mutation | External result uncertain | Query operation ledger |
| Recovery rebuilds different bytes | Candidate not retained | Resume stored artifact |
| Repeated test failures continue | Retry category too broad | Inspect original failure |
A focused operator exercise
Retain a synthetic candidate outside the agent workspace and record its digest before deployment. Interrupt the agent during a local build, registry publication, remote mutation, and acceptance check as separate exercises. For each case, identify whether work resumes, re-executes, or requires external-state inspection under the installed Jenkins step and retry configuration.
Make the remote action store a stable operation identifier and queryable progress. Lose the agent immediately after the target accepts the operation, then recover on another agent. Confirm the replacement checks the target result before repeating any non-idempotent action. A new workspace should not erase knowledge of a completed release or justify rebuilding different bytes under the old identity.
Repeat a controller restart during a resumable and a documented nonresumable step in the test pipeline. Capture final artifact, target state, operation count, and failure evidence. The exercise passes when infrastructure recovery preserves the release contract and operators know exactly when retry is safe, when continuation is appropriate, and when reconciliation is required first.
Give operators a recovery decision
The runbook should explain when to retry a stage, when to resume a retained candidate, and when to inspect or repair the target first. Include the operation identifier and queries needed to establish current state. Keep the original failure visible after a later attempt succeeds.
Clouvenix can help design Jenkins stages around durable artifacts and observable external actions. The outcome is recovery from worker loss without making the release's side effects uncertain.
Frequently asked questions
Is an agent-related failure always safe to retry?
No. The failure category explains why Jenkins lost execution, not whether an external action completed. Review the commands inside the retry block and reconcile any operation whose acknowledgement may have been lost.
Should the whole pipeline restart after controller maintenance?
Use the supported durability and resume behaviour for the installed steps. Where a step cannot resume, retry the appropriate safe boundary and retain the same candidate identity rather than rebuilding or redeploying without state checks.
Explore Clouvenix devops automation for implementation and operational handover.