Retrying a step repeats its operation, not its intention
A workflow can coordinate several remote operations. If one call times out after the remote service has accepted it, repeating the call may repeat the business effect. Distinguish a request that definitely never reached its target from an operation whose outcome is uncertain. That distinction belongs in workflow design before an incident.
Google provides different retry policies for idempotent and non-idempotent steps. An idempotent operation can be repeated without changing the intended result beyond the first successful application. Choosing the idempotent retry policy does not make an unsafe operation idempotent; the downstream contract must provide that property.
Prepare a step-level execution contract
- Inventory each step's read or write behavior, external effect and stable operation identifier. Define when the result becomes durable and observable.
- Classify errors as retryable, terminal or uncertain. Select bounded retries and backoff for the actual operation rather than attaching one policy to every remote call.
- Rehearse timeout after acceptance, invalid input and partial workflow completion using non-delivering test dependencies. Retain a recovery owner and reconciliation path.
For beginners, restarting an entire workflow can repeat steps that already succeeded. Recovery may need to resume or reconcile a specific business operation instead of re-running the complete sequence from its beginning.
Illustrative incident: an external request succeeds before timeout
A workflow asks a document service to create an export. The service records the export, but its response is delayed until the workflow's request times out. A retry without a stable operation key creates another export. This is an illustrative scenario. Raising the maximum retry count increases attempts without resolving uncertain completion.
Diagnose the step and its downstream record
Capture execution identity, failing step, error category and the business operation key. Compare that attempt with the downstream service's durable record. Keep request payloads and bearer credentials out of ordinary workflow logs.
| Observation | Evidence | Recovery boundary |
|---|---|---|
| Request rejected as invalid | Validated input and remote error | Correct input, do not blindly retry |
| Remote result exists after timeout | Stable operation key and result record | Reuse or reconcile result |
| Transient failure with safe repeated operation | Retry policy and bounded attempts | Controlled retry |
| Earlier steps already changed state | Execution history and business records | Partial-completion recovery |
Do not classify every HTTP error as transient. An unchanged invalid permission or malformed payload generally needs a specific correction, while an uncertain external effect needs evidence before repetition.
Repair repeat safety and verify partial recovery
For the illustrative export, add a stable operation key supported by the document service or an equivalent durable deduplication design. Define how the workflow retrieves the existing result after an uncertain response. Preserve the distinction between a failed call and a failed business operation.
Test repeated execution of the same step, a workflow restart after partial completion and a downstream timeout after acceptance. Confirm exactly which records and external effects exist afterward. If an operation cannot be safely repeated or reversed, provide a reviewed manual reconciliation path rather than an unlimited automated retry loop.
Reliability and workflow SOP
Measure completed business workflows, unresolved uncertain steps and duplicate effects alongside execution errors. An illustrative internal objective could require every state-changing step to have an approved repeat or reconciliation contract before release. It is an application control, not a Workflows SLA or an exactly-once promise for arbitrary external services.
The SOP should retain execution, step, operation key, durable outcome, retry policy and recovery owner. Escalate unresolved external effects before replay. Review the contract whenever a new step is added, because a previously safe read-only workflow can acquire different recovery requirements as soon as it performs a write.
See Clouvenix engineering support for scope based on documented requirements and acceptance evidence.
Technical reference
Google Cloud: retry steps. Reviewed 19 October 2026; check prerequisites against the product configuration and software version in use.