Context
A task queue can deliver work reliably while the application performs the business action twice. The difficult moment is often a lost response: a handler completes an operation, but the queue does not receive a successful acknowledgement. A later attempt is reasonable from the queue's perspective. The application must decide what repeating the operation means.
Consider an illustrative document service that creates a signed agreement and sends it to a customer. The handler finishes, then loses its connection before returning. If the retry creates another agreement and sends another message, the customer sees a duplicate even though the infrastructure is recovering as designed. Queue reliability and business idempotency are different requirements.
Accept the delivery contract explicitly
Google documents that Cloud Tasks can execute a task more than once and does not provide general task execution ordering. Queue configuration controls dispatch rates, concurrency, and retry behaviour. These controls help manage processing, but they do not make an external business action happen exactly once. 12
Write the handler contract around a stable business operation identity. A task name can help manage task creation, but task-name deduplication is not a permanent ledger of completed business effects. The operation identity should remain useful after an individual queue item has disappeared.
Put a durable decision before the side effect
Create or claim an operation record using an atomic mechanism appropriate to the database. Record who requested the work, the permitted tenant, the immutable inputs or their trusted reference, and the current state. A repeated delivery should load that same record rather than invent a new operation.
A simple “check whether completed, then execute” sequence can race when two handlers run together. Use a transaction, uniqueness constraint, or another correct coordination mechanism to establish ownership. Also define how a worker whose claim expires is prevented from completing stale work after a newer worker takes over. An expiring lock alone is not a full correctness argument.
Handle the external system's uncertainty
If the downstream API supports an idempotency key, use a stable key derived from the business operation and preserve the provider's response reference. If it does not, define a reconciliation lookup or a human review state for ambiguous results. A timeout after sending a request is not proof that the downstream action never happened.
Keep a distinction between “not attempted,” “attempted with unknown outcome,” and “confirmed complete.” These states may require different retry decisions. Automatically moving every exception back to a clean pending state hides uncertainty and can repeat financial or customer-visible actions.
Acknowledge at the correct boundary
Return the queue's expected success response only when the handler has completed the promised unit of work or durably transferred responsibility through a deliberate design. Returning success before a transient in-memory job finishes can lose work if the process exits. Holding a request open for an unbounded job can create timeouts and duplicate attempts.
Choose a unit small enough to complete within the target's operating limits. For large batches, create child operations with their own identities and track overall completion. The parent can report partial progress without pretending that every item succeeded because one HTTP handler returned a success status.
Bound retries and dispatch pressure
Tune queue concurrency against the target's useful capacity, including database connections and downstream rate limits. A queue draining faster is not an improvement if the target rejects most attempts. Set retry limits and backoff according to the operation's deadline, and make terminal failures visible in an application-owned failure workflow.
Separate retryable infrastructure failures from invalid input and revoked authorization. A task created yesterday may no longer be permitted to act on a customer account today. Revalidate the authorization conditions that must still hold at execution time rather than treating possession of a queued payload as permanent permission.
Test the ambiguous moments deliberately
In staging, stop a handler before the external call, after the external call but before recording completion, and after recording completion but before acknowledgement. Deliver the same task concurrently. The expected outcome should be one business result or an explicit unresolved state, never silent duplicate effects.
Inspect the operation ledger and downstream records after each test. A response code alone cannot demonstrate correctness. Include a worker restart and a late retry after the original request deadline so the recovery policy is exercised beyond the easiest success path.
Give support a useful completion view
Display the requested operation, current state, latest attempt, confirmed external reference, and next permitted action. Avoid exposing raw task payloads containing secrets. An operator replay should target the original business operation and explain whether it will reconcile, retry, or create a deliberately new operation.
For example, resending a document notification may be safe while creating a second signed agreement is not. Make those separate support actions. The interface should not offer one broad retry button that repeats every stage indiscriminately.
A focused operator exercise: A lost-acknowledgement exercise
Use a harmless downstream test operation that returns a stable reference. Complete the downstream action, then deliberately interrupt the handler before its queue acknowledgement. Deliver the same business operation again and inspect the durable ledger, downstream record, and customer-facing status. The result should remain one confirmed operation or an explicit unresolved state requiring reconciliation.
Next, deliver the same operation concurrently to two workers. Verify ownership is established atomically and that a stale worker cannot complete after its authority has expired. Keep these outcomes separate from a normal retry test that fails before any external side effect, since the ambiguous boundary is the harder correctness case.
Quick diagnostic map
| Evidence | Review | Acceptance boundary |
|---|---|---|
| Operation identity | Store a stable business key | Reuse it across delivery attempts |
| Lost acknowledgement | Repeat a completed request in a sandbox | Confirm the effect is not repeated |
| Outcome record | Reconcile external action with ledger state | Define uncertain-result recovery |
Frequently asked questions
Does a unique task name guarantee one business action forever? No. Durable operation tracking and downstream behaviour still determine that result.
Clouvenix can review the task handler and its external effects together, helping teams use retries as a recovery mechanism while retaining a clear, auditable business outcome.
For a scoped review of this workflow, discuss the relevant Clouvenix engineering service.