Context
A queue worker can appear healthy while repeatedly sending the same invoice, reserving the same stock, or calling the same partner API. This often happens when several SQS messages arrive in one Lambda invocation: some complete successfully, one fails, and the application treats the entire invocation as failed. Understanding the batch boundary helps a startup protect customers while controlling unnecessary processing. The important business question is which individual operations have durably completed.
Understand the two acknowledgements
AWS supports partial batch responses through the event source mapping setting ReportBatchItemFailures. The handler returns the message identifiers that failed so successful records do not need to be retried because of another record's error. An exception escaping the handler still makes the complete batch fail. With FIFO queues, stop after the first failure and include failed and unprocessed messages in the response to preserve ordering. These mechanics are documented in the linked Lambda guide.
An acknowledgement is not evidence that every external action happened exactly once. A worker can update a database and lose its connection before reporting success. When that message returns, the worker needs a way to recognise the previous business operation. Give an invoice, fulfilment, or subscription change a stable operation identifier that survives retries. Transport message identifiers are useful for diagnostics; the business identifier decides whether repeating the operation is safe.
An illustrative invoice incident
Imagine a finance application processing four invoice requests. Three rows are valid, while one references a customer account that has been removed. A broad exception handler aborts the invocation after some invoices have already been created. The next delivery repeats those successful requests. Increasing Lambda memory might make the cycle faster, but it does not correct the acknowledgement contract.
Investigate one operation from receipt to durable result. Record its business identifier, message identifier, attempt, handler version, database outcome, and external provider reference. Compare repeated attempts against the invoice table. Do not log the full invoice payload merely to obtain correlation; a small set of identifiers usually provides clearer evidence and avoids unnecessary exposure of customer details.
Establish the processing contract
Separate validation failures, transient dependency failures, and uncertain outcomes. A malformed request needs investigation or a defined rejection workflow. A temporary database connection failure may justify retry. A payment provider timeout after submission is an uncertain outcome: first reconcile using the provider's operation reference rather than assuming nothing happened.
Decide when the local result becomes authoritative. For a database-only action, a unique constraint and transaction can protect the operation identifier. For an external action, use a provider-supported idempotency mechanism where available and retain its result. Do not mark the operation complete before required durable work has finished. Equally, avoid holding a database transaction open throughout a slow remote request without reviewing the consequences.
Implement and deploy both halves
Review the active event source mapping, not just the function configuration in source control. Enable partial reporting on the intended mapping and ensure the deployed handler produces the expected failure response. A handler change without the matching integration setting leaves a hidden mismatch. Conversely, a configuration change cannot rescue code that simply throws after processing an otherwise mixed batch.
Process each standard-queue record inside an explicit error boundary. Collect failed identifiers and return them after processing the batch. Keep shared setup failures separate: if the application cannot initialise its database client, treating every record as successful would lose work. For FIFO processing, implement and test the stop-on-failure rule instead of copying a standard-queue parallel loop into the handler.
Run a useful failure exercise
Prepare a small synthetic batch with independently identifiable operations. Make one record fail before any durable effect. Confirm that successful records stay complete and the failed record becomes eligible for another attempt. Then repeat the exercise with a failure after the database commit but before the response is returned. This second test checks the business deduplication design, which a happy-path invocation cannot demonstrate.
Test a complete handler failure as well. Remove access to a test dependency, invoke the worker, and inspect the resulting retries and alarms. Restore access and observe recovery without manually deleting messages. A recovery drill should show that the queue can make progress after the fault is removed, while already completed customer operations remain unchanged.
Control backlog and poison messages
Partial responses improve isolation, but a permanently invalid message still needs ownership. Define a dead-letter investigation process with a reason code, responsible team, and replay decision. Replaying a message after correcting its data is a business operation; preserve its original operation identifier so recovery does not bypass deduplication.
Watch message age, failed operation counts, repeated attempts per business identifier, and the rate of completed work. A low Lambda error count can hide individual failures when the handler deliberately reports them in its response. Publish an application metric for record-level failures and compare it with queue progress. Otherwise, the function dashboard may look reassuring while customers wait.
Quick diagnostic map
| Evidence | Interpretation | Next check |
|---|---|---|
| Whole batch repeats | Handler or mapping mismatch | Inspect failure response |
| One operation repeats | Commit acknowledgement uncertain | Read operation ledger |
| FIFO order breaks | Processing continued after failure | Return unprocessed identifiers |
A focused operator exercise
Use a dedicated test queue and four synthetic operations with stable business identifiers. Give each operation a known expected database result, and make the third fail before commit. Keep the handler version and event source mapping configuration in the exercise record. This makes it possible to compare record-level processing with invocation-level status without involving customer data.
Repeat the third operation after repairing its test dependency, then repeat one already completed operation deliberately. Confirm the operation ledger distinguishes recovery from duplication. Next, interrupt the handler after a successful commit and verify the replacement attempt recognises the completed result. Record the exact database evidence rather than relying on a log message printed before commit.
Finally, inspect the dead-letter path with one permanently invalid record. The operator should identify its business owner, failure reason, correction requirement, and permitted replay method. Keep successful records out of that repair population. The exercise passes when every synthetic operation has one explained final disposition and the worker can recover without deleting unresolved work.
Hand over an actionable operating procedure
The runbook should identify the event source mapping, handler release, deduplication store, dead-letter destination, and reconciliation query. Include a sample failed operation and the evidence needed before replay. State who may change retry behaviour and who approves an external action whose previous result is uncertain.
For a global SaaS team, this gives support staff a practical answer to “was my invoice processed?” without requiring them to interpret Lambda invocation statistics. Clouvenix can help review the queue worker, implement the operation ledger, and rehearse recovery against controlled failures. A successful review ends with per-operation evidence and a repeatable recovery path.
Explore Clouvenix devops automation for implementation and operational handover.