Context
DynamoDB Streams records table changes, and Lambda can process those records in batches. A failing record can delay progress while already successful work is attempted again. Recovery needs both stream-checkpoint knowledge and application idempotency. Increasing function concurrency alone will not repair a batch that repeatedly fails on the same input.
Capture the mapping and processing contract
Record the table stream ARN, event-source mapping UUID, Lambda version or alias, batch settings, retry policy, maximum record age and any discard destination. Identify the business effect produced by each change, such as updating a search document or sending a downstream event.
Use sanitized records in a test stream. Record identifiers and sequence numbers as diagnostic evidence without copying sensitive item images into a general incident channel. Establish how the business will reconcile changes that cannot be processed before stream retention expires.
1. Locate the backlog and first repeat failure
Align Lambda IteratorAge, duration, errors and throttles with application logs. Group errors by shard and sequence number where available. Repeated failure on the same sequence suggests a record or deterministic code problem; general slowness across shards suggests a different capacity or dependency issue.
Review event-source filtering and mapping state as well as the function. A successful manual function test does not prove the stream mapping is polling or that its response settings match the handler's return value.
2. Verify the partial-batch contract
By default, Lambda advances the checkpoint only when a batch succeeds. To report selected failures, the mapping must enable ReportBatchItemFailures, and the handler must return the expected structure containing failed sequence identifiers. Returning that structure without enabling the mapping feature does not activate partial failure handling.
For DynamoDB streams, the lowest reported failed sequence defines where retry processing resumes. Records beyond that point can therefore be attempted again. Keep downstream effects idempotent even when the handler reports partial success. Review how bisect-on-error interacts with the chosen handler behavior in a controlled test.
3. Distinguish recovery choices with evidence
| Evidence | Likely issue | Corrective action |
|---|---|---|
| Same sequence fails repeatedly | Poison record or deterministic bug | Fix handler or quarantine through policy |
| Partial response returned, whole batch repeats | Mapping feature missing or response invalid | Inspect response types and shape |
| Age grows while errors stay low | Slow processing or throttled capacity | Inspect duration and dependencies |
| Records discarded without business record | Recovery path incomplete | Validate destination and reconciliation |
An explicitly illustrative scenario: a product-indexing function rejects a newly introduced field type. One record repeatedly blocks a shard while other product changes wait. The team fixes parsing in a test deployment, confirms correct partial-response behavior with a deliberately failing sample, and uses the product ID as an idempotency boundary for index updates. It then verifies that age declines and affected products match the source table.
Correct, verify and protect the recovery window
Choose retry and maximum-age settings with the business owner. Infinite-looking retry settings do not extend DynamoDB Streams' finite retention. Validate that discarded-record handling is configured and that its consumers can identify missing business updates.
After correction, compare iterator age with new event arrival, confirm the failed sequence progresses, and reconcile representative source items with downstream state. Do not replace a mapping with LATEST merely to clear a backlog; creation and update polling are eventually consistent and may miss intended work under that selection.
The SOP should include record ownership, safe samples, idempotency keys, retention deadlines, discard-destination checks and reconciliation procedures. Review these after schema or handler changes, before the next malformed record becomes an incident.
Explore Clouvenix managed infrastructure for implementation and operational handover.
Technical reference
Partial-batch responses, mapping parameters, and DynamoDB integration behavior describe checkpoints and retries.