The subscriber application is not the forwarder
A Pub/Sub dead-letter policy can forward messages that remain undeliverable after approximately the configured number of attempts. The forwarding work uses the Pub/Sub service agent for the subscription's project. Granting only the application subscriber access to the dead-letter topic does not give this service agent the permissions it needs.
The service agent must be able to publish to the dead-letter topic and acknowledge messages on the source subscription as documented. Delivery-attempt behavior depends on a correctly configured forwarding path and is not a strict business retry counter. A dead-letter topic also needs a subscription if operators want to receive and investigate the forwarded messages.
Prepare the source subscription's project number, the dead-letter topic project, policy settings, and read access to both resources' IAM policies. Identify an owner for investigating exceptions and a retention period for those messages. Use a harmless test message that cannot trigger production side effects.
Verify the forwarding chain
- Inspect
gcloud pubsub subscriptions describe SOURCE_SUBSCRIPTION --project=SOURCE_PROJECT. Record the topic, dead-letter destination, maximum-delivery-attempt setting, and retention behavior. Confirm the destination's full resource name, particularly when it lives in another project. - Identify the source project's Pub/Sub service agent in its documented
[email protected]form. Read the destination topic IAM policy and source subscription IAM policy. Match the forwarding roles to this identity rather than a similarly named runtime account. - Confirm that a dedicated investigation subscription is attached to the dead-letter topic. Inspect its permissions, retention, and monitoring. A topic without a usable consumer path is a poor operational inbox, even if forwarding itself succeeds.
- Test the path with a controlled failure case in a nonproduction subscription or approved isolated flow. Observe source delivery attempts, forwarding, and the wrapper attributes that identify the originating subscription. Do not expect an exact attempt count as the sole pass criterion.
Illustrative scenario
A fictional billing importer repeatedly rejects one invalid test message. The team configured a dead-letter topic and granted its application account publisher access, but forwarding does not behave as expected. Reading the IAM policies shows that the Pub/Sub service agent lacks the required destination permission. After a narrowly scoped correction, the message appears on the investigation subscription with its source context intact.
| Observation | Inspect | Correction |
|---|---|---|
| Attempts continue without forwarding | Service-agent permissions and policy | Fix the documented forwarding grants |
| Topic exists but operators see no messages | Attached investigation subscription | Create the intended consumption path |
| Attempt count differs from expectation | Approximate counting behavior | Use outcome evidence, not a hard counter |
| Cross-project forwarding fails | Full resource names and destination IAM | Bind the source project's service agent |
Verify exception handling
Read the forwarded test message through the investigation consumer and confirm source attribution, payload integrity, and safe handling of sensitive fields. Verify that the source subscription stops treating the forwarded message as ordinary pending work. Make sure the investigation tool does not automatically replay invalid messages into the original workload. A forwarding success is an exception requiring a decision, not a completed business transaction.
Operating procedure
Monitor dead-letter arrivals, oldest investigation age, and forwarding-related permission failures. Assign each exception a reason and owner. Repair the underlying input or application condition before approved replay, preserving original business identifiers. Review IAM when moving topics across projects or replacing subscriptions, because the forwarding service agent and resource scopes can change.
Explore Clouvenix managed infrastructure for implementation and operational handover.
Technical reference
dead-letter topics, failure handling, and subscription properties.