Context
SNAT lets private backends share a public address for outbound connections. Azure Load Balancer assigns port capacity to backends according to its outbound configuration. One busy instance can exhaust its allocation while other instances remain healthy. An overall load balancer chart can conceal that imbalance.
Prerequisites are the affected backend IP, a failure window in UTC, read access to outbound rules and metrics, and application connection telemetry. Confirm that Load Balancer actually provides the outbound path. A NAT gateway, instance public IP or network appliance changes the investigation.
1. Identify the failing connection pattern
Record the destination host, resolved address, port and protocol. Separate failure to establish a new connection from a timeout on an established connection. Compare affected and unaffected backends during the same interval. An inbound health probe succeeding does not prove that the backend has outbound capacity.
Read the outbound rule and associated frontend addresses. Check the backend pool's actual size and intended maximum scale. Explicit port allocation needs to support that fleet size. Preserve this configuration before tuning it, including any inbound rule that influences outbound translation behavior.
2. Split the metrics by backend
Compare Allocated SNAT Ports and Used SNAT Ports with backend IP and protocol dimensions. Inspect SNAT Connection Count with its connection-state dimension during the failures. Use an aggregation and time grain that exposes peaks instead of averaging them across a quiet period.
Correlate those charts with connection creation rate, active sockets and requests per backend. A worker that creates a fresh HTTP client for every message produces different demand from one that reuses a connection pool. Check whether one destination dominates. Capture failed request identifiers so downstream rejection can be distinguished from source connection failure.
3. Correct demand or allocation deliberately
If unnecessary connection churn is responsible, reuse supported client pools and close resources correctly. Bound retry attempts and worker concurrency. A slow partner must not trigger an unbounded pile of additional connections. Verify idle connection behavior with the destination owner.
For legitimate demand, review outbound allocation, frontend capacity and maximum backend count together. A NAT gateway may suit a design needing shared dynamic capacity. Coordinate partner allowlists for every possible new public address before moving traffic. Test a controlled worker group rather than changing the entire egress design during diagnosis.
Diagnostic checkpoints
| Observation | Suspected boundary | Confirm with |
|---|---|---|
| One backend approaches allocation | Uneven demand or churn | Connection creation rate |
| Every backend fails to one partner | Destination issue | Partner connection logs |
| Inbound works, outbound fails | Egress capacity | SNAT connection state |
| Failure follows scale-out | Port plan mismatch | Rule and pool count |
Illustrative scenario
A hypothetical invoice worker opens a new connection for each upload. Only its busiest instance fails intermittently. Splitting port metrics by backend reveals allocation pressure hidden by the fleet average. The investigation then compares connection reuse with the required peak demand. This fictional case makes no claim about measured customer improvements.
Verification and operations
Repeat an approved representative workload with the same destination mix and concurrency. Acceptance requires completed business operations, bounded retry behavior and per-backend headroom during the peak. Keep rollback instructions for address and allowlist changes.
The SOP should record the egress owner, frontend addresses, partner allowlists and maximum tested pool size. Alert on failed SNAT connections alongside application errors. Review allocation before increasing worker concurrency or scaling limits, because both can invalidate a previously adequate port budget.
Explore Clouvenix managed infrastructure for implementation and operational handover.
Technical reference
Outbound SNAT and metrics.