Context
A service accepts established clients but new connections occasionally stall during a burst. One possible boundary is the listener's accept queue: completed TCP connections wait there until the application accepts them. That queue is different from incomplete TCP handshakes and from HTTP requests already being processed by workers.
Prepare the destination address, port, network namespace and a timestamped failing connection. Identify the application owner and expected burst behavior. These checks should remain observational on production systems. Load generation belongs in an approved representative environment because an uncontrolled connection test can create the pressure being investigated.
Inspect the relevant queue
- Run
ss -ltnin the correct network namespace and locate the affected listener. Read the installedssdocumentation for its listening-socket queue columns. Capture repeated snapshots during the failing period rather than a single quiet observation. Verify whether a reverse proxy or socket-activation unit owns the listener instead of the application process you expected. - Read
sysctl net.core.somaxconnandsysctl net.ipv4.tcp_max_syn_backlog. Compare them with the application's configured listen backlog. The kernel documentation distinguishes the completed-connection backlog from the incomplete-handshake queue. Changing the latter does not directly enlarge an application's completed accept queue, and the application can request less than the system cap. - Inspect relevant TCP extended statistics with a supported tool such as
nstat, if available. Compare timestamped changes in listen-overflow and listen-drop counters with the client failures. These are broader host or namespace signals, so correlate them with the specific listener and workload before attributing every increase to one service. - Examine application acceptance behavior. CPU saturation, a blocked event loop, worker startup or a synchronized dependency wait can keep the program from accepting connections quickly. Increasing the queue may absorb a short burst while leaving a sustained inability to accept untouched.
An illustrative burst mismatch
In an illustrative internal API, a scheduled fleet job causes many clients to reconnect at the same instant. The listener queue fills briefly while workers are busy starting request handlers. Existing sessions continue successfully. The team tests client-side jitter and accept-path improvements before deciding whether a larger service backlog is appropriate for the remaining legitimate burst.
| Observation | Diagnostic question | Direction |
|---|---|---|
| Queue grows only during bursts | Can the service drain after the burst? | Evaluate burst absorption |
| Queue stays near its limit | Why is acceptance persistently slow? | Investigate application capacity |
| Handshakes fail before completion | Is the incomplete queue affected? | Inspect handshake-specific evidence |
| Host counters rise, listener is quiet | Is another listener responsible? | Refine attribution |
Correct and verify the cause
Tune the application and kernel settings together only after confirming their relationship. Preserve the previous values and avoid copying a large universal setting from another workload. The kernel specifically cautions that aborting connections on overflow can harm clients; treat such behavior changes as separate decisions.
Retest with the same representative burst, measuring successful connections, connect latency, queue occupancy and application response quality. A larger queue that merely extends waiting time is not a complete performance improvement. Verify that memory use and recovery after the burst remain acceptable.
Maintain connection-level visibility
Monitor new-connection success separately from request success on reused connections. Document listener ownership, namespace, application backlog and expected reconnect patterns. Include client retry coordination in the SOP when synchronized reconnects matter. Clouvenix server performance can help connect queue evidence to application behavior.
Technical reference
Kernel TCP settings and Linux socket interface manual collection.