Context
A request can spend more time resolving a hostname than talking to the destination. In Kubernetes, the Pod's resolver configuration may try search-domain combinations before an external name. The effect depends on the name, resolver library, application runtime, and caching. A quick lookup from the node does not necessarily reproduce what happens inside the application container.
Consider an illustrative service that calls several external APIs during each request. After moving into Kubernetes, connection establishment becomes inconsistent. The APIs themselves respond normally, but traces show time before the TCP connection begins. That is a reason to investigate name resolution with the actual Pod configuration rather than immediately increasing HTTP timeouts.
Inspect the Pod's effective resolver settings
Kubernetes DNS troubleshooting guidance starts with a test Pod and inspection of resolv.conf, including nameserver, search paths, and options such as ndots. It also covers CoreDNS resources, permissions, and common host-resolver issues. Namespace and Pod DNS configuration influence the names the workload can resolve. 1
Read the settings from the affected Pod or an appropriately equivalent diagnostic context. Record dnsPolicy, dnsConfig, host networking, namespace, image, and runtime. A diagnostic image with a different resolver library may answer a useful question without reproducing the application's exact lookup behaviour.
Separate DNS time from later connection phases
Instrument lookup duration independently from TCP, TLS, and server response time where the runtime permits it. Capture whether the application requests IPv4, IPv6, or both and whether it reuses existing connections. A reused connection may hide DNS entirely, while a new client object can trigger repeated lookups.
Compare successful and slow requests to the same hostname. Look for cache expiry, deployment events, or a periodic burst of new processes. Do not assume every slow outbound request is DNS-related simply because the hostname appears in the error. A timeout after TLS has completed points elsewhere.
Observe search expansion instead of guessing
Use a controlled lookup to inspect which names are queried and in what order. A name with fewer dots than the configured threshold may be tried with search suffixes before being treated as absolute, depending on the resolver. This can multiply queries, especially when external names are resolved frequently without effective caching.
An absolute DNS lookup can help isolate search expansion during diagnosis. Do not blindly add a trailing dot to every application URL: TLS hostname validation, HTTP Host handling, proxies, and application libraries need compatible behaviour. Keep DNS experiments distinct from production URL changes unless the complete request path is tested.
Check the cluster DNS path
Inspect CoreDNS health, restarts, resource saturation, query latency, errors, and upstream behaviour. Confirm the DNS service and endpoints are correct. If several namespaces are affected at once, shared DNS capacity or upstream resolution becomes more plausible than an isolated application setting.
Review network policy and the actual resolver destination. DNS may use UDP and TCP, and node-local DNS designs can change the path. A policy written around an assumed CoreDNS Pod address may not match the deployed architecture. Test the path used by the workload rather than copying an allow rule from a different cluster.
Examine application caching carefully
Runtime DNS caching can improve performance but also delay adoption of changed addresses. Determine whether the application uses the system resolver, an asynchronous library, a custom DNS client, or a long-lived connection pool. Each can have different caching and refresh behaviour.
Do not replace every hostname with a fixed IP as a performance fix. That can break failover, certificate identity, load balancing, or private endpoint changes. If a local cache or node-local DNS component is introduced, establish its ownership, failure behaviour, and monitoring. A new cache adds a new operating dependency.
Change scope before changing the whole cluster
When the evidence points to search expansion, evaluate a targeted Pod DNS configuration or application lookup change in staging. Test short internal service names as well as external fully qualified names. Lowering a threshold can alter lookup order, so a successful external API test alone is incomplete.
If CoreDNS capacity is constrained, tune it from measured query rate and latency, with appropriate availability. Keep query logging bounded because it can produce substantial volume and expose sensitive hostnames. Use logs to answer a specific question, then return to the normal observability policy.
Validate under realistic connection churn
Run a controlled workload with cold starts, connection reuse, and the normal mix of internal and external names. Record query counts per application operation, DNS latency, resolution errors, and successful request latency. Repeat after the change using the same hostnames and runtime.
An illustrative acceptance target might require both a lower high-percentile lookup time and unchanged internal service resolution. The actual threshold should come from the application's latency objective. Keep a test for address changes so a performance improvement does not create an undetected recovery delay later.
A focused operator exercise: An internal-and-external lookup comparison
Select one short internal service name and one approved external hostname. Measure lookups from the application runtime before and after a targeted DNS configuration change, including a cold process and a warm process. Capture actual query behaviour where authorized so the result distinguishes search expansion from cache reuse.
Then change an address in a controlled test zone and verify the application eventually uses the new answer under its intended caching contract. This second step prevents a latency optimization from introducing an unmeasured recovery delay. Keep the resolver library, Pod configuration, hostname forms, and connection-pool behaviour with the results so another engineer can reproduce the comparison.
Quick diagnostic map
| Evidence | Review | Acceptance boundary |
|---|---|---|
| Resolver input | Inspect Pod search domains and ndots | Reproduce the original lookup form |
| Query expansion | Compare internal and external names | Measure retries and search suffix attempts |
| DNS service | Inspect latency and error evidence | Avoid changing global settings from one sample |
Frequently asked questions
Why does a node lookup work while a Pod lookup fails? They can use different resolvers, search paths, policies, and network routes.
Clouvenix can trace DNS from the application runtime through the cluster resolver, helping teams choose a focused correction instead of masking lookup delays with longer request deadlines.
For a scoped review of this workflow, discuss the relevant Clouvenix engineering service.