Context
A large Linux server can have free memory and spare CPUs while an application suffers from poor memory locality. NUMA places processors and memory into nodes with different access costs. Investigate thread placement and memory allocation together before pinning a process or disabling the system's normal balancing behavior.
Record the topology visible to the workload
Identify sockets, NUMA nodes, CPUs per node, available memory, and the application's allowed CPU and memory sets. Tools such as lscpu, numactl --hardware where installed, and the relevant cgroup configuration provide complementary views.
In a VM, record virtual NUMA topology and the hypervisor's placement model where available. The guest's view may differ from physical host layout. A change in VM size or vCPU count can alter topology and invalidate an earlier affinity assumption.
Compare placement with memory allocation
Use numastat and supported per-process views to inspect allocation across nodes. Correlate those observations with where the process's busy threads actually execute. Memory on another node can be legitimate; the question is whether the access pattern causes a measurable problem.
Counters such as allocation hits and misses have defined meanings and can be cumulative. Do not turn an old nonzero miss count into a diagnosis of current remote-access latency. Capture interval behavior and application performance under a comparable workload.
Include local-node memory pressure. A process can be restricted to a subset of nodes even when the whole machine has free memory. CPU affinity, cpusets, container placement, and memory policy can combine to create a constrained allocation domain.
Examine the application's allocation pattern
First-touch behavior can matter when one startup thread initializes a large memory region later accessed by many worker threads. Database buffers, in-memory analytics, and JVM heaps may have workload- or runtime-specific NUMA guidance.
Check the software vendor's supported configuration before changing affinity or memory policy. Hard binding can improve one benchmark while reducing scheduling flexibility or causing local-node pressure during a peak. Interleaving can spread allocations but is not a universal improvement for every access pattern.
Review automatic NUMA balancing and runtime settings as separate variables. Changing host policy, thread count, huge-page configuration, and application heap size together produces an experiment that is difficult to interpret.
An illustrative large-memory regression
A service moves to a larger multi-node VM and becomes slower despite gaining resources. Its startup path initializes a large shared structure on one node, while worker threads execute across several nodes. A controlled profile shows increased memory-access cost during the main operation.
The team compares a supported initialization or placement change using the same dataset. It measures throughput, tail latency, CPU behavior, and allocation distribution. The accepted configuration also survives workload growth and a restart; a single favorable average is not treated as a permanent topology rule.
Quick diagnostic map
| Evidence | Hypothesis | Next check |
|---|---|---|
| Threads and hot memory on different nodes | Locality mismatch | Process allocation and CPU placement |
| One node pressured; system memory free | Restricted allocation domain | cpuset and memory policy |
| Regression after VM resize | Topology changed | Virtual NUMA and runtime settings |
| Affinity improves one task; hurts another | Workload-specific tradeoff | Representative mixed-load test |
A focused operator exercise
Choose a representative memory-intensive task with a fixed dataset and correctness check. Record topology, process affinity, cgroup restrictions, runtime configuration, and allocation statistics. Run the baseline repeatedly after a controlled startup sequence.
Change one supported placement or allocation behavior and repeat the same workload. Include a peak and the expected neighboring activity. Compare completion, latency, local-node pressure, and allocation evidence. Restore the previous policy if the benefit is not reproducible or if another critical workload deteriorates.
Keep tuning tied to a supported configuration
Document the exact machine topology and software version behind the decision. Revalidate after hardware migration, hypervisor changes, VM resize, or runtime upgrades. NUMA tuning should explain a measured bottleneck, not become an unexplained set of permanent boot flags.
Monitor application symptoms and node-level resource balance together. A healthy-looking machine-wide average can conceal a constrained subset, while a visually uneven allocation can still be correct for the workload.
Frequently asked questions
Is remote memory always a problem?
No. Evaluate its cost in the actual workload and topology. A nonzero counter alone does not establish harmful latency.
Should I pin every database process to one node?
Follow the engine's supported guidance and test capacity as well as performance. Hard binding can create a smaller failure and memory-pressure domain.
Continue the investigation
- VMware CPU ready: distinguish scheduling delay from guest CPU usage
- Transparent Huge Pages: Test Allocation Latency Tradeoffs
Browse the cloud, server and infrastructure guide library for related operational questions.
Explore Clouvenix Server & Website Performance Troubleshooting to discuss implementation, validation and operational handover.