Context
Tail sampling can retain traces based on what happened across a request, such as an error or high latency. That decision requires trace spans to reach the appropriate sampler and remain available long enough. A policy saying keep errors does not guarantee every error trace survives routing, buffering, or earlier sampling.
Map the complete trace path
Record SDK sampling, propagation, agents, gateways, load balancing, tail-sampling processors, exporters, and backend ingestion. Identify where spans can be dropped before the tail decision. A tail sampler cannot reconstruct spans that an upstream head sampler never recorded or exported.
Verify trace context crosses service and queue boundaries according to the instrumentation contract. Missing context can create separate traces that no sampling policy can reunite automatically. Use controlled request identities rather than sensitive payloads to follow test traffic.
Keep related spans together
Stateful tail sampling needs an appropriate trace-affinity design. Randomly distributing spans from one trace across independent sampler instances can leave each with an incomplete view. Use the supported trace-ID-aware routing architecture for the deployed Collector and load-balancing components.
Review what happens during scaling and membership changes. A routing change can affect in-flight traces while new traces use the new distribution. Test rollout behavior rather than assuming steady-state affinity proves upgrade continuity.
Distinguish routing availability from decision correctness. A sampler can receive traffic and report healthy while its local view lacks the error span that should trigger retention.
Budget decision time and memory
The decision wait must fit the trace-duration and span-arrival pattern. A short wait can decide before a slow downstream span appears. A much longer wait retains more active state and needs a deliberate memory budget.
Estimate active trace population from arrival rate and retention time, then measure actual memory with the real span count and attribute sizes. For an illustrative arrival rate of 2,000 traces per second and a 20-second decision interval, about 40,000 arriving trace identities can overlap before accounting for processing, late spans, and implementation details.
Treat configuration values such as num_traces as capacity controls whose exact behavior must be checked against the installed processor version. Monitor dropped or evicted trace state, late spans, queue pressure, and Collector restarts. A nominal capacity estimate is not a substitute for the processor's operational metrics.
An illustrative missing-error trace
A team configures error-based retention but sees an incomplete trace for an important failure. Spans were routed randomly to several sampler replicas, and the error span arrived at a different instance from the request's main spans.
The team corrects trace-affinity routing and tests a delayed error span across a Collector rollout. It also verifies upstream sampling and exporter loss. The acceptance result accounts for retained trace completeness and known loss conditions rather than only the configured error policy.
Quick diagnostic map
| Evidence | Boundary to test | Next check |
|---|---|---|
| Trace fragments differ by sampler | Affinity failure | Trace-ID routing |
| Slow traces lose late spans | Decision window mismatch | Arrival distribution and late-span metrics |
| Collector memory rises with traffic | State budget pressure | Active traces and span sizes |
| Error span never reaches gateway | Earlier loss or instrumentation | SDK sampling and exporter path |
A focused operator exercise
Generate a small known trace set containing normal requests, errors, long operations, and deliberately delayed spans. Record expected trace and span identities, then verify the backend's retained results against the sampling policy.
Increase load within a bounded test and exercise sampler scaling, restart, and downstream exporter delay. Measure memory, dropped state, late spans, and retained completeness. Keep source and test data nonsensitive. Document which failures the architecture can recover from and which produce expected telemetry loss.
Operate sampling as a data-quality system
Version sampling policies and evaluate them against a repeatable trace set. Keep a controlled diagnostic route when appropriate for critical investigations, with access and retention suited to the data. Reducing stored trace volume is useful only when the retained evidence still answers the intended operational questions.
Frequently asked questions
Can tail sampling recover a span removed by head sampling?
No. It can decide only from the spans that reach its processing path.
Should I increase decision_wait whenever traces are incomplete?
First inspect routing, propagation, and earlier loss. A longer wait addresses only certain arrival patterns and increases state requirements.
Continue the investigation
- OpenTelemetry queues: persist telemetry and test recovery
- Application Insights trace gaps: review sampling across service boundaries
Browse the cloud, server and infrastructure guide library for related operational questions.
Explore Clouvenix OpenTelemetry Instrumentation Services to discuss implementation, validation and operational handover.
Technical reference
Tail-Based Sampling with service.criticality | OpenTelemetry