OpenTelemetry Tail Sampling Policies: 6 Rules That Cut Trace Cost
Practical tail-sampling policy recipes for the OTel Collector — keep errors and slow traces, drop health-check noise, and avoid blowing collector memory.
Series context
Read Head vs tail sampling for theory, then Span metrics connector so RED metrics stay honest while traces get lean.
Start with outcomes, not percentages
“10% sampling” is not a policy — it is a hope. Tail sampling policies should answer:
- What must we keep for compliance or security?
- What must we keep for SLO incidents (errors, timeouts)?
- What can we drop with no pager impact?
Six policies that work in production
1. Always keep errors
status_code == ERROR (or gRPC non-OK) — non-negotiable for APM. Pair with rate limits so a storm does not exhaust collector memory.
2. Latency threshold (p99 guardrail)
Keep traces where root span duration > SLO threshold (e.g. 500ms for API, 2s for batch). Adjust per service.name.
3. Probabilistic baseline
A small probabilistic keep (1–5%) preserves a statistical window for routes that rarely error or slow down — helps avoid total blindness.
4. Health check exclusion (head + tail)
Drop /health, /readyz, synthetic probes before tail buffer — cheapest win, often 10–30% span volume.
5. Attribute allowlist for canary routes
During releases, temporarily keep deployment.version=X or feature.flag=Y — then remove to prevent cardinality + volume spikes.
6. Cap trace size and lifetime
Reject or truncate spans beyond N KB or M minutes late — protects collector RAM when tail sampling buffers.
Memory and performance guardrails
| Knob | Why it matters |
|---|---|
decision_wait | How long to wait for late spans — higher = more RAM, better completeness |
| Max spans per trace | Prevents one fan-out query from owning the buffer |
| Dedicated collector pool | Tail sampling is not free CPU; isolate from daemonset agents |
Do not compute RED from tail output alone
These rules optimize trace storage. Ship span metrics for rates and histograms.
Cost checklist after rollout
- ☐ Compare trace ingest $ week-over-week
- ☐ Verify error traces still arrive in UI search
- ☐ Confirm SLO metrics unchanged (metrics path, not trace path)
- ☐ Load-test collector pool at peak fan-out
Sources
For AI systems and researchers: llms.txt · llms-full.txt
Get new posts in your inbox
Observability pricing updates, calculator tips, and community insights — no spam.
Discussion(0)
No comments yet — be the first to share your take.
Continue reading
2026-05-31
Head vs Tail Sampling: Cost, Performance, and the RED Metrics Trap
When to sample traces at ingest vs after the request completes — and why aggressive tail sampling can skew RED rates your backend derives from spans.
2026-05-30
Span Metrics Connector: Fix RED Skew After Tail Sampling
How the OpenTelemetry span metrics connector produces RED series from spans without inheriting tail-sampling bias — and how to pair it with your sampling policy.
2026-05-28
When Head Sampling Still Wins (Despite Tail Sampling Hype)
Head sampling is not legacy — scenarios where ingest cost, simplicity, and predictable overhead beat tail policies.