Head vs Tail Sampling: Cost, Performance, and the RED Metrics Trap

Why sampling exists at all

Every span you retain has a bill: ingest, indexing, storage, and query. At scale, 100% trace retention is rarely worth the cost. Sampling chooses which traces survive — but when you make that choice changes both cost and what your metrics mean.

Two dominant models:

Model	Decision point	Typical cost profile	Debug story
Head sampling	Before the request finishes	Predictable, lowest ingest	You may drop the one slow trace you needed
Tail sampling	After spans complete	Higher collector CPU/memory; better signal per kept trace	Keeps errors and outliers — if rules are right

Head sampling: pros, cons, and cost impact

Head sampling flips a coin (or applies a percentage) when a trace starts. Unselected traces never hit your backend.

Pros

• Lowest ingest cost — you never pay to ship spans you will discard.
• Simple mental model — “we keep 10% of traffic.”
• Minimal collector work — no need to buffer full traces in memory.

Cons

• Blind to tail events — a timeout at 29s looks identical to a fast 200 if you dropped the trace at the edge.
• Biased RED if you derive metrics from stored traces — error and duration histograms reflect the sampled population, not production.
• Bad for rare failures — a 0.1% error rate may disappear entirely at 1% head sampling.

Cost tip: Head sampling is appropriate for high-volume, homogeneous paths (health checks, static assets) where you mainly need volume trends, not per-route SLO proof.

Tail sampling: pros, cons, and performance tradeoffs

Tail sampling waits until a trace is complete (or times out), evaluates rules, then forwards a subset.

Pros

• Keeps errors, high latency, and specific routes while dropping “happy path” noise.
• Better incident debugging — you are far more likely to retain the trace that explains a pager.
• Higher effective signal per dollar than naive head sampling for APM use cases.

Cons

• Collector memory and CPU — buffering spans until policy runs; large traces and long timeouts hurt.
• Delayed export — tail decisions add latency before backends see data.
• Operational complexity — policies must be maintained (latency thresholds, status codes, attribute matches).

Performance tip: Cap trace size and clock skew tolerance in the collector. Run tail sampling on a dedicated pool of collector replicas so application pods are not competing for the same memory limits.

Rule-of-thumb cost math

Suppose 1M spans/minute at $0.30 per million ingested (illustrative vendor rate):

Strategy	Effective keep rate	Approx. ingested spans/min	Relative ingest $
No sampling	100%	1,000,000	100%
10% head	10%	100,000	10%
5% tail (errors + p99 only)	~5–8%	50,000–80,000	5–8%

Tail sampling often lands near head sampling on cost — but with much better debuggability. The hidden cost is running the policy, not the per-span price.

Combining policies (what mature teams do)

1. Head sample noise at the SDK or agent (e.g. 10% on health checks only).
2. Tail sample at the collector with explicit status_code=ERROR, latency thresholds, and service.name allowlists.
3. Never compute production SLO error rates from the tail-sampled trace store alone.

That skew hits exactly the metrics platform owners trust for autoscaling and release gates: Rate, Errors, Duration derived from trace-derived span metrics.

What to do instead of trusting skewed RED

• Treat trace stores as debug archives, not billing-grade metric sources.
• Emit native metrics (histograms/counters) from the app or collector for SLOs.
• Or use a span metrics connector that understands sampling and aggregates before bias accumulates — covered in the next article in this series.

Quick decision checklist

Question	Lean head	Lean tail
Need guaranteed error exemplars?	No	Yes
Traffic mostly uniform and cheap to lose?	Yes	No
Collector memory budget tight?	Yes	No
Using traces as sole RED source?	Avoid both without correction	Avoid without correction

Closing: performance vs truth

Tail sampling wins performance per dollar for investigations. Head sampling wins simplicity and lowest ingest. Neither is free when backends turn spans into RED time series — conditional retention introduces skew into rates and histograms that look authoritative on dashboards.

Continue the series: How the span metrics connector fixes RED skew after tail sampling — then model the tradeoff in our observability cost calculator.

Sources & further reading

• OpenTelemetry: Tail sampling — when decisions happen after trace completion
• OpenTelemetry Collector: tail_sampling processor — policy configuration reference
• Grafana Tempo: Trace metrics — how backends derive metrics from traces