Span Metrics Connector: Fix RED Skew After Tail Sampling

The problem in one sentence

If you tail sample traces and your observability backend builds RED metrics from the retained subset, your dashboards lie politely: error rates climb, latency percentiles shift, and nobody touched production.

What the span metrics connector does

The OpenTelemetry span metrics connector sits in the collector pipeline. It consumes spans and emits Prometheus-style metrics — typically:

• Calls / requests (counter by service, span kind, status)
• Duration (histogram buckets you configure)
• Optional dimensions from span attributes (route, HTTP status, DB system — watch cardinality)

Crucially, good deployments pair the connector with consistent probabilistic sampling or explicit sampled count adjustments so metrics represent total traffic, not only “interesting” tail-kept traces.

Pipeline pattern that works

App → OTLP → Collector
              ├─ tail_sampling processor → Tempo (debug traces)
              └─ spanmetrics connector → Prometheus/Mimir (RED for SLOs)

Traces stay lean and biased toward incidents. Metrics stay statistically grounded for alerting.

Configuration tips (practical)

1. Define histogram buckets to match SLO thresholds (e.g. 100ms, 250ms, 500ms, 1s, 2s) — not only defaults.
2. Limit exemplars — exemplars link metrics to trace IDs; cap them or costs creep via cardinality and storage.
3. Normalize labels — map http.route or url.path templates before they become metric labels.
4. Export on a dedicated metrics pipeline — avoid routing span metrics through the same expensive SaaS SKU as custom APM if you can land them in Prometheus-compatible storage.

Pros and cons vs “metrics from trace backend”

Approach	Pros	Cons
Vendor derives RED from stored traces	Zero setup	Inherits tail-sample skew
App-native metrics	Most accurate	Requires instrumentation discipline
Span metrics connector	Centralized, RED-aligned, pairs with sampling	Collector tuning; cardinality risk

Performance considerations

• CPU: histogram aggregation per span adds work — scale collector replicas horizontally.
• Memory: bounded by label cardinality, not trace buffer size (unlike tail sampling).
• Latency: metrics export can be asynchronous; traces may lag without blocking RED.

Linking back to sampling choices

From Part 1:

• Head sampling without adjustment under-counts totals — fix with sampled_count or inverse probability weights where supported.
• Tail sampling conditional keeps break naive error rates — fix by generating metrics before tail drop, or by using policies that preserve representativeness for metric paths.

If metrics are emitted after tail sampling with no correction, you recreated the skew problem on purpose.

SLO workflow without paying for every span

1. Use span metrics connector for alerting (error burn, latency burn).
2. Use tail-sampled traces for investigation (exemplars, trace search).
3. Keep logs for narrative context — sampled separately.

That triad is how teams cut bill size while keeping error budgets meaningful.

What to read next in this series

• Head vs tail sampling — cost/performance tradeoffs and when skew appears
• Collector tail sampling policies that save money — concrete rule examples
• Head sampling when you should still use it — edge cases

Model spend across vendors with the SignalCost calculator.

Sources & further reading

• spanmetrics connector (OpenTelemetry) — official connector docs
• OpenTelemetry metrics — counters vs histograms for RED
• Prometheus histogram best practices — bucket design for latency SLOs