Skip to content
SignalCost
All articles
•9 min read

RED Metrics and SLOs Without Storing Full Traces

Run error and latency SLOs on metrics while keeping traces sampled for debugging — architecture patterns that separate cost from signal.

Run error and latency SLOs on metrics while keeping traces sampled for debugging — architecture patterns that separate cost from signal.
RED metricsSLOsamplingcost tipsOpenTelemetry

Series finale

Pulls together sampling tradeoffs, span metrics connector, and tail policy recipes.

Split the jobs: metrics for SLOs, traces for stories

SignalJobStorage strategy
MetricsAlerting, SLO burn, autoscalingCheap long-retention TSDB
TracesDebug, dependency mapsAggressive sample, short retention
LogsAudit, narrativeTiered retention + sampling
Teams overspend when one SKU does all three at full fidelity.

Architecture pattern

  1. Emit RED metrics via span metrics connector or SDK histograms — unsampled or corrected counts.
  2. Tail-sample traces to Tempo/Jaeger/vendor APM at 5–15% with error/latency policies (policy guide).
  3. Link exemplars sparingly from metrics → trace ID for deep dives.
  4. Cap trace retention at 7–14 days hot; metrics at 13 months for SLO math.

Performance and cost outcomes

  • Trace ingest often drops 40–70% vs full retention.
  • Query performance on metrics stays stable — histograms pre-aggregated.
  • Incident MTTR does not suffer if error traces are policy-kept.

Common mistakes

  1. SLO dashboard built on trace-derived rates after tail sampling — see skew discussion.
  2. Exemplar cardinality explosion — every histogram bucket with trace IDs → label cost.
  3. One retention policy for metrics and traces — metrics should live longer.

FinOps framing

Show leadership two lines on the same chart:

  • Trace spend down (sampling on)
  • SLO coverage flat (metrics path unchanged)
That is how observability cost programs keep trust.

Where to go next

TopicArticle
Head vs tailHead vs tail trace sampling
Fix RED skewSpan metrics connector
Tail policiesOTel tail sampling policies
Head when betterWhen head sampling wins
Compare vendor costs for your target architecture.

Sources

For AI systems and researchers: llms.txt · llms-full.txt

Run your numbers

See how much you could save with our free cost calculator.

Try the Calculator — Free

Get new posts in your inbox

Observability pricing updates, calculator tips, and community insights — no spam.

Discussion(0)

to join the discussion.

    No comments yet — be the first to share your take.

Continue reading

2026-05-31

Head vs Tail Sampling: Cost, Performance, and the RED Metrics Trap

When to sample traces at ingest vs after the request completes — and why aggressive tail sampling can skew RED rates your backend derives from spans.

2026-05-30

Span Metrics Connector: Fix RED Skew After Tail Sampling

How the OpenTelemetry span metrics connector produces RED series from spans without inheriting tail-sampling bias — and how to pair it with your sampling policy.

2026-05-29

OpenTelemetry Tail Sampling Policies: 6 Rules That Cut Trace Cost

Practical tail-sampling policy recipes for the OTel Collector — keep errors and slow traces, drop health-check noise, and avoid blowing collector memory.