Why 70% of Your Observability Spend Is Wasted (And How to Fix It)

The $2 Billion Waste Problem

According to Chronosphere's observability cost research, roughly 70% of enterprise observability spend goes to storing logs that are never queried. Gartner notes that observability has become a distinct discipline as organizations consolidate monitoring stacks — yet many still pay for 10–20+ tools simultaneously.

Let that sink in: most companies are paying more for observability data they don't use than for data they do.

This isn't because engineers are careless. It's a structural problem:

1. Default instrumentation is verbose — SDKs and frameworks emit everything by default
2. Fear of missing data — "We might need that log someday" keeps the firehose running
3. No cost attribution — nobody knows which team or service is driving costs
4. Pricing opacity — complex billing models make it hard to trace spend to value

How to Audit Your Observability Spend

Step 1: Map Your Current Costs

Start with a simple inventory:

Tool	Annual Cost	Primary Use	Data Types
?	?	?	?

Include every observability-related tool: APM, logging, metrics, alerting, synthetics, status pages, incident management. Many teams are surprised to find they're spending more than they thought.

Step 2: Measure Data Utilization

For each data source, answer:

• When was it last queried? (Check your platform's usage analytics)
• Which dashboards use it? (Audit for stale dashboards)
• Which alerts depend on it? (Map alert → data source)
• Would an incident investigation need it? (Be honest)

Categorize each source:

Category	Definition	Action
Critical	Queried daily, used in active alerts/dashboards	Keep, optimize retention
Useful	Queried weekly, needed for investigations	Keep, reduce retention
Dormant	Not queried in 30+ days	Archive to cold storage
Waste	Never queried, no alerts, no dashboards	Drop entirely

Step 3: Calculate Waste Percentage

For most organizations, the breakdown looks like:

• Critical: 15-25% of data volume
• Useful: 10-20% of data volume
• Dormant: 20-30% of data volume
• Waste: 30-50% of data volume

If your numbers are similar, you're paying for 2-3x more data than you actually need.

The Five Biggest Sources of Waste

1. Health Check and Heartbeat Logs

Every load balancer, Kubernetes probe, and uptime checker generates success logs every few seconds. These create enormous volume with near-zero diagnostic value.

Fix: Filter these at the collector level. You already have alerts for when they fail — you don't need logs for when they succeed.

2. DEBUG and TRACE Level Logs in Production

Development log levels left on in production are the most common form of waste. A single service emitting DEBUG logs can generate 10-100x the volume of INFO-only logging.

Fix: Enforce INFO as the production default. Allow dynamic level changes for incident debugging only, with auto-revert after a time window.

3. Redundant Full-Body Request/Response Logging

APIs that log entire request and response bodies for every call generate massive volumes. At 5KB per request and 10,000 RPM, that's 72GB/day from one service.

Fix: Log request/response bodies only for errors. For successful requests, log only the metadata (method, path, status, duration).

4. Stale Dashboards and Unused Alerts

Gartner notes that most enterprises have dashboards that haven't been viewed in 60+ days and alerts that fire but are routinely ignored (Gartner: observability). Each one keeps data pipelines running and storage allocated.

Fix: Audit dashboards and alerts quarterly. Delete anything not viewed or acted on in 60 days. If someone needs it later, it can be recreated.

5. Duplicate Data Across Tools

If you're sending the same logs to Splunk (for security), Datadog (for APM), and S3 (for compliance), you're storing the same data 3x. Each copy has separate ingest, storage, and query costs.

Fix: Use an observability pipeline (OpenTelemetry Collector, Vector, or Cribl) to route data to a single destination per use case. Send security logs to your SIEM only. Send APM data to your observability platform only.

The Quick Win Checklist

Do these 7 things this week for immediate savings:

1. ☐ Drop health check and heartbeat success logs
2. ☐ Set all production services to INFO log level
3. ☐ Truncate request/response bodies to metadata only
4. ☐ Delete dashboards not viewed in 60+ days
5. ☐ Disable alerts with >50% ignore rate
6. ☐ Normalize URL paths in metric labels to fix cardinality
7. ☐ Set hot retention to 14 days (with explicit exceptions only)

Most teams that complete this checklist see a 20-40% cost reduction within one billing cycle.

Build a Cost-Conscious Culture

The long-term fix isn't just technical — it's cultural:

• Make costs visible: Show observability spend per team on an internal dashboard
• Set team budgets: Give each team an observability budget and let them manage it
• Reward efficiency: Celebrate teams that reduce costs without losing visibility
• Shift ownership: Move observability budget under engineering, not IT operations

When engineers see the cost of their telemetry decisions, they naturally start making better ones.

What's Next?

1. Run the cost calculator to benchmark your current spend
2. Complete the 7-point quick win checklist above
3. Read our guide to cutting costs by 50% for deeper strategies

Sources & further reading

• Chronosphere: observability cost management — research on telemetry waste and spend attribution
• Gartner: observability — industry definition and consolidation trends
• FinOps Foundation — cloud financial management practices applicable to observability budgets