Observability build vs buy: Datadog, Grafana, and the 3‑year TCO

Observability build vs buy should be framed as a three‑year capital allocation problem, not a checkbox on a procurement form. Most technical leaders treat Datadog, Grafana Cloud, Honeycomb, and Elastic as feature sets; the actual decision is a dollars-and‑engineers tradeoff with concrete breakpoints.

Direct answer: Buy hosted monitoring if your annual SaaS bill is under $120k or you have fewer than 2 dedicated SRE/observability engineers; build when your SaaS line item exceeds $250k/yr, your retention needs exceed ~20 TB/month, or you require strict data residency or custom ingest pipelines. For a 3‑year horizon, a typical self-hosted stack requires 1.5–3.0 FTEs and ~$40k–$120k/yr infrastructure but can cut recurring SaaS spend by 30–60% after year two.

Concrete stakes: a single senior SRE (loaded) is roughly $200k–$240k/year in the U.S.; a mid-market Datadog deployment for 200 hosts with 10 TB/month logs and APM can be $120k–$350k/year depending on retention and high-cardinality metrics. A self-hosted Grafana/Prometheus/Loki/Cortex stack for the same scale will need ~2 FTEs and $40k–$80k/yr in AWS/GCP infra plus $0–$25k/yr in commercial licenses.

Observability build vs buy: a concrete three‑year model

Start with the predictable line items. Hosted monitoring bills have three levers you control: hosts/agents, metrics cardinality, and log volume/retention. Datadog, New Relic, and Splunk publish per-host and per-GB tiers; for example, 200 hosts at $15/host/mo is $36,000/yr for infrastructure metrics alone. Add APM at $0.10–$0.50/trace or $20–$40/host/mo and logs at $0.10–$0.30/GB-ingested depending on retention, and you quickly hit $120k–$350k/yr.

Self-hosted costs split differently: human capital and infra. A conservative production team needs 1.5–2.5 FTEs to run Prometheus/Cortex for high-cardinality metrics, Grafana for visualization, and Loki/Tempo for logs/traces. Using a $220k loaded salary, 2 FTEs is $440k/yr or $1.32M over three years. Add infrastructure—EKS nodes, S3/Cold storage, EBS, load balancers—at $3k–$8k/month ($36k–$96k/yr), plus backup and disaster recovery. Over three years that’s $108k–$288k.

Quotable numbers: S3 storage costs $0.023/GB‑month; 20 TB retained hot is $460/mo or $5,520/yr. Logs compressed to S3 for long-term retention at 50 TB/month raw equals ~$1,150/mo or $13,800/yr in storage alone. If your observability retention policy forces you to keep 6–12 months of logs, storage line items become the dominant cost regardless of vendor.

Break-even math: if hosted SaaS is $250k/yr and self-hosted humans+infra cost $500k/yr, you’re behind in year one but may cross over by year two or three as SaaS grows with ingest and retention. In many real cases the crossover point is when SaaS spend exceeds ~60–70% of a single senior SRE’s loaded cost ($130k–$170k/yr), because one SRE can architect significant operational efficiencies in a self-hosted stack that reduce marginal costs later.

Buy hosted observability until the bill approaches the cost of one senior SRE or your retention needs force long‑term storage; after that, self-hosting is a financial lever, not a vanity project.

What this means for a CTO: thresholds, risk, and hybrid patterns

You control three knobs when choosing: cost profile (opex vs capex), engineering runway (are 1–3 SREs available?), and non‑financial constraints (compliance, data sovereignty, feature parity). If you’re a 20‑engineer startup with no dedicated SRE, you should not build. A $30k–$120k/yr hosted bill buys you paging configuration, managed upgrades, and integrations you won’t produce reliably in months.

If you run a platform with 100+ engineers, 1,000+ services, or strict retention/PII rules, plan to invest. For example: if your logs retention policy is 12 months at 20 TB/month, S3 costs alone are ~$66k/yr—this shifts the calculus toward self-hosting or a hybrid model where you self-host long-term storage and use hosted providers for short-term high-cardinality analysis.

Adopt a hybrid approach when you need fast time-to-value and longer-term cost control: use Datadog, Grafana Cloud, or Honeycomb for 7–30 day hot windows and operational alerts, and backfill long‑term retention to self-hosted S3/warehouse (compression + cold storage). You’ll pay two bills short-term, but you cap hosted bill growth and avoid retention shock when your data volume scales 3–10×.

3-step checklist to decide (hosted monitoring vs self-hosted)

1. Calculate your current and projected annual SaaS observability spend and compare it to one senior SRE’s loaded cost; if SaaS > 60% of that salary, model a build scenario for 3 years.
2. Quantify retention and legal constraints: if you need >20 TB/month hot retention or strict residency (GDPR/PCI/HIPAA), weight self-hosting or vendor with on‑prem options higher.
3. Model human effort: if you can staff 1.5–3.0 dedicated observability engineers for the duration of the migration, your probability of success rises sharply; otherwise choose hosted and invest the saved time into product differentiation.

Key operational tradeoffs: a self-hosted Cortex cluster lowers marginal metric costs but increases mean time to repair for ingest spikes; Grafana Enterprise adds access controls and reporting but costs $10k–$50k/yr; Grafana Cloud or Elastic Cloud can defer engineering headcount for 12–24 months at a premium of 25–80% versus self-hosted infra.

Vendor lock and switching costs matter: migrating logs/metrics/traces out of Datadog or Splunk is nontrivial—expect months of re-ingest, historical data mapping, and a migration cost equal to 10–30% of your annual SaaS bill. That switching friction is a legitimate reason to stay with hosted if the SaaS bill is under your break-even threshold.

OSS and standards reduce risk: instrument with OpenTelemetry from day one so you can switch backends without re-instrumenting services. OpenTelemetry reduces switching friction and lowers incremental migration costs from months to weeks in practice.

When you build, design for composability: separate ingest, real-time processing, and long-term storage. Use object storage (S3/GS) for cold logs, a metrics long-term store like Cortex with bucketed retention, and a query layer (Grafana) that can federate both. This architecture keeps marginal costs predictable and enables a later move back to hosted telemetry if desired.

Final operational rule: prioritize signal cost over raw volume. High-cardinality labels and low-value log noise are where hosted bills explode. Invest an engineer to reduce cardinality and noise before you buy a bigger plan—this reduces both SaaS and infra spend by 20–40%.

Key takeaways

1. Buy hosted monitoring if your annual SaaS spend is under $120k or you lack 1.5 dedicated SREs; it buys speed and reduces operational risk.
2. Build when SaaS spend exceeds ~$250k/yr, retention needs exceed ~20 TB/month, or you require strict data residency—self-hosting typically shows positive TCO by year two or three.
3. Use OpenTelemetry and a hybrid architecture to limit vendor lock, cap hosted growth, and keep migration paths open.
4. Apply human-capital math: one senior SRE (~$200k–$240k/yr loaded) is the right unit to compare against recurring SaaS spend when modeling a 3‑year decision.

Observability build vs buy is a solvable financial decision, not a philosophical one. Buying is the right choice when you need speed, a small team, and predictable Opex; building is the right choice when your data volumes, retention policy, or compliance requirements make hosted pricing untenable. Use the thresholds above, instrument with OpenTelemetry, and treat retention as the dominant cost lever—do that and you convert observability from a runaway expense into a predictable platform investment.