Event streaming architecture: Kafka vs. Kinesis vs. Pulsar

Event streaming architecture is the single infrastructure choice that most often gets mis-scoped as "just a queue." The primary mistake is equating raw throughput with total cost of ownership: a 100k messages/sec producer is a different problem than 100k small messages/day with multi-day retention and cross-region replication.

Direct answer: If you need low-ops, AWS-only deployments with predictable throughput under 50k events/sec and 7–14 day retention, Amazon Kinesis will generally be the cheapest operational path at <$2k/month for typical mid-market workloads; if you need enterprise features (connectors, schema registry, transactions) and 99.99% SLAs, Confluent Cloud or AWS MSK is the right buy at $3k–$15k/month; if you need sub-10ms local latency and single-region high-throughput, Redpanda or self-hosted Kafka can cut tail latency by 3× but costs shift to people (expect a 1–2 FTE SRE burden, ~$200k/yr).

Three concrete stakes: egress and retention bill line items blow up quickly — at $0.09/GB egress, moving 10TB/month costs roughly $900/month; a 30-day retention storing 1TB/day equals 30TB of storage and multi-thousand-dollar monthly bills. Second, developer time: a single misconfigured retention can trigger reprocessing jobs that pull down to 30% of engineering bandwidth for a week. Third, SLAs: noisy neighbor incidents on shared streams create 3–8 hour outages unless you invest in isolation (dedicated brokers or shards).

Event streaming architecture: guarantees, latency, and operational surface

Start with delivery guarantees. Apache Kafka supports idempotent producers and transactions that enable end-to-end exactly-once semantics for many stream processing topologies. Amazon Kinesis Data Streams provides at-least-once delivery and relies on client-side de‑duplication for idempotency. Apache Pulsar offers topic-level retention and tiered storage, and its architecture separates serving from storage — which simplifies geo-replication but changes latency characteristics.

Latency and throughput trade-offs map to architecture. In practice, managed Kafka offerings like Confluent Cloud or AWS MSK typically deliver 10–50ms publish-to-broker latency inside a region and 50–200ms end-to-end depending on consumers. Redpanda and tuned self-hosted Kafka can push single-digit-millisecond tail latency for local consumers. Amazon Kinesis typical write latency is 50–300ms and is region-optimized for durability and availability rather than sub-10ms tail latency.

Operational surface and cost. A small MSK deployment (3 brokers with moderate storage) runs in the low-thousands per month; Confluent Cloud starts around $3k/mo for production clusters with connectors and schema registry. Amazon Kinesis shard pricing is roughly $0.015/shard-hour (≈$11/shard-month) plus PUT costs, which means modest costs for predictable workloads. Self-hosted Kafka introduces hardware, networking, monitoring, and people costs: expect 1 SRE at $180k–$220k/yr to manage a cluster reliably, plus 2–3 smaller incidents per quarter that consume cross-functional engineering time.

Ecosystem and connector story matters when you actually ship. Kafka's ecosystem (Kafka Connect, Kafka Streams, ksqlDB, Confluent Connectors) accelerates integrations with databases, S3, and analytics tooling; Kinesis leans on AWS-native integrations (Firehose, Lambda, Glue, Kinesis Data Analytics) and reduces vendor-surface area if you are already on AWS. Pulsar's function framework and tiered storage make it attractive when you require low-cost cold storage for long retention at petabyte scale.

Pick a stream not for raw throughput but for the level of operational responsibility you can staff: the tool that fits your team's headcount wins more often than the tool with the highest benchmark.

What this means for a CTO or technical founder

You must convert product requirements into three measurable knobs: peak events/sec, retention days, and recovery RTO/RPO. If your product needs <50k events/sec, 7–14 day retention, single-region analytics, and you want minimal SRE, default to Amazon Kinesis or Google Pub/Sub. That choice buys you fallback integrations and reduces the need for a dedicated streaming SRE on day one.

If you need exactly-once processing across multiple stateful stream processors, require connectors to enterprise data warehouses, or expect multi-region active-active traffic, choose Confluent Cloud or AWS MSK. Budget $3k–$12k/month for managed Kafka in production and plan for 0.5–1.5 FTE of platform engineering to own schema governance, connector ops, and incident runbooks.

For sub-10ms tail latency, very high throughput in a single region, or when you need to keep egress low for cost reasons, consider Redpanda or a tuned self-hosted Kafka cluster. Accept that you are trading platform spend for headcount: the marginal cost is not just hardware but ~1 SRE ($180k/yr) plus observability (Prometheus+Grafana, prod pipelines, superset of alerts).

Key takeaways — practical rules to apply today

1) Map your 95th percentile throughput, retention, and multi-region needs before choosing a provider; 2) If you have fewer than 15 engineers and no 24/7 ops, prefer AWS Kinesis or managed Confluent to avoid a 1–2 FTE ops tax; 3) If exactly-once semantics and connector breadth matter, pay for managed Kafka ($3k–$12k/mo) rather than betting on client-side de-duplication; 4) If you need sub-10ms tail latency, invest in Redpanda/self-hosted Kafka and budget an SRE; 5) Treat egress at $0.09/GB and storage at scale as first-order budget items.

A short evaluation checklist you can run this week: 1) measure production 95th-percentile event size and events/sec for one busy day, 2) calculate 30-day storage and monthly egress at $0.09/GB, 3) simulate burst 2×–5× the baseline and see how your candidate handles retention and backpressure, and 4) run a dry incident-resolve drill to measure time-to-recovery.

Choosing an event streaming architecture is not binary. It is a set of economic and operational trade-offs. You can buy isolation and features at $3k–$15k/month with managed providers, or you can reduce unit cost by shifting responsibility onto people and tooling; that shift is expensive in teams and often invisible in early budgets. Set the decision by a staffing and SLA equation, not a benchmark chart.