Event streaming architecture: Kafka vs. Kinesis vs. Pulsar

Event streaming architecture is the single infrastructure choice that most often gets mis-scoped as "just a queue." The primary mistake is equating raw throughput with total cost of ownership: a 100k messages/sec producer is a different problem than 100k small messages/day with multi-day retention and cross-region replication.

Direct answer: If you are on AWS, want no brokers to run and can make consumers idempotent, start with Amazon Kinesis Data Streams: in provisioned mode a shard costs $0.015 per hour (about $11 a month) and takes up to 1 MB/s or 1,000 records/s of writes (AWS pricing, AWS quotas). If you need exactly-once processing across topics, Kafka Connect or Kafka Streams, choose managed Kafka (Confluent Cloud or Amazon MSK). If you need very long retention on cheap storage or built-in geo-replication, look at Pulsar. Self-hosting any of them moves the cost from the vendor bill to the people who run it.

Three stakes are easy to underestimate. Data transfer: AWS charges $0.09/GB for the first 10 TB a month sent from EC2 to the internet (AWS EC2 pricing), so shipping 10 TB a month to consumers outside AWS costs roughly $900 before any streaming fees. Retention: Kinesis keeps data for 24 hours by default and can keep it for up to 365 days at extra cost, so every extra day of replay is a line item you choose. Duplicates: at-least-once systems redeliver records after producer retries and consumer restarts (AWS); if downstream writes are not idempotent, a single restart can double-count orders or payments.

Event streaming architecture: guarantees, latency, and operational surface

Start with delivery guarantees. Since Kafka 0.11 (2017), Apache Kafka has idempotent producers and transactions that write to several partitions and commit consumer offsets atomically, which is what makes exactly-once processing in Kafka Streams possible (Confluent). Amazon Kinesis Data Streams can deliver a record more than once, and AWS tells applications to embed a primary key and de-duplicate downstream (AWS). In Apache Pulsar the brokers are stateless and messages are stored in Apache BookKeeper, and clusters can replicate to other regions with built-in geo-replication (Apache Pulsar docs).

Latency and throughput follow from the architecture, and published benchmarks rarely match your message sizes, batching and consumer code. Kafka and Pulsar let you tune batching, replication and acknowledgement settings for lower latency or higher durability. Kinesis caps each shard at 1 MB/s or 1,000 records/s of writes and 2 MB/s of reads, so throughput scales by adding shards (AWS quotas). Measure end-to-end latency on your own workload before you commit.

Operational surface and cost. Kinesis in provisioned mode bills $0.015 per shard-hour plus $0.014 per million PUT payload units of 25 KB, in US East (AWS pricing). One shard written at its full 1,000 small records per second for a month is about 2.6 billion payload units: roughly $37 in PUT charges plus $11 for the shard. Managed Kafka is priced by cluster or by throughput, so get a quote for your own peak load and retention rather than relying on a rule of thumb. Self-hosting Kafka or Pulsar adds servers, storage, monitoring and on-call time. For scale, the US median wage for software developers was $135,980 in May 2025 (BLS), so even part of an engineer's year spent on a cluster can outweigh the managed-service bill.

Ecosystem and connector story matters when you actually ship. Kafka's ecosystem (Kafka Connect, Kafka Streams, ksqlDB, Confluent Connectors) accelerates integrations with databases, S3, and analytics tooling; Kinesis leans on AWS-native integrations (Firehose, Lambda, Glue, Kinesis Data Analytics) and reduces vendor-surface area if you are already on AWS. Pulsar's function framework and tiered storage make it attractive when you require low-cost cold storage for long retention at petabyte scale.

Pick a stream not for raw throughput but for the level of operational responsibility you can staff: the tool that fits your team's headcount wins more often than the tool with the highest benchmark.

What this means for a CTO or technical founder

Convert product requirements into three measurable knobs: peak events per second, retention days, and recovery targets (RTO/RPO). If you are on AWS, your peak fits a manageable number of shards, a day to a week of replay is enough, and your consumers can de-duplicate, default to Amazon Kinesis (or Google Pub/Sub on Google Cloud). That buys you native integrations and no brokers to patch.

If you need exactly-once processing across multiple stateful stream processors, connectors to data warehouses and databases, or multi-region replication, choose managed Kafka (Confluent Cloud or Amazon MSK) and price it on your real throughput and retention. Someone still has to own schema governance, connector operations and incident runbooks, so name that owner before you launch.

If you need the lowest possible latency in a single region, very high throughput, or full control over where data lives, consider a self-hosted Kafka-compatible cluster such as Apache Kafka or Redpanda. You trade the vendor bill for headcount: hardware is only part of the cost, alongside monitoring (for example Prometheus and Grafana), upgrades, capacity planning and on-call.

Key takeaways — practical rules to apply today

1) Map your 95th-percentile throughput, retention and multi-region needs before choosing a provider. 2) If nobody on the team will carry a pager for brokers, prefer a managed service (Kinesis, Pub/Sub, Confluent Cloud or MSK). 3) If exactly-once semantics and connector breadth matter, use Kafka transactions rather than hand-built de-duplication. 4) If you use Kinesis, design consumers to be idempotent from day one, because AWS documents duplicate delivery. 5) Treat data transfer ($0.09/GB out of AWS for the first 10 TB) and retention as first-order budget items.

A short evaluation checklist you can run this week: 1) measure 95th-percentile event size and events per second for one busy day, 2) work out the shard count or cluster size, retention storage and monthly data transfer at your provider's published prices, 3) replay a burst of 2× to 5× your baseline against each candidate and watch backpressure and consumer lag, and 4) run an incident drill that restarts consumers and check that no record is double-counted.

Choosing an event streaming architecture is not binary. It is a set of economic and operational trade-offs: managed services charge you for isolation and features, while self-hosting lowers unit cost by moving the work onto your team, a cost that rarely shows up in early budgets. Decide on staffing and recovery targets, not a benchmark chart. If you want help sizing it, Eltherion projects start as low as $1,000; tell us what you are building.