Apache Pulsar

3.x

stableCost: highTeam: seniorLatency: tens of msDurability: strong

Summary

Distributed, multi-tenant pub-sub and streaming platform that separates the serving layer (stateless brokers) from the storage layer (Apache BookKeeper), natively supports both queue-style and stream-style consumption on the same topic, and tiers cold segments to object storage without a separate compaction process.

Primary Use Case

Event streaming and CDC transport where Kafka's throughput and replay model are needed but broker/storage coupling, multi-tenant isolation, or built-in geo-replication are also required, and the team can support a two-component (broker plus BookKeeper) operational model in exchange for those properties.

Workload Fit

event streaminghigh throughput writescdc pipelineasync decoupling

Strengths

Best for

·Event streaming requiring both queue-style and stream-style consumption on the same topic without duplicating infrastructure
·Multi-tenant platforms needing native namespace-level resource isolation between unrelated workloads on one cluster
·CDC and high-throughput pipelines where independent scaling of storage versus broker capacity is valuable

Excels when

·Both competing-consumer (queue) and consumer-group (stream) semantics are needed from the same underlying topic
·Multi-tenant isolation between workloads on a shared cluster is a hard requirement, not an application-layer convention
·Broker failure recovery time matters more than avoiding a second operational component (BookKeeper)

Architectural advantages

·Broker/storage separation means broker failure requires no data re-replication, unlike Kafka's ISR catch-up
·Multiple subscription types (exclusive, shared, failover) on one topic serve both queue and stream consumption patterns natively
·Native tiered storage to object storage without a bolt-on compaction process
·First-class multi-tenancy (tenants, namespaces) for resource isolation on a shared cluster

When to Avoid

Avoid when

·Team size or maturity cannot support two operationally distinct components (brokers and BookKeeper): this is a higher operational floor than Kafka, not a lower one
·The workload is a simple work queue at low-to-medium volume: the multi-tenant, dual-consumption-model complexity is unwarranted
·The organization already has deep Kafka operational expertise and no specific need Pulsar's broker/storage split addresses

Common misuses

·Treating Pulsar as a drop-in Kafka replacement without validating consumer API and semantic differences, notably subscription types versus consumer groups
·Using shared subscriptions and then relying on message ordering: shared mode is explicitly unordered, competing-consumer delivery
·Under-sizing the BookKeeper ensemble because Kafka experience suggests broker replica count alone determines durability

Consistency & Transactions

Consistency modeleventual

ACID compliantNo

Supports transactionsYes

Scaling

Characteristics

horizontal writehorizontal readsharded

Operational burdenhigh

Typical read latency5 ms

Typical write latency5 ms

Read scalability

Exclusive and failover subscriptions provide Kafka-consumer-group-style parallel reads across partitions; shared subscriptions provide RabbitMQ-style competing-consumer fan-out within a single partition, a mode Kafka consumer groups do not support (a Kafka partition binds to exactly one consumer within a group).

Write scalability

Write throughput scales horizontally by adding partitions and bookies independently: because storage is decoupled from the broker layer, storage capacity and broker throughput scale on separate axes, unlike Kafka where both are bound to the same broker's local disk.

Failure Behavior

Known failure modes

·BookKeeper ensemble under-replication: if too few bookies acknowledge a write, durability drops below the configured ensemble/write-quorum/ack-quorum settings
·Broker failure requires no data movement (data lives in BookKeeper, not on the broker), but a bookie failure does require re-replication, analogous to Kafka's ISR catch-up
·Shared subscription mode trades strict ordering for competing-consumer fan-out: message order across the topic is not preserved once more than one consumer is attached
·Two-component operational model (brokers plus bookies) doubles the failure surface an operator must monitor compared to Kafka's single-component brokers

Bottlenecks

·BookKeeper write-quorum/ack-quorum configuration bounds write latency the same way Kafka's acks=all and min.insync.replicas do
·Two-component capacity planning (brokers and bookies) adds a coordination dimension Kafka's single-component model does not have

Degradation patterns

·Bookie under-replication during node loss requires re-replication before durability targets are restored, similar in effect to Kafka under-replicated partitions
·Shared-subscription consumer imbalance causes uneven per-consumer load since dispatch is round-robin, not partition-affinity-based

Recovery considerations

·Broker replacement requires no partition data movement since storage lives in BookKeeper, a faster recovery path than Kafka broker replacement
·Bookie replacement does require re-replication of the segments it held, bounded by ensemble size and network bandwidth
·Geo-replication (built-in) provides cross-cluster recovery options Kafka requires MirrorMaker or a third-party tool to achieve

Operational Pitfalls

·Running brokers and bookies without understanding the broker/storage separation: capacity planning must account for both layers independently
·Using shared subscriptions where strict per-key ordering is required: shared mode is explicitly competing-consumer, not ordered
·Under-provisioning BookKeeper ensemble size for the required durability level: default settings are not automatically production-safe

Architecture Guidance

Common topology roles

event streamcdc transportasync busevent log

Migration notes

·From Kafka to Pulsar: partition-level ordering and consumer-group semantics carry over conceptually, but subscription type (exclusive/shared/failover) must be chosen deliberately, it has no direct Kafka equivalent
·From RabbitMQ to Pulsar: shared subscriptions provide the closest equivalent to a competing-consumers work queue, but Pulsar retains a replayable log underneath, unlike RabbitMQ's consumption-destructive queue

Advisor Guidance

Info

When: scenario needs both queue-style and stream-style consumption from the same event data

Pulsar's subscription types (exclusive, shared, failover) support both consumption models on one topic; evaluate before standing up separate Kafka and RabbitMQ deployments for the two use cases

Warning

When: scenario has team_maturity below senior

Pulsar's two-component operational model (brokers plus BookKeeper) requires at least the same operational maturity as Kafka, and adds a second component to monitor: consider a managed offering (StreamNative Cloud) to reduce ops burden

Comparison Factors

operational complexity

High: two distinct components (brokers, BookKeeper) to operate, a higher floor than Kafka's single-component model

high

throughput

Very high: comparable to Kafka at the high end, with independent broker/storage scaling

high

consumption model flexibility

Highest of the broker profiles: exclusive, shared, and failover subscriptions on one topic cover both queue and stream semantics

high

broker failure recovery

Faster than Kafka: broker replacement requires no partition data movement since storage lives in BookKeeper

high

Managed Cloud Options

StreamNative CloudDataStax Astra Streaming

Enables Patterns

write ahead log cdcevent sourcingcompeting consumers

Basis

Mature, production-deployed technology (Apache top-level project) with documented broker/storage separation and subscription-type semantics; smaller operational deployment base than Kafka, so benchmark and incident-pattern documentation is less extensive

Sources & Claims

Pulsar separates the serving layer (stateless brokers) from the storage layer (Apache BookKeeper); a broker holding a partition's traffic can fail and be replaced without any partition data movement, because the data was never stored on that broker's local disk

pending

official documentation · Apache Pulsar documentation, Concepts and Architecture: Broker and BookKeeper

messaging-knowledge-completion batch, added alongside SQS and Google Pub/Sub profiles

A single Pulsar topic supports multiple subscription types concurrently: exclusive (single consumer), shared (round-robin competing consumers, unordered), and failover (one active consumer with automatic standby promotion), letting one topic serve both queue-style and stream-style consumption

pending

official documentation · Apache Pulsar documentation, Concepts and Architecture: Subscriptions

messaging-knowledge-completion batch