Apache Pulsar vs Kafka: Architecture, Trade-offs, and When to Choose Each

Kafka is the default choice for event streaming. It has been in production since 2011, runs at extraordinary scale at companies like Netflix, LinkedIn, and Uber, and has a mature ecosystem of managed services, connectors, and stream processors that no other system can match. If you are picking a message broker today with no specific constraints, Kafka is the reasonable choice.

Pulsar makes a different set of architectural bets. Its compute-storage separation, first-class multi-tenancy, and built-in geo-replication address real operational pain points that Kafka requires bolt-on solutions to solve. For specific workloads — multi-tenant SaaS platforms, globally distributed applications, and systems that need independent scaling of compute and storage — Pulsar is not just competitive with Kafka; it is architecturally superior.

This post covers both systems in depth: how they store messages, how consumers work, what multi-tenancy and geo-replication look like in practice, and where the ecosystem gaps between them actually bite. The goal is not to declare a winner but to give you enough to make an informed decision for a specific problem.

Core Architecture

Kafka: brokers own everything

Kafka’s data model is the commit log. Topics are divided into partitions. Each partition is an ordered, append-only sequence of records stored on broker disk. The partition is the fundamental unit of parallelism, replication, and ordering.

Each partition has a leader broker and zero or more follower replicas. All producer writes go to the leader. Followers replicate asynchronously. The in-sync replica set (ISR) is the subset of followers that are caught up to the leader within replica.lag.time.max.ms (default 30 seconds). When the leader fails, a new leader is elected from the ISR — ensuring no data loss for messages that were written with acks=all.

Storage is coupled to brokers. When a partition’s leader sits on a specific broker, that broker’s disk holds the data. Scaling the cluster means adding brokers and then rebalancing partitions across them — a process that copies partition data between brokers and can saturate network I/O for minutes or hours on large clusters.

Kafka 4.0, released in March 2025, completed the removal of ZooKeeper (KIP-500). The KRaft metadata mode — where a subset of brokers run a Raft-based metadata quorum — is now mandatory. This eliminates the operational burden of a separate ZooKeeper ensemble and raises the practical partition limit from roughly 100K (ZooKeeper bottleneck) to approximately 1.9M. Existing clusters still on ZooKeeper must migrate before upgrading to 4.x.

Consumer coordination uses consumer groups. Consumers with the same group.id share partition assignments; each partition is assigned to at most one consumer in a group at any time. Kafka 4.0 also shipped KIP-848, the next-generation consumer rebalance protocol, which replaces the stop-the-world rebalance with incremental cooperative rebalancing — significantly reducing latency spikes during group membership changes.

Pulsar: compute and storage are separate

Pulsar separates the two responsibilities that Kafka bundles into a broker. Pulsar brokers handle message routing, subscription management, and the producer/consumer protocol. Apache BookKeeper — a separate cluster of nodes called bookies — handles durable storage.

Messages are written to BookKeeper ledgers. A ledger is an immutable, append-only sequence of entries. By default, each ledger is striped across three bookies with a write quorum of two: a write completes when any two of the three bookies acknowledge it. There is no leader bookie for a ledger — the write-quorum model means the failure of any one bookie is transparent to writes in progress. A new ledger is opened on a different set of bookies if a bookie goes offline.

This separation has direct operational implications. Pulsar brokers are stateless. When a broker fails, its topic ownership is transferred to another broker in seconds without any data movement — the data is in BookKeeper, untouched. Adding brokers scales compute capacity; adding bookies scales storage capacity. These happen independently, which eliminates the fundamental coupling problem that makes Kafka cluster operations expensive.

The tradeoff is complexity. A Pulsar cluster requires three subsystems: brokers, BookKeeper, and ZooKeeper (for cluster metadata and leader election). Kafka 4.x needs only brokers. For smaller deployments, Pulsar’s operational surface is meaningfully larger.

The storage rebalancing gap

The practical consequence of Kafka’s coupled architecture surfaces most clearly during rebalancing. When you add brokers to a Kafka cluster, the new brokers are empty. You must explicitly move partitions to them using the partition reassignment tooling. Moving a partition means copying its data from the current broker to the new one over the network, then updating the leader assignment. For a cluster with terabytes of partition data, this process runs for hours and must be carefully rate-limited to avoid starving producers and consumers of I/O.

In Pulsar, adding a bookie does not require moving existing data. New ledgers are created on whatever bookie set BookKeeper selects based on load; old ledgers stay where they are. The cluster rebalances incrementally as new messages arrive, with no manual intervention and no bulk data transfer.

Topic and Subscription Models

Kafka: partition ownership

Kafka’s consumer model is built on partition ownership. Within a consumer group, each partition has exactly one assigned consumer at a time. Ordering within a partition is preserved. Ordering across partitions is not — consumers process their assigned partitions independently.

Message acknowledgment is cumulative. When a consumer commits offset N, it implicitly acknowledges every message from offset 0 through N. There is no mechanism to acknowledge message 500 without acknowledging 499. This matters when processing is non-sequential: if a consumer processes messages 501 through 1000 successfully but fails on 499, it must reprocess everything from 499 onward after a reset.

Dead letter queue support does not exist in Kafka’s protocol. If a consumer fails to process a message, it must implement retry and DLQ logic in application code, typically by writing failed messages to a separate topic and consuming from that topic on a delayed schedule.

Pulsar: four subscription types

Pulsar separates the concept of a subscription (a named position in the topic) from the consumers attached to it. A topic can have multiple subscriptions simultaneously, each maintained at an independent position. This means multiple independent consumer groups can consume the same topic without any coordination, and each group sees every message — true fan-out at the broker level rather than at the application level.

The four subscription types define how messages are dispatched to the consumers attached to a subscription:

Exclusive: A single consumer receives all messages in order. A second consumer attempting to join fails. Useful for ordered, single-consumer processing.

Failover: Multiple consumers can attach, but only one is active at a time. The standby takes over automatically if the active consumer disconnects. Preserves ordering. Useful for leader-follower consumer patterns.

Shared: Messages are dispatched round-robin to any consumer in the subscription. Ordering is not preserved. Scales horizontally without partition constraints. Acts like a work queue. This is the Pulsar equivalent of a load-balanced consumer pool.

Key_Shared: Messages with the same key are always routed to the same consumer. Different keys are distributed across consumers. Ordering is preserved within a key. This is roughly equivalent to Kafka’s partition-per-key model, but without the operational requirement to pre-define partition counts. Adding a consumer triggers a key reassignment (with a brief redelivery window), but does not require any partition metadata changes.

Pulsar supports per-message acknowledgment. A consumer can acknowledge message 500 without acknowledging 499. Unacknowledged messages are redelivered after a configurable timeout. Negative acknowledgment (NACK) allows a consumer to signal that a message should be redelivered immediately rather than waiting for the timeout. Built-in dead letter topic support routes messages to a configurable DLQ after a maximum number of redelivery attempts — no application code required.

Multi-Tenancy

Kafka does not have native multi-tenancy. Topics exist in a flat namespace. ACLs can restrict which clients can produce or consume from which topics, but there is no resource isolation, quota enforcement, or configuration inheritance between tenants. A single aggressive producer can saturate the cluster for everyone else unless you manually configure per-client throttling. The standard workaround for true isolation is separate Kafka clusters per tenant — which is expensive and operationally heavy.

Pulsar’s multi-tenancy model is hierarchical: tenant → namespace → topic.

A tenant is the top-level isolation boundary. A namespace is a logical grouping within a tenant. Topics inherit configuration from their namespace. Policies set at the namespace level — retention, TTL, replication, authentication, authorization, rate limits — apply automatically to every topic in that namespace without per-topic configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Create a tenant
pulsar-admin tenants create customer-a \
  --allowed-clusters us-east,eu-west

# Create namespaces for different environments
pulsar-admin namespaces create customer-a/production
pulsar-admin namespaces create customer-a/staging

# Set retention on the production namespace
pulsar-admin namespaces set-retention customer-a/production \
  --size 10G --time 7d

# Set a backlog quota — stop accepting new messages if backlog exceeds 5GB
pulsar-admin namespaces set-backlog-quota customer-a/production \
  --limit 5G --policy producer_exception

# Enable geo-replication for production namespace
pulsar-admin namespaces set-replication-clusters \
  --clusters us-east,eu-west customer-a/production

A SaaS platform can give each customer a tenant with dedicated namespaces, quotas, and replication policies — all from a single Pulsar cluster. The same configuration in Kafka requires separate clusters per customer or substantial manual per-topic work with no policy inheritance.

Geo-Replication

Kafka: MirrorMaker 2

Kafka’s cross-cluster replication tool is MirrorMaker 2 (MM2), built on Kafka Connect. MM2 consumes from a source cluster and produces to a target cluster. It handles consumer group offset translation — a non-trivial problem because offsets are cluster-specific and do not map 1:1 across clusters.

Setting up MM2 requires deploying and operating a Kafka Connect cluster, configuring source and target connector instances, monitoring consumer lag across the replication boundary, and handling the offset translation mapping that lets consumers fail over from one cluster to another without re-reading all messages. Active-active replication (both clusters accepting writes, both replicating to each other) is possible but requires careful handling of message provenance to prevent replication loops. Topic names on the target cluster are prefixed by convention (e.g., us-east.orders), which means consumer code must be cluster-aware.

MM2 works, but it is an operationally distinct system layered on top of Kafka rather than an integral part of it. In practice, teams that need robust geo-replication with Kafka often upgrade to Confluent Replicator (commercial) or build significant operational tooling around MM2.

Pulsar: built-in

Pulsar’s geo-replication is part of the broker, not an external tool. Enabling replication for a namespace is a single command:

1
2

pulsar-admin namespaces set-replication-clusters \
  --clusters us-east,eu-west customer-a/production

Once enabled, every message produced to a topic in this namespace is asynchronously replicated to all configured clusters. Topics have the same name in every cluster — no namespace prefixing. Message deduplication across regions is handled automatically using producer sequence IDs. The replication cursor is an internal subscription that tracks which messages have been replicated to each remote cluster.

Consumers on either cluster consume from the same topic name. A consumer that fails over from us-east to eu-west begins consuming from roughly the position it left off. The “roughly” is important: replication cursor positions are snapshotted periodically, not per-message, so a failover can result in re-reading a small number of already-processed messages. Idempotent consumers handle this correctly; non-idempotent consumers need to account for it.

The operational difference between the two approaches is significant at scale. Pulsar’s built-in replication means there is one system to monitor instead of two (Kafka + MM2/Connect). It means there is no separate consumer group offset translation layer to debug. It means topic names are consistent across regions. For teams building globally distributed applications where replication is a requirement, not an afterthought, Pulsar starts with a meaningful advantage.

Tiered Storage

Both systems support offloading cold data to object storage (S3, GCS, Azure Blob), but from different starting points.

Pulsar has had tiered storage since early in its lifecycle. BookKeeper ledgers are the unit of offload: when a namespace’s offload policy triggers (by data age or total size), sealed ledgers are copied to object storage and removed from BookKeeper. Brokers serve reads transparently from either BookKeeper (hot) or object storage (cold) — applications see no difference. Configuration is per-namespace:

1
2
3
4
5
6
7

pulsar-admin namespaces set-offload-policies \
  --driver s3 \
  --s3-bucket my-archival-bucket \
  --s3-region us-east-1 \
  --offload-threshold-in-bytes 1073741824 \   # Offload when namespace exceeds 1GB
  --offload-deletion-lag-in-millis 86400000   # Delete BookKeeper copy after 24h
  customer-a/production

Kafka tiered storage (KIP-405) reached general availability in Kafka 3.9, released in March 2025. It works similarly — recent log segments stay on broker disks (hot), older segments are offloaded to remote storage (cold). Configuration is per-topic or globally via broker properties. This is a real and welcome addition, but it is newer than Pulsar’s implementation with fewer production deployments behind it.

The cost implication of tiered storage is material. BookKeeper or EBS costs roughly $0.08–0.10/GB-month at typical cloud pricing. S3 costs $0.023/GB-month — a 4x reduction for cold data. For topics with long retention requirements (audit logs, compliance data, analytics feeds), tiered storage is the difference between a storage bill that scales linearly with retention and one that does not.

Schema Registry

Kafka has no built-in schema registry. The most common solution is the Confluent Schema Registry — an open-source JVM service that stores schemas, assigns schema IDs, and validates producer payloads. Producers embed the schema ID in each message; consumers retrieve the schema by ID to deserialize. The Confluent Schema Registry is operationally separate from Kafka: it requires its own deployment, HA configuration, and monitoring. AWS Glue Schema Registry is the managed alternative for MSK deployments.

Pulsar includes schema management in the broker. Schemas are associated with topics and stored in ZooKeeper alongside other topic metadata. The broker enforces schema compatibility — a producer that attempts to write data incompatible with the topic’s current schema receives an error:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

from pulsar.schema import AvroSchema, Record, Field

class Order(Record):
    order_id = Field(str)
    amount   = Field(float)
    status   = Field(str)

producer = client.create_producer(
    topic='customer-a/production/orders',
    schema=AvroSchema(Order)
)

Compatibility modes (BACKWARD, FORWARD, FULL, ALWAYS_COMPATIBLE) are configured per-topic and enforced server-side. A schema evolution that breaks BACKWARD compatibility is rejected at the broker before it can affect consumers.

The Confluent Schema Registry is a capable system and teams that already run it have no compelling reason to replace it. But for teams evaluating Kafka or Pulsar without existing infrastructure, Pulsar’s zero-overhead schema management is a genuine simplification. There is no separate service to deploy, no additional HA to configure, and no separate monitoring to maintain.

Pulsar Functions and the Stream Processing Gap

Pulsar Functions are lightweight stateful compute units that run on Pulsar brokers or in Kubernetes pods. They consume from one or more input topics, apply a transformation or filter, and produce to an output topic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

public class FilterFunction implements Function<Order, Order> {
    @Override
    public Order process(Order order, Context context) throws Exception {
        if (order.getAmount() > 1000.0) {
            context.getCounter("high-value").increment(1);
            return order;    // Pass through to output topic
        }
        return null;         // Drop the message
    }
}

Pulsar IO extends this with a connector framework for moving data into and out of external systems: Kafka, Kinesis, Cassandra, Elasticsearch, JDBC, S3, and others.

The honest assessment: Pulsar Functions are not a Kafka Streams replacement. Kafka Streams is a mature, full-featured stream processing library with support for joins, windowed aggregations, state stores with changelog topics, and exactly-once processing semantics. ksqlDB adds SQL-based stream processing on top. These tools, combined with Apache Flink and its Kafka connector, represent a stream processing ecosystem that is years ahead of what Pulsar Functions can do. If complex stream processing — multi-stream joins, time windowing, stateful aggregations — is a primary use case, Kafka’s ecosystem wins clearly.

Ecosystem and Operational Maturity

The most honest thing to say about Pulsar’s ecosystem relative to Kafka’s is that the gap is real and it matters.

Kafka has over 500 community and vendor-maintained connectors through Confluent Hub. Kafka Streams is deployed at enormous scale. ksqlDB is production-ready. Strimzi, the CNCF Kafka operator for Kubernetes, has years of production use and a large contributor community. AWS MSK, Confluent Cloud, and Aiven for Kafka give teams managed options with different cost and feature trade-offs. The talent pool is large — most data engineers who have worked in the last five years have Kafka experience.

Pulsar has roughly 30 built-in connectors. StreamNative is the primary commercial backer. StreamNative Cloud and Datastax Astra Streaming are the managed options, with fewer regions and less operational history than MSK. The Pulsar Kubernetes operator exists but is less mature than Strimzi. Fewer engineers have deep Pulsar experience, which matters during incidents.

This is not an argument against Pulsar. It is an argument for weighing the operational risk of choosing a less-mature system against the architectural advantages. For teams that have the operational capacity to run Pulsar in production and whose workloads align with its strengths, the advantages are real. For teams that are already stretched thin and need to minimize operational risk, Kafka’s mature ecosystem and larger support community are a genuine advantage.

Performance Characteristics

Kafka’s sequential disk write model — appending to partition logs, leveraging the OS page cache — produces very low write latency. With tuned configuration (acks=all, linger.ms=5, batch.size=32KB, compression.type=snappy), Kafka achieves write throughput in the millions of messages per second on a three-node cluster. Read latency for recent messages served from the page cache is sub-millisecond.

Pulsar’s write path adds a network hop: the broker receives the message, sends it to BookKeeper bookies, and waits for the write quorum to acknowledge before confirming to the producer. This adds 1–3ms compared to Kafka’s local disk write under equivalent conditions. However, Pulsar’s stateless broker model produces more consistent tail latency under load. When a Kafka broker is under pressure — a partition leader is handling a rebalance, a follower is catching up, disk I/O is saturated — latency spikes are common. Pulsar brokers can shed load more gracefully because they do not own data.

For the vast majority of workloads, the difference between 2ms and 4ms average write latency does not matter. Where it matters is at the extreme: high-frequency financial trading, real-time bidding, and similar latency-critical applications where Kafka’s local disk advantage is meaningful. For everything else — event-driven microservices, log aggregation, metrics pipelines, IoT ingestion — both systems are fast enough that the choice should be made on architecture and operational grounds, not raw throughput numbers.

Configuration Reference

Kafka producer (high throughput)

1
2
3
4
5
6
7

acks=all
linger.ms=5
batch.size=32768
buffer.memory=67108864
compression.type=snappy
max.in.flight.requests.per.connection=5
enable.idempotence=true

acks=all waits for the full ISR to acknowledge each batch — maximum durability. linger.ms=5 accumulates messages for 5ms before sending, increasing batch fill. max.in.flight.requests.per.connection=5 allows 5 in-flight batches per connection while keeping ordering guarantees when enable.idempotence=true.

Kafka consumer

1
2
3
4
5

group.id=my-consumer-group
enable.auto.commit=false
max.poll.records=500
isolation.level=read_committed
session.timeout.ms=30000

enable.auto.commit=false with manual offset commits gives exactly-once processing semantics when combined with a transactional output. isolation.level=read_committed ensures consumers only see messages from committed transactions.

Pulsar producer (high throughput)

1
2
3
4
5
6
7
8
9

Producer<byte[]> producer = client.newProducer()
    .topic("tenant/namespace/topic")
    .enableBatching(true)
    .batchingMaxMessages(1000)
    .batchingMaxPublishDelayMicros(10_000)   // Batch for up to 10ms
    .maxPendingMessages(50_000)
    .compressionType(CompressionType.SNAPPY)
    .blockIfQueueFull(true)
    .create();

Pulsar consumer (Key_Shared)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

Consumer<byte[]> consumer = client.newConsumer()
    .topic("tenant/namespace/topic")
    .subscriptionName("my-subscription")
    .subscriptionType(SubscriptionType.Key_Shared)
    .ackTimeout(2, TimeUnit.MINUTES)
    .negativeAckRedeliveryDelay(5, TimeUnit.SECONDS)
    .deadLetterPolicy(DeadLetterPolicy.builder()
        .maxRedeliverCount(5)
        .deadLetterTopic("tenant/namespace/topic-dlq")
        .build())
    .subscribe();

Key_Shared with a built-in DLQ handles the retry-and-exhaust pattern that requires manual implementation in Kafka.

Decision Framework

The honest summary: Kafka is the safer, lower-risk choice for most teams in 2026. Its ecosystem depth, talent availability, and managed service options mean that problems are well-documented and solutions are readily available. The cases where Pulsar is the stronger architectural choice are specific: you need genuine multi-tenancy from a single cluster, you need geo-replication that works reliably without MirrorMaker complexity, or your storage and compute scaling requirements diverge significantly. If those are your actual requirements — not hypothetical future ones — Pulsar is worth the operational investment. If they are not, Kafka is the right choice.