DBRaven
Pattern · messaging

Publisher-Subscriber

mature

Summary

Producers publish events to a topic or channel without knowing subscribers. All interested subscribers receive every event, decoupling producers from consumers at both the semantic and temporal level.

Problem

Multiple downstream consumers need to react to the same events, but the producer should not need to know who they are or maintain direct connections to each one. Adding a new consumer must not require modifying the producer.

Description

In pub-sub, the publisher declares that an event occurred (e.g., "order.placed") and broadcasts it to a topic. Any subscriber that has declared interest in that topic receives a copy. Multiple subscribers receive the same event independently : this is the core distinction from competing consumers, where only one consumer processes each message.

Kafka durable pub-sub: subscribers are consumer groups. Each consumer group receives all events independently. Events are retained on disk for a configurable retention period, so a subscriber added after the fact can replay the full history. This is the most operationally robust pub-sub model for event-driven architectures.

Redis pub-sub: ephemeral. No message persistence. If a subscriber is offline when an event is published, that event is permanently lost to that subscriber. Suitable for real-time fanout (live dashboards, notification delivery) where missing an event is acceptable. Not suitable for event sourcing or audit trails.

SNS/SQS fan-out: SNS topic fans out to multiple SQS queues. Each SQS queue is a competing-consumer pool for one subscriber group. Combines pub-sub semantics (multiple independent consumers) with durable queuing per subscriber. The standard AWS pattern for reliable multi-consumer fanout.

Temporal decoupling: a publisher does not need a subscriber to be online to publish. Events accumulate in the topic. This allows subscribers to be deployed, updated, or restarted independently without the publisher being aware.

Tradeoffs

Producer-consumer decoupling
+0.9

Producer has zero knowledge of subscribers; adding consumers requires no producer change

Event replay and auditability
+0.7

Kafka enables full event history replay; Redis pub-sub provides none

Operational complexity
-0.4

Kafka requires partition management, consumer group monitoring, and retention tuning

At-least-once delivery overhead
-0.3

Each subscriber independently tracks offset; each must handle duplicate delivery

Storage cost
-0.2

Kafka log retention scales with event volume and retention period

When to use

Multiple independent consumers need to react to the same event

Pub-sub delivers each event to all subscribers without producer modification

Producers and consumers evolve independently and must be decoupled

Temporal and semantic decoupling is the primary value of pub-sub

Event replay is needed for new consumers or data recovery

Kafka-backed pub-sub with log retention supports replay; design for this if replay is a requirement

When not to use

Only one consumer should process each message

Use competing consumers if messages must be processed exactly once by one worker

Message delivery must survive broker restart with zero configuration overhead

Ephemeral pub-sub (Redis) loses messages on broker restart; use Kafka or SNS/SQS for durability

Operational Requirements

mandatory

Configure topic retention period to match the longest expected consumer catchup window

A consumer offline for 48 hours must be able to replay 48 hours of events; retention must cover this

mandatory

Monitor consumer group lag per topic-partition

Growing lag indicates a consumer cannot keep up with the publish rate

mandatory

Assign unique consumer group IDs per independent subscriber

Shared group IDs cause competing-consumer semantics instead of pub-sub semantics

recommended

Define schema and versioning strategy for events

Multiple consumers on one topic creates coupling through the event schema; use schema registry for Kafka

Characteristics

Scales on
write
Implementation complexitylow
Operational complexitymedium
Scaling ceilingKafka: each consumer group scales independently to partition count. A topic with 20 subscribers (consumer groups) handles 20× independent consumption without affecting the producer. Throughput ceiling is partition throughput × partition count. Redis pub-sub: single-threaded delivery; fanout to thousands of subscribers in one Redis instance becomes CPU-bound on the broker.

Technologies

Canonical

kafkasns

Alternatives

redisgoogle pub subrabbitmq

Relationships

Evolves to

event sourcing

Complements

competing consumersevent sourcingwrite ahead log cdc

Basis

Core event-driven pattern with mature implementations across all major messaging platforms

Related Architecture Knowledge

Inbound: affects this entity

SupportsTechnology
nats
Grounded

NATS provides sub-millisecond pub/sub messaging with subject hierarchy and wildcard subscriptions, enabling publish-subscribe communication between services.

Full relationship →

Used In Architecture Scenarios

Publisher-Subscriber: DBRaven