Competing Consumers
matureSummary
Multiple consumer instances read from the same queue or topic; each message is processed by exactly one consumer, scaling processing throughput by adding consumers without changing the producer.
Problem
A single consumer process cannot keep up with message arrival rate. Processing throughput must scale independently of the producer without requiring coordination between consumers.
Description
In the competing consumers pattern, a pool of consumer instances subscribes to the same queue or consumer group. The broker delivers each message to exactly one consumer: the first to claim it. This provides horizontal scaling of message processing: double the consumers, roughly double the processing throughput, up to the partition count (Kafka) or message arrival rate (AMQP/SQS).
Kafka consumer groups: partition assignment is the unit of parallelism. Each partition is consumed by exactly one consumer in the group. Maximum concurrency = number of partitions. Adding consumers beyond the partition count is idle capacity. Rebalancing occurs when consumers join or leave, causing a brief processing pause.
AMQP (RabbitMQ): consumers compete at the queue level via basic.get or basic.consume with basic.ack. Prefetch count (basic.qos) controls how many unacknowledged messages a consumer can hold: set too high and a slow consumer hogs messages; set too low and throughput suffers from round-trip overhead.
SQS: visibility timeout creates a soft lock: a consumer receives a message and it becomes invisible to other consumers for the visibility timeout period. If the consumer does not delete the message within that window, it reappears in the queue for another consumer. This implements at-least-once delivery.
At-least-once delivery is the common guarantee, and it is what competing consumers itself provides: any consumer in the pool can pick up a redelivered message, so duplicates are a normal, expected event, not an edge case. Getting to effectively-once processing needs a separate mechanism at the consumer, idempotent handling of a redelivered message, concretely built with the inbox pattern (a local dedup record and the business logic committed in one transaction). The outbox pattern is not a substitute for this: outbox solves a different problem, reliable delivery from producer to broker without a dual-write gap, and does nothing about a competing consumer processing the same message twice. Reliable production and idempotent consumption are complementary, not alternative paths to the same guarantee.
Tradeoffs
Linear scaling of throughput with consumer count, up to partition ceiling
Consumer pool is stateless and easy to scale; broker handles coordination
Global ordering sacrificed; only per-partition ordering preserved (Kafka)
At-least-once delivery requires idempotent consumers or deduplication logic
Parallelism bounded by partition count; over-provisioning consumers wastes resources
When to use
Message processing is idempotent or at-least-once delivery is acceptable
Competing consumers typically provide at-least-once delivery; duplicate processing must be safe
Per-message ordering is not required across the full queue
Kafka preserves order within a partition; across partitions, ordering is not guaranteed
Tasks are CPU-bound or I/O-bound and parallelizable
The pattern's value is parallel execution; non-parallelizable tasks don't benefit
Processing throughput exceeds what a single consumer can sustain
Add consumers when queue depth grows monotonically or consumer lag is non-zero
When not to use
Strict global ordering is required across all messages
Kafka guarantees order within a partition, not globally; single-consumer or keyed routing required for total ordering
Consumers hold shared mutable state requiring coordination
Competing consumers are most effective when stateless; shared state requires external coordination and defeats horizontal scaling
Operational Requirements
Implement idempotent message processing or use idempotency keys
At-least-once delivery guarantees duplicate delivery on consumer restart or timeout
Set consumer prefetch / max in-flight messages to match processing rate
Too high causes head-of-line blocking on slow messages; too low wastes throughput
Monitor consumer group lag per partition (Kafka) or queue depth (SQS)
Lag growth is the primary signal that consumers are insufficient for the load
Configure dead-letter queue (DLQ) for messages that exhaust retries
Poisoned messages that fail repeatedly must be routed to DLQ to prevent queue stall
Characteristics
Technologies
Canonical
Alternatives
Relationships
Complements
Basis
Foundational messaging pattern with decades of operational history across Kafka, RabbitMQ, and SQS
Related Architecture Knowledge
Inbound: affects this entity
Kafka consumer groups are the canonical implementation of the competing consumers pattern. Each consumer group member is assigned an exclusive subset of topic partitions, ensuring each message is processed by exactly one consumer within the group while enabling horizontal scaling up to the partition count.
Tradeoffs
- ·Partition ceiling bounds parallelism; over-partitioned topics add controller overhead
- ·Rebalancing pauses processing; eager vs cooperative rebalancing protocols trade pause duration for complexity
- ·At-least-once delivery requires idempotent consumers to handle redelivery on rebalance
NATS JetStream consumer groups distribute messages across multiple consumer instances, implementing competing consumers with at-least-once delivery.
Full relationship →Used In Architecture Scenarios
Multi-Tenant SaaS
A multi-tenant API gateway providing authentication, distributed rate limiting, request routing, payload transformation, and per-tenant usage analytics for API publishers. The hot path: authentication check, rate limit evaluation, and routing decision: must complete in under 1ms using Redis-only data structures to avoid proxying latency dominating upstream service response time. PostgreSQL stores tenant configuration, subscription plans, and API key definitions. Kafka receives API usage events for downstream billing and analytics. Configuration changes (rate limit updates, routing rule edits) must propagate to all gateway replicas without restart.
Multi-Tenant SaaS
A multi-tenant developer tooling platform providing CI/CD pipeline execution, log aggregation, code analysis, and dependency scanning across isolated tenant organizations. Tenant isolation is the primary correctness constraint: a security boundary violation between tenants is a critical incident, not a performance event. PostgreSQL row-level security enforces data isolation; Redis manages job queues and distributed locks; Elasticsearch indexes pipeline log output for search; Kafka delivers webhook events to tenant-registered endpoints; MinIO stores pipeline artifacts. Resource quota enforcement prevents any single tenant's burst from affecting others.
Write-Heavy Application
A durable background job execution platform where jobs are enqueued via API and executed by competing consumer worker pools. PostgreSQL is the durable job store, job definitions, retry state, scheduling metadata, and dead-letter records persist in PostgreSQL with ACID guarantees, surviving any worker or queue infrastructure failure. Redis tracks in-flight job state (which worker claimed which job, visibility timeout lease expiry) to enable fast lease checks without PostgreSQL queries on the hot path. Temporal provides workflow orchestration for multi-step jobs that require coordination across multiple execution stages, with built-in state machine semantics and durable activity execution. Kafka carries job completion events to downstream consumers (analytics, billing triggers, notification fan-out). Backpressure between the job enqueue rate and worker execution rate prevents runaway job accumulation when workers are degraded.
Write-Heavy Application
A high-rate device telemetry ingestion architecture designed for millions of devices emitting metrics at 1–60 second intervals. Kafka absorbs device writes as an ingestion buffer, decoupling device-facing ingest endpoints from the storage write path so that downstream storage pressure never propagates back to devices. TimescaleDB provides time-series storage with automatic chunk partitioning by time range, native compression, and continuous aggregate views for rollup queries. ClickHouse serves as the OLAP layer for device fleet analytics queries. Redis caches last-known device state (current readings per device) for real-time alerting queries that must not scan historical storage. Backpressure on the Kafka consumer side prevents storage write throughput from being overwhelmed by burst ingestion events from device reconnect storms.
AI / RAG Application
A low-latency feature serving platform for ML model inference, providing both batch (offline) and real-time (online) feature access with strict training-serving consistency. Redis serves the hot feature cache with p99 latency targets below 5ms for online inference requests; PostgreSQL provides point-in-time feature lookups for offline training jobs with temporal consistency guarantees; Cassandra stores high-cardinality feature entities at write scale beyond PostgreSQL's single-primary ceiling; Kafka streams feature computation events from online feature pipelines to update the cache; Qdrant stores vector features for embedding-based model inputs and similarity lookups; ClickHouse serves feature analytics and drift monitoring across training dataset populations. Feature versioning is first-class: every feature value is tagged with a pipeline_version and computed_at timestamp to support model reproducibility and training-serving skew diagnosis.
Event-Driven System
A multi-channel notification delivery architecture that accepts upstream business events (order placed, payment received, comment posted, threshold alert triggered) and routes them to per-channel delivery workers (push via FCM/APNs, email via SendGrid, SMS via Twilio, in-app via WebSocket). Kafka carries raw business events from upstream producers. RabbitMQ handles per-channel fan-out with separate exchanges and queues per delivery channel, isolating email queue backlog from push notification delivery. PostgreSQL provides durable notification state tracking (sent, failed, bounced, suppressed). Redis enforces per-user rate limiting (notification frequency caps to prevent fatigue) and stores deduplication tokens to prevent duplicate sends across retry attempts. The inbox pattern on the consumer side ensures idempotent delivery even when Kafka produces duplicate events.
Analytics Pipeline
A metrics, logs, and traces ingestion and query platform built to absorb the telemetry output of a production system fleet: including the telemetry volume spikes that accompany the incidents the platform is meant to detect. ClickHouse stores metrics data with automatic time-based rollup via continuous materialized views; TimescaleDB provides complementary time-series storage for high-cardinality alert evaluation; Kafka buffers the ingestion stream against downstream write pressure, decoupling ingest acceptance rate from storage write throughput; Elasticsearch serves log full-text search and structured field filtering; Redis caches dashboard query results and active alert state for sub-100ms alert evaluation latency. The alert engine evaluates threshold and anomaly rules against pre-computed materialized views, not raw data, to bound alert evaluation cost independent of ingestion volume.
Event-Driven System
A video and audio streaming architecture where content ingestion triggers an async multi-variant transcoding pipeline, CDN delivery handles 95%+ of playback traffic, and Cassandra absorbs the write volume of per-user viewing history. Kafka decouples upload events from transcoding workers; MinIO stores raw and encoded assets; Redis maintains playback session state and view counters. The architecture must handle upload spikes without blocking delivery, and cache cold starts without cascading database load.