Architecture Review: Two-Sided Marketplace Platform

One partition (a database shard, a Kafka topic partition, a Redis hash slot) receives traffic so far above its peers that it saturates while the others sit idle. Aggregate capacity looks healthy, but the hot partition throttles or lags, and everything routed to it degrades. The cause is skew in how keys map to partitions, and the fix depends on whether the skew is spread across many keys or concentrated in one.

Affects 1 node. (Marketplace Mixed)

A failure or degradation in one service causes increased load, held resources, or error propagation in its callers, which in turn degrade their callers, until the failure front propagates through the entire dependency graph and brings down services with no direct dependency on the original failure point.

Affects 0 nodes

Seed 'Read Hotspot Saturation' identifies 3 metrics relevant to hot_partition. Execution preview confirms this risk manifests under modelled load.

Metrics to instrument: partition_qps, p99_latency_ms, cache_hit_rate

Trigger: Checkout failures from payment provider unavailability causing full transaction rollback and user-facing errors; need for asynchronous notification delivery; listing search performance insufficient from PostgreSQL full-text search. Migrate from 'Monolithic marketplace application with single database' to 'Event-driven marketplace with Kafka + saga-based checkout flow'. Introduce the outbox pattern and Kafka event publishing before the saga orchestration layer. Validate event delivery reliability before building compensating transaction logic on top of it.

Saga implementation requires designing and testing all compensation paths: untested compensation logic fails exactly when it is most needed; Kafka adds operational complexity that small teams may not be ready to operate

Trigger: Listing search p99 > 1s; faceted navigation (category + price + location + rating) not supportable in PostgreSQL without full-table scans; ranking algorithm requiring feature vectors that PostgreSQL cannot efficiently support. Migrate from 'PostgreSQL full-text search for listing discovery' to 'Elasticsearch for listing search with CDC-based indexing'. Run the Elasticsearch index in shadow mode (populated but not serving traffic) for 2 weeks before cutting over search traffic. Validate that listing updates propagate within the expected SLA window during the shadow period.

Elasticsearch index must be initially populated from PostgreSQL before CDC takes over: initial sync must complete without data loss; Search index staleness during CDC consumer lag must be explicitly communicated in the search UX

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: Redis cache miss spike visible in monitoring; PostgreSQL query rate spiking for listing reads despite stable write volume; p99 listing API latency > 500ms during traffic spike events

Bottleneck: Cache TTL expiry on hot listings during peak traffic: all concurrent requests bypass cache simultaneously. Evolution: Implement staggered TTL jitter on listing cache entries; use probabilistic early refresh (refresh before TTL expiry when remaining TTL < 20% and request rate is high); implement single-flight/request coalescing at the application layer to collapse concurrent cache misses into a single database read

Signal: Saga compensation events appearing in order event log; checkout p99 > 2s; pg_locks showing contended rows on inventory_reservations table; idempotency key conflicts increasing in payment service logs

Bottleneck: Concurrent checkout transactions competing for the same inventory rows; saga timeout thresholds too aggressive. Evolution: Increase inventory reservation table partition count; tune saga step timeout to 2x the observed p99 for each step under load; implement a per-listing checkout serialization queue to prevent N concurrent sagas competing for the same inventory