Architecture Review: E-Commerce Order Platform

Concurrent writers to the same rows serialize behind each other's row locks, so latency is set not by the work a transaction does but by how long it waits for the writers ahead of it. On a hot row the queue depth, and therefore the tail latency, grows with concurrency while throughput flattens. Blocked writers hold connections open, so a single contended row can drain the connection pool as a secondary failure.

Affects 0 nodes

A failure or degradation in one service causes increased load, held resources, or error propagation in its callers, which in turn degrade their callers, until the failure front propagates through the entire dependency graph and brings down services with no direct dependency on the original failure point.

Affects 0 nodes

Seed 'Thundering Herd (Cache Stampede) Risk Probe' identifies 2 metrics relevant to thundering_herd.

Metrics to instrument: error_rate, p95_latency_ms

Trigger: Payment provider timeout causing full checkout rollback and user-facing error; fulfillment system outage causing checkout to fail synchronously rather than queue the fulfillment work; inability to replay failed fulfillment notifications after downstream system recovery. Migrate from 'Synchronous checkout with direct database payment insert and synchronous payment API call' to 'Saga-orchestrated checkout with outbox-based fulfillment events'. Implement the outbox pattern and fulfill-via-event flow first, before the full saga orchestrator. Validate that fulfillment events are reliably delivered across payment provider failures before implementing compensation logic.

Saga compensation paths for every checkout failure scenario must be designed before deployment: partial saga implementation is more dangerous than no saga; Idempotency keys for the payment provider must be persisted before the payment call, not after: a crash between these two events causes a payment charge with no local record

Trigger: Product search p99 > 1s; faceted navigation (category + price range + brand + availability) requiring full-table scans in PostgreSQL; ranking by popularity or relevance score not achievable in PostgreSQL without full-table aggregation. Migrate from 'PostgreSQL full-text search for product discovery' to 'Elasticsearch for product search with CDC-based catalog indexing'. Run Elasticsearch in shadow mode for 2 weeks before serving traffic. Compare search result sets between PostgreSQL full-text and Elasticsearch for a representative query sample. Validate CDC latency meets the acceptable price staleness window.

Elasticsearch index must be seeded from PostgreSQL before CDC takes over: the initial bulk import must complete without catalog updates being lost during the window; Search result prices are eventually consistent with PostgreSQL: checkout must re-validate price at order creation, not trust the search result price

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: PostgreSQL pg_locks showing high RowExclusiveLock contention on inventory_items for specific sku_ids; checkout p99 > 2s for contended SKUs; deadlock errors appearing in application logs during sale events; effective checkout throughput for hot SKUs well below per-request checkout latency would predict

Bottleneck: Concurrent saga checkout attempts competing for the same inventory row via row-level locking. Evolution: Introduce a per-SKU checkout serialization queue at the application layer : all concurrent checkout requests for the same SKU are queued and processed serially, converting lock contention into queue latency. Alternatively, use PostgreSQL advisory locks with non-blocking trylock: requests that cannot acquire the lock immediately return a "sold out" response rather than queuing. For very high flash sale volumes, pre-allocate inventory slots (reserve N slots per sale event, each slot is a row with one reservation) to spread lock contention across N rows instead of one.

Signal: Checkout p99 tracking payment provider p99 almost linearly; connection pool utilization on the payment service rising during payment provider slowdowns; circuit breaker trip events appearing in payment service metrics; saga timeout events correlated with payment provider latency spikes

Bottleneck: Checkout saga holding a database connection and an inventory reservation open for the duration of the payment provider call: payment latency directly amplifies connection pool pressure. Evolution: Decouple the payment step from the synchronous checkout saga: reserve inventory and create the order record synchronously, then process payment asynchronously. The customer receives an "order confirmed, payment processing" state immediately; the payment step runs as a separate saga step triggered by an event. This reduces the synchronous checkout latency to the inventory reservation time, not the payment provider round-trip time.