Architecture Review: Streaming Media Platform

Message queue or event stream consumer processing rate falls below producer write rate, causing consumer lag to grow unboundedly: eventually leading to increased end-to-end latency, producer backpressure, data expiry, or queue resource exhaustion.

Affects 2 nodes. (Event Streaming, Slow Consumer)

When a popular cached key expires or a service recovers from downtime, all requests that were waiting or arrive simultaneously miss the cache and hit the origin database concurrently, producing a request spike that can overwhelm the database within seconds.

Affects 1 node. (Redis)

Seed 'Queue Consumer Backlog' identifies 4 metrics relevant to queue_backlog_accumulation.

Metrics to instrument: queue_depth, consumer_lag_seconds, consumer_throughput

Trigger: Upload API p99 exceeding 30 seconds due to in-process transcoding blocking the response; upload timeouts reported by client applications during large file uploads. Migrate from 'Synchronous transcoding in the upload request handler (blocking API response)' to 'Async transcoding via Kafka topic with competing consumer workers'. Accept the upload, write the raw asset to MinIO, publish a transcoding_requested event to Kafka, and return 202 Accepted immediately. Status polling or webhook callbacks communicate availability. This is the correct model from day one for any content > a few megabytes.

Clients must handle the content-available state transition asynchronously : polling or webhook delivery of transcoding completion is required and must be built before removing synchronous behavior; Duplicate transcoding jobs if the Kafka producer retries without idempotency configured: implement idempotent producer and deduplication by upload_id at the worker to prevent wasted compute on duplicate work

Trigger: PostgreSQL viewing history table exceeding 500M rows; write latency on history inserts affecting OLTP transaction throughput on the shared primary; history queries scanning large time ranges causing sequential scan pressure. Migrate from 'Viewing history in PostgreSQL' to 'Viewing history in Cassandra'. Model the Cassandra schema around the three or four concrete read patterns (user's history ordered by time, content's viewer list, resume position by user+content). Everything else goes to an analytics pipeline, not Cassandra.

Cassandra schema must encode all required query access patterns at design time; ad-hoc queries that were easy in PostgreSQL SQL require pre-defined tables in Cassandra; Migration requires a dual-write period with history being written to both stores simultaneously; validating equivalence before decommissioning PostgreSQL history is non-trivial at high write volume

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: Kafka consumer group lag on the transcoding topic growing during peak upload hours; content availability delay > 10 minutes for newly uploaded videos; transcoding worker CPU consistently above 85% across all instances

Bottleneck: Transcoding consumer group undersized relative to peak upload volume. Evolution: Increase transcoding consumer instances up to the transcoding topic partition count; tune partition count to match the maximum desired worker parallelism (set this at topic creation, not after lag appears); implement per-uploader upload rate limits to smooth burst input; consider priority queuing so premium-tier content does not wait behind bulk ingest jobs

Signal: MinIO GET request rate spikes > 10x baseline immediately after content publish or CDN invalidation; MinIO p99 latency > 500ms; CDN miss ratio > 5% on popular content

Bottleneck: CDN cache miss storm on first-play of new or recently-updated content. Evolution: Implement origin request coalescing (single origin fetch per CDN node per object, queue subsequent requestors for the in-flight response); pre-warm CDN edges for anticipated high-traffic content before publish; add rate limiting at the origin gateway to cap per-second origin requests per content_id