Streaming Media Platform

Topology Notes

• ·Upload path: API gateway → MinIO (raw asset write) → Kafka (transcoding_requested event published). The API returns 202 after the Kafka publish succeeds. The raw asset write to MinIO must complete before the Kafka event is published to guarantee the worker can access the file. If MinIO is unavailable, the upload fails and no Kafka event is published: no orphaned jobs.
• ·Transcoding path: Kafka consumer workers pull transcoding_requested events, fetch the raw asset from MinIO, produce HLS/DASH variant streams, and write the encoded variants back to MinIO under versioned paths. Workers publish transcoding_completed to Kafka on success, or transcoding_failed on error after exhausting retries. Failed jobs must dead-letter and alert: they do not silently drop.
• ·Playback path: CDN handles > 95% of GET requests for segment files by serving from edge cache. Cache miss hits the MinIO origin via a signing proxy. CDN cache TTL for HLS segments is 24–72 hours; playlist files (*.m3u8) carry shorter TTL (5–60 minutes) since they reference the current variant set.
• ·Redis holds two key types: playback session state (viewer_id + content_id → position, quality, session metadata; TTL 4h) and view counters (content_id → approximate view count using Redis HyperLogLog for deduplication). Playback session writes happen every 30 seconds of playback; do not use Redis as the long-term storage for sessions: flush completed sessions to PostgreSQL.
• ·Cassandra keyspace is partitioned by (user_id, content_id) for the resume position table (one row per user-content pair) and by (content_id, date_bucket) for the viewing history table. Time bucketing by day prevents partition size unboundedness for high-traffic content. SELECT queries targeting history must always include a date range to avoid full-partition scans.