Apache Flink

Best for

• ·Stateful streaming with exactly-once guarantees where event ordering and late data handling are first-class concerns
• ·Fraud detection and real-time alerting using Flink CEP (Complex Event Processing) for pattern matching across event sequences
• ·CDC pipeline processing where upstream database changes must be transformed, enriched, and applied to downstream systems with transactional guarantees
• ·Real-time aggregation over time windows (tumbling, sliding, session) that feed dashboards, feature stores, or materialized views
• ·ETL workflows that process bounded datasets (batch mode) and streaming data through the same operator graph

Excels when

• ·Workload requires exactly-once end-to-end semantics: Flink's two-phase commit checkpoint protocol is one of the few streaming systems that genuinely provide this guarantee
• ·State size per job exceeds what fits in JVM heap and RocksDB state backend with incremental checkpoints provides cost-effective large-state storage
• ·Event-time processing is required: watermarks and allowed lateness enable correct aggregation over out-of-order event streams without application-level buffering
• ·Pipeline complexity justifies the investment: joins between streams, enrichment against broadcast state, time-correlated pattern matching

Avoid when

• ·Team has fewer than 2–3 engineers with Flink or Spark Streaming operational experience: the operational surface (checkpoint tuning, backpressure debugging, state evolution) is not suitable for teams new to distributed streaming
• ·Latency requirement is sub-100ms end-to-end: exactly-once checkpoint-based processing introduces latency proportional to checkpoint interval; use Kafka Streams or custom consumer logic for millisecond-latency requirements
• ·Workload is simple stream filtering or format conversion without stateful aggregation: Kafka Streams or simple consumer processes are adequate and far simpler to operate
• ·Job topology will change frequently: each topology change requires a savepoint and replay from the checkpoint; rapid iteration is expensive

Read scalability

Known failure modes

• ·Checkpoint pressure: slow or unavailable sinks cause checkpoint barriers to accumulate in the operator pipeline; checkpoint latency increases until the checkpoint timeout is exceeded and the job enters a restart loop
• ·State backend RocksDB compaction pressure: write-heavy stateful jobs generate L0 sstables faster than RocksDB compaction can merge them; read amplification increases and checkpoint size grows
• ·Watermark stalling: a single idle or slow source partition holds back the global watermark; all event-time windows that depend on the watermark advancing stop emitting results
• ·Backpressure cascade: a slow sink subtask propagates backpressure upstream through the operator DAG; the entire job slows to the pace of the slowest sink
• ·Savepoint/checkpoint incompatibility after job upgrade: changes to operator UIDs, state schemas, or serializer versions break savepoint restore: state schema evolution requires explicit compatibility planning
• ·Exactly-once transaction accumulation: at high sink parallelism, Kafka exactly-once sinks hold open transactions for the entire checkpoint interval; exceeding Kafka's transaction timeout causes commits to fail

Bottlenecks

• ·Sink throughput: the slowest sink subtask determines the throughput ceiling for the entire job; external system write latency becomes the job throughput limiter
• ·Checkpoint storage I/O: at high parallelism with large state, checkpoint uploads to S3 consume network bandwidth proportional to state size times checkpoint frequency
• ·RocksDB compaction contention: write-heavy stateful operators cause compaction threads to compete with processing threads for CPU; pin compaction threads to isolated cores on high-throughput jobs
• ·Watermark propagation delay through idle partitions: a Kafka partition that receives no events holds back the watermark for all subtasks; configure idle source timeout to advance watermarks past idle partitions
• ·JobManager single point of coordination: job submission, checkpoint coordination, and failure recovery all go through the JobManager; HA requires ZooKeeper or Kubernetes leader election

Degradation patterns

• ·Checkpoint duration creep: as state grows over the job lifetime, checkpoint upload time increases; monitoring checkpoint size trend identifies impending checkpoint timeout before it causes job restarts
• ·Backpressure cascade from sink overload: a temporarily slow downstream system causes upstream operators to fill their network buffers; processing latency increases across the entire job DAG
• ·Watermark stall from idle partitions: in multi-source jobs, a source that stops producing events freezes the global watermark; all event-time windows stop emitting until the idle source advances

Common topology roles