Batch ETL Pipeline
burst writeSummary
Periodic large-volume data movement from source systems to analytical targets. High throughput over sustained windows; not latency-sensitive. Primary constraints are source read impact, destination write throughput, and job completion SLO.
Example Systems
- ·Nightly billing aggregation job
- ·Data warehouse ETL from production PostgreSQL
- ·ML feature store population from transaction database
- ·End-of-month regulatory reporting pipeline
- ·S3 data lake ingestion from operational databases
- ·Analytics database backfill from event stream
Characteristics
Capacity
Access Patterns
Recommended Patterns
Patterns to Avoid
Basis
Standard workload type with well-understood operational characteristics; ETL pipeline design patterns are extensively documented
Related Architecture Knowledge
Outbound: this entity affects
Batch ETL workloads that process historical time series data benefit from rollup pre-aggregation to reduce the data volume that must be scanned for each pipeline run.
Full relationship →Used In Architecture Scenarios
Multi-Tenant SaaS
A multi-tenant developer tooling platform providing CI/CD pipeline execution, log aggregation, code analysis, and dependency scanning across isolated tenant organizations. Tenant isolation is the primary correctness constraint: a security boundary violation between tenants is a critical incident, not a performance event. PostgreSQL row-level security enforces data isolation; Redis manages job queues and distributed locks; Elasticsearch indexes pipeline log output for search; Kafka delivers webhook events to tenant-registered endpoints; MinIO stores pipeline artifacts. Resource quota enforcement prevents any single tenant's burst from affecting others.
Write-Heavy Application
A durable background job execution platform where jobs are enqueued via API and executed by competing consumer worker pools. PostgreSQL is the durable job store, job definitions, retry state, scheduling metadata, and dead-letter records persist in PostgreSQL with ACID guarantees, surviving any worker or queue infrastructure failure. Redis tracks in-flight job state (which worker claimed which job, visibility timeout lease expiry) to enable fast lease checks without PostgreSQL queries on the hot path. Temporal provides workflow orchestration for multi-step jobs that require coordination across multiple execution stages, with built-in state machine semantics and durable activity execution. Kafka carries job completion events to downstream consumers (analytics, billing triggers, notification fan-out). Backpressure between the job enqueue rate and worker execution rate prevents runaway job accumulation when workers are degraded.
Event-Driven System
A video and audio streaming architecture where content ingestion triggers an async multi-variant transcoding pipeline, CDN delivery handles 95%+ of playback traffic, and Cassandra absorbs the write volume of per-user viewing history. Kafka decouples upload events from transcoding workers; MinIO stores raw and encoded assets; Redis maintains playback session state and view counters. The architecture must handle upload spikes without blocking delivery, and cache cold starts without cascading database load.