Cold Start Latency

degraded

Summary

Newly started service instances handle their first requests significantly slower than steady-state instances, causing latency spikes when load balancers route traffic to cold pods before they are warmed up.

Description

Cold start latency affects any service that requires a warm-up period before reaching steady-state performance. The gap between cold and warm performance varies by runtime and implementation:

JVM services: class loading and JIT compilation dominate. On startup, all application classes must be loaded from disk and verified. JIT compilation is triggered by execution: the first N invocations of a method run in interpreted mode (slow) before the JIT decides to compile the hot path. For a complex service, the JIT warm-up window can extend 30–120 seconds. JVM TieredStopAtLevel=1 (C1 JIT only) can reduce warm-up time at the cost of peak throughput.

In-process cache: request-response caches, computed lookup tables, and configuration caches are empty on startup. The first requests populate the cache while experiencing full database latency. A service that normally serves 95% of requests from cache will serve 100% from database during the warm-up window.

Connection pool warm-up: HikariCP initializes connections lazily by default. The first requests compete for the initial connection establishment (5–50ms per connection). Set minimumIdle = maximumPoolSize to eagerly initialize all connections on startup.

Kubernetes interaction: Kubernetes readiness probes check whether the pod is ready to receive traffic. A readiness probe that returns healthy before JIT warm-up is complete routes traffic to a cold pod. The load balancer treats the cold pod as equivalent to warm pods, sending it full traffic share immediately. The correct behavior is to delay the readiness probe success until after warm-up: either by implementing a synthetic warm-up sequence in the readiness handler or by using a startup probe with a separate (longer) timeout.

Serverless cold start: AWS Lambda, Google Cloud Run, and similar platforms start new instances on demand. Cold start latency includes: container init, runtime bootstrap, and dependency initialization: typically 200ms–2s depending on language and package size. JVM Lambdas with heavy Spring frameworks can take 5–10 seconds to cold start.

Characteristics

Propagationisolated

Time to detectVisible within seconds of a new pod receiving traffic, as p99 latency spikes on that instance. In aggregate metrics (cluster-level p99), detectable during rolling deployments when cold pod fraction is significant.

Blast radiusCold start latency primarily affects the new or restarted instance. However, under load, slow responses from a cold instance increase latency for the requests it serves. If many instances are restarted simultaneously (rolling deploy, cluster restart), a significant fraction of traffic can hit cold instances during the deployment window, causing cluster-wide latency spikes. Retried slow requests also add load to warm instances.

Triggers

·Kubernetes pod restart or scale-out event routes traffic to new pod before warm-up
·Lambda or serverless function invoked after idle period
·Service restart after deployment with no readiness gate
·Autoscaling event adds new instances that receive immediate full traffic share

Detection Signals

latency spikelog errors

Mitigation Strategies

Implement synthetic warm-up in readiness probepreventscomplexity: medium

Before reporting ready, execute a synthetic sequence of requests that exercises the hot code paths: triggering JIT compilation, populating in-process caches, and establishing connection pool connections. Only then return 200 from the readiness endpoint. This delays traffic routing until the instance is actually warm.

Configure HikariCP to eagerly initialize connection poolpreventscomplexity: low

Set minimumIdle = maximumPoolSize to establish all connections on startup rather than lazily. connectionInitSql can run a lightweight ping query on each new connection to verify the connection is live before adding it to the pool.

Use Kubernetes startup probe for slow-starting containerspreventscomplexity: low

Configure a startup probe with a longer failureThreshold * periodSeconds budget (e.g., 120 seconds total). The startup probe gates the liveness and readiness probes from activating until the container has had enough time to initialize. Prevents the liveness probe from killing a slow- starting pod.

Enable class data sharing (CDS) for JVM servicescomplexity: medium

JVM Class Data Sharing and AppCDS pre-compile the class loading metadata into a shared archive that is memory-mapped on startup, reducing class loading time by 30–50%. Requires generating the archive as part of the container build process.

Recovery Steps

1.Check per-instance latency metrics: is one instance significantly slower than peers?
2.Correlate latency spike with pod restart or deployment event timestamp
3.If cold start is the cause, the instance will warm up naturally within 30–120s
4.For immediate relief, add a warm-up endpoint call to the deployment pipeline before pod goes live
5.Review readiness probe configuration: ensure it is not approving traffic before warm-up completes

Estimated recovery time: Self-resolving within 30–120 seconds as JIT compilation and cache population complete. For Lambda cold starts, resolved within the first invocation. With readiness probe warm-up, the issue is prevented entirely at the cost of slightly longer deployment time.

Affected Systems

Patterns

health check patternconnection pooling

Technologies

kafkaelasticsearch

Basis

Common operational failure in JVM and containerized environments; Kubernetes startup probe and HikariCP documentation cover mitigations

Run This Failure

Blast radius analysis for this failure mode within each scenario that carries it.

API Gateway Platform →ML Feature Serving Platform →

Related Architecture Knowledge

Inbound: affects this entity

MitigatesPattern

blue green deployment

Grounded

Blue-green deployment pre-warms the new environment (connections, caches, JIT) before traffic shifts, eliminating cold-start latency that would otherwise occur during in-place deployments.

Full relationship →

Used In Architecture Scenarios

API Gateway Platformhigh

Multi-Tenant SaaS

A multi-tenant API gateway providing authentication, distributed rate limiting, request routing, payload transformation, and per-tenant usage analytics for API publishers. The hot path: authentication check, rate limit evaluation, and routing decision: must complete in under 1ms using Redis-only data structures to avoid proxying latency dominating upstream service response time. PostgreSQL stores tenant configuration, subscription plans, and API key definitions. Kafka receives API usage events for downstream billing and analytics. Configuration changes (rate limit updates, routing rule edits) must propagate to all gateway replicas without restart.

ML Feature Serving Platformexpert

AI / RAG Application

A low-latency feature serving platform for ML model inference, providing both batch (offline) and real-time (online) feature access with strict training-serving consistency. Redis serves the hot feature cache with p99 latency targets below 5ms for online inference requests; PostgreSQL provides point-in-time feature lookups for offline training jobs with temporal consistency guarantees; Cassandra stores high-cardinality feature entities at write scale beyond PostgreSQL's single-primary ceiling; Kafka streams feature computation events from online feature pipelines to update the cache; Qdrant stores vector features for embedding-based model inputs and similarity lookups; ClickHouse serves feature analytics and drift monitoring across training dataset populations. Feature versioning is first-class: every feature value is tagged with a pipeline_version and computed_at timestamp to support model reproducibility and training-serving skew diagnosis.