Post-Mortem Framework · Operational: Cold Start Latency

Cold Start Latency

SEV-4, Near-Miss

Isolated propagation · operational · Affects 2 scenario(s)

Severity Classification

Classified as DEGRADED based on failure mode severity. This failure mode appears in 2 known architecture scenarios, indicating widespread relevance.

Propagation Chain

Origin component

Cold Start Latency begins at the source component. Trigger: Kubernetes pod restart or scale-out event routes traffic to new pod before warm-up.

Immediate (T+0) · Signal: Latency Spike

Blast Radius

Cold start latency primarily affects the new or restarted instance. However, under load, slow responses from a cold instance increase latency for the requests it serves. If many instances are restarted simultaneously (rolling deploy, cluster restart), a significant fraction of traffic can hit cold instances during the deployment window, causing cluster-wide latency spikes. Retried slow requests also add load to warm instances.

Contributing Factors

Trigger Condition: Kubernetes pod restart or scale-out event routes traffic to operational

This operational trigger enables Cold Start Latency: Kubernetes pod restart or scale-out event routes traffic to new pod before warm-up

Trigger Condition: Lambda or serverless function invoked after idle periodoperational

This operational trigger enables Cold Start Latency: Lambda or serverless function invoked after idle period

Trigger Condition: Service restart after deployment with no readiness gateoperational

This operational trigger enables Cold Start Latency: Service restart after deployment with no readiness gate

Mitigation Gaps

MEDIUMKnown mitigator 'blue_green_deployment' not in runbook

Add 'blue green deployment' to the runbook. Blue-green deployment pre-warms the new environment (connections, caches, JIT) before traffic shifts, eliminating cold-start latency that would otherwise occur during in-place deployments.

Remediation Plan

ImmediateCheck per-instance latency metrics: is one instance significantly slower than pe

Check per-instance latency metrics: is one instance significantly slower than peers?

Effort: Minutes to hours (on-call response)

ImmediateCorrelate latency spike with pod restart or deployment event timestamp

Correlate latency spike with pod restart or deployment event timestamp

Effort: Minutes to hours (on-call response)

ImmediateIf cold start is the cause, the instance will warm up naturally within 30–120s

If cold start is the cause, the instance will warm up naturally within 30–120s

Effort: Minutes to hours (on-call response)

Short-TermImplement synthetic warm-up in readiness probe

Before reporting ready, execute a synthetic sequence of requests that exercises the hot code paths: triggering JIT compilation, populating in-process caches, and establishing connection pool connections. Only then return 200 from the readiness endpoint. This delays traffic routing until the instance is actually warm.

Effort: 1 day to 1 week

Short-TermConfigure HikariCP to eagerly initialize connection pool

Set minimumIdle = maximumPoolSize to establish all connections on startup rather than lazily. connectionInitSql can run a lightweight ping query on each new connection to verify the connection is live before adding it to the pool.

Effort: 1 day to 1 week

Short-TermUse Kubernetes startup probe for slow-starting containers

Configure a startup probe with a longer failureThreshold * periodSeconds budget (e.g., 120 seconds total). The startup probe gates the liveness and readiness probes from activating until the container has had enough time to initialize. Prevents the liveness probe from killing a slow- starting pod.

Effort: 1 day to 1 week

Short-TermAdd alerting for documented detection signals

Configure alerts for: latency spike, log errors. Set thresholds to fire at 70% of critical level to allow response before full failure.

Effort: 1–3 days

Long-TermEliminate cross-scenario Cold Start Latency exposure

Cold Start Latency affects 2 architecture scenarios (API Gateway Platform, ML Feature Serving Platform). Design a shared mitigation strategy or a platform-level safeguard that prevents this failure mode from manifesting across all affected services.

Effort: 1–3 months

This post-mortem framework is derived from structured architecture knowledge. It provides an evidence-grounded starting point, not a substitute for a live incident review conducted by the team closest to the system. Adjust remediation priorities based on actual runtime observations.