Health Check Pattern
matureSummary
Expose a dedicated endpoint that reports service health and critical dependency status so that load balancers and orchestrators can route traffic away from unhealthy instances and restart failed ones.
Problem
Without health checks, load balancers route traffic to crashed, deadlocked, or dependency-starved service instances, causing request failures that are invisible until user-facing error rates spike.
Description
A health check endpoint is the foundational observability contract between a service and its runtime environment (Kubernetes, ECS, load balancer, service mesh). It answers one question: is this instance capable of serving traffic right now?
Two distinct check types serve different purposes:
Liveness check: is the process alive and not deadlocked? Returns 200 if the process is running and responsive. Kubernetes uses liveness probe results to decide whether to restart the container. A liveness check should be extremely cheap: a simple "I'm alive" response with no dependency calls. A liveness check that calls external dependencies can cause healthy pods to be restarted when the dependency is degraded.
Readiness check: is the service ready to serve requests? Checks that all critical dependencies are reachable and the service has completed startup (e.g., database migrations, cache warming, configuration loading). Kubernetes uses readiness probe results to decide whether to include the pod in the load balancer endpoint pool. An unready pod is removed from traffic but is NOT restarted. Readiness checks should only test dependencies that are on the critical path of every request.
Health checks must NOT be: - Expensive: a health check that runs a complex query causes database load
proportional to the number of instances in the cluster. On a 50-pod cluster
with 10-second intervals, that is 5 health-check queries per second just
for observability.
- Overly broad: checking non-critical dependencies (analytics service, email
provider) causes the service to appear unhealthy when only non-critical
dependencies are degraded. This removes instances from traffic unnecessarily.
- Optimistic: a health check that always returns 200 regardless of state
defeats the purpose and provides false confidence.
Cascading health failure risk: if ServiceA's readiness check calls ServiceB, and ServiceB is slow, ServiceA's readiness check times out. Kubernetes marks ServiceA as unready and removes it from the load balancer. If enough ServiceA pods become unready due to ServiceB degradation, ServiceA effectively goes down even though it could serve requests that do not depend on ServiceB. Design readiness checks to use short timeouts (< 1s) and only check dependencies that are truly blocking for all requests.
Startup probe: Kubernetes also supports a startup probe used during initial container startup, with a separate (usually longer) timeout to accommodate slow initialization. This prevents the liveness probe from killing a slow- starting container before it has had time to initialize.
Tradeoffs
Removes failed or unready instances from traffic before users experience errors
Overly broad readiness checks can cascade dependency failures into service unavailability
Health checks add baseline load on dependencies proportional to cluster size and probe frequency
Health check failures are an early signal of dependency degradation before user-facing errors spike
When to use
Service runs in an environment that supports health-based routing (Kubernetes, ECS, ALB)
Health checks are only actionable when the infrastructure can react to them
Service has external dependencies that can fail independently
Readiness checks should verify critical dependency connectivity to prevent routing to broken instances
Service startup time is variable or involves slow initialization
Startup probes prevent liveness timeouts during slow initialization; readiness gates prevent premature traffic routing
When not to use
Health check logic is more expensive than serving actual requests
A health check that queries the database with a complex query adds database load at cluster scale
Operational Requirements
Separate liveness and readiness checks with different scopes
Liveness: is the process alive (no dependency calls). Readiness: are critical dependencies reachable.
Set health check timeout to be a fraction of the probe period
A 10s probe period with a 9s timeout means one slow response causes the probe to fail; use 1-2s timeouts
Only check dependencies on the critical request path in readiness probes
Non-critical dependencies failing should not mark the service as unready
Include health check status in structured logs and expose as a metric
Health check failure history aids post-incident analysis; time-series metrics show degradation trends
Characteristics
Technologies
Canonical
Alternatives
Relationships
Complements
Basis
Foundational pattern documented in Kubernetes, AWS, and service mesh documentation; universally applied in containerized systems
Related Architecture Knowledge
Inbound: affects this entity
Health checks validate the new blue-green environment before traffic shift, ensuring the switch is only made when the new version is confirmed ready to serve requests.
Full relationship →