Observability Must Be Designed In, Not Added After
“A system instrumented after a production incident captures what happened last time : it cannot reliably reveal the failure modes you haven't encountered yet or the slow degradations that compound over weeks. ”
The three pillars of observability: metrics, logs, and traces: address orthogonal questions: what happened in aggregate, what happened event by event, and why it happened across services. Retrofitting distributed tracing to a mature service requires instrumenting every request path, dependency call, and background job. The cardinality trap in logging and the bimodal latency problem in metrics cannot be solved after the fact without significant re-engineering. Services shipped without observability are services you cannot operate safely at scale.
Why It Matters
The most dangerous failure mode for observability is unknown unknowns. A service with p50 and p95 latency metrics will not surface a bimodal latency distribution where 1% of requests take 10x longer than the rest. That 1% is often a specific failure path : a slow database query, a cache miss cascade, a network partition to one availability zone. p50/p95 will show normal until p99 customers are screaming. By then, you don't have the traces to know why.
Distributed traces are the hardest observability component to retrofit. Adding tracing to a mature microservices system requires propagating trace context through every HTTP client, every message queue producer and consumer, every background job scheduler, every gRPC call, and every internal service boundary. This is weeks of engineering work per service and requires coordination across every team that owns an upstream or downstream dependency. Starting at service inception costs an hour; retrofitting costs a sprint per service.
The cardinality trap is the silent cost escalator. A logging system that emits JSON with user_id, session_id, and request_id in every log line creates a high-cardinality field explosion that makes Elasticsearch indexing prohibitively expensive or simply broken. CloudWatch Logs Insights queries on high-cardinality fields can cost thousands of dollars per query at scale. The right cardinality decisions: which fields belong in metrics, which belong in structured logs, which belong in trace spans: must be made at design time when they are cheap, not at incident time when they are urgent.
Failure Modes
- ·Bimodal latency distribution invisible because only p50/p95 metrics are tracked, not p99/p999
- ·Distributed trace context lost at a message queue boundary because consumers were not instrumented
- ·Log cardinality explosion causing index failures or cost overruns in log aggregation systems
- ·Silent degradation accumulating over days undetected because no trend alert exists for gradual drift
- ·Post-incident forensics impossible because log retention was set to 7 days and the incident was 10 days ago
- ·Alert fatigue from mis-tuned thresholds causing the team to ignore the alert that matters
Amplification Risks
- ⚡A trace gap at one service boundary makes the entire distributed trace incomplete: the weakest link determines trace fidelity
- ⚡High-cardinality logging at one service can saturate the shared log aggregation pipeline, degrading observability for all services
- ⚡Alert fatigue from one misconfigured service erodes trust in the entire alerting system
Temporal Behavior
- ⟳Slow degradations over days or weeks are only visible if trend-based alerts are configured in addition to threshold alerts
- ⟳Observability debt accumulates silently: each uninstrumented code path is a future incident without a trace
- ⟳Log retention windows create a temporal blindness boundary: incidents that surface weeks later may have no forensic data
Boundary Implications
- ◈Every service boundary is an observability boundary: trace context must be explicitly propagated, not assumed
- ◈The operational responsibility boundary for observability must include both development (instrument the code) and platform (maintain the infrastructure)
- ◈Consistency boundaries in the data model create observability gaps when read replicas lag behind and metrics don't reflect the lag
Topology
- ·Every service-to-service edge in the topology is a trace propagation boundary: uninstrumented edges break distributed traces
- ·Every async consumer is a trace context gap unless explicitly designed to carry trace headers through the queue
- ·Observability infrastructure (Prometheus, Jaeger, ELK) is a topology dependency that itself has availability and failure modes
Scaling
- ·Log volume and metric cardinality scale with traffic: observability costs compound at scale if not designed with cardinality in mind
- ·Distributed tracing sampling rates must be tuned at scale: 100% sampling becomes prohibitively expensive above moderate traffic
- ·Alert signal-to-noise ratio degrades as systems scale unless alert thresholds are maintained in proportion to traffic growth
Resilience
- ·Systems with comprehensive observability recover faster from incidents because root cause is identified sooner
- ·Without p99/p999 latency metrics, SLA violations accumulate undetected until customer escalations arrive
- ·Trace coverage across all service boundaries is a resilience investment: it reduces mean time to detect and mean time to resolve
Governance Implications
- ·Every new service must ship with at minimum request count, error rate, and p99 latency before going to production
- ·Distributed tracing instrumentation must be a launch requirement, not a backlog item
- ·Log schema must be reviewed for cardinality before service launch to prevent cost explosion at scale
- ·Alert coverage must be audited regularly: services with zero alerts are almost certainly under-instrumented
Evolution Implications
- ·Adding a new service to an existing traced system requires trace propagation at every new service boundary
- ·Migrating from one log aggregation system to another requires validating that cardinality assumptions still hold
- ·As traffic grows, observability cost grows: sampling strategy must evolve with traffic volume
Mitigation Patterns
- →Ship every service with the three minimum signals: request count, error rate, p99 latency: before the first production deployment
- →Add structured JSON logging with trace_id, service, and request_id from day one: retrofitting this is expensive
- →Propagate OpenTelemetry trace context through all HTTP clients, message queue producers/consumers, and gRPC calls at service inception
- →Design log schemas with cardinality budget: distinguish high-cardinality fields (trace span attributes) from low-cardinality fields (log lines)
- →Set retention policies deliberately: metrics 13 months minimum for year-over-year comparison, traces 7-30 days, logs 30-90 days
Cross-References