Architecture Review: Healthcare Records Platform

Asynchronous replicas fall behind the primary under write load and serve reads from an older version of the data. Reads keep succeeding, so nothing errors; what breaks is one of three specific consistency guarantees (read-after-write, monotonic reads, or consistent prefix), each with a distinct user-visible anomaly.

Affects 1 node. (Read Replica)

Concurrent writers to the same rows serialize behind each other's row locks, so latency is set not by the work a transaction does but by how long it waits for the writers ahead of it. On a hot row the queue depth, and therefore the tail latency, grows with concurrency while throughput flattens. Blocked writers hold connections open, so a single contended row can drain the connection pool as a secondary failure.

Affects 1 node. (Write-Heavy Transactional)

Seed 'Replication Lag Under Write Burst' identifies 4 metrics relevant to replication_lag_cascade. Execution preview confirms this risk manifests under modelled load.

Metrics to instrument: replication_lag_seconds, stale_read_rate, replica_wal_apply_rate

Trigger: HIPAA audit requirement exposed during external security review; inability to reconstruct which practitioner accessed a patient record and when; audit log gaps found during incident investigation (application-layer logging not guaranteed to capture all access paths, including background jobs and admin tools). Migrate from 'Mutable clinical records with application-layer audit logging' to 'Event-sourced clinical records with atomic audit event + outbox writes'. Define a "record_state_bootstrap" event type for all existing clinical records as of the migration cutover date. These events carry the current state with a note that pre-migration history is unavailable. This satisfies HIPAA reconstruction requirements for post-migration access while being honest about pre-migration gaps.

Historical records before the migration cutover cannot be event-sourced retroactively without synthetic "initial_state" events: document the boundary date explicitly and include it in audit reports; The transition requires a period of dual-write (old mutable path + new event path) with reconciliation to validate equivalence before decommissioning the mutable-only path

Trigger: FHIR events being published to Kafka but corresponding clinical record transactions rolling back, resulting in phantom events being consumed by downstream clinical systems; or Kafka publish failures causing clinical transactions to roll back and block charting workflows. Migrate from 'Inline Kafka publish inside clinical transaction (dual-write)' to 'Outbox pattern with CDC relay for FHIR event delivery'. The outbox pattern is non-negotiable for FHIR delivery. The dual-write pattern creates phantom clinical events when transactions roll back: in a healthcare context, a phantom "medication_administered" event on a downstream system is a patient safety incident. The outbox guarantees at-least-once delivery aligned with transaction commit; downstream consumers must dedupe on event ID so replays do not create duplicate clinical events.

Outbox relay introduces delivery lag (< 5s under normal load): downstream systems must tolerate this latency and must not assert synchronous availability of FHIR events as part of the clinical transaction commit path; FHIR message construction errors in the relay must dead-letter and alert rather than silently dropping: a lost FHIR event can mean a downstream system has no record of a clinical event

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single Redis Node / Sentinel Cluster → Distributed Redis Cluster (Consistent Hash Ring)

Migration complexity: medium. Rollback: complex.

Signal: Audit log table growing at > 500K rows/day; INSERT p99 on audit_log > 20ms; autovacuum unable to keep up with dead tuple accumulation from UPDATE operations on the audit log's index pages

Bottleneck: Audit log receiving one row per record access creates I/O contention with clinical record writes on the same PostgreSQL primary. Evolution: Partition the audit_log table by month using PostgreSQL declarative partitioning; child partitions allow VACUUM to operate on bounded table segments without scanning the entire history; index each partition independently to keep index size proportional to partition row count rather than total log size

Signal: pg_locks showing RowExclusiveLock waits on clinical_records or encounter_notes during shift-change peak hours; write p99 > 100ms; occasional deadlock errors in application logs correlated with concurrent addenda writes to the same encounter

Bottleneck: Multiple clinical staff members writing addenda to the same encounter simultaneously, or two processes updating encounter status concurrently. Evolution: Implement optimistic locking with an encounter version column; reject concurrent writes with a conflict error and require the client to reload and retry; this eliminates lock waits by failing fast rather than waiting; ensure the application presents a clear conflict resolution UI: in a clinical context, silent overwrites of concurrent edits are a patient safety risk, not just a data integrity issue