Use Healthcare Records Platform as the Foundational Architecture Pattern
Deterministic ADR derived from topology, simulation, and advisor intelligence for Healthcare Records Platform. Traceable to YAML knowledge entities.
Context
Healthcare records demand a fundamentally different correctness model than most business applications. HIPAA requires detecting and reporting unauthorized access within 60 days. Clinical decisions made on stale or incorrect record state can cause patient harm. Records cannot be deleted: only amended with explicit attribution and reason. A patient's state at any historical point must be reconstructable for legal and clinical review. Inter-facility interoperability requires HL7 FHIR event streaming to external systems that may have different data models and availability characteristics. The architecture must enforce access control at the data layer (not just the application layer) because a misconfigured application code path must not expose records to unauthorized practitioners. Every record read, write, and amendment must be captured in a tamper-evident audit log before the operation is acknowledged. Primary operational risks include: HIPAA audit log gap: if the audit log write fails independently of the clinical record write (dual-write without atomicity), access events can be lost entirely; lost audit events are a HIPAA breach and may not be detectable until an external audit surfaces the gap; the outbox pattern guarantees atomicity but must be applied to audit event writes, not just downstream FHIR events; RLS policy coverage gap on new tables: every new PostgreSQL table that stores patient-identifiable data must have a corresponding row-level security policy before deployment; a DDL migration that creates a table without RLS exposes all records to any authenticated database role, bypassing application-layer access control; Schema migration lock during active clinical hours: a table-level DDL lock on clinical_records, encounter_notes, or audit_log during peak clinical use blocks all concurrent reads and writes; in an EHR context this is a patient safety event, not just a performance degradation.
Decision
We will adopt the **Healthcare Records Platform** architecture pattern. This is a expert-complexity architecture appropriate for teams at platform engineering team level or above. The advisor rates this pattern as 'advanced' operational maturity.
Rationale
An electronic health record (EHR) architecture built around strict auditability, HIPAA compliance, and append-only correctness. Clinical records are mutable by design (amendments, addenda) but corrections must be explicitly attributed, not silently overwritten. Event sourcing provides a reconstructable audit log; PostgreSQL row-level security enforces patient-level access control at the database layer; Kafka streams HL7 FHIR events to downstream clinical systems. The architecture must support breach detection, access auditing, and state reconstruction at any historical point: not just current state retrieval. Core technology stack: postgresql, kafka, redis.
Accepted Tradeoffs
- ⚠Event sourcing provides perfect reconstruction fidelity and HIPAA-grade audit completeness, but every clinical operation writes two rows (event + projection update), doubling per-transaction I/O; at 100,000 clinical events per day this is manageable, but the write amplification must be modeled when sizing the PostgreSQL primary
- ⚠PostgreSQL row-level security enforces access control at the database layer even if application logic is bypassed, but RLS adds query planning overhead on complex policies; patient-level policies on large tables require testing with production-scale data before rollout to validate that EXPLAIN plans remain index-based
- ⚠Kafka FHIR event streaming enables real-time downstream system integration and replay capability, but HL7 FHIR message construction from internal events adds a transformation layer that must stay synchronized with internal schema evolution; a FHIR message schema that does not match the consuming system's expectation causes silent message rejection, not an immediate error
- ⚠Synchronous replication (synchronous_commit = remote_apply) provides RPO = 0 for clinical records but adds per-write latency (typically 2–10ms on LAN); clinical workflows that issue 5–10 sequential database writes per encounter amplify this into 10–100ms of added latency per encounter save operation
Risks
Asynchronous replicas fall behind the primary under write load and serve reads from an older version of the data. Reads keep succeeding, so nothing errors; what breaks is one of three specific consistency guarantees (read-after-write, monotonic reads, or consistent prefix), each with a distinct user-visible anomaly.
Concurrent writers to the same rows serialize behind each other's row locks, so latency is set not by the work a transaction does but by how long it waits for the writers ahead of it. On a hot row the queue depth, and therefore the tail latency, grows with concurrency while throughput flattens. Blocked writers hold connections open, so a single contended row can drain the connection pool as a secondary failure.
An ALTER TABLE or other DDL statement takes an ACCESS EXCLUSIVE lock that conflicts with every other lock type, including a plain SELECT's ACCESS SHARE. Once that lock request is waiting, every later query on the table queues behind it too, so a DDL statement that is merely waiting, not yet running, is enough to take the table's entire traffic down.
Two or more transactions each hold a lock the other needs, forming a cycle in the lock wait-for graph that no participant can escape on its own. The database breaks the cycle by aborting one transaction, surfacing a serialization-class error the application must catch and retry. Under sustained contention, naive immediate retries re-enter the same cycle and amplify it into a retry storm.
Production configuration diverges from the intended state through manual changes, partial rollouts, and environment-specific overrides, causing failures that are intermittent, hard to reproduce, and require cross-node comparison to diagnose.
Alternatives Considered
AI Retrieval-Augmented Generation Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Healthcare Records Platform is a better fit for the identified workload profile.
Analytics Data Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Healthcare Records Platform is a better fit for the identified workload profile.
API Gateway Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Healthcare Records Platform is a better fit for the identified workload profile.
Audit and Compliance Platform shares core technology (kafka, postgresql) with the chosen architecture but applies different structural patterns; Healthcare Records Platform is a better fit for the identified workload profile.
Scaling Thresholds
Signals indicating the architecture is approaching its scaling limits:
Tier 1: Audit Log Write Throughput
Signal: Audit log table growing at > 500K rows/day; INSERT p99 on audit_log > 20ms; autovacuum unable to keep up with dead tuple accumulation from UPDATE operations on the audit log's index pages
Evolution: Partition the audit_log table by month using PostgreSQL declarative partitioning; child partitions allow VACUUM to operate on bounded table segments without scanning the entire history; index each partition independently to keep index size proportional to partition row count rather than total log size
Tier 2: Concurrent Encounter Write Lock Contention
Signal: pg_locks showing RowExclusiveLock waits on clinical_records or encounter_notes during shift-change peak hours; write p99 > 100ms; occasional deadlock errors in application logs correlated with concurrent addenda writes to the same encounter
Evolution: Implement optimistic locking with an encounter version column; reject concurrent writes with a conflict error and require the client to reload and retry; this eliminates lock waits by failing fast rather than waiting; ensure the application presents a clear conflict resolution UI: in a clinical context, silent overwrites of concurrent edits are a patient safety risk, not just a data integrity issue
Tier 3: FHIR Event Streaming Throughput
Signal: Kafka consumer lag growing on FHIR event topics; downstream clinical systems reporting stale data; outbox table accumulating unprocessed rows > 10,000 at rest
Evolution: Increase outbox relay consumer parallelism; partition Kafka FHIR topics by patient_id to maintain per-patient event ordering while enabling parallel processing; profile FHIR message construction for CPU-intensive transformation paths (e.g., terminology code mapping) and consider caching terminology lookups in Redis
Tier 4: Multi-Facility Read Replica Distribution
Signal: PostgreSQL primary I/O > 70% from read queries during morning rounds (when all staff are querying overnight encounter summaries simultaneously); read replica replication lag > 5s during peak read periods
Evolution: Direct all clinical summary and dashboard reads to the read replica via CQRS routing; ensure read replica has synchronous_standby_names configured to receive writes at most 5 seconds behind primary; audit queries specifically should read from primary (not replica) to guarantee audit log completeness is not affected by replication lag
Migration Path
Mutable clinical records with application-layer audit logging → Event-sourced clinical records with atomic audit event + outbox writes
HIPAA audit requirement exposed during external security review; inability to reconstruct which practitioner accessed a patient record and when; audit log gaps found during incident investigation (application-layer logging not guaranteed to capture all access paths, including background jobs and admin tools)
Inline Kafka publish inside clinical transaction (dual-write) → Outbox pattern with CDC relay for FHIR event delivery
FHIR events being published to Kafka but corresponding clinical record transactions rolling back, resulting in phantom events being consumed by downstream clinical systems; or Kafka publish failures causing clinical transactions to roll back and block charting workflows
All facilities sharing a single PostgreSQL cluster → Per-facility database with cross-facility patient index and record linkage
Facility acquisition or merger; compliance requirement for data residency (state or country-level); single-cluster I/O saturation as facility count grows beyond 5–10 concurrent clinical sites
Operational Requirements
- Minimum team maturity: Platform Engineering Team: This scenario has expert operational complexity. It is recommended for Platform Engineering Team teams or higher.
- Runbooks and alerting for high-severity risks: 4 high-severity risks identified. Each requires a documented runbook, alerting threshold, and on-call response procedure before running in production.
- Event stream operations expertise: This architecture includes event stream infrastructure (Kafka, Kinesis, or similar). Operations requires consumer group management, partition assignment, dead-letter handling, and lag monitoring.
- Cache sizing and eviction policy configuration: Redis or equivalent cache requires correct maxmemory configuration, eviction policy selection (allkeys-lru is common), and cold-start warming strategy after restarts.