Architecture Review: Financial Ledger Platform

Concurrent writers to the same rows serialize behind each other's row locks, so latency is set not by the work a transaction does but by how long it waits for the writers ahead of it. On a hot row the queue depth, and therefore the tail latency, grows with concurrency while throughput flattens. Blocked writers hold connections open, so a single contended row can drain the connection pool as a secondary failure.

Affects 1 node. (Write-Heavy Transactional)

A failover mechanism promotes a new leader without confirming the old one has stopped, so two nodes simultaneously believe they hold the primary role and both accept writes. The two histories diverge, and when the partition that triggered the failover heals, one set of committed transactions must be discarded.

Affects 1 node. (Two-Phase Commit (2PC)). 1 mitigation identified

Seed 'Lock Contention Risk Probe' identifies 2 metrics relevant to lock_contention.

Metrics to instrument: error_rate, p95_latency_ms

Trigger: Audit requirement to reconstruct account state at any historical point in time; compliance requirement for complete transaction history; inability to explain why a balance is what it is from current state alone. Migrate from 'Mutable account balance table with no event history' to 'Event sourced ledger with append-only events and projected balance view'. Bootstrap historical state as a single "initial_balance" event per account. Run the event-sourced model in parallel with the mutable model for 30 days before decommissioning the mutable path.

Historical event reconstruction is only possible from the migration cutover date: pre-migration history must be bootstrapped as initial balance events; Projected balance views must be kept consistent with the event log; any discrepancy indicates a bug that must be caught in testing

Trigger: Kafka publish failures causing financial transaction rollbacks; or Kafka publish succeeding but transaction rolling back, causing phantom events downstream. Migrate from 'Synchronous Kafka publish in transaction (dual-write pattern)' to 'Outbox pattern with CDC relay to Kafka'. The outbox pattern is non-negotiable for financial systems. The synchronous dual-write is an anti-pattern that creates phantom events on rollback.

Outbox relay introduces a delivery lag (typically < 5s); downstream consumers must handle this eventual delivery, not assume synchronous availability; Relay stall must trigger an immediate operational alert: unmonitored stall is a consistency incident

Simulation demonstrates critical degradation of redis, postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Simulation demonstrates critical degradation of postgresql

Without a runbook, recovery from this failure mode will be ad-hoc

Evolution from Unified OLTP + Analytics on PostgreSQL → Separated OLTP (PostgreSQL) + OLAP (ClickHouse/Snowflake)

Migration complexity: medium. Rollback: always.

Evolution from Single-Node PostgreSQL → Partitioned PostgreSQL

Migration complexity: high. Rollback: rarely.

Signal: pg_locks shows contended rows on accounts table; write p99 > 50ms; deadlock errors in application logs; pg_stat_activity showing many transactions waiting for RowExclusiveLock on the same account rows

Bottleneck: Concurrent debit/credit transactions competing for the same account row versions. Evolution: Implement optimistic locking with version column and retry; or queue concurrent updates for the same account entity through an account-scoped serialization queue at the application layer; or partition the accounts table by account range

Signal: Write p99 > 100ms with synchronous_commit = remote_apply; replica WAL apply lag visible in pg_stat_replication; network jitter between primary and replica causing write latency spikes correlating with replication ACK delays

Bottleneck: Synchronous replication write-ahead wait amplifying network latency for every committed transaction. Evolution: Co-locate primary and replica in the same availability zone for lowest replication RTT; tune wal_sender_timeout and recovery_min_apply_delay; evaluate whether synchronous_commit = on (durable to primary WAL only) is acceptable for your regulatory risk model