Financial Payment Platform
Payment processing with idempotency-first design: Redis idempotency store prevents duplicate charges, PostgreSQL append-only ledger is the authoritative record, the outbox pattern guarantees at-least-once payment processor delivery (the Redis idempotency store deduplicates for effectively-once), and ClickHouse provides immutable audit log for compliance.
Description
Payment systems have a binary correctness requirement: a charge must execute exactly once. Too few executions leave revenue on the table. Too many executions charge customers twice. Neither is acceptable. This composition achieves exactly-once semantics through three complementary mechanisms:
1. Idempotency key (client-supplied): before processing any charge request, the system
checks Redis for the idempotency key. If found, return the cached result immediately
without re-executing the charge. This handles client retries.
2. Append-only ledger: charges are never updated or deleted. Balance = sum of all
credit and debit transactions for an account. This eliminates a class of race
conditions where concurrent updates overwrite each other.
3. Outbox + saga pattern for external payment processor: the charge record and outbox
event are written in a single ACID transaction. The outbox is drained to Kafka by
CDC, and the payment processor is called by a saga executor. If the processor call
fails, the saga retries from the outbox: the local ledger record persists and is
reconciled after processor confirmation.
The audit log in ClickHouse and S3 serves dual purposes: operational analytics for finance (revenue metrics, chargeback analysis) and compliance archival with immutable retention (S3 Object Lock, 7-year retention for financial records).
Use Cases
- ·Payment processing platforms handling 10k–500k transactions/day
- ·Subscription billing systems with recurring charges
- ·Marketplace payment splitting and escrow
- ·Internal ledger systems for credits, refunds, and adjustments
- ·Financial platforms requiring SOC 2 or PCI-DSS compliance audit trail
Scale Profile
Entry Point
Any volume: idempotency is mandatory from day one regardless of scale
Sweet Spot
10k–500k transactions/day, sub-100ms P95 payment initiation latency
Scaling Ceiling
PostgreSQL ledger handles ~5k TPS with proper partitioning (partition by account_id range). Beyond this, consider Vitess or CockroachDB for horizontal ledger scale.
Typical RPS
10–5k TPS for payment initiation
Architecture Nodes (9)
Merchant APIs, mobile clients, and internal billing systems. Must supply an idempotency_key with every charge request. Key must be unique per business operation, not per retry.
Handles charge initiation, refund requests, and status queries. First step: idempotency key check against Redis. On HIT: return cached result. On MISS: proceed to ledger write + outbox.
Idempotency key store. Key: idempotency:{client_id}:{idempotency_key}. Value: serialized response. TTL: 24 hours. SET NX (set-if-not-exists) prevents race conditions between concurrent requests with the same key.
Append-only financial ledger. Charges are INSERT-only: never UPDATE or DELETE. Balance computed as SUM(amount) WHERE account_id = ? over transaction log. Partitioned by account_id range. Includes outbox table for saga events.
Durable event transport for saga execution. CDC reads outbox table → Kafka. Topics: payment.initiated, payment.confirmed, payment.failed, payment.refunded. Exactly-once delivery required.
Stateful saga coordinator. Consumes payment.initiated events. Calls external payment processor. On success: publishes payment.confirmed + updates ledger. On failure: retries with exponential backoff, publishes payment.failed after max retries. Circuit breaker on processor unavailability.
Third-party payment processor (Stripe, Adyen, Braintree). Network call with its own idempotency key. Saga executor re-uses the internal charge_id as processor idempotency key. Failures trigger retry with same idempotency key.
Immutable audit log of all payment operations. Receives events from Kafka. ReplacingMergeTree on (charge_id, version). Enables sub-second queries for fraud detection and finance reconciliation.
Long-term compliance archive with S3 Object Lock (WORM mode). Daily export of ClickHouse audit events as Parquet. Retained for 7 years per financial regulatory requirements.
Dependencies (9)
4 critical path edges. Failure on these directly degrades user-facing requests.
Charge requests
Client submits charge with idempotency_key in header. API returns synchronously with charge result or 202 Accepted if processing is async.
Timeout: 10s
Idempotency key check
First operation on every charge request. SET NX with 24h TTL. If key already exists: return cached response immediately. If not: proceed with charge processing.
Timeout: 100ms
Ledger write + outbox
Single ACID transaction: INSERT into charges table + INSERT into outbox table. Both succeed or both fail. If idempotency key was just set and this transaction fails, the idempotency key is deleted to allow retry.
Timeout: 3s
Outbox → payment.initiated
CDC reads outbox table. Publishes payment.initiated events to Kafka. Exactly-once via Kafka transactional producer + idempotent broker.
Saga event consumption
Saga executor consumes payment.initiated. Idempotent: checks if charge already processed before calling payment processor. Stores saga state in dedicated table in ledger_db.
Processor charge call
Network call to external payment processor. Uses charge_id as processor idempotency key. Retry with same key on timeout. Circuit breaker after 5 consecutive failures.
Timeout: 30s
Saga outcome events
Publishes payment.confirmed or payment.failed. Downstream consumers update UI, trigger fulfillment, or handle failure compensation.
Audit event ingestion
All payment events flow to ClickHouse audit log. Append-only. Used for reconciliation, fraud detection, and operational reporting.
Daily compliance export
Daily Parquet export to S3 Object Lock bucket. Files are immutable after write. Retained per regulatory schedule.
Failure Propagation
How a failure in one component cascades through the system. Each path documents the mechanism and the mitigation that breaks the chain.
Mechanism
Payment processor downtime causes saga executor calls to timeout. Circuit breaker opens after 5 failures. New charges queued in Kafka outbox. Client sees pending status until processor recovers. Retries replay automatically via Kafka consumer on circuit breaker close.
Mitigation
Circuit breaker with exponential backoff retry. Max retry window: 24 hours. Alert on circuit breaker open. Display pending status to users with estimated resolution time.
Mechanism
Redis downtime causes idempotency check to fail. Two options: (1) reject all requests (safest: no risk of duplicate charges), (2) proceed without idempotency check (risk of duplicates on client retry). Choice depends on risk tolerance.
Mitigation
Fail-safe: reject charge requests when idempotency store is unavailable. Return 503 with Retry-After. Redis Sentinel or Cluster for HA. Idempotency store downtime is a P1 incident.
Mechanism
High-value account with many concurrent charges causes row-level lock contention. Saga executor updating the same account serializes. P99 latency rises. Under extreme load, lock wait timeout cascades.
Mitigation
Advisory locks per account_id for serialized charge processing on same account. Partition ledger by account_id to limit lock scope. Statement timeout at 5s.
Scaling Transitions
Inflection points where this architecture begins to degrade and what the recommended evolution looks like.
PostgreSQL primary write throughput saturates. Ledger INSERT throughput and WAL generation exceed single-instance capacity.
Recommended Action
Partition ledger by account_id range across multiple PostgreSQL instances. Introduce Vitess for transparent horizontal sharding.
Single Redis instance becomes hot under global payment volume. Idempotency key TTL management and memory pressure.
Recommended Action
Redis Cluster with consistent hashing on client_id. Geo-replicated for multi-region deployments.
Patterns Applied
Architectural Notes
- ·Two-phase commit (2PC) across the ledger and payment processor is explicitly avoided. 2PC requires the payment processor to participate in the distributed transaction, which external APIs cannot guarantee. Saga with compensation is the correct model.
- ·The idempotency key window (24 hours) must match your client retry policy. If clients retry after 24 hours, duplicate charges are possible. Extend TTL if business requires longer retry windows.
- ·Never UPDATE a charge row in the ledger. If a charge is reversed, INSERT a new reversal record. Append-only ledger eliminates UPDATE-based race conditions.
- ·S3 Object Lock WORM mode is a compliance requirement for financial records in many jurisdictions. Test the Object Lock policy in staging before production: it cannot be disabled after enabling.
Confidence
StrongIdempotency + append-only ledger + outbox/saga is the documented approach by Stripe, Braintree, and major payment platforms. Pattern is well-specified in distributed systems literature.