Search Platform
Separate write and read paths for search. PostgreSQL is the authoritative store. Elasticsearch is an eventually consistent projection maintained by a Kafka-driven indexer. Redis caches frequent queries with short TTLs. Index staleness of 1–30 seconds is expected and documented.
Description
Search systems require a fundamental architectural choice: treat search as a primary datastore (operationally complex) or as a derived, eventually consistent index (operationally simple, with documented consistency trade-offs). This composition takes the latter approach.
The authoritative data lives in PostgreSQL. Elasticsearch is a read-optimized projection of that data, built and maintained by a dedicated indexing pipeline. The decoupling between write path (PostgreSQL) and read path (Elasticsearch) allows each to be scaled, tuned, and maintained independently.
The indexer consumes domain events from Kafka, applies field mappings and denormalization, and bulk-indexes into Elasticsearch. Redis caches the most frequent search queries with a 30–60 second TTL, reducing Elasticsearch load by 40–70% for popular queries with stable result sets.
The critical user-facing trade-off: documents are not immediately searchable after write. A product created at T=0 appears in search results at T+1s to T+30s. This is acceptable for most search use cases and must be explicitly communicated to the engineering team consuming the search API.
Use Cases
- ·Product catalog search with faceted filtering
- ·Full-text document or content search
- ·Enterprise search across internal knowledge bases
- ·Job board or marketplace listing search
- ·Search over datasets from 10M to 1B indexed documents
Scale Profile
Entry Point
1M indexed documents or 1k search queries/hour
Sweet Spot
10M–500M documents, 10k–100k queries/hour
Scaling Ceiling
Elasticsearch handles billions of documents with proper cluster sizing. Query throughput ceiling is hardware-dependent: typically 10k–50k queries/second per cluster.
Typical RPS
100–10k search queries/second
Architecture Nodes (9)
Users performing search queries via web or mobile interfaces.
Handles all write (create/update/delete entities) and read (search queries) requests. Write path: writes to PostgreSQL. Read path: checks Redis cache, falls back to Elasticsearch.
Authoritative source of truth for all entities. Search index is derived from this. Full entity data with all fields. The only store that receives writes.
Polls the outbox table in PostgreSQL and publishes entity change events to Kafka. Alternatively replaced by Debezium CDC. Tracks last published offset. Ensures at-least-once event delivery.
Event transport for entity changes. One topic per entity type (products, articles, users). Partitioned by entity_id for ordering. Retention: 7 days.
Kafka consumer that transforms entity events into Elasticsearch documents. Applies field mappings, denormalization, and bulk indexing. Uses Elasticsearch bulk API with batch size 500 and flush interval 5 seconds.
Distributed inverted index for full-text and structured search. Primary search store. Index per entity type with explicit mapping. 3-node cluster minimum for production. Shard count: 1 shard per 50M documents.
Query result cache. Cache key: SHA256(query_string + filters + sort + page). TTL: 30–60 seconds. Invalidated on bulk re-index. Hit rate target: 50–70% for production search workloads.
Elasticsearch cluster administration and index monitoring. Not in the request path. Used by engineering to monitor index health, shard allocation, and query profiling.
Dependencies (9)
3 critical path edges. Failure on these directly degrades user-facing requests.
Write and search requests
All client requests. Writes mutate PostgreSQL. Reads query Redis then Elasticsearch.
Timeout: 10s
Entity writes
All entity create/update/delete operations. Outbox event written in same transaction. Returns immediately: does not wait for Elasticsearch indexing.
Timeout: 3s
Query cache lookup
Cache-aside on search read path. Checks Redis first. On HIT: return cached results. On MISS: query Elasticsearch, populate cache with result + TTL.
Timeout: 50ms
Search queries
Elasticsearch Query DSL requests on cache miss. Full-text search with term filters, range filters, and aggregations. Query timeout: 2000ms.
Timeout: 3s
Outbox events
Outbox relay polls outbox table every 100ms. Reads uncommitted events, publishes to Kafka, marks as published. Maintains at-least-once delivery guarantee.
Entity change events
Events published to per-entity-type Kafka topics. Schema: {event_id, entity_type, entity_id, operation, payload, occurred_at}.
Index update events
Search indexer consumes entity events. Builds Elasticsearch document from event payload. Bulk indexes in batches of 500 with 5-second flush.
Bulk index operations
Elasticsearch bulk API. Uses upsert (_index with doc_as_upsert). Delete events issue Elasticsearch delete by ID. Index refresh interval: 1 second (default).
Timeout: 10s
Cluster monitoring
Kibana reads cluster health, shard allocation, and index stats. Not in production request path.
Failure Propagation
How a failure in one component cascades through the system. Each path documents the mechanism and the mitigation that breaks the chain.
Mechanism
Elasticsearch JVM heap exhaustion from large query result sets or unbounded aggregations. GC pauses cause query timeout. Full OOM causes node failure. With 3-node cluster, replica shards continue serving but reduced capacity.
Mitigation
Set heap to 50% of node RAM (max 32GB). Circuit breaker limits on aggregation field data. Avoid returning >10k documents per query (use search_after pagination). Monitor heap usage alert at 75%.
Mechanism
Indexer crash causes consumer lag growth. Search results become increasingly stale. Redis cache returns stale cached results. When indexer recovers, bulk replay can spike Elasticsearch write throughput, causing index merge pressure.
Mitigation
Monitor consumer lag. Alert at 1-minute lag for near-real-time use cases. Implement indexer concurrency with multiple partitions. Throttle replay speed to avoid Elasticsearch saturation.
Mechanism
PostgreSQL failure takes down write path. Search read path continues: Elasticsearch and Redis are independent. Outbox relay stops polling. Indexer consumer lag begins growing.
Mitigation
PostgreSQL HA with automated failover. Read path degradation is zero. Write path resumes after promotion.
Scaling Transitions
Inflection points where this architecture begins to degrade and what the recommended evolution looks like.
Shard size exceeds 50GB recommended ceiling. Query routing across too many shards increases coordination overhead. Merge pressure from bulk indexing degrades query latency.
Recommended Action
Implement index lifecycle management (ILM). Time-based index rotation. Hot-warm-cold tier architecture for cost optimization.
Redis cache eviction rate rises. Cache effectiveness drops below 40%. Hot key contention on popular queries.
Recommended Action
Migrate to Redis Cluster. Shard by query hash prefix. Consider CDN-level caching for completely public search results.
Patterns Applied
Architectural Notes
- ·Index staleness (1–30s) is a feature, not a bug: communicate this SLA explicitly in API documentation. Attempting to make search immediately consistent requires synchronous indexing which kills write throughput.
- ·Never let Elasticsearch be the source of truth. If Elasticsearch data is corrupted or an index is dropped, it must be fully rebuildable from PostgreSQL by replaying all events.
- ·Elasticsearch split-brain prevention requires a minimum cluster size of 3 nodes. Two-node clusters are dangerous: network partition causes both nodes to elect themselves as master.
- ·The query cache TTL should match your data mutation rate. Highly mutable data (prices, inventory) needs 5–10s TTL. Static content (articles) can tolerate 60–300s.
Confidence
StrongOutbox-driven search indexing to Elasticsearch is widely deployed at Shopify, Airbnb, and major SaaS platforms. Pattern is documented in multiple engineering blogs.