Elasticsearch
8.xSummary
Distributed full-text search and analytics engine built on Apache Lucene, designed for near-real-time indexing, complex search queries, and log analytics.
Primary Use Case
Full-text search over large document corpora, application log aggregation and analysis (via ELK stack), and near-real-time analytics on unstructured data.
Workload Fit
Strengths
Best for
- ·Full-text search over large document sets requiring relevance ranking
- ·Log aggregation and analysis where structured queries are impractical
- ·Faceted navigation with real-time aggregation (e-commerce search, dashboards)
- ·Supplementing a primary OLTP store with a search-optimized read path
Excels when
- ·Query patterns require full-text relevance scoring that SQL LIKE cannot provide
- ·Ingestion volume makes per-row RDBMS indexing uneconomical
- ·Multiple search facets and aggregations run simultaneously
Architectural advantages
- ·Inverted index provides sub-second search across billions of documents
- ·Distributed sharding handles data sets that exceed single-node capacity
- ·Aggregation API enables analytics queries that would be expensive in OLTP databases
When to Avoid
Avoid when
- ·Data requires strict ACID consistency or transactional writes
- ·Primary access pattern is key-value lookup: Elasticsearch has higher overhead than Redis/DynamoDB
- ·Team cannot maintain a dedicated Elasticsearch cluster: operational complexity is high
Common misuses
- ·Using Elasticsearch as the primary datastore: it is a search index, not a source of truth
- ·Indexing every field in every document: use selective field mapping to control index size
- ·Creating a new index per day without ILM: leads to shard explosion over time
Consistency & Transactions
Scaling
Read scalability
Index shards are distributed across nodes. Search requests fan out to all relevant shards and are aggregated at the coordinating node. Adding nodes scales both read throughput and query parallelism.
Write scalability
Documents are indexed into primary shards with asynchronous replication to replicas. Bulk indexing achieves high ingestion rates; refresh intervals control near-real-time visibility latency.
Failure Behavior
Known failure modes
- ·Shard explosion: too many small shards exhausts heap and degrades cluster stability
- ·GC pressure: large heaps or high query concurrency causes stop-the-world GC pauses
- ·Split-brain (pre-7.x): network partition could lead to two active master nodes
- ·Mapping explosion: dynamic mapping on high-cardinality fields creates unbounded field count
- ·Hot thread saturation: expensive aggregations starve search threads
Bottlenecks
- ·Coordinating node aggregation overhead scales with shard count and result set size
- ·JVM heap pressure under high query concurrency causes GC pauses
- ·Refresh interval (default 1s) creates indexing-to-search latency floor
Degradation patterns
- ·Shard count growth without ILM causes cluster state size to exceed master node capacity
- ·Query-heavy load with insufficient heap causes circuit breaker trips and 429 errors
- ·Mapping conflicts from dynamic mappings block new document ingestion
Recovery considerations
- ·Index snapshots to object storage are the primary backup mechanism
- ·Shard recovery after node failure requires network-intensive shard reallocation
- ·Cluster yellow/red state during recovery degrades search availability until rebalancing completes
Operational Pitfalls
- ·Setting JVM heap >50% of system RAM: ES relies on OS file cache for performance
- ·Over-sharding: each shard has fixed overhead; hundreds of shards per node degrade performance
- ·Not configuring ILM: unbounded index growth exhausts disk without lifecycle policies
- ·Using dynamic mappings for production indices: uncontrolled field creation causes mapping explosions
Architecture Guidance
Common topology roles
Migration notes
- ·From PostgreSQL full-text search: Elasticsearch relevance scoring and faceting are significantly more capable, but require data synchronization pipeline
- ·To OpenSearch: API-compatible fork, but plugin ecosystem and some enterprise features differ
- ·Adding Elasticsearch as a secondary index: requires CDC or dual-write to keep in sync with primary datastore
Advisor Guidance
When: scenario uses Elasticsearch as a primary datastore
Elasticsearch is a search index, not a source of truth: add a durable primary store and sync to ES
When: scenario has full_text_search or log_analytics workload
Configure ILM policies from day one to prevent shard explosion as data grows
When: scenario uses dynamic mappings on high-cardinality fields
Define explicit index mappings: dynamic mapping on high-cardinality fields causes mapping explosions and cluster instability
Comparison Factors
operational complexity
High: JVM tuning, shard management, ILM, and cluster topology require dedicated expertise
search capability
Very high: full-text relevance, faceting, aggregations, and geo-search in one engine
consistency
Eventual: not suitable as a source of truth for transactional data
cost
High: cluster size scales with data volume and query concurrency requirements
Managed Cloud Options
Enables Patterns
Basis
Well-documented at scale; operational characteristics widely published across industry
Learning Modules
CQRS Operational Tradeoffs
How separating the command and query models enables independent scaling, what consistency guarantees CQRS gives up, how read model staleness propagates, and when the operational complexity is justified.
Search Engine Internals
How inverted indexes enable full-text search, how BM25 relevance scoring ranks results, and how Elasticsearch's indexing pipeline, refresh cycle, and shard architecture determine operational behavior under load.
Evolution Paths
Simulations
Related Architecture Knowledge
Outbound: this entity affects
Elasticsearch implements sharding natively: every index is divided into primary shards distributed across nodes, with replica shards providing redundancy. Sharding is not optional: all Elasticsearch data exists within a shard.
Tradeoffs
- ·Fixed shard count at creation requires capacity planning before data ingestion begins
- ·Cross-shard queries fan out to all shards and merge results on the coordinating node: too many shards add merge overhead
- ·Replica shards double storage and I/O for writes: fewer replicas during bulk indexing, restore after load
Used In Architecture Scenarios
Read-Heavy Application
A CMS for publishing and serving structured content: articles, documentation, product pages, and localized variants: where read APIs serve 50–100x more traffic than editorial write APIs. PostgreSQL stores the content graph (articles, authors, categories, taxonomy) and workflow state (draft, in-review, scheduled, published). Redis caches published content objects for read APIs. Elasticsearch powers full-text content search with faceting and relevance ranking. Cache invalidation on publish must be fast and complete; N+1 query patterns on content relationship traversal are the dominant database performance risk during reads.
Multi-Tenant SaaS
A multi-tenant developer tooling platform providing CI/CD pipeline execution, log aggregation, code analysis, and dependency scanning across isolated tenant organizations. Tenant isolation is the primary correctness constraint: a security boundary violation between tenants is a critical incident, not a performance event. PostgreSQL row-level security enforces data isolation; Redis manages job queues and distributed locks; Elasticsearch indexes pipeline log output for search; Kafka delivers webhook events to tenant-registered endpoints; MinIO stores pipeline artifacts. Resource quota enforcement prevents any single tenant's burst from affecting others.
Marketplace Platform
An e-commerce order lifecycle platform handling cart, checkout, payment, fulfillment, and returns across a mixed read/write workload where product discovery is read-heavy, checkout is write-transactional, and fulfillment is event-driven. The saga pattern orchestrates multi-step checkout: reserve inventory → charge payment → confirm order → notify fulfillment. PostgreSQL owns order records and inventory with row-level locking; Redis holds session state and cart contents with sub-millisecond access; RabbitMQ delivers fulfillment notifications with dead-letter handling; Elasticsearch serves product search and order history with faceted navigation. CQRS separates the write command path from the read model: the order read model is denormalized for fast order history queries without joining across domain tables.
Analytics Pipeline
A metrics, logs, and traces ingestion and query platform built to absorb the telemetry output of a production system fleet: including the telemetry volume spikes that accompany the incidents the platform is meant to detect. ClickHouse stores metrics data with automatic time-based rollup via continuous materialized views; TimescaleDB provides complementary time-series storage for high-cardinality alert evaluation; Kafka buffers the ingestion stream against downstream write pressure, decoupling ingest acceptance rate from storage write throughput; Elasticsearch serves log full-text search and structured field filtering; Redis caches dashboard query results and active alert state for sub-100ms alert evaluation latency. The alert engine evaluates threshold and anomaly rules against pre-computed materialized views, not raw data, to bound alert evaluation cost independent of ingestion volume.
Search-Heavy Application
A content platform architecture centered on Elasticsearch for full-text search, faceted navigation, and ranked results, with PostgreSQL as the transactional source of truth and Redis for session management and hot content caching. WAL-based CDC maintains index freshness by streaming PostgreSQL changes into Elasticsearch asynchronously. The core tension is between search index freshness, query performance, and index maintenance cost under high write volume.
Marketplace Platform
A two-sided marketplace architecture serving buyers, sellers, listings, transactions, search, and notifications from a shared infrastructure, where multiple independent domains must coordinate without tight coupling. Event sourcing captures every state transition; the saga pattern orchestrates multi-step transactions (create order, reserve inventory, charge payment, notify seller) with compensating transactions for partial failures. Kafka decouples domain event publication from consumption; RabbitMQ handles notification fanout; Elasticsearch serves listing search; Redis caches listing display and session state.