Shards and Replicas in Elasticsearch

Elasticsearch, built on top of Apache Lucene, offers a powerful distributed system that enhances scalability and fault tolerance. This distributed nature introduces complexity, with various factors influencing performance and stability.

Key among these are shards and replicas, fundamental components that require careful management to maintain an efficient Elasticsearch cluster. This article delves into what shards and replicas are, their impact, and the tools available to optimize their configuration.

Elasticsearch indexes can grow to enormous sizes, making data management challenging. To handle this, an index is divided into smaller units called shards. Each shard is a separate Apache Lucene index, containing a subset of the documents from the main Elasticsearch index. This division helps keep resource usage in check, as Lucene indexes have a maximum document limit of approximately 2.1 billion.

Large shards can be inefficient, making operations like moving indices across machines time-consuming and resource-intensive. Splitting data across multiple shards distributed across different machines allows for manageable chunks, reducing risks and improving efficiency. However, finding the right balance in the number of shards is crucial. Too few shards can slow down query execution, while too many can consume excessive memory and disk space, impacting performance.

Setting Up Shards

When creating an index, you define the number of shards, a decision that cannot be changed without reindexing the data. For instance, you might set up an index as follows:

PUT /sensor
{
    "settings": {
        "index": {
            "number_of_shards": 6,
            "number_of_replicas": 2
        }
    }
}

Generally, each shard should hold between 30-50GB of data. For example, if you expect to accumulate around 300GB of logs daily, an index with 10 shards would be appropriate.

Shard States

Shards can exist in various states:

• Initializing: The initial state before the shard becomes usable.
• Started: The shard is active and ready to receive requests.
• Relocating: The shard is being moved to another node, often due to disk space issues.
• Unassigned: The shard has not been assigned, typically due to node failure or index restoration.

To view shard states and metadata, use the following command:

GET _cat/shards

For specific indices:

GET _cat/shards/sensor

Replicas are copies of shards, enhancing data redundancy and search performance. Each replica resides on a different node from the primary shard, ensuring data availability even if a node fails. While replicas help distribute search queries for faster processing, they consume additional memory, disk space, and compute power.

Unlike primary shards, the number of replicas can be adjusted at any time. However, the number of nodes limits the number of replicas that can be effectively utilized. For instance, a cluster with two nodes cannot support six replicas; only one replica will be allocated. A cluster with seven nodes, however, can accommodate one primary shard and six replicas.

Optimizing Shards and Replicas

Optimization involves monitoring and adjusting configurations as index dynamics change. For time series data, newer indices are usually more active, necessitating different resource allocations than older indices. Tools like the rollover index API can automatically create new indices based on size, document count, or age, helping maintain optimal shard sizes.

For older, less active indices, techniques like shrinking (reducing the number of shards) and force merging (reducing Lucene segments and freeing space) can decrease memory and disk usage.

1. Plan Shard Count at Index Creation

• Determine the appropriate number of shards based on expected data volume (e.g., 30-50GB per shard).
• Set shard count at index creation since it cannot be changed without reindexing.

2. Balance Shard Size

• Avoid too large shards to prevent inefficiencies in data movement and processing.
• Ensure shards are not too small, as excessive shards can increase memory and disk overhead.

3. Set an Appropriate Number of Replicas

• Use replicas to enhance data redundancy and search performance.
• Adjust the number of replicas based on the number of available nodes (n + 1 rule for n replicas).

4. Monitor Shard States Regularly

• Use _cat/shards API to check shard states and ensure they are in optimal states (e.g., STARTED).

5. Use Rollover API for Dynamic Indices

• Implement rollover indices for time series or growing datasets to keep shard sizes manageable.

6. Optimize Older Indices

• For less active indices, use shrinking to reduce the number of shards.
• Employ force merging to consolidate Lucene segments and free up resources.

7. Distribute Shards Evenly Across Nodes

• Ensure primary and replica shards are on different nodes to prevent data loss from node failure.
• Balance shard distribution to avoid overloading specific nodes.

8. Monitor Cluster Health

• Use Elasticsearch monitoring tools or third-party solutions (e.g., Elastic Stack, Prometheus) to track cluster performance and resource utilization.

Shards and replicas form the backbone of Elasticsearch’s distributed architecture. Understanding and optimizing their configuration is critical for maintaining a robust and high-performing Elasticsearch cluster. By effectively managing shards and replicas, you can ensure better scalability, fault tolerance, and overall performance of your Elasticsearch deployment.