relationship between cluster, node, index, shard, segment#

Explanations of basic terminology:

• An Elasticsearch cluster has multiple nodes, for example data nodes, ML nodes, etc.
• A node is a JVM instance that is running Elasticsearch.
• An index is a collection of documents, an index can have multiple primary shards and replica shards.
• A shard is placed in a node in the Elasticsearch cluster.
• A shard is a Apache Lucene index
• A Lucene index consists of multiple segments (internal structure used by Lucene)

You can use the cat API to get the info about nodes/index/shard/segments:

GET _cat/nodes?v=true

GET _cat/indices/my-index

GET _cat/shards/my-index

GET _cat/segments/my-index

# or you can also use the following api to get segments info about an index
GET my-index/_segments

ref:

• https://discuss.elastic.co/t/relation-between-shards-and-nodes/104562
• Elasticsearch node: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html
• cat node API: https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-nodes.html
• understanding segments in Elastic: https://stackoverflow.com/q/15426441/6064933

shards and replia (primary and replica)#

A index can have multiple primary shards and replica shards. Primary shard can accept read/write requests, while replica shards can only accept read requests.

The cat shards api can be used to check the status of shards:

# v=true will show the column header
GET _cat/shards/my_index?v=true&h=index,shard,prirep,state,docs,unassigned.for,unassigned.reason&s=state

number of shards and number of replicas#

For how to set proper number of shards, refer to official doc

Constraints for the number of replica: for a primary shard, its replica shards can not be in the same node, also between those replica shards, they can not be in the same node. This effectively means that the number of replica must be less or equal to num_node - 1. For example, if you have 3 nodes, if primary1 is in node 1, then its replica shards can only take node 2 and node 3. If you break this constraint, and set the number of replica to larger value, when you check the info of this index, you will see that its health status is yellow instead of green.

GET _cat/indices/my_index?v

If you check the shard info about this index (GET _cat/shards/my_index), you will see that some replica has UNASSIGNED status:

The number of shards is a static index setting and can only be set at index creation time. The number of replicas is a dynamic setting that can be changed dynamically for a index without interrupting search and indexing request. You can set the number of shards and replicas using the index creation api:

DELETE my_index

PUT my_index
{
  "settings": {
    "index.number_of_shards": "1",
    "index.number_of_replicas": "2"
  }
}

As explained, the number of replica is a dynamic setting, you can change the value after index creation with index-update-setting api:

PUT my_index/_settings
{
  "settings": {
    "index.number_of_replicas": "1"
  }
}

When you decrease the number of replicas, Elastic will delete the extra replicas. When you increase the number of replicas, Elastic will automatically copy the primary shards to suitable node. For some time, you will see the index status is yellow. If you use the cat-shard API, you will see that the state for the replica shards is INITIALIZING. After some time, the state of these replica shards become STARTED, and the index status becomes green.

ref:

• Elastic shard and replica guide: https://www.elastic.co/search-labs/blog/elasticsearch-shards-and-replicas-guide

shard write and read model#

When we do indexing operation for an index, the operation is first done on primary shards, then synced to replica shards. If you have a large number of documents to index, this is usually slower than only updating the primary shards. So the Elastic official doc recommends to set the number of replica to 0 for initial large load. After indexing, you can set the number of replica to its original value, Elastic will then sync the changes under the hood.

Having multiple replica helps Elastic to prevent data loss and also let Elastic to handle more search request, because it can distribute the read operation to one of the node holding the replica shards. When Elastic receive search/read request, the request will be routed to nodes that contains the relevant data, see shard-routing.

explain why shard is unassigned or assigned to a certain node#

If you see that a shard is unassigned in the cat-shard API and want to get more detailed info. The cluster allocation explain API can explain why a shard is unassigned or assigned. The API: https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html

GET _cluster/allocation/explain
{
  "index": "my_custom_index",
  "shard": 0,
  "primary": true
}

For the index parameter, it seems we can not use alias, and we have to use the actual index name. shard refers to the shard number. primary refers whether this is a primary or replica shard.

Note that when we want to explain for unassigned shard, we should not use the current_node:

To explain an unassigned shard, omit this parameter.

References#

• no shard available exception: https://stackoverflow.com/a/54019924/6064933
• Elastic shards and replicas: https://stackoverflow.com/q/15694724/6064933