ElasticSearch index has thousands of shards

Question

A request to the _search endpoint returns the following:

{
    "took": 303,
    "timed_out": false,
    "num_reduce_phases": 12,
    "_shards": {
        "total": 6121,
        "successful": 6121,
        "skipped": 0,
        "failed": 0
    },

The number of shards is increasing periodically. What could cause the number of shards to increase like this? I do not see any evidence of the shards being explicitly added.

Answer 1

Shards are set up on index creation, depending on the elasticsearch version the defaults may change.

For the last version it is 1 primary shard and 1 replica, meaning each index will use 2 shards. You can change the number of replicas after creating the index, but not the number of primaries.

Then for example if you have 2 primary shards and 3 replicas index you will have a total of 8 shards ( 2 pri + 2 pri * 3 rep = 8)

Common reasons of seeing shards increasing it is you are ingesting 1 index a day. Many ingestion tools will use the current day as index name, causing a new index generating every day.

You can read more about shards here:

https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster