Elasticsearch - Whats the ideal Shard config for 4 node cluster

Question

I have 4 servers installed and running ES. I am looking to setup 2 shards and corresponding replica (1 to 1 shard).

My challenge is, do I need to make 2 nodes as masters and the other 2 nodes as just node datas?

The plan is

Node A acts as Master with 2 Primary Shards = Replica is Node B
Node C acts as Master with 2 Primary Shards = Replica is Node D

Is this an ideal configuration or is there a better alternative. Also since they are all clustered, when datas are pushed to the cluster, would either of the master node take responsibility to distribute the shard between the 2 master nodes?

If I make all 4 nodes both master and data, which config settings will make node A primary shard and node B the replica or which config will tell node A that its replica is node B. Same for nodes C & D.

Thanks

Answer 1

You have two separate problems here:

• Cluster Topology

  It is recommended to have exactly 3 master nodes in an Elasticsearch cluster. You need this to increase resiliency towards node failures and avoid split brain problems.

  An Elasticsearch node can act both as master as well as data node. Note that if a node is set to be a master node but not data node, it cannot store any indexed data (read shards). Hence depending on how much data you want to index, you can set one, two, three or even all four nodes as data nodes.

• Data Topology

  The number of primary and replica shards again depend on how much data you want to index and the disk capacity of the data nodes. If unsure, you can start with the default settings of 5 primary shards and 1 replica shard.

Shards will only be present in data nodes (doesn't matter if they are also master nodes). Regarding balancing shards between the data nodes, you don't need to worry about it; master node will take care of it.