How to setup elastic cluster for huge amount of data?

Question

I am asked to setup elastic search cluster for about 100 TB of text Data! I already know how to do searches and aggs in elastic, but I really don't know how to setup cluster with more than one node for such big data! I mean how many masters, zoe keepers, CDs, ... ? or Do I need to have one dedicate server for activeMQ? ...

Is there any document for explaining that?

Answer 1

ES is a distributed system and creating a cluster with 1 node or 1000 node doesn't make much difference.

What you can do in your case, is to have some master node and more data nodes to create a big cluster.

The master node is responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes.

Data nodes hold the shards that contain the documents you have indexed. Data nodes handle data related operations like CRUD, search, and aggregations. These operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more data nodes if they are overloaded.

You can choose master nodes of small size(if they don't hold data) and data nodes are of large size.

Below is the configuration for master node.

http.port: 9200
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
cluster.name: elasticsearch_hobbes ## note this cluster name must be same for all the es nodes in the same cluster
node.name: "elasticsearch_001_master"// give 002 for other master node
node.master: true
**node.data: false (This master node will not hold the data)**
path.data: /usr/local/var/elasticsearch/
path.logs: /usr/local/var/log/elasticsearch/
discovery.zen.ping.multicast.enabled: false

And below is the config for data node.

cluster.name: elasticsearch_hobbes
node.name: "node2"
node.master: false
node.data : true
http.port: 9201
discovery.zen.ping.multicast.enabled: false
script.engine.groovy.inline.aggs: on
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

You can then go to KOPF plugin of your master node, by clicking the http://localhost:9200/_plugin/kopf/#!/cluster And see the below screen, which shows all the three nodes in the cluster.

Note:- Please follow https://github.com/lmenezes/elasticsearch-kopf to install KOPF plugin. And let me know if you face any problem setting up the cluster.