每天约200G日志的Elasticsearch集群设计

Question

I've created ES cluster (version 5.4.1) with 4 data nodes, 3 master, one client node (kibana).

The data nodes are r4.2xlarge aws instance (61g memory, 8vCPU) with 30G memory allocated for the ES JAVA.

We're writing around 200G of logs every day and keep it for the last 14 days.

I'm looking for recommendations to our cluster to improve the cluster performance, especially the search performance (kibana).

More data nodes? more client nodes? bigger nodes? more replica's? anything that can improve the performance is an option.

Is there anyone with something close to this design or loads? I'll be happy to hear about other designs and loads.

Thanks, Moshe

Answer 1

1. How many shards are you using? The default of 5? That would even be a pretty good number. Depending on who you ask a shard should be between 10G and 50G; with a logging use-case probably rather on the 50GB side.
2. Which queries do you want to speed up? Do they mainly target recent data or long time-spans? If you are mainly interested in recent data, you could use different node types in a hot-warm architecture. More power to the nodes with recent data and less data; the bulk of older and less frequently accessed data on less powerful nodes.
3. Generally you'll need to find your bottleneck. I'd get the free monitoring plugin and take a look at how both Kibana and Elasticsearch are doing.

Wild guess: You are limited on IO. Prefer local disks over EBS, prefer SSDs over spinning disks, and if you can, get as many IOPS as you can afford for that use-case.