Elasticsearch缓慢的搜索查询性能

Question

I'm having a lot of issues tuning Elasticsearch to give a high search query performance. These are my specs:

ES Setup: Version: 0.90.0, 2 nodes (m3.2xlarge aws intances) in cluster, 32GB RAM each, 50% allocated to ES_HEAP_SIZE, no swapping

Data: 75MM documents, 25 fields each

Queries made for benchmark: Multimatch query against 5 text fields

I've tried everything mentioned here and here

Upto a 30 requests/sec input query frequency, the response time stays less than 1s. Above that 30+ requests/sec, the performance plummets and response time increases to 50s. While this happens JVM Heap is stable (around 7-8 in bigdesk) and GC is also stable. However, the CPU rapidly increases and is at 800% (8-core) and load average is very high 16. The hot threads keep switching between search and scoring functions like BooleanScorer2.nextDoc, BooleanQuery.createWeight, DisjunctionSumScorer.advance, BufferedIndexInput.refill and such

Question: Could you help me find out why performance plummets after 30 req/sec and how to resolve this by changing the cluster configuration if possible.

Thanks in advance!

Answer 1

I know you're seeing CPU-bound behavior, but are you seeing any I/O spikes around when you start having performance issues?

If you're storing your index on EBS volumes I wouldn't be surprised to start seeing I/O saturation with a test like yours. M3 instances have fast local (ephemeral) SSD volumes, and if you're tuning for responsiveness, you should make sure your index is stored locally.

https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing

I realize this doesn't speak directly to the CPU issue, but anything you can do to make a single query more responsive (including filtering/etc...) will boost your throughput.