Elasticsearch Java using very high CPU and MEMORY

Question

Elasticsearch suddenly stopped working due high CPU usage, and now when I restart it, it keeps using around 100% CPU and 58% Memory it doesn't drop down. There is around 1.300.000 data linked to Elasticsearch. Using Linux server Ubuntu 15.04

default/elasticsearch

ES_HEAP_SIZE=2g (half of my memeory)
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited

limits.conf

elasticsearch - nofile 65535
elasticsearch - memlock unlimited

elasticsearch.yml

bootstrap.mlockall: true

curl http://localhost:9200/_nodes/hot_threads?pretty=true

::: {elasticnode}{wrdxzFcXT0aTXnp4R5KFbA}{127.0.0.1}{localhost/127.0.0.1:9300} Hot threads at 2016-03-01T10:24:09.884Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 93.7% (468.4ms out of 500ms) cpu usage by thread 'elasticsearch[elasticnode][clusterService#updateTask][T#1]' 10/10 snapshots sharing following 29 elements sun.nio.fs.UnixNativeDispatcher.readdir(Native Method) sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.readNextEntry(UnixDirectoryStream.java :168) sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.hasNext(UnixDirectoryStream.java:201) org.elasticsearch.index.translog.Translog$OnCloseRunnable.handle(Translog.java:726) org.elasticsearch.index.translog.Translog$OnCloseRunnable.handle(Translog.java:713) org.elasticsearch.index.translog.ChannelReference.closeInternal(ChannelReference.java:67) org.elasticsearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java: 64) org.elasticsearch.index.translog.TranslogReader.close(TranslogReader .java:143) org.apache.lucene.util.IOUtils.close(IOUtils.java:97) org.elasticsearch.index.translog.Translog.close(Translog.java:425) org.apache.lucene.util.IOUtils.close(IOUtils.java:97) org.apache.lucene.util.IOUtils.close(IOUtils.java:84) org.elasticsearch.index.engine.InternalEngine.closeNoLock(InternalEngine.java:990) org.elasticsearch.index.engine.Engine.close(Engine.java:1147) org.apache.lucene.util.IOUtils.close(IOUtils.java:97) org.apache.lucene.util.IOUtils.close(IOUtils.java:84) org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:837) org.elasticsearch.index.IndexService.closeShardInjector(IndexService.java:443) org.elasticsearch.index.IndexService.removeShard(IndexService.java:416) org.elasticsearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClus terStateService.java:743) org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterSt ateService.java:364) org.elasticsearch.indices.cluster.IndicesClusterStateServ ice.clusterChanged(IndicesClusterS tateService.java:164) org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalCluste rService.java:600) org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterServ ice.java:762) org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPriorit izedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231) org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPriorit izedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745)

0.1% (263.4micros out of 500ms) cpu usage by thread 'elasticsearch[elasticnode][transport_clie
nt_timer][T#1]{Hashed wheel timer #1}' 10/10 snapshots sharing following 5 elements java.lang.Thread.sleep(Native Method)

org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445) org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) java.lang.Thread.run(Thread.java:745)

Query example. This is the code in elasticsearch.php :

public function advancedquery($type, $query) {
    return $this->call($type . '/_search', 'POST', $query);
}

Any idea what is this about? I can provide any additional information as necessary.

Answer 1

Based on the commentary. You have a cronjob running every 10 minutes that has node.js running a file in JavaScript . Based on your question, you have the JS file pinging a PHP file on the same server. On the same server you have a Java core running your posix stuff.

The answer to your CPU overload question is:

Choose one system. You have three environments running on the same instance. Java and PHP run on Apache , Node.JS on NGINX . And you shouldn't even cross Java and PHP. That's the problem. It's like running AutoCad, Final Cut Pro, and Unreal Game Engine all at the same time... doesn't work like that.

Condense you system into one language of your choice, supported by the AWS SDK's of course.