ElasticSearch消耗100％CPU

Question

After some period of time (sometimes minutes, sometimes days), my app start consuming 100% CPU. As I see from VisualVM, it's always happening at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink class.

And also, as I see from logs, it throws following exception:

NioClientSocketPipelineSink:internalWarn:105 - Unexpected exception in the selector loop.
java.nio.channels.CancelledKeyException
    at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
    at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:69)
    at java.nio.channels.SelectionKey.isConnectable(SelectionKey.java:318)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:369)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Btw, i'm not sure where is a problem.

Is it ElasticSearch issue? Or embedded Netty? Or my server/app configuration?

I'm using Ubuntu 64 11.10, Sun Java 1.6.0_26-b03 and Elastic Search 0.18.5

---

Update

Seems that it was because I've tried to open and close connection for every request to ElasticSearch. And it was, i guess, some kind of concurrency issue.

Fixed by opening a connection to Elastic Search just once, on startup.

Answer 1

Can you check with latest netty version (3.2.7.Final) and see if it fix it ? I think we fixed a bug there which could cause an infinity loop in the selector and so consume 100 % cpu.