Hazelcast Out of memory issue

Question

Frequent out of memory issue in Hazelcast 3.3 , What I wonder is node has enough memory and CPU, 16GB Ram, 8 Core processor, still this error is happening. Its a single node setup.

com.hazelcast.core.HazelcastException: Client is shutting down!!!
    at com.hazelcast.client.connection.nio.ClientConnection.innerClose(ClientConnection.java:302)
    at com.hazelcast.client.connection.nio.ClientConnection.close(ClientConnection.java:355)
    at com.hazelcast.client.connection.nio.ClientConnection.close(ClientConnection.java:230)
    at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.shutdown(ClientConnectionManagerImpl.java:249)
    at com.hazelcast.client.ClientOutOfMemoryHandler$ClientHelper.closeSockets(ClientOutOfMemoryHandler.java:54)
    at com.hazelcast.client.ClientOutOfMemoryHandler$ClientHelper.cleanResources(ClientOutOfMemoryHandler.java:45)
    at com.hazelcast.client.ClientOutOfMemoryHandler.onOutOfMemory(ClientOutOfMemoryHandler.java:34)
    at com.hazelcast.instance.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:143)
    at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handleOperationError(BasicOperationService.java:827)
    at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:703)
    at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$400(BasicOperationService.java:675)
    at com.hazelcast.spi.impl.BasicOperationService.runOperationOnCallingThread(BasicOperationService.java:229)
    at com.hazelcast.spi.impl.BasicInvocation.doInvokeLocal(BasicInvocation.java:286)
    at com.hazelcast.spi.impl.BasicInvocation.doInvoke(BasicInvocation.java:272)
    at com.hazelcast.spi.impl.BasicInvocation.invoke(BasicInvocation.java:232)
    at com.hazelcast.spi.impl.BasicInvocationBuilder.invoke(BasicInvocationBuilder.java:45)
    at com.hazelcast.client.impl.client.PartitionClientRequest.process(PartitionClientRequest.java:60)
    at com.hazelcast.client.impl.ClientEngineImpl$ClientPacketProcessor.processRequest(ClientEngineImpl.java:418)
    at com.hazelcast.client.impl.ClientEngineImpl$ClientPacketProcessor.run(ClientEngineImpl.java:357)
    at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:546)
    at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:439)
    at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:433)
    at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:408)
    at ------ End remote and begin local stack-trace ------.(Unknown Source)
    at com.hazelcast.client.spi.impl.ClientCallFuture.resolveResponse(ClientCallFuture.java:192)
    at com.hazelcast.client.spi.impl.ClientCallFuture.get(ClientCallFuture.java:134)
    at com.hazelcast.client.spi.impl.ClientCallFuture.get(ClientCallFuture.java:111)
    at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:110)
    at com.hazelcast.client.proxy.ClientMapProxy.get(ClientMapProxy.java:180)

For Every new run, this happens after (around) 15 mins of run.

After this, continuously We are getting the following error,

java.util.concurrent.RejectedExecutionException: Internal executor rejected task: java.util.concurrent.FutureTask@5463e258, because client is shutting down...
    at com.hazelcast.client.spi.impl.ClientExecutionServiceImpl$1.rejectedExecution(ClientExecutionServiceImpl.java:60)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
    at com.hazelcast.client.spi.impl.ClientExecutionServiceImpl.submitInternal(ClientExecutionServiceImpl.java:81)
    at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.getOrConnect(ClientConnectionManagerImpl.java:339)
    at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.tryToConnect(ClientConnectionManagerImpl.java:280)
    at com.hazelcast.client.spi.impl.ClientInvocationServiceImpl.send(ClientInvocationServiceImpl.java:120)
    at com.hazelcast.client.spi.impl.ClientInvocationServiceImpl.invokeOnRandomTarget(ClientInvocationServiceImpl.java:61)
    at com.hazelcast.client.spi.ProxyManager.initialize(ProxyManager.java:197)
    at com.hazelcast.client.spi.ProxyManager.getOrCreateProxy(ProxyManager.java:180)
    at com.hazelcast.client.HazelcastClient.getDistributedObject(HazelcastClient.java:461)
    at com.hazelcast.client.HazelcastClient.getMap(HazelcastClient.java:314)
    at com.hazelcast.client.HazelcastClientProxy.getMap(HazelcastClientProxy.java:97)

Please, If there any configuration set up I need to add???

For INFO :

There is no eviction policy added, Its default as NONE. Am using the MAP STORE for persistence policy with write delay as 1 second.

Answer 1

Since you don't have a support contract, this is a best value answer.

1. You have a single node setup, as wrong as a single node cluster sounds, as wrong it is to run Hazelcast that way. Hazelcast is not designed for single member clusters (and it doesn't make sense either).

2. The amount of heap is set to 16GB. What is your expected data value? I mean you cannot store 16GB of data in 16GB heap.

3. Have you tried to get a heap-dump when your JVM ran OOME? Have you looked into it to see what is eating up your memory? (as mentioned look for "how to find memory leaks")

4. If you have no eviction selected, you won't get eviction, even with a MapStore.

5. Have you tried different GC combinations or tried to look at the GC at all? There are different kinds of GC errors.

6. It seems like your client is running OOME so maybe the problem is not the amount of data at all. But since there is no further information, it is hard to reason.