Boot2Docker Hazelcast and ulimit

Question

I try to create a development environment for the software we develop. For this I use Docker and Boot2docker because I work on mac (Yosemite). We use Hazelcast in the product, but I am having a problem when I "dockerize" it. At the first connection to Hazelcast here's what happens:

INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Hazelcast Community Edition 3.2.2 (20140527) starting at Address[172.17.0.2]:5601
déc. 24, 2014 2:11:27 PM com.hazelcast.system
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Copyright (C) 2008-2014 Hazelcast.com
déc. 24, 2014 2:11:27 PM com.hazelcast.core.LifecycleService
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Address[172.17.0.2]:5601 is STARTING
déc. 24, 2014 2:11:27 PM com.hazelcast.instance.Node
AVERTISSEMENT: [172.17.0.2]:5601 [akio] [3.2.2] No join method is enabled! Starting standalone.
déc. 24, 2014 2:11:27 PM com.hazelcast.core.LifecycleService
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Address[172.17.0.2]:5601 is STARTED
déc. 24, 2014 2:12:30 PM com.hazelcast.nio.SocketAcceptor
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Accepting socket connection from /127.0.0.1:55736
déc. 24, 2014 2:12:30 PM com.hazelcast.nio.TcpIpConnectionManager
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] 5601 accepted socket connection from /127.0.0.1:55736
déc. 24, 2014 2:12:30 PM com.hazelcast.client.AuthenticationRequest
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Received auth from Connection [/127.0.0.1:55736 -> null] live=true, client=true, type=JAVA_CLIENT, successfully authenticated
déc. 24, 2014 2:12:30 PM com.hazelcast.nio.SocketAcceptor
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Accepting socket connection from /172.17.0.2:57088
déc. 24, 2014 2:12:30 PM com.hazelcast.nio.TcpIpConnectionManager
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] 5601 accepted socket connection from /172.17.0.2:57088
déc. 24, 2014 2:12:30 PM com.hazelcast.client.AuthenticationRequest
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Received auth from Connection [/172.17.0.2:57088 -> null] live=true, client=true, type=JAVA_CLIENT, successfully authenticated
déc. 24, 2014 2:12:31 PM com.hazelcast.partition.InternalPartitionService
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Initializing cluster partition table first arrangement...
déc. 24, 2014 2:13:42 PM com.hazelcast.nio.TcpIpConnection
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Connection [Address[172.17.0.2]:57088] lost. Reason: Socket explicitly closed
déc. 24, 2014 2:13:42 PM com.hazelcast.nio.TcpIpConnection
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Connection [Address[127.0.0.1]:55736] lost. Reason: Socket explicitly closed
déc. 24, 2014 2:13:42 PM com.hazelcast.initializer
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Destroying node initializer.
déc. 24, 2014 2:13:42 PM com.hazelcast.instance.Node
INFOS: [172.17.0.2]:5601 [akio] [3.2.2] Hazelcast Shutdown is completed in 111 ms.
java.lang.OutOfMemoryError: unable to create new native thread
déc. 24, 2014 2:13:42 PM com.hazelcast.concurrent.lock.operations.LockOperation
GRAVE: [172.17.0.2]:5601 [akio] [3.2.2] unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
    at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.runAsynchronous(BasicInvocation.java:617)
    at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.set(BasicInvocation.java:668)
    at com.hazelcast.spi.impl.BasicInvocation.notify(BasicInvocation.java:303)
    at com.hazelcast.spi.impl.BasicPartitionInvocation.notify(BasicPartitionInvocation.java:28)
    at com.hazelcast.spi.impl.ResponseHandlerFactory$LocalInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:144)
    at com.hazelcast.spi.impl.BasicOperationService.processOperation(BasicOperationService.java:385)
    at com.hazelcast.spi.impl.BasicOperationService.access$300(BasicOperationService.java:102)
    at com.hazelcast.spi.impl.BasicOperationService$BasicOperationProcessorImpl.process(BasicOperationService.java:754)
    at com.hazelcast.spi.impl.BasicOperationScheduler$PartitionThread.process(BasicOperationScheduler.java:276)
    at com.hazelcast.spi.impl.BasicOperationScheduler$PartitionThread.doRun(BasicOperationScheduler.java:270)
    at com.hazelcast.spi.impl.BasicOperationScheduler$PartitionThread.run(BasicOperationScheduler.java:245)

déc. 24, 2014 2:13:42 PM com.hazelcast.spi.OperationService
AVERTISSEMENT: [172.17.0.2]:5601 [akio] [3.2.2] While sending op error... op: com.hazelcast.concurrent.lock.operations.LockOperation@4ff5ed2a, error: java.lang.OutOfMemoryError: unable to create new native thread
com.hazelcast.spi.exception.ResponseAlreadySentException: NormalResponse already sent for callback: BasicInvocation{ serviceName='hz:impl:lockService', op=com.hazelcast.concurrent.lock.operations.LockOperation@4ff5ed2a, partitionId=50, replicaIndex=0, tryCount=100, tryPauseMillis=500, invokeCount=1, callTimeout=10000, target=Address[172.17.0.2]:5601}, current-response: : com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
    at com.hazelcast.spi.impl.ResponseHandlerFactory$LocalInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:141)
    at com.hazelcast.spi.impl.BasicOperationService.handleOperationError(BasicOperationService.java:531)
    at com.hazelcast.spi.impl.BasicOperationService.processOperation(BasicOperationService.java:404)
    at com.hazelcast.spi.impl.BasicOperationService.access$300(BasicOperationService.java:102)
    at com.hazelcast.spi.impl.BasicOperationService$BasicOperationProcessorImpl.process(BasicOperationService.java:754)
    at com.hazelcast.spi.impl.BasicOperationScheduler$PartitionThread.process(BasicOperationScheduler.java:276)
    at com.hazelcast.spi.impl.BasicOperationScheduler$PartitionThread.doRun(BasicOperationScheduler.java:270)
    at com.hazelcast.spi.impl.BasicOperationScheduler$PartitionThread.run(BasicOperationScheduler.java:245)

After searching on google this error can be caused due to a boundary process the user can control. On after my research, should be used "ulimit -u" on the host (Boot2Docker), but it does not offer the option.

Does someone already encountered the problem? If so, you're able to get around it?

PS: Docker host = final CentOS 6.6 java version "1.7.0_60" Hazelcast = 3.2.2

Answer 1

I'm not sure what does not offer the option "ulimit -u" but normally you run this on linux shell. You have to configure the soft and hard limit for open files inside the docker container and probably also on the host machine. Never used docker myself but this is a common linux problem, nothing docker not Hazelcast specific.