Netty server hangs not accepting any connection

Question

We've developed a Netty(3.5.11) based IM server that uses our custom protocol.

The following is the order in which handlers are added to the pipeline.

objChannelPipeline.addLast("nettyLoggingHandler", objFrameworkLoggingHandler);
objChannelPipeline.addLast("ipFilter", objCustomIPFilterHandler);
objChannelPipeline.addLast("idleHandler", objIdleStateHandler);
objChannelPipeline.addLast("loggingHandler", objLoggingHandler);
objChannelPipeline.addLast("frameDecoder",objDelimiterBasedFrameDecoder);
objChannelPipeline.addLast("messageDecoder", new CustomProtocolDecoderHandler());
objChannelPipeline.addLast("groupOrder", executionHandler);
objChannelPipeline.addLast("ProtocolMultiplexer", objRegistrationHandler);

The "ProtocolMultiPlexer" handler is replaced with a suitable "ProtocolHandler" after we find the protocol from the register message that is obtained from the clients.

The ipFilterHandler looks into a table in MYSQL database which contains blacklisted IP and decides if the connection from remote IP is to be processed or not.

Issue : After every few random number of days, the server stops processing any messages. We could recreate this issue by performing a load test and killing all the connections to the mysql server.When all the MYSQL processes where killed, all the netty threads except the boss thread seemed to be hung. The server was accepting connection requests but, further processing of messages where not taking place. We thought that the issue was solved when we spotted that we had not added MYSQL "connectTimeout" and "socketTimeout" values. After adding these values and we tried again to repeat our test by killing all MYSQL process under load we did not find any threads entering hung state.

After deploying the server with the above changes in production, we encountered a similar error but this time, even the boss thread along with all other "Netty" threads have entered a hung state. The only thread that is working is a cleaner thread from our DBPool( http://www.snaq.net/java/DBPool/ ). No Netty threads are logging anything and all the threads seemed to be hung. I was not able to get thread dump. Any help is appreciable

Thanks

Answer 1

Netty handlers are not supposed to perform long blocking operations like reading from a database. Do it from a separate thread, as it is proposed in http://netty.io/4.0/guide/#faq.4