TThreadPoolServer Clients org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe

Question

I am encountering an issue when the thrift client is tested with high load.

Setup: Thrift Server is a TThreadPoolServer. It performs and operation which takes only 1 -5 ms.Thrift Client connects to the server with a new connection each time, calls the operation once and closes the connection.

When the load is at 160 TPS, the client starts to throw the following exception

org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe  at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) ~[libthrift-0.9.1.jar:0.9.1]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65) ~[libthrift-0.9.1.jar:0.9.1]
at com.ebay.traffic.email.delstats.thrift.interfaces.GSCassandreServices$Client.send_insertRow(GSCassandreServices.java:84) ~[GSCassandreServices$Client.class:na]
at com.ebay.traffic.email.delstats.thrift.interfaces.GSCassandreServices$Client.insertRow(GSCassandreServices.java:75) ~[GSCassandreServices$Client.class:na]
at com.ebay.traffic.email.delstats.thrift.impl.client.SimpleThriftClient.insertRowViaCQL(SimpleThriftClient.java:90) ~[SimpleThriftClient.class:na]
at com.ebay.traffic.email.delstats.entity.EventsEntityQueryHandlerImpl.insertEvents(EventsEntityQueryHandlerImpl.java:97) [EventsEntityQueryHandlerImpl.class:na]
at com.ebay.app.raptor.delstats.resources.StatsServiceResource.registerEvents(StatsServiceResource.java:306) [StatsServiceResource.class:na]
at sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0]
at java.lang.reflect.Method.invoke(Method.java:602) ~[na:2.6 (08-23-2013)]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [jersey-server-1.8.jar:1.8]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [jersey-server-1.8.jar:1.8]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [jersey-server-1.8.jar:1.8]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) [jersey-server-1.8.jar:1.8]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.8.jar:1.8]
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [jersey-server-1.8.jar:1.8]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.8.jar:1.8]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [jersey-server-1.8.jar:1.8]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) [jersey-server-1.8.jar:1.8]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) [jersey-server-1.8.jar:1.8]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) [jersey-server-1.8.jar:1.8]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) [jersey-server-1.8.jar:1.8]

Please note that connections succeed mostly, but I lose around 10% total connections due to this error

I checked netstat on the server and the connections are getting closed for every single request(Not many open connections at a time) and also there are no exceptions/errors on the server at all. Is this a limitation on the thrift server?

Thanks, Gopi

Answer 1

My load opened lot of connections on the server which was more that desired. I could see more than 5000 sockets opened in the server side at a moment. As EJP suggested, the best way to do deal with this is to use a thread pooled client which worked from me. This restricts the connections to the server and the server was able to accept all the connections.

Ways to reproduce the issue:

• Have a Thrift interface which does nothing
• Start TThreadPoolServer
• Run a simple Java client with 100 threads and each of this client has a for loop running indefinitely - The For Loop should create a new socket, call the thrift interface and then close the socket connection(before the end of loop)