CXF, Camel and Spring thread blocking issue

Question

Issue is with Camel and CXF (endpoints and client) and multiple requests firing at once.

Here's the exception we are getting:

java.net.SocketException: Unexpected end of file from server
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
    at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:266)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1543)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1513)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1318)
    at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
    at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:632)
    at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
    at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:570)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:479)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:382)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:335)
    at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:135)

I have tried hitting the endpoint directly with SoapUI load tests and can not reproduce the problem. It only happens going through the spring jaxws:client bean.

One request at a time from the client works fine and we never see the error, only when we try to do multiple.

Also performed a thread dump while the threads are waiting, and all of the requests (5 of them) look like the following:

http-listener-1(1)" daemon prio=5 tid=0x00007fa2843e8800 nid=0x9d07 in Object.wait() [0x000000018a266000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000001708b27e8> (a org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream)
    at org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.getHttpResponse(AsyncHTTPConduit.java:562)
    - locked <0x00000001708b27e8> (a org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream)
    at org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.getResponseCode(AsyncHTTPConduit.java:674)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1543)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1513)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1318)
    at org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.close(AsyncHTTPConduit.java:383)
    at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
    at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:632)
    at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
    - locked <0x00000001704175f8> (a org.apache.cxf.phase.PhaseInterceptorChain)
    at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:570)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:479)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:382)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:335)
    at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:135)

I also tried using the ASync client, while the stacktrace and thread dump looked different, the result was the same.

This also seems to be happening only on our dev instances (Mac laptops) and we are running this inside Glassfish 4.0 with JDK 1.7.x (1.7u65 on my machine, slightly different version on others)

One of the camel routes, although it seems to affect all of them (and they are setup the sameway)

from("cxf:/structure?serviceClass=" + StructureEndpoint.class.getName() + "&loggingFeatureEnabled=true")
            .routeId("structure")
            .to("log:com.test.camel")
            .recipientList(simple("direct:structure:${header.operationName}"));

When this happens I never see the camel/cxf log output enabled by loggingFeatureEnabled, and so we don't see our .to("log:*") output either. It just never gets here.

Reproducable Test:

SoapUI, simple load test. Start/end threads at 5 Start test, get errors for first 5 requests. The test goes through our frontend, which uses the cxf client to hit the backend camel/cxf services.

Hitting camel/cxf services directly does not cause any issues.

Envirnoment Test Update

We have 2 other environments running on Linux, both on JDK 1.7.0u60, same version of glassfish that do not exhibit this behavior. All machines affected are OS X only. (will be testing a windows box shortly)

Answer 1

After some isolation and testing of a smaller sample that I was able to test with, found the problem.

Testing with the smaller app I was able to deploy to do different servers locally and everything worked fine. Even tried deploying the services to Tomcat and the frontend on Glassfish, still no problems.

As soon as I put front and back on the same server, it gets stuck if I sent it 5 requests at the same time. I also tried using JAX-WS generated (instead of CXF) services, but they had the same problem, ruling out issues with CXF specific issue.

Root Problem

Default http thread pool for Glassfish maxes at 5. So, as soon as we tried doing 5 or more requests at the same time....blamo! And technically if we tried 5 requests, we were really doing 10. 5 to the resources, which would then go out and do 5 to the backend themselves.

Solution

A couple things will solve this. Increase thread pool size in glassfish. 5 seems small anyways, so that will need tuning no matter what.

Another is to simply have the two applications on different servers, which is going to be how production is configured anyways, but probably adds to much overhead for local development.

Outstanding Questions

The only left unknown, is why we do not see this on our linux qa/dev environments. The difference with those is they have apache sitting in front of them, where our local instances do not.