并发多线程请求的 Java OKHTTP SocketTimeoutException

Question

My Java application uses a variety of proxy servers to collect data from a particular domain. The special application requires the following procedure:

• Loading a URL through a specific proxy
• Wait 5 seconds
• Loading the next url through the same proxy

So that the loading of the information (due to the 5 second pause) doesn't take forever, I work with a total of 400 threads. Each of these threads uses its own proxy server, ie also with its own OKHTTP client:

MyHTTPClient = new OkHttpClient.Builder()
.connectTimeout(7, TimeUnit.SECONDS)
.writeTimeout(7, TimeUnit.SECONDS)
.readTimeout(7, TimeUnit.SECONDS)
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ProxyIP, ProxyPort)))
.proxyAuthenticator((Route route, Response response) - > {
    return response.request().newBuilder().header("Proxy-Authorization", Credentials.basic(ProxyUser, ProxyPass)).build();
})
.sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0])
.hostnameVerifier((String hostname, SSLSession session) - > true)
.addNetworkInterceptor(new LoggingInterceptor())
.retryOnConnectionFailure(true)
.build();

Each thread has to use its own proxy, so each thread has its own OKHTTP client. In total there are 400 OKHTTP clients.

Each thread now processes its queries as follows:

 while (true) {
     MyAnswer = MyHTTPClient.newCall(
         new Request.Builder().url(https://www.example.com)
         .addHeader("referer", SomeReferrer)
         .addHeader("cache-control", "no-cache")
         .addHeader("pragma", "no-cache")
         .get().build())
     .execute();

     Body = MyAnswer.body().string();
     MyAnswer.body().close();

     Thread.sleep(5000);
 }

And that works great too, for about a minute. Then suddenly the following error messages appear for each individual request on every single thread:

java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.waitForConnect(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:107)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.base/java.net.Socket.connect(Socket.java:608)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:120)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
at okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.kt:261)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:201)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at okio.InputStreamSource.read(JvmOkio.kt:90)
at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:129)
... 22 more

---

WHAT I TRIED SO FAR

I've done a bit of research and found out that there might be some kind of limit in terms of concurrent connections. I found out, that whenever this error occurs, there are no idle connections in the connection pool:

MyHTTPClient.connectionPool().idleConnectionCount() //Always 0 when the Timeout-Exception occurs

So i tried to increase the size of the connection pool for every single okhttpclient but that seemed to have absolutely no effect:

.connectionPool(new ConnectionPool(5000, 100, TimeUnit.MILLISECONDS))

I also tried to increase and decrease the timeout-values of the httpclient and that actually seemed to make a difference in terms of how long the application would run without this error but it actually didn't prevent this to happen.

I also tried to increase the amount of simultaneous connections and decrease the amount of time a connection stays in "TIME_WAIT" status as mentioned here: https://docs.microsoft.com/en-us/answers/questions/482793/tcpip-cuncurrent-connections.html

I also tried to add a Dispatcher to the OKHTTPClient, as follows:

Dispatcher MyDispatcher = new Dispatcher(Executors.newCachedThreadPool());
MyDispatcher.setMaxRequests(9999999);
MyDispatcher.setMaxRequestsPerHost(9999999);

---

If you have any ideas about what could be the cause or what else I could try, please let me know. I am also happy to send you further information, if needed.

Thank you in advance!

Answer 1

Create one OkHttpClient initially and then use OkHttpClient.newBuilder() to customize the proxy. This will cause the clients to share thread pools and connection pools. That might not fix your problem on its own, but it'll make it more efficient.

  final OkHttpClient sharedHttpClient = new OkHttpClient.Builder()
    .connectTimeout(7, TimeUnit.SECONDS)
    .writeTimeout(7, TimeUnit.SECONDS)
    .readTimeout(7, TimeUnit.SECONDS)
    .sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0])
    .hostnameVerifier((String hostname, SSLSession session) - > true)
    .addNetworkInterceptor(new LoggingInterceptor())
    .retryOnConnectionFailure(true)
    .build();

MyHTTPClient = sharedHttpClient.newBuilder()
    .proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ProxyIP, ProxyPort)))
    .proxyAuthenticator((Route route, Response response) - > {
    return response.request().newBuilder().header("Proxy-Authorization", Credentials.basic(ProxyUser, ProxyPass)).build();
})
    .build();

Answer 2

I have met a similar situation last day. Here are my ideas about these problem:

1. The first step is to find out which kind of timeout you have occurred. Since there are several types of timeout scene in httpclient. Such as:
   • timeout to fetch a connection from connectionPool
   • timeout to establish a socket connection
   • timeout to do wite and read operation

I have a little trick to accomplish this goal: Change the settings of timeout in OkHttpClient.Builder() , let different kind of timeout vary from each one others. And then print the actual timeout duration in log, so you can find out which one it truely match.

2. Check if there is any abnormal tcp connection. Since you have use connection pool in your program, it may not mantains too much connections at the same time. You can check the number of connection by:

# each state of connection
netstat -n| awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
# connection to specifed port
netstat -ntap|grep :PORT

3. Check the response time in server side. Depends on your web container, you can see if there is any request take too long to return. But it may not be your answer, since you have confirm another computor doesn't have the same issue.

4. Check the network itself. Use this script to check if any package take too long( -w param specified the timeout duration) to return:

for i in {1..10000}; do nc -vnz IP PORT -w 1 2>&1 |grep "timed out"; done

That's all, hope that may help you.