Read time out in Apache HTTP Client but request runs fine with curl

Question

. 。 Answers to this question are eligible for a +50 reputation bounty. TomStroemer wants to to this question. 。

• I have a PUT requests that is done by a dependency scanner we are using
• I'm behind a corporate proxy
• Request is done with Apache HTTP Client 4
• I get a "read time out" after the configured timeout
• I recreated the request with curl and this works fine (with the same proxy) and finishes within seconds
• It works fine with Apache HTTP Client without the proxy.

So I'm trying to find out what config is missing in the HTTP client config or what else could be the error. Unfortunately neither our networking team nor the product vendor could help me out:-(

This is the curl request:

curl --connect-timeout 60 -m 60 -v -X PUT --data-binary @test.json 
https://someblackduckserver/api/developer-scans/12345 
-H "Content-type: application/vnd.blackducksoftware.developer-scan-1-ld-2+json" 
-H "X-BD-DOCUMENT-COUNT: 1" 
-H "Accept: application/vnd.blackducksoftware.scan-4+json" 
-H "X-BD-MODE: append" 
-H "X-BD-RAPID-SCAN-MODE: ALL" -H "X-BD-VERSION-NAME: scan-test" 
-H "User-Agent: synopsys_detect/8.1.0-SNAPSHOT BlackDuckCommon/65.0.0 ..." 
-H "X-BD-PROJECT-NAME: scan-test" 
-H "Authorization: Bearer ..... left out ....."

Where test.json is a json file with about 1.3MB.

Minified code when using Apache HTTP Client:

RequestBuilder requestBuilder = RequestBuilder.create("PUT");
URIBuilder uriBuilder = new URIBuilder(new URI("https://someblackduckserver/api/developer-scans/12345"));
requestBuilder.setUri(uriBuilder.build());

Charset bodyEncoding = StandardCharsets.UTF_8;
requestBuilder.setCharset(bodyEncoding);

requestBuilder.addHeader("Content-type", "application/vnd.blackducksoftware.developer-scan-1-ld-2+json");
requestBuilder.addHeader("X-BD-DOCUMENT-COUNT", "1");
requestBuilder.addHeader("Accept", "application/vnd.blackducksoftware.scan-4+json");
requestBuilder.addHeader("X-BD-MODE", "append");
requestBuilder.addHeader("X-BD-RAPID-SCAN-MODE", "ALL");
requestBuilder.addHeader("X-BD-VERSION-NAME", "scan-test");
requestBuilder.addHeader("User-Agent", "synopsys_detect/8.1.0-SNAPSHOT BlackDuckCommon/65.0.0 ...");
requestBuilder.addHeader("X-BD-PROJECT-NAME", "scan-test");
requestBuilder.addHeader("Authorization", "Bearer ..... left out .....");

HttpEntity httpEntity = new FileEntity(new File("test.json"));
requestBuilder.setEntity(httpEntity);

HttpUriRequest request = requestBuilder.build();

HttpContext httpContext = new BasicHttpContext();

HttpClientBuilder clientBuilder = HttpClientBuilder.create();

RequestConfig.Builder defaultRequestConfigBuilder = RequestConfig.custom().setCookieSpec(CookieSpecs.STANDARD);
// set our proxy
defaultRequestConfigBuilder.setProxy(new HttpHost("..... (left out)", 8080));

// set timeout
int timeoutInSeconds = 60;
defaultRequestConfigBuilder.setConnectTimeout(timeoutInSeconds * 1000);
defaultRequestConfigBuilder.setSocketTimeout(timeoutInSeconds * 1000);
defaultRequestConfigBuilder.setConnectionRequestTimeout(timeoutInSeconds * 1000);

clientBuilder.setDefaultRequestConfig(defaultRequestConfigBuilder.build());

// for testing: ignore ssl
HostnameVerifier hostnameVerifier;
SSLContext sslContext = SSLContextBuilder.create().loadTrustMaterial(new TrustAllStrategy()).build();
hostnameVerifier = new NoopHostnameVerifier();
SSLConnectionSocketFactory connectionFactory = new SSLConnectionSocketFactory(sslContext, hostnameVerifier);
clientBuilder.setSSLSocketFactory(connectionFactory);

try (CloseableHttpClient client = clientBuilder.build()) {

    CloseableHttpResponse closeableHttpResponse = client.execute(request, httpContext);

    System.out.println(closeableHttpResponse);
}

I removed some sensitive details from the request.

Any idea what I'm missing here or what could go wrong?

Logs from a sample the call:

2022-07-25 10:38:38.201 [main] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default
2022-07-25 10:38:38.206 [main] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
2022-07-25 10:38:38.207 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {tls}->http://our.proxy.ip.address:8080->https://someblackduckserver.com:443][total available: 0; route allocated: 0 of 2; total allocated: 0 of 20]
2022-07-25 10:38:38.259 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {tls}->http://our.proxy.ip.address:8080->https://someblackduckserver.com:443][total available: 0; route allocated: 1 of 2; total allocated: 1 of 20]
2022-07-25 10:38:38.260 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Opening connection {tls}->http://our.proxy.ip.address:8080->https://someblackduckserver.com:443
2022-07-25 10:38:38.261 [main] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connecting to /our.proxy.ip.address:8080
2022-07-25 10:38:38.301 [main] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connection established 10.17.10.44:34272<->our.proxy.ip.address:8080
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> CONNECT someblackduckserver.com:443 HTTP/1.1
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: someblackduckserver.com
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.13 (Java/11.0.13)
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "CONNECT someblackduckserver.com:443 HTTP/1.1[\r][\n]"
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: someblackduckserver.com[\r][\n]"
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.13 (Java/11.0.13)[\r][\n]"
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
2022-07-25 10:38:38.346 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 200 Connection established[\r][\n]"
2022-07-25 10:38:38.346 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Proxy-Agent: Proxy-Vendor-Proxy/1.0[\r][\n]"
2022-07-25 10:38:38.346 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"
2022-07-25 10:38:38.347 [main] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 200 Connection established
2022-07-25 10:38:38.347 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Proxy-Agent: Proxy-Vendor-Proxy/1.0
2022-07-25 10:38:38.347 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Tunnel to target created.
2022-07-25 10:38:38.432 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Enabled protocols: [TLSv1.3, TLSv1.2]
2022-07-25 10:38:38.440 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Enabled cipher suites:[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2022-07-25 10:38:38.440 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Starting handshake
2022-07-25 10:38:39.252 [main] DEBUG jdk.event.security - X509Certificate: ... our ssl inspect certificate ...
2022-07-25 10:38:39.252 [main] DEBUG jdk.event.security - X509Certificate: ... our internal certificate ...
2022-07-25 10:38:39.275 [main] DEBUG jdk.event.security -  TLSHandshake: someblackduckserver.com:8080, TLSv1.2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, 1942523025
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Secure session established
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory -  negotiated protocol: TLSv1.2
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory -  negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory -  peer principal: CN=*.someblackduckserver.com, O="Synopsys, Inc.", L=Mountain View, ST=California, C=US
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory -  peer alternative names: [*.someblackduckserver.com]
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory -  issuer principal: ... our ssl inspect certificate ...
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 60000
2022-07-25 10:38:39.277 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Executing request PUT /api/developer-scans/12345 HTTP/1.1
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> PUT /api/developer-scans/12345 HTTP/1.1
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-type: application/vnd.blackducksoftware.developer-scan-1-ld-2+json
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> X-BD-DOCUMENT-COUNT: 1
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept: application/vnd.blackducksoftware.scan-4+json
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> X-BD-MODE: append
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> X-BD-RAPID-SCAN-MODE: ALL
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> X-BD-VERSION-NAME: rapid-scan-test-igl-2
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: synopsys_detect/8.1.0-SNAPSHOT BlackDuckCommon/65.0.0 (Eclipse Foundation 11.0.12 amd64 Windows 10 10.0)
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> X-BD-PROJECT-NAME: project_name
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Authorization: Bearer ...
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Length: 287381
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: someblackduckserver.com
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "PUT /api/developer-scans/12345 HTTP/1.1[\r][\n]"
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-type: application/vnd.blackducksoftware.developer-scan-1-ld-2+json[\r][\n]"
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-BD-DOCUMENT-COUNT: 1[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept: application/vnd.blackducksoftware.scan-4+json[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-BD-MODE: append[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-BD-RAPID-SCAN-MODE: ALL[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-BD-VERSION-NAME: rapid-scan-test-igl-2[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: synopsys_detect/8.1.0-SNAPSHOT BlackDuckCommon/65.0.0 (Eclipse Foundation 11.0.12 amd64 Windows 10 10.0)[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-BD-PROJECT-NAME: project_name[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Authorization: Bearer ...[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Length: 287381[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: someblackduckserver.com[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"

2022-07-25 10:38:39.423 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> .... all the logs for file content ....

2022-07-25 10:39:39.483 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[read] I/O error: Read timed out"
2022-07-25 10:39:39.483 [main] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: Close connection
2022-07-25 10:39:39.483 [main] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: Shutdown connection
2022-07-25 10:39:39.483 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Connection discarded
2022-07-25 10:39:39.483 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {tls}->http://our.proxy.ip.address:8080->https://someblackduckserver.com:443][total available: 0; route allocated: 0 of 2; total allocated: 0 of 20]
2022-07-25 10:39:39.484 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection manager is shutting down
2022-07-25 10:39:39.484 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection manager shut down
Exception in thread "main" java.net.SocketTimeoutException: Read timed out
    at java.base/java.net.SocketInputStream.socketRead0(Native Method)
    at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
    at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
    at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
    at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
    at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1318)
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:959)
    at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:84)
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.example.BdTest.main(BdTest.java:93)

Answer 1

May be not a solution but a workaround if it works. I wrote my own Http client that is much simpler than Apache Http client. The reason I wrote it is that in many cases you just need a simplistic functionality so using a thin library lessens complexity and allows you to debug it if need be and it would allow you easier way to get to the bottom of the problem. So, may be you can give it a shot. At may even work or if not than it would be much easier to debug than Apache Http client. My Http client is part of MgntUtils Open-Source library written and maintained by me. Here is HttpClient Javadoc. The library can be obtained as Maven artifact from Maven Central and from Github (including source code and Javadoc)