Connection reset - Java (underlying socket status remain established) Azure VM

Question

I am debugging one problem of connection reset and need some help.

Here is the background

Using java version 8, apache httpClient 4.5.2

I have a following program, which runs successfully on windows 10, 7 but end up with connection reset on Azure windows server 2016 VM.

import java.io.IOException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import org.apache.commons.codec.binary.Base64;
import org.apache.http.Header;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.util.EntityUtils;





public class TestConnectionReset
{
  static PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager();
  static  {
    connManager.setMaxTotal(10);
    connManager.setDefaultMaxPerRoute(2);
  }
  public static void main(String[] args) throws ClientProtocolException, IOException, InterruptedException {
    while (true) {
      HttpClientBuilder clientBuilder = HttpClientBuilder.create();
      RequestConfig config = RequestConfig.custom().setConnectTimeout(1800000).setConnectionRequestTimeout(1800000)
        .setSocketTimeout(1800000).build();
      clientBuilder.setDefaultRequestConfig(config);
      clientBuilder.setConnectionManager(connManager);
      String userName = "xxxxx";
      String password = "xxxxx";
      String userNamePasswordPair = String.valueOf(userName) + ":" + password;

      String authenticationData = "Basic " + new String((new Base64()).encode(userNamePasswordPair.getBytes()));

      HttpPost post = new HttpPost("https://url/rest/oauth/token");
      Map<String, String> requestBodyMap = new HashMap<String, String>();
      requestBodyMap.put("grant_type", "client_credentials");

      String req = getFormUrlEncodedBodyFromMap(requestBodyMap);

      StringEntity stringEntity = new StringEntity(req);
      post.setEntity(stringEntity);
      post.setHeader("Authorization", authenticationData);
      post.setHeader("Content-Type", "application/x-www-form-urlencoded");

      CloseableHttpClient closeableHttpClient = clientBuilder.build();
      HttpResponse response = closeableHttpClient.execute(post);
      Header[] hs = response.getAllHeaders();
      for (Header header : hs) {
        System.out.println(header.toString());
    }
      System.out.println(EntityUtils.toString(response.getEntity()));
      Thread.sleep(10*60*1000L);
    } 
  }


  public static String getFormUrlEncodedBodyFromMap(Map<String, String> formData) {
    StringBuilder requestBody = new StringBuilder();
    Iterator<Map.Entry<String, String>> itrFormData = formData.entrySet().iterator();
    while (itrFormData.hasNext()) {
      Map.Entry<?, ?> entry = (Map.Entry)itrFormData.next();
      requestBody.append(entry.getKey()).append("=").append(entry.getValue());
      if (itrFormData.hasNext()) {
        requestBody.append("&");
      }
    } 
    return requestBody.toString();
  }
}

I am using pooling httpclient connection manager. 1st request in 1st time loop execution succeeded but subsequent iteration of for loop with next request fails.

My findings

If we see underlying socket connection on windows 10, after 1st request socket goes into CLOSE_WAIT state and next request executes with closing the existing connection and creating new connection.

Actually server closes the connection in duration of 5 minutes. But windows 10 able to detect it and re-initiate the connection when next request is triggered.

Now, on windows server 2016, I can see that netstat shows socket ESTABLISHED state. Means connection is ready to use and in that, it picks up the same connection and finally server has already closed it so results into connection reset error.

I suspect its an environmental issue, where server 2016 is keeping socket ESTABLISHED even after server has terminated it, but on windows 10 socket status changed to CLOSE_WAIT.

Help on this is much appreciated

Answer 1

Finally got the root cause,

Its issue with microsoft azure. They are using SNAT and closing outbound TCP connections after 4 minute idle time. This wasted my 5 days to figureout.

Means if you are connected with server with keep-alive and hope that you can reuse the connection till server time out and sends FIN. But before that if idle period reaches to 4 minutes, azure kills it. BOOM!!. Worst part is, it is not even notifying server or client with RST, means violating TCP and questioning its reliability.

 clientBuilder.setKeepAliveStrategy(new ConnectionKeepAliveStrategy() {

        @Override
        public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
            // TODO Auto-generated method stub
            return 3*60*1000;
        }
    });

Using above code, I managed to close connection on 3 minute expiry and close it before azure kills it.