Missing [FIN, ACK] and connection reset

Question

I have two questions, both below after the explanation of the problem.

I'm trying to develop proxy chaining support for HTTP requests using Apache's Java async HTTP client library. This is done mostly by providing some custom SchemeIOSessionStrategy classes. I'm seeing some problems with [FIN, ACK] sequences and missing connection resets when more than one proxy is configured.

The code being developed will use the Apache library exclusively if no proxies are configured. If more than one proxy is configured the custom SchemeIOSessionStrategy classes provide the setup to connect to the proxies.

To connect through a proxy, the code will open a connection to the first proxy server then issue a CONNECT request. If the proxy server is the last in the chain it will connect to the target server otherwise it will connect to the next proxy server in the chain. Once all the setup is complete the HTTP request is sent using the Apache library.

For a configuration with one proxy the following applies: client --> proxy 1 --> HTTP server

With two proxy servers: client --> proxy 1 --> proxy 2 --> HTTP server

The goal is to reuse connections whenever possible. When going to a particular HTTP server, the server closes the connection after some idle time. If requests are executed using the same connection, the code will go through the reconnect process.

All the HTTP requests used are the same in all scenarios. The client on both Linux and Mac OS are running the same version of Java 7. When configured with only one proxy, everything works fine. The HTTP server sends [FIN,ACK] to the proxy. The proxy closes it's connection to the HTTP server by replying with [FIN, ACK]. The proxy then closes the connection to the HTTP client by sending [FIN, ACK]. The client responds with [FIN, ACK] and the proxy responds with ACK to complete the close.

When configured with two proxies, things get off. The HTTP server sends [FIN, ACK] to proxy 2. Proxy 2 closes it's connection to the HTTP server correctly by responding with [FIN, ACK]. Proxy 2 then sends [FIN, ACK] to proxy 1, which correctly closes the connection to proxy 2 by replying with [FIN, ACK]. Proxy 1 then sends [FIN, ACK] to the HTTP client. The HTTP client responds with ACK but never sends FIN. This occurs if the client is running on Linux or Mac OS.

My first question is why might the client not send the closing [FIN, ACK] to proxy1? I can't find any pending writes or other data that I could find.

If a request is then attempted using the same connection, proxy 1 sends a RST response indicating the connection is not usable. If the client is running on Mac OS, this results in the client getting a connection reset exception. Reconnect is successful after this and the request is processed.

If the client is running on Linux, the client never gets the connection reset exception and the client hangs on a Channel.read(). My second question, is why would Linux not present the client with the connection reset exception.

Currently testing on HTTP proxies, the connection to the proxies is as follows:

    HttpRequest request = null;

    String uri = _proxyHost.getTargetHost().getHostName() + ":" + _proxyHost.getTargetHost().getPort();
    ProtocolVersion protocolVersion = new ProtocolVersion("HTTP", 1, 0);
    request = new BasicHttpRequest("CONNECT", uri, protocolVersion);

    if ((_proxyHost.getPrincipal() != null) && !_proxyHost.getPrincipal().isEmpty()) {
        StringBuilder builder = new StringBuilder(_proxyHost.getPrincipal());

        if ((_proxyHost.getCredentials() != null) && !_proxyHost.getCredentials().isEmpty()) {
            builder.append(":");
            builder.append(_proxyHost.getCredentials());
        }

        String encodedAuthString = DatatypeConverter.printBase64Binary(builder.toString().getBytes("UTF-8"));
        request.addHeader("Proxy-Authorization", "Basic " + encodedAuthString);
    }

    request.addHeader("Pragma", "No-Cache");
    request.addHeader("Proxy-Connection", "Keep-Alive");
    request.addHeader("Connection", "Keep-Alive");

To write the connect request:

    StringBuilder builder = new StringBuilder();
    RequestLine requestLine = request.getRequestLine();

    builder.append(requestLine.getMethod());
    builder.append(" ");
    builder.append(requestLine.getUri());
    builder.append(" ");
    builder.append(requestLine.getProtocolVersion().toString());
    builder.append(CRLF);

    for (Header header : request.getAllHeaders()) {
        builder.append(header.toString());
        builder.append(CRLF);
    }

    builder.append(CRLF);

    ByteBuffer byteBuffer = ByteBuffer.wrap(builder.toString().getBytes());
    ioSession.channel().write(byteBuffer);

The response is ready using ioSession.channel().read(). ioSession is an org.apache.http.nio.reactor.IOSession object.

Since it's an HTTP proxy reads are handled by ioSession.channel().read(dst) and writes by ioSession.channel().write(src).

To execute a request:

        HttpAsyncClientBuilder builder = HttpAsyncClients.custom().setRedirectStrategy(new LaxRedirectStrategy());
        ConnectingIOReactor ioReactor = IOReactorFactory.getInstance().createConnectingReactor();
        Registry<SchemeIOSessionStrategy> registry = generateRegistry(config);

        builder.setConnectionManager(new PoolingNHttpClientConnectionManager(ioReactor, registry));

        if (config.getProxy() != null) {
            proxyHost = new ProxyHost(config.getProxy());
            builder.setProxy(proxyHost);
        }

        CloseableHttpAsyncClient client = builder.build();
        client.start();

    RequestConfig.Builder configBuilder = RequestConfig.custom();
    configBuilder.setRedirectsEnabled(true);
    requestBuilder.setConfig(configBuilder.build());

    client.execute(requestBuilder.build(), context, handler);

The client gets hung with the following stack trace:

Thread 15996: (state = IN_NATIVE) - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int) @bci=4 (Compiled frame) - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114 (Compiled frame) - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=48 (Compiled frame) - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234 (Compiled frame) - com.....io.proxy.impl.PassThroughChannel.read(java.nio.ByteBuffer, java.nio.channels.ByteChannel) @bci=28, line=42 (Compiled frame) - com.....io.proxy.impl.HttpProxyIOSession$HttpProxyInternalByteChannel.read(java.nio.ByteBuffer) @bci=105, line=220 (Compiled frame) - org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(java.nio.channels.ReadableByteChannel) @bci=30, line=1 64 (Compiled frame) - org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(java.nio.channels.ReadableByteChannel) @bci=5, line=136 (Compiled frame) - org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(org.apache.http.nio.NHttpClientEventHandler) @bci=38, line=241 (Compiled frame) - org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(org.apache.http.impl.nio.DefaultNHttpClientConnection) @bci=5, line=73 (Compiled frame) - org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(java.lang.Object) @bci=5, line=37 (Compiled frame) - org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(org.apache.http.nio.reactor.IOSession) @bci=32, line=113 (Compiled frame) - org.apache.http.impl.nio.reactor.BaseIOReactor.readable(java.nio.channels.SelectionKey) @bci=11, line=159 (Compiled frame) - org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(java.nio.channels.SelectionKey) @bci=45, line=338 (Compiled frame) - org.apache.http.impl. nio.reactor.AbstractIOReactor.processEvents(java.util.Set) @bci=28, line=316 (Compiled frame) - org.apache.http.impl.nio.reactor.AbstractIOReactor.execute() @bci=80, line=277 (Compiled frame) - org.apache.http.impl.nio.reactor.BaseIOReactor.execute(org.apache.http.nio.reactor.IOEventDispatch) @bci=13, line=105 (Interpreted frame) - org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run() @bci=8, line=584 (Interpreted frame) - java.lang.Thread.run() @bci=11 (Interpreted frame)

Answer 1

My first question is why might the client not send the closing [FIN, ACK] to proxy1? I can't find any pending writes or other data that I could find.

If the client has received a FIN from the peer and hasn't sent one, the client port will be in the CLOSE-WAIT state, waiting for the client-local application to close its socket. The client is probably still attempting connection-pooling and will discover the close next time it goes to use that connection.

Why this is different with different numbers of proxies in the chain is a mystery, unless it affects the Connection: header somehow. Maybe the Apache library sends one?

If the client is running on Linux, the client never gets the connection reset exception and the client hangs on a Channel.read(). My second question, is why would Linux not present the client with the connection reset exception.

Pass. Are you sure the client received the RST ?