如何为java HttpURLConnection流量启用线路记录？

Question

I've used Jakarta commons HttpClient in another project and I would like the same wire logging output but using the "standard" HttpUrlConnection.

I've used Fiddler as a proxy but I would like to log the traffic directly from java.

Capturing what goes by the connection input and output streams is not enough because the HTTP headers are written and consumed by the HttpUrlConnection class, so I will not be able to log the headers.

Answer 1

According to Sun's HttpURLConnection source there is some logging support via JUL .

Setup (adjust path as required):

-Djava.util.logging.config.file=/full/path/to/logging.properties

logging.properties:

handlers= java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = FINEST
sun.net.www.protocol.http.HttpURLConnection.level=ALL

This will log to the console, adjust as required to eg log to a file.

Example output:

2010-08-07 00:00:31 sun.net.www.protocol.http.HttpURLConnection writeRequests
FIN: sun.net.www.MessageHeader@16caf435 pairs: {GET /howto.html HTTP/1.1: null}{User-Agent: Java/1.6.0_20}{Host: www.rgagnon.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
2010-08-07 00:00:31 sun.net.www.protocol.http.HttpURLConnection getInputStream
FIN: sun.net.www.MessageHeader@5ac0728 pairs: {null: HTTP/1.1 200 OK}{Date: Sat, 07 Aug 2010 04:00:33 GMT}{Server: Apache}{Accept-Ranges: bytes}{Content-Length: 17912}{Keep-Alive: timeout=5, max=64}{Connection: Keep-Alive}{Content-Type: text/html}

Note that this prints only headers without body.

See http://www.rgagnon.com/javadetails/java-debug-HttpURLConnection-problem.html for details.

There is also system property -Djavax.net.debug=all . But it's mainly useful for SSL debugging .

Answer 2

I've been able to log all SSL traffic implementing my own SSLSocketFactory on top of the default one.

This worked for me because all of our connections are using HTTPS and we can set the socket factory with the method HttpsURLConnection.setSSLSocketFactory .

A more complete solution that enables monitoring on all sockets can be found at http://www.javaspecialists.eu/archive/Issue169.html Thanks to Lawrence Dol for pointing in the right direction of using Socket.setSocketImplFactory

Here is my not ready for production code:

public class WireLogSSLSocketFactory extends SSLSocketFactory {

    private SSLSocketFactory delegate;

    public WireLogSSLSocketFactory(SSLSocketFactory sf0) {
        this.delegate = sf0;
    }

    public Socket createSocket(Socket s, String host, int port,
            boolean autoClose) throws IOException {
        return new WireLogSocket((SSLSocket) delegate.createSocket(s, host, port, autoClose));
    }

    /*
    ...
    */

    private static class WireLogSocket extends SSLSocket {

        private SSLSocket delegate;

        public WireLogSocket(SSLSocket s) {
            this.delegate = s;
        }

        public OutputStream getOutputStream() throws IOException {
            return new LoggingOutputStream(delegate.getOutputStream());
        }

        /*
        ...
        */

        private static class LoggingOutputStream extends FilterOutputStream {
            private static final Logger logger = Logger.getLogger(WireLogSocket.LoggingOutputStream.class);
            //I'm using a fixed charset because my app always uses the same. 
            private static final String CHARSET = "ISO-8859-1";
            private StringBuffer sb = new StringBuffer();

            public LoggingOutputStream(OutputStream out) {
                super(out);
            }

            public void write(byte[] b, int off, int len)
                    throws IOException {
                sb.append(new String(b, off, len, CHARSET));
                logger.info("\n" + sb.toString());
                out.write(b, off, len);
            }

            public void write(int b) throws IOException {
                sb.append(b);
                logger.info("\n" + sb.toString());
                out.write(b);
            }

            public void close() throws IOException {
                logger.info("\n" + sb.toString());
                super.close();
            }
        }
    }
}

Answer 3

Solution#1: Use Decorator Pattern

You will have to use Decorator pattern on HttpURLConnection class to extend it's functionality. Then override all HttpURLConnection method(s) and delegate operation to Component pointer as well as capture required information and log it.

Also make sure you override parent class URLConnection.getOutputStream() : OutputStream and URLConnection.html#getInputStream() : InputStream methods to return decorated OutputStream and InputStream objects as well.

.

Solution#2: Use custom, in-memory http proxy

Write a simple http proxy server and have it start in it's separate thread during application startup and initialization. See Example simple proxy server .

Have your application configured to use above HTTP proxy for all your requests. See configuring Java to use Proxies .

Now all your traffic is going through above proxy, just like how it happens in fiddler. Hence you have access to raw http stream "from client to server" as well as "back from server to client". You will have to interpret this raw information and log it as required.

Update: Use HTTP Proxy as Adapter to SSL based Web Server.

  == Client System =============================== 
  |                                              | 
  |    ------------------------------            | 
  |   |                              |           | 
  |   |    Java process              |           | 
  |   |                       ----   |           | 
  |   |        ----------    |    |  |           | 
  |   |       |          |    -O  |  |           | 
  |   |       |  Logging |        |  |           | 
  |   |       |   Proxy <---HTTP--   |    -----  | 
  |   |       |  Adapter |           |   |     | | 
  |   |       |  Thread o------------------>   | | 
  |   |       |        o |           |   |     | | 
  |   |        --------|-            |   | Log | | 
  |   |                |             |    -----  | 
  |    ----------------|-------------            | 
  |                    |                         | 
  =====================|========================== 
                       |                           
                       |                           
                     HTTPS                         
                      SSL                          
                       |                           
  == Server System ====|========================== 
  |                    |                         | 
  |    ----------------|----------------         | 
  |   |                V                |        | 
  |   |                                 |        | 
  |   |   Web Server                    |        | 
  |   |                                 |        | 
  |    ---------------------------------         | 
  |                                              | 
  ================================================ 

Answer 4

In Linux you can run the VM under strace:

strace -o strace.out -s 4096 -e trace=network -f java ...

Answer 5

I don't think you can do that automatically, but you could subclass FilterOutputStream and FilterInputStream with the HttpUrlConnection 's output and input streams as parameters. Then as bytes are written/read, log them as well as pass them through to the underlying streams.

Answer 6

What about using AspectJ to insert a Pointcut to add logging advice around a method? I believe AspectJ can weave it's way into private/protected methods.

It appears that sun.net.www.protocol.http.HttpURLConnection.writeRequest may call sun.net.www.http.HttpClient.writeRequest which takes the MessageHeader Object as an input so that would be your target.

In the end this might work but will be awfully fragile and only work on the Sun JVM; and really you could only trust the exact version you are using.

Answer 7

On the off chance, that you're only interested in getting at the content on the wire (headers, body etc), you might want to give wireshark a go.

This has the advantage of not having to change any code, though if enabling logging through code was what you're after, this answer is not applicable.

Answer 8

To refresh with Java 8 environment:

Following @sleske answer

System.setProperty("javax.net.debug","all");

worked for me out of the box.

Also was @weberjn suggestion of

strace -o strace.out -s 4096 -e trace=network -f java

but not useful if handling with SSL traffic as it dumps encoded stream.

All other code tricks did not work for me, but maybe not trying enough hard.