Jsoup http logging

Question

Is there a way to log the http request and response? Let's assume the below request

Connection.Response res = Jsoup.connect("LOGIN_URL_HERE")
            .data("user", "USER", "pass", "PASS")
            .method(Connection.Method.POST)
            .execute();

How can I log the http request and response? Please mind that I want the HTTP and not just the HTML that will be parsed.

Answer 1

By default jsoup uses a implementation of java.net.HttpURLConnection So I suppose you need to turn on logging for that implementation (probably: sun.net.www.protocol.http.HttpURLConnection) or for java.net .

There is a system property that will enable logging for java net utils

-Djavax.net.debug=all

Answer 2

As Jsoup lacks logging (version I'm using: 1.12.1 ) and using the -Djavax.net.debug=all JVM argument logs are too verbose, the best way I found is to decorate the HttpConnection class, so one can customize what is logged. To achive this, the execute method call needs to be surrounded by logging the properties of the Connection.Request and Connection.Response .

Sample implementation using SLF4J :

import org.jsoup.Connection;
import org.jsoup.helper.HttpConnection;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;

public class DiagnosticConnection extends HttpConnection {
    static final Logger LOG = LoggerFactory.getLogger(DiagnosticConnection.class);

    @Override
    public Connection.Response execute() throws IOException {
        log(this.request());
        Connection.Response response = super.execute();
        log(response);

        return response;
    }

    public static Connection connect(String url) {
        Connection connection = new DiagnosticConnection();
        connection.url(url);
        return connection;
    }

    private static void log(Connection.Request request) {
        LOG.info("========================================");
        LOG.info("[url] {}", request.url());
        LOG.info("== REQUEST ==");
        logBase(request);
        LOG.info("[method] {}", request.method());
        LOG.info("[data] {}", request.data());
        LOG.info("[request body] {}", request.requestBody());
    }

    private static void log(Connection.Response response) {
        LOG.info("== RESPONSE ==");
        logBase(response);
        LOG.info("[code] {}", response.statusCode());
        LOG.info("[status msg] {}", response.statusMessage());
        LOG.info("[body] {}", response.body());
        LOG.info("========================================");
    }

    private static void logBase(Connection.Base<?> base) {
        LOG.info("[headers] {}", base.headers());
        LOG.info("[cookies] {}", base.cookies());
    }

}

When using the decorator, instead of Jsoup.connect(<URL>) you should use DiagnosticConnection.connect(<URL>)

Answer 3

Based on Gergely Toth response, I have created my own LoggerHttpConnection and I'm working with that.

import android.util.Log
import org.jsoup.Connection
import org.jsoup.helper.HttpConnection
import org.jsoup.nodes.Document
import org.jsoup.parser.Parser
import java.io.InputStream
import java.net.Proxy
import java.net.URL
import javax.net.ssl.SSLSocketFactory

class LoggerHttpConnection private constructor(
    private val delegate: HttpConnection,
    private val saveFile: Boolean
) : Connection {

    private val tag = "LoggerHttpConnection"

    companion object {
        fun connect(url: String, saveFile: Boolean = false): LoggerHttpConnection {
            return LoggerHttpConnection(
                HttpConnection.connect(url) as HttpConnection,
                saveFile
            )
        }
    }

    private fun log(request: Connection.Request): String {
        Log.i(tag, "========================================")
        var line = "[url] ${request.url()}"
        var log = "$line\n\n== REQUEST ==\n"
        Log.i(tag, line)

        Log.i(tag, "== REQUEST ==")
        log += logBase(request)

        line = "[method] ${request.method()}"
        log += "$line\n"
        Log.i(tag, line)

        for (data in request.data()) {
            line = "[data] ${data.key()}=${data.value()}"
            log += "$line\n"
            Log.i(tag, line)
        }

        line = "[request body] ${request.requestBody()}"
        log += "$line\n"
        Log.i(tag, line)

        return log
    }

    private fun log(response: Connection.Response): String {
        var line = ""
        var log = "\n== RESPONSE ==\n"

        Log.i(tag, "== RESPONSE ==")
        log += logBase(response)

        line = "[code] ${response.statusCode()}"
        log += "$line\n"
        Log.i(tag, line)

        line = "[status msg] ${response.statusMessage()}"
        log += "$line\n"
        Log.i(tag, line)

        line = "[body] ${response.body()}"
        log += "$line\n"
        Log.i(tag, line)

        Log.i(tag, "========================================")

        return log
    }

    private fun logBase(base: Connection.Base<*>): String {
        var line = ""
        var log = ""
        for (header in base.headers()) {
            line = "[header] ${header.key}=${header.value}"
            log += "$line\n"
            Log.i(tag, line)
        }
        for (cookie in base.cookies()) {
            line = "[cookie] ${cookie.key}: ${cookie.value}"
            log += "$line\n"
            Log.i(tag, line)
        }
        return log
    }

    override fun execute(): Connection.Response {
        var logs = log(request())
        val response = delegate.execute()
        logs += log(response)
        if (saveFile)
            logs.saveToFile("request_log") //do something to save your log in a file if its necesary
        return response
    }

    override fun ignoreContentType(ignoreContentType: Boolean): Connection {
        delegate.ignoreContentType(ignoreContentType)
        return this
    }

    override fun postDataCharset(charset: String?): Connection {
        delegate.postDataCharset(charset)
        return this
    }

    override fun get(): Document {
        return delegate.get()
    }

    override fun post(): Document {
        return delegate.post()
    }

    /** Continue implementing necessary methods for Connection */

}

Now just declare your request using LoggerHttpConnection instead Jsoup and everything will work

Connection.Response res = LoggerHttpConnection.connect("LOGIN_URL_HERE")
        .data("user", "USER", "pass", "PASS")
        .method(Connection.Method.POST)
        .execute();