使用Jsoup登录和导航

Question

I am trying to login to a website using JSoup, my goal is scrape some data from the website but I am having some problems with the logging in/navigating.

See the code below for how the code currently looks like.

    try {
        Connection.Response response = Jsoup.connect("https://app.northpass.com/login")
                .method(Connection.Method.GET)
                .execute();

        response = Jsoup.connect("https://app.northpass.com/login")
                .data("educator[email]", "email123")
                .data("educator[password]", "password123")
                .cookies(response.cookies())
                .method(Connection.Method.POST)
                .execute();

        // Go to new page
        Document coursePage = Jsoup.connect("https://app.northpass.com/course")
                .cookies(response.cookies())
                .get();

        System.out.println(groupPage.title());

    } catch (IOException e) {
        e.printStackTrace();
    }

I have also tried adding

.data("commit", "Log in")

and

.userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21")

without any success.

The error I get is as follow:

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=500, URL=https://app.northpass.com/login

From what I have read on other threads, people suggest using a userAgent (which, as said above, I have already tried). Thanks in advance for any help.

Answer 1

If you look at the network traffic when you attempt a login in your browser you'll see that an additional item of data is sent: authenticity_token . This is a hidden field in the form.

You will need then to extract that from the initial response and send it with the POST request:

try {
    Connection.Response response = Jsoup.connect("https://app.northpass.com/login")
            .method(Connection.Method.GET)
            .execute();

    //I can't test this but will be something like
    //see https://jsoup.org/cookbook/extracting-data/selector-syntax
    Document document = response.parse();
    String token = document.select("input[hidden]").first().val();

    response = Jsoup.connect("https://app.northpass.com/login")
            .data("educator[email]", "email123")
            .data("educator[password]", "password123")
            .data("authenticity_token", token)
            .cookies(response.cookies())
            .method(Connection.Method.POST)
            .execute();

    // Go to new page
    Document coursePage = Jsoup.connect("https://app.northpass.com/course")
            .cookies(response.cookies())
            .get();

    System.out.println(groupPage.title());

} catch (IOException e) {
    e.printStackTrace();
}