使用Jsoup无法成功登录此页面

Question

I've been struggling to login into this page so I could scrape the private

learn.sun.ac.za/my

page for a good while now. I've searched through multiple SO posts and tried to apply the advice from each to no effect.

attempt 1

String url = "https://sso-prod.sun.ac.za/cas/login?service=http%3A%2F%2Flearn.sun.ac.za%2Flogin%2Findex.php";
String userAgent = "Mozilla/5.0";

            Connection.Response response = Jsoup.connect(url)
                    .userAgent(userAgent)
                    .method(Connection.Method.GET)
                    .execute();

            response = Jsoup.connect(url)
                    .userAgent(userAgent)
                    .cookies(response.cookies())
                    .data("action", "login")
                    .data("username", "MYUSERNAME")
                    .data("password", "MYPASSWORD")
                    .method(Connection.Method.POST)
                    //.followRedirects(true)
                    .execute();            

            Document doc = Jsoup.connect("http://learn.sun.ac.za/my")
                    .cookies(response.cookies())
                    .userAgent(userAgent)
                    .get();

            System.out.println(doc.title());

output: "Single Sign on | Corporation"

indicating that it did not login.

From advice in other posts I monitored the traffic out via chrome and added all the headers from there to the code

String url = "https://sso-prod.sun.ac.za/cas/login?service=http%3A%2F%2Flearn.sun.ac.za%2Flogin%2Findex.php";
String userAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36";

            Connection.Response response = Jsoup.connect(url)
                    .userAgent(userAgent)
                    .method(Connection.Method.GET)
                    .execute();

            response = Jsoup.connect(url)
                    .userAgent(userAgent)
                    .cookies(response.cookies())
                    .header("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
                    .header("accept-encoding", "gzip, deflate")
                    .header("accept-language", "en-US,en;q=0.8")
                    .header("cache-control", "max-age=0")
                    .header("connection", "keep-alive")
                    .header("content-length", "114")
                    .header("content-type","application/x-www-form-urlencoded")
                    .header("dnt", "1")
                    .header("host", "sso-prod.sun.ac.za")
                    .header("origin", "https://sso-prod.sun.ac.za" )
                    .header("referer", "https://sso-prod.sun.ac.za/cas/login?service=http%3A%2F%2Flearn.sun.ac.za%2Flogin%2Findex.php")
                    .header("upgrade-insecure-requests", "1")
                    .data("action", "login")
                    .data("username", "MYUSERNAME")
                    .data("password", "MYPASSWORD")
                    .data("lt", "LT-3042474-9t3oldTU1253G6HVqFffHgMWxnYXdg")
                    .data("execution", "e1s1" )
                    .data("_eventId", "submit")
                    .method(Connection.Method.POST)
                    //.followRedirects(true)
                    .execute();            

            Document doc = Jsoup.connect("http://learn.sun.ac.za/my")
                    .cookies(response.cookies())
                    .userAgent(userAgent)
                    .get();

            System.out.println(doc.title());

which had the same result. What I did after was print out the actual html code and found that there was no login error message anywhere in the code which means I screwed up somewhere and havent actually submitted the form?

this is what a successful chrome login looks like

Answer 1

That's because your LT value is wrong - you should get the value for each new session, and not to send some value like you do. Look at the HTML of the page:

The selector for the DIV that contains the value is div.controls:nth-child(1) .
So the steps are -
Load the page
Get the value of LT Add it to your POST request.