Using JSOUP to Login to ConEd website

Question

I have read extensively about how do this and I have tried a number of different variations, but I can't get it to work.

Basically, I just want to login to the ConEdison website and scrape my billing history. Here is what I have:

Connection.Response loginForm = Jsoup.connect("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng")
                    .data("_LASTFOCUS","")
                    .data("_EVENTTARGET","")
                    .data("_EVENTARGUMENT","")
                    .data("_VIEWSTATE", viewState)
                    .data("_EVENTVALIDATION", eventValidation)
                    .data("ctl00$Main$Login1$UserName", username)
                    .data("ctl00$Main$Login1$Password", password)
                    .data("ctl00$Main$Login1$LoginButton", "Sign In")
                    .userAgent("Mozilla/5.0")
                    .method(Method.POST)
                    .execute();

            Map<String, String> loginCookies = loginForm.cookies();

            Document document = Jsoup.connect("https://apps.coned.com/CEMyAccount/CSOL/BillHistory.aspx?lang=eng")
                    .cookies(loginCookies)
                    .get();

            Elements data = document.select("table.ctl00_Main_lvBillHistory_Table1");

            //checking if it found the right page
            System.out.println("document: " + document);
            //checking if it found the table
            System.out.println("data: " + data);

I know the information is correct (though I don't know if I really need to pass the data parameters with no values).

I am not getting any errors, just printing out the login page ( https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng )

Any help would be greatly appreciated.

Thanks

EDIT

So, I am now convinced that I was not able to get to the internal page, because after the POST to https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng , 3 cookies are set, but then it sends GET requests to https://apps.coned.com/cemyaccount/SessionTransfer.aspx?dir=2asp&url=https://apps.coned.com/csol/MainHome.asp?src=DOTNET then to https://apps.coned.com/csol/SessionTransfer.asp?dir=2asp&guid=3c413f48-d2eb-434a-896b-f9c4eb100714&url=https://apps.coned.com/csol/MainHome.asp?src=DOTNET&frm= for additional cookies before going to the homepage

Does anyone know how I can follow all these redirects and get the cookies in the end?

Here is what I currently have, but I cannot get the cookies from the POST call.

Response response = Jsoup
                        .connect("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng")
                        .method(Method.GET)
                        .execute();

                Map<String, String> cookies = response.cookies();
                cookies.put("NSC_DpoFe_Bqqt-TTM-pme", response.cookie("NSC_DpoFe_Bqqt-TTM-ofx"));
                System.out.println("response cookies: " + cookies);

                response = Jsoup
                        .connect("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng")
                        .header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
                        .header("Accept-Encoding", "gzip, deflate")
                        .header("Accept-Language", "en-US,en;q=0.8")
                        .header("Connection", "keep-alive")
                        .cookies(cookies)
                        .header("Host", "apps.coned.com")
                        .referrer("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng&login=0")
                        .userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36")
                        .data("_LASTFOCUS", "")
                        .data("_EVENTTARGET", "")
                        .data("_EVENTARGUMENT", "")
                        .data("_VIEWSTATE", viewState)
                        .data("_EVENTVALIDATION", eventValidation)
                        .data("ctl00$Main$Login1$UserName", username)
                        .data("ctl00$Main$Login1$Password", password)
                        .data("ctl00$Main$Login1$LoginButton", "Sign In")
                        .followRedirects(false)
                        .method(Method.POST)
                        .execute(); 

                System.out.println("post cookies: " + response.cookies());
                cookies.putAll(response.cookies());
                System.out.println("response cookies: " + cookies);

                response = Jsoup
                        .connect("https://apps.coned.com/cemyaccount/SessionTransfer.aspx?dir=2asp&url=https:"
                                + "//apps.coned.com/csol/MainHome.asp?src=DOTNET")
                        .cookies(cookies)
                        .followRedirects(false)
                        .method(Method.GET)
                        .execute();

                cookies.putAll(response.cookies());
                System.out.println("response cookies: " + cookies);
                String guid = response.header("location");

                response = Jsoup
                        .connect("https://apps.coned.com/csol/SessionTransfer.asp?dir=2asp&guid="
                                + guid + "&url=https://apps.coned.com/csol/MainHome.asp"
                                + "?src=DOTNET&frm=")
                        .cookies(cookies)
                        .method(Method.GET)
                        .execute();

                cookies.putAll(response.cookies());
                System.out.println("response cookies: " + cookies);

                Document dataPage = Jsoup
                        .connect("https://apps.coned.com/CEMyAccount/CSOL/BillHistory.aspx?lang=eng")
                        .cookies(cookies)
                        .get();

                System.out.println("data page: " + dataPage);

                Elements data = dataPage.select("table.ctl00_Main_lvBillHistory_Table1");

                System.out.println("data: " + data);

In the output I get all the cookies, except the POST cookies which are blank.

Answer 1

• Open Development tools (press F12 ).
• Select Network tab option.
• Up and left there is a round button. If it's not red, click it. This will record the traffic.
• Visit the login page. Enter your credentials and login.
• Check what happens in Development tools. There is a table there, showing all the files that were received.
• Check the Type column. Search for the row that has the value document . Select it. This will open a new screen.
• Select Headers and scroll down until you locate the Request Headers . There you will find the request that was made from your browser. You will find there all the values that were send to the server.
• Search for the parameters you need. Take the values and hardcode them to your code. Use the same user-agent (just in case) and in general try to imitate this request with your code.

All the steps above are for the Chrome browser.

Answer 2

答案非常简单-下划线的标头有2个下划线，我只使用了1。