[英]Using JSOUP to Login to ConEd website
I have read extensively about how do this and I have tried a number of different variations, but I can't get it to work. 我已经广泛阅读了有关此操作的信息,并尝试了许多不同的变体,但无法使其正常工作。
Basically, I just want to login to the ConEdison website and scrape my billing history. 基本上,我只想登录ConEdison网站并抓取我的帐单记录。 Here is what I have:
这是我所拥有的:
Connection.Response loginForm = Jsoup.connect("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng")
.data("_LASTFOCUS","")
.data("_EVENTTARGET","")
.data("_EVENTARGUMENT","")
.data("_VIEWSTATE", viewState)
.data("_EVENTVALIDATION", eventValidation)
.data("ctl00$Main$Login1$UserName", username)
.data("ctl00$Main$Login1$Password", password)
.data("ctl00$Main$Login1$LoginButton", "Sign In")
.userAgent("Mozilla/5.0")
.method(Method.POST)
.execute();
Map<String, String> loginCookies = loginForm.cookies();
Document document = Jsoup.connect("https://apps.coned.com/CEMyAccount/CSOL/BillHistory.aspx?lang=eng")
.cookies(loginCookies)
.get();
Elements data = document.select("table.ctl00_Main_lvBillHistory_Table1");
//checking if it found the right page
System.out.println("document: " + document);
//checking if it found the table
System.out.println("data: " + data);
I know the information is correct (though I don't know if I really need to pass the data parameters with no values). 我知道信息是正确的(尽管我不知道是否真的需要传递没有值的数据参数)。
I am not getting any errors, just printing out the login page ( https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng ) 我没有任何错误,只是打印了登录页面( https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng )
Any help would be greatly appreciated. 任何帮助将不胜感激。
Thanks 谢谢
EDIT 编辑
So, I am now convinced that I was not able to get to the internal page, because after the POST to https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng , 3 cookies are set, but then it sends GET requests to https://apps.coned.com/cemyaccount/SessionTransfer.aspx?dir=2asp&url=https://apps.coned.com/csol/MainHome.asp?src=DOTNET then to https://apps.coned.com/csol/SessionTransfer.asp?dir=2asp&guid=3c413f48-d2eb-434a-896b-f9c4eb100714&url=https://apps.coned.com/csol/MainHome.asp?src=DOTNET&frm= for additional cookies before going to the homepage 因此,我现在确信我无法进入内部页面,因为在POST到https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng之后 ,设置了3个cookie,但随后它将GET请求发送到https://apps.coned.com/cemyaccount/SessionTransfer.aspx?dir=2asp&url=https://apps.coned.com/csol/MainHome.asp?src=DOTNET,然后发送到https: //apps.coned.com/csol/SessionTransfer.asp?dir=2asp&guid=3c413f48-d2eb-434a-896b-f9c4eb100714&url=https://apps.coned.com/csol/MainHome.asp?src=DOTNET&frm=进入首页之前的Cookie
Does anyone know how I can follow all these redirects and get the cookies in the end? 有谁知道我如何遵循所有这些重定向并最终获得cookie?
Here is what I currently have, but I cannot get the cookies from the POST call. 这是我目前拥有的,但是我无法从POST调用中获取Cookie。
Response response = Jsoup
.connect("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng")
.method(Method.GET)
.execute();
Map<String, String> cookies = response.cookies();
cookies.put("NSC_DpoFe_Bqqt-TTM-pme", response.cookie("NSC_DpoFe_Bqqt-TTM-ofx"));
System.out.println("response cookies: " + cookies);
response = Jsoup
.connect("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng")
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
.header("Accept-Encoding", "gzip, deflate")
.header("Accept-Language", "en-US,en;q=0.8")
.header("Connection", "keep-alive")
.cookies(cookies)
.header("Host", "apps.coned.com")
.referrer("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng&login=0")
.userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36")
.data("_LASTFOCUS", "")
.data("_EVENTTARGET", "")
.data("_EVENTARGUMENT", "")
.data("_VIEWSTATE", viewState)
.data("_EVENTVALIDATION", eventValidation)
.data("ctl00$Main$Login1$UserName", username)
.data("ctl00$Main$Login1$Password", password)
.data("ctl00$Main$Login1$LoginButton", "Sign In")
.followRedirects(false)
.method(Method.POST)
.execute();
System.out.println("post cookies: " + response.cookies());
cookies.putAll(response.cookies());
System.out.println("response cookies: " + cookies);
response = Jsoup
.connect("https://apps.coned.com/cemyaccount/SessionTransfer.aspx?dir=2asp&url=https:"
+ "//apps.coned.com/csol/MainHome.asp?src=DOTNET")
.cookies(cookies)
.followRedirects(false)
.method(Method.GET)
.execute();
cookies.putAll(response.cookies());
System.out.println("response cookies: " + cookies);
String guid = response.header("location");
response = Jsoup
.connect("https://apps.coned.com/csol/SessionTransfer.asp?dir=2asp&guid="
+ guid + "&url=https://apps.coned.com/csol/MainHome.asp"
+ "?src=DOTNET&frm=")
.cookies(cookies)
.method(Method.GET)
.execute();
cookies.putAll(response.cookies());
System.out.println("response cookies: " + cookies);
Document dataPage = Jsoup
.connect("https://apps.coned.com/CEMyAccount/CSOL/BillHistory.aspx?lang=eng")
.cookies(cookies)
.get();
System.out.println("data page: " + dataPage);
Elements data = dataPage.select("table.ctl00_Main_lvBillHistory_Table1");
System.out.println("data: " + data);
In the output I get all the cookies, except the POST cookies which are blank. 在输出中,我获得了所有cookie,但POST cookie是空白的。
Network
tab option. Network
选项卡选项。 Type
column. Type
列。 Search for the row that has the value document
. document
的行。 Select it. Headers
and scroll down until you locate the Request Headers
. Headers
并向下滚动,直到找到“ Request Headers
。 There you will find the request that was made from your browser. All the steps above are for the Chrome browser. 以上所有步骤均适用于Chrome浏览器。
答案非常简单-下划线的标头有2个下划线,我只使用了1。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.