在抓取网站之前使用JSoup登录

Question

I'm trying to use Jsoup to login to a website and then scrape the contents. I've already read a few similar questions, and figured that I need to do a POST and pass all the input parameters. I've done just that, but it still doesn't work for some reason. Any help greatly appreciated.

HTML form:

<form method="post" action="https://login.yahoo.com/config/login?" name="login_form" id="login_form" onsubmit="return hash2(this)">
<input type="hidden" name="sessionId" id="sessionId" value="IyLZEAs5n9RP">
<input type="hidden" name=".tries" value="1">
<input type="hidden" name=".src" value="spt">
<input type="hidden" name=".md5" value="">
<input type="hidden" name=".hash" value="">
<input type="hidden" name=".js" value="">
<input type="hidden" name=".last" value="">
<input type="hidden" name="promo" value="">
<input type="hidden" name=".intl" value="us">
<input type="hidden" name=".lang" value="en-US">
<input type="hidden" name=".bypass" value="">
<input type="hidden" name=".partner" value="">
<input type="hidden" name=".u" value="516l00ha52emg">
<input type="hidden" name=".v" value="0">
<input type="hidden" name=".challenge" value="HupRQ9x1ptIRHP1H8P9eYBbAlofE4YsoSQ--">
<input type="hidden" name=".yplus" value="">
<input type="hidden" name=".emailCode" value="">
<input type="hidden" name="pkg" value="">
<input type="hidden" name="stepid" value="">
<input type="hidden" name=".ev" value="">
<input type="hidden" name="hasMsgr" value="0">
<input type="hidden" name=".chkP" value="Y">
<input type="hidden" name=".done" value="http://hockey.fantasysports.yahoo.com/hockey/27381/startingrosters?.scrumb=0">
<input type="hidden" name=".pd" value="spt_ver=0&c=&ivt=&sg=">
<input type="hidden" name=".ws" id=".ws" value="0">
<input type="hidden" name=".cp" id=".cp" value="0">     
<input type="hidden" name="nr" value="0">
    <input type="hidden" name="pad" id="pad" value="6">
<input type="hidden" name="aad" id="aad" value="6">

<div id='inputs'>
                <input name='login' id='username' type="text" maxlength='96' tabindex='1' aria-required="true" placeholder="Yahoo ID" autocorrect="off" value="">
                <input name='passwd' id='passwd' type='password' maxlength='64' tabindex='2' aria-required="true" placeholder="Password" autocorrect="off" value="">
                <div id="captchaDiv"></div>
        </div>

My code:

Connection.Response loginForm = Jsoup.connect(url).method(Connection.Method.GET)
                                .execute();

Document doc = Jsoup.connect(url).data("sessionId", "IyLZEAs5n9RP").data(".tries", "1").data(".src", ".spt").data(".md5", "").data(".hash", "").data(".js", "").data(".last", "").data("promo", "").data(".intl", "us").data(".lang", "en-US").data(".bypass", "").data(".partner", "").data(".u", "515l00ha52emg").data(".v", "0").data(".challenge", "HupRQ9x1ptIRHP1H8P9eYBbAlofE4YsoSQ--").data(".yplus", "").data(".emailCode", "").data("pkg", "").data("stepid", "").data(".ev", "").data("hasMsgr", "0").data(".chkP", "Y").data(".done", "http://hockey.fantasysports.yahoo.com/hockey/27381/startingrosters?.scrumb=0").data(".pd", "spt_ver=0&c=&ivt=&sg=").data(".ws", "0").data(".cp", "0").data("nr", "0").data("pad", "6").data("aad", "6")
        .data("login", "MYEMAIL").data("passwd", "MYPASSWORD")
        .cookies(loginForm.cookies()).post();

System.out.println(doc.title());

When running this, it still prints the login title though. Sorry for the bad one-line formatting, but there are a lot of parameters and their values aren't pertinent to the question. I put the final few parameters on new lines.

Looking at the parameters, I can see that stuff like "sessionId" will be different depending on the session. I can see that being a problem. Would that mean that I'd have to first save the value of that particular sessionId and pass that value to my Jsoup.connect ?

Answer 1

It is not enough with hardcoding the form parameters! You must read the parameter values from the first page and the pass them in the next post along user/pass and cookies.

Hope it helps.