登录基于PHP的网站并抓取数据-问题

Question

I am creating a 3rd party java application (desktop) that needs to connect to a php-based site and log in to gather pertinent data. There is no accessible web service, no API, and every user will have their own secure login. The site uses dojo (if that matters), and I am using Java HttpClient to send the post.

HttpPost httppost = new HttpPost("https://thewebsite.net/index/login"); // .php ?
// Create a new HttpClient and Post Header
HttpClient httpclient = new DefaultHttpClient();

//initialize the response string    
String nextpage = "";

try {
    // Add nvps
    List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(3);
    nameValuePairs.add(new BasicNameValuePair("", ""));
    nameValuePairs.add(new BasicNameValuePair("login", "USER"));
    nameValuePairs.add(new BasicNameValuePair("", ""));
    nameValuePairs.add(new BasicNameValuePair("pass", "PASSWORD"));
    nameValuePairs.add(new BasicNameValuePair("Submit", ""));

    httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs));

HttpResponse response = httpclient.execute(httppost);
userID = EntityUtils.toString(response.getEntity());

System.out.println(nextpage);
httppost.releaseConnection();
}
...

Now, the issue I'm having is that the response given to me is a validation jscript for the user / pass fields through dojo.

<script type='text/javascript'> 
dojo.require("dojox.validate._base"); 

function validate_RepeatPassword(val, constraints)
{
    var isValid = false; 

    if(constraints)  { 
        var otherInput =  dijit.byId(constraints[0]); 
        if(otherInput) { 
        var otherValue = otherInput.value; 
            isValid = (val == otherValue); 
        } 
    } 
    return isValid; 
}

</script>

I simply want to connect, parse an html response, and close the connection.

When I use firebug, I get this as the post method, but I can't seem to get it to run: Referer https://thewebsite.net/index/login Source login=USER&pass=PASSWORD

When I use the HttpPost client to construct a direct post url without namevaluepairs:

HttpPost httppost = new HttpPost("https://thewebsite.net/index/login?login=USER&pass=PASSWORD"); 

, I get an error response that states "the user and pass fields cannot be left blank."

My question is: Is there a direct method to log in that is simpler that I'm missing that will allow me to successfully continue past log in?

Thanks - I love the SO community; hope you can help.

Answer 1

I think best library for doing this is jsoup

Connection.Response res = 
Jsoup.connect("https://thewebsite.net/index/login?login=USER&pass=PASSWORD")
.method(Method.POST)
.execute();

After this you need to make verification also. You need to read cookies, request parameters and header parameters and this will work.

Answer 2

I didn't end up using your exact code (with the post parameters), but JSoup was the fix.

here's what I used:

`res = Jsoup.connect("https://thewebsite.net/index/login")
    .data("login", User).data("pass", Pass)
    .userAgent("Chrome").method(Method.POST).execute();

//then I grabbed the cookie and sent the next post for the data

Document t = res.parse(); //for later use
SessionID = res.cookie("UNIQUE_NAME");

//the JSON
Connection.Response driverx =     Jsoup.connect("https://thewebsite.net/datarequest/data").cookie("UNIQUE_NAME",SessionID).userAgent("Chrome").method(Method.POST).execute();`