从Web网址检索整页源

Question

I have a minor project that I am working on where I am scraping information from webpages. As a beginning step I began looking at the page source from

http://www.walmart.com/search/search-ng.do?search_query=camera&ic=16_0&Find=Find&search_constraint=0

After analyzing what I needed to do I attempted to retrieve that same page information using two methods that were both unsuccessful

First I tried a simple request using Jsoup which looks like the following

    Document doc;
    try {
        doc = Jsoup.connect("http://www.walmart.com/search/search-ng.do?search_query=camera&ic=16_0&Find=Find&search_constraint=0").get();

        System.out.println(doc);

    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

Which brought up some page information but not the actual page source which includes all of the search results

Then I tried and Apache Commons http solution which looks like

    String url = "http://www.walmart.com/search/search-ng.do?search_query=camera&ic=16_0&Find=Find&search_constraint=0";
    DefaultHttpClient httpclient = new DefaultHttpClient();
    HttpPost request = new HttpPost(url);

        HttpResponse response;
        try {
            response = httpclient.execute(request);
            StatusLine status = response.getStatusLine();
            String responseString = EntityUtils.toString(response.getEntity());

            System.out.println(status);
            System.out.println(responseString);

        } catch (ClientProtocolException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

but I keep receiving a page permanently moved status.

So far it seems that Jsoup is my best option for moving forward. I believe that issue with not receiving all of the search results has to do with the scripts that are on the page not running when called by Jsoup's get function.

How would I get all of the page information so that I can begin retrieving information from the search results.

Answer 1

Jsoup does not support execution of javascript, meaning that you wont be able to parse dynamically generated HTML. Simply put, Jsoup does not simulate a browser environment, but is a pure parser.

I would suggest that you instead use HtmlUnit which is a "GUI-less browser for Java programs". It has support for javascript execution, and can be used to generate the HTML source you then later might want to parse easier with Jsoup.

HtmlUnit can be found here .