HTML抓取网站加载错误的Jsoup Java

Question

I'm trying to run a script to pull information from a site however when I compare the actual website to the site that my program shows it is not the same.

some examples of what is missing is the beginning !doctype and the companies' info http://www.manta.com/mb_43_E7_24/manufacturing/minnesota

I'm not sure if javascript is part of the issue, i tried turning it off and it still worked, but i also noticed there is a lot of javascript in it; no login is required for the website. Maybe cookies?(I don't know much about cookies)

String keyword = "http://www.manta.com/mb_43_E7_24/manufacturing/minnesota.php";
Document doc = Jsoup.connect(keyword).referrer("http://www.google.com").userAgent("Mozilla/5.0 (Windows; U;     WindowsNT 5.1; en-US; rv1.8.1.6) Gecko/20070725 Firefox/2.0.0.6").get();
System.out.Println(doc.toString());

Above is the code im using

Any ideas why it is failing to load my page the way that my browser does? At one point I had it working but I accidently broke it

And do you have any recommendations to a solution if this solution would not be a reasonable one to pull information from a website?

I PUT SOME MORE WORK IN AND FOUND THAT IT WORKS FOR http://www.manta.com/ but not if I add the suffex String /mb_43_E7_24/manufacturing/minnesota.php

Is the suffex in anyway involved?

Or might it be the site temporarily banning me for too many requests?

Answer 1

Jsoup does not execute/ render the Javascript. HTMLUnit has a headless browser which renders the full page and returns the content as a String. Selenium is useful as well. Selenium has WebDrivers for Firefox, Chrome, IE, and HTMLUnit. I have used the below code to execute the Javascript and return the html. I have found this useful for news sites I wish to scrape who use Javascript for the comments section.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;

public class Test{

  private WebDriver driver;
  private String output; 

  public Document getDocument(String input) {
    driver = new HTMLUnitDriver(true); //the param true turns on javascript.
    driver.get(input);
    output = driver.getPageSource();
    driver.quit();
    return Jsoup.parse(output);
  }
}

The above code should be enough to get started...