Jsoup.parse Mobile url

Question

I am new to Jsoup and I am trying to download a mobile website using Jsoup.parse().Code below works fine for a normal URL but not for mobiles.What's wrong with it ?

Code:

 private static Document downloadDocument(String url, String referer, int timeout) {
    if (url.isEmpty() || url == null) {
        return null;
    }
    if (referer.isEmpty() || referer == null) {
        //default to google.
        referer = "http://www.google.com";
    }
    Document document;
    try {
        document = Jsoup.parse(new URL(url), timeout);
    } catch (IOException e) {
        //TODO - Remove System.out.println - Memory Issue.
        System.out.println("Sorry, unable to download document");
        return null;
    }
    return document;
}

Stack trace is as follows:

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403, URL=http://m.careerbuilder.com/
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:449)
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:424)
    at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:178)
    at org.jsoup.helper.HttpConnection.get(HttpConnection.java:167)
    at org.jsoup.Jsoup.parse(Jsoup.java:183)
    ...

Answer 1

The website you want to parse checks the user agent and does not accept the default one (which is Java/jdk_version ). So you should use a "fake" user agent, like this:

Document html = Jsoup.connect("http://m.careerbuilder.com").userAgent("Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36").get();
System.out.println(html);

Where Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36 is the user agent of Chrome 32.0.1667.0