搜寻可能需要登录的网站-JSOUP

Question

I am trying to scrape a website that possibly requires authentication. When I try the following code I get an error :

org.jsoup.UnsupportedMimeTypeException: Unhandled content type. Must be text/*, application/xml, or application/xhtml+xml. Mimetype=application/json; charset=utf-8, URL= https://sso.mims.com/Account/Signin at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:547) at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:493) at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:205) at com.aiingov.proc.MedScraper.main(MedScraper.java:49)

public static void main(String[] args) throws IOException {

String url = "https://sso.mims.com/Account/Signin";
            String userAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36";

            Connection.Response response = Jsoup.connect(url).userAgent(userAgent)
                    .method(Connection.Method.GET)
                    .execute();

            response = Jsoup.connect(url)
                    .cookies(response.cookies())
                    .data("action", "login")
                    .data("login", "xxxxx")
                    .data("password", "xxxxx")
                    .data("auto_login", "1")
                    .userAgent(userAgent)
                    .method(Connection.Method.POST)
                    .followRedirects(true)
                    .execute();           

            Document document = Jsoup.connect("https://www.mims.com/india/drug/info/abacavir/abacavir?type=full&mtype=generic")
                    .cookies(response.cookies())
                    .userAgent(userAgent)
                    .get();

            System.out.println(document);

            Elements elements = document.body().select("*");

               for (Element element : elements) {
                   System.out.println(element.ownText());
               }

Without the login code in place I get the following output:

You will be redirected to your destination shortly.

How do I fix this?

Answer 1

Try using the ignoreContentType method.

 Jsoup.connect(url).ignoreContentType(true);//chain any other methods

Description of the method from the JSoup Docs:

Ignore the document's Content-Type when parsing the response. By default this is false, an unrecognised content-type will cause an IOException to be thrown. (This is to prevent producing garbage by attempting to parse a JPEG binary image, for example.) Set to true to force a parse attempt regardless of content type.