使用 Jsoup.connect() 时会发生什么？ 为什么这么慢？

Question

I'm using the following line to load document

Document doc = Jsoup.connect("http://www.some.site.with.lotsof.images/")
        .header("Accept-Encoding", "gzip, deflate")
        .userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0")
        .maxBodySize(0)
        .timeout(600000).get();  // So Slow (~10 Seconds)
Elements lyricList = doc.getElementsByClass("some-class");

I need only the src of the images. So i need only the plain html text to be loaded.

Is the line slow because of loading images from the URL?

I mean does Jsoup.connect() wait for the whole page to be loaded along with the images?

Answer 1

instead of using Jsoup for fetching and parsing, try combining OkHttp for fetching and Jsoup for parsing

OkHttpClient okHttp = new OkHttpClient();
Request request = new Request.Builder().url("https://example.com").get().build();
Document doc = Jsoup.parse(okHttp.newCall(request).execute().body().string());

it made a great difference in my case, here are the average results for a simple benchmark I ran:

okHttp+Jsoup: 283ms

Jsoup: 476ms

Answer 2

Jsoup connection might become slow because of:

• your internet connection speed or
• CPU usage (Some other program is eating up memory!) or
• the respond speed of the web server you are accessing.

I've been scraping thousands of pages and the above three ( especially the third one ) have been the most likely problems to slow down Jsoup.connect() . In your case I believe it's the web server you are trying to connect to that is slowing down your connection because Jsoup does not wait for images to load, it gets you whatever the initial html response from the server is.