使用 JSOUP 读取 HTML 页面并创建文本文件

Question

I am trying to read the IMDB list of top 50 movies. The code is working fine, but it stops reading at number 43 out of a list of 50.

public class FetchData {

public static void main(String[] args) {
    try {
        // TODO code application logic here
        Document doc = Jsoup.connect("https://www.imdb.com/list/ls053181721/").userAgent("Mozilla/17.0").get();
        Elements temp = doc.select("div.lister-item-content");

        int i=0;
        File file = new File("C:\\Demo Java\\IMDBList.txt");
        FileWriter writer = new FileWriter(file);
        for(Element movieList : temp) {
            i++;
            System.out.println(i+" "+movieList.getElementsByTag("a").first().text());
            writer.write(+i+". "+movieList.getElementsByTag("a").first().text().toString()+"\n");

        }
        writer.close();
    } catch (IOException ex) {
        Logger.getLogger(FetchData.class.getName()).log(Level.SEVERE, null, ex);
    }

}

Answer 1

The html document you are loading through Jsoup does not load entirely, as the it exceeds the default maximum body size of 1MB. You need to increase the maximum allowed body size of the request in order to load the complete document.

Document doc = Jsoup.connect("https://www.imdb.com/list/ls053181721/")
                    .userAgent("Mozilla/17.0")
                    .maxBodySize(0)
                    .get();

Note: Adding maxBodySize(0) allows unlimited size.

Please Refer: https://jsoup.org/apidocs/org/jsoup/Connection.html#maxBodySize-int-