使用 jsoup 进行网页抓取仅返回表格的一部分

Question

I am new at coding.
I'm trying to webscrape a table with a list of funds from a broker's website. The code is working fine but ut is returning only part of the list (a bit more then the first half of the list), and I can't find out why

I've already checked the html strucutre and the tags and everything seems to be right...

int count = 0;
String URL = "https://institucional.xpi.com.br/investimentos/fundos-de-investimento/lista-de-fundos-de-investimento.aspx";

try {
    Document doc = Jsoup.connect(URL).userAgent("Mozilla/17.0").get();
            
for (Element table: doc.select("#tableTodos tr")) {
    Elements tds = table.getElementsByTag("td");
    if (tds.size() > 0) {
        count++;
        System.out.println(count + " - " + tds.get(2).text());
}

This is the final part of the return

138 - Kapitalo Kappa FIN FIC FIM
139 - Kapitalo Tarkus FIC FIA
140 - Kinea Atlas II FIM
141 - Kinea Chronos FIM
142 - Kinea RF Absoluto FI LP
143 - Leblon Ações FIC FIA
144 - Legacy Capital Advisory FIC FIM
145 - Legg Mason Clearbridge US Large Cap Growth FIA IE
146 - Legg Mason Martin Currie European Absolute Alpha FIM IE
147 - Mauá Capital Ações FIC FIA

It goes only to 147, the table at the website has more than 300 rows...

Answer 1

you should search the elements in the table by "tr" tag and not td. It will give you all the rows in the table. then, in each row, search for the td and print it's text.

EDIT 1:

    ChromeOptions chromeOptions = new ChromeOptions();
    chromeOptions.addArguments("--headless");
    chromeOptions.addArguments("--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.2840.100 Safari/537.36");
    driver = new ChromeDriver(chromeOptions);
    chromeDriver.get("https://institucional.xpi.com.br/investimentos/fundos-de-investimento/lista-de-fundos-de-investimento.aspx");
    List<WebElement> elements = chromeDriver.findElement(By.xpath("//*[@id=\"tableTodos\"]")).findElements(By.tagName("tr"));
    System.out.println(elements.get(200).getText());

EDIT 2:

Add maxBodySize yo your get call:

Document doc = Jsoup.connect(URL).timeout(0).maxBodySize(0).get();