jsoup not scraping all elements?

Question

jsoup

When scraping data from website that has 200 elements , the Output is only the first 49 or 50 elements of the 200 elements , why ? - how can i solve this prob. to get all 200 elements data ?``

Document d = Jsoup.connect("https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2047675.m570.l1313.TR0.TRC0.H0.Xsilver.TRS0&_nkw=silver&_sacat=0").timeout(6000).get();
Elements ele = d.select("div#ResultSetItems");

for (Element element : ele.select("li.shic")) {
    String img_url = element.select("li").attr("listingid");
    System.out.println(img_url);
}

Answer 1

The other elements are fetched on demand by Javascript as is common these days, so they're invisible to JSoup. There is no way to have JSoup perform those fetches, so you're going to have to come up with a better way than scraping to get that data. I suggest you look at API options that EBay offers .