Jsoup: How Do I Load More Webpage Content in Java?

Question

I am currently trying to parse titles from news articles from yahoo news using Jsoup. It works great except for the fact that I only get a limited number of said news articles.

The problem I am trying to tackle is how to get the news that are only loaded once you scroll to the bottom of the webpage when you are in your browser. Is there a way to load those with Jsoup and then parse?

Thanks!

Answer 1

can you give me a reference link that jsoup does not support javascript..cant seem to find one.

Some links from SO itself:

"Jsoup does not support javascript and it does not emulate a browser(...)"
Mosty Mostacho - Getting Jsoup to support dynamically generated html by JavaScript

"JSoup is an HTML parser (...) it's completely unaware of any content that is added to the DOM by Javascript after the initial page load."
fvu - Page content is loaded with javascript and Jsoup doesn't see it

"JSoup (...) is not a web browser. It excels at parsing static html content. It does not run any javascript, so you are out of luck."
luksch - Jsoup get dynamically generated HTML

"JSoup does not process Javascript."
NK - Extracting “hidden” HTML with Jsoup