Jsoup not seeing some text on website

Question

Currently I am making a program (in Java) that grabs all the streamers on twitch (Videogame streaming site) from a given URL eg and lists them into a text file using Jsoup.

However, no matter what I try, it seems like I can't get the streamer's names. After a while I discovered that the page source for some reason does not contain the streamer's names which I think could be the problem?

Here is my code currently.

public static void main(String[] args) throws IOException {
    int i = 0;

    PrintWriter streamerwriter = new PrintWriter("streamer.txt", "UTF-8");
    Document doc = Jsoup.connect(https://www.twitch.tv/directory/game/Hearthstone%3A%20Heroes%20of%20Warcraft).get();
    Elements streamers = doc.getElementsByClass("js-profile-link");

    for (Element streamer : streamers) {
        i++;
        System.out.println(i + "." + streamer.text());
        streamerwriter.println(i + "." + streamer.text());
    }

    streamerwriter.close();
}

Any help would be greatly appreciated.

Answer 1

you don't need to parse webpage.Because twitch has an api to select streamers.

https://streams.twitch.tv/kraken/streams?limit=20&offset=0&game=Hearthstone%3A+Heroes+of+Warcraft&broadcaster_language=&on_site=1

so you should parse json data

if you want to know why you don't see streamers in jsoup because of lazy loading.because that part you want to parse is loaded lazily.You should know that lazy request and parse that url by jsoup which i found and wrote up.(twitch api)

Please check this question out: how to use Jsoup in site that has lazyload scrollLoader.js