Java get javascript content from webpage

Question

I'm looking to get data from Betfair live stats of football. I have found a link to the stats part of the webpage, which is here:

https://wab-visualisation.performgroup.com/csb/index.html?wbuserId=0&token=f6eb1c7fff3ad3ca5153b159ac4b8b69fcb117ec7578893e5c9f9f6a608503fe4e64ebad85c7e4faa8ddea4e8ce70e99ce88551b77b8fe02616b56bf89af950ad89984a1cfcc8eb589dd1679b8204e8e0983fdff5c7599d00cabd805c0bd18aaf29485b0c122413c89ad5f7e70684e5f1609dea97f05286919f137baab64d36b&width=374&height=214&cssdiff=https%3a%2f%2fassets.cdnbf.net%2fstatic%2fdatavis%2fbf-css%2fbetfair1.css&flash=y&streamonly=true&partnerId=7&statsswitch=false&lang=en&defaultview=stats&version=1.15

(The page changes regularly, so please check https://www.betfair.com/sport/inplay and select any match, then get the source of the "match panel" on the left)

However, when I try to scrape this with JSoup, it doesn't seem to return anything.

    Document doc = Jsoup.connect("https://wab-visualisation.performgroup.com/csb/index.html?wbuserId=0&token=f6eb1c7fff3ad3ca5153b159ac4b8b69fcb117ec7578893e5c9f9f6a608503fe4e64ebad85c7e4faa8ddea4e8ce70e99ce88551b77b8fe02616b56bf89af950ad89984a1cfcc8eb589dd1679b8204e8e0983fdff5c7599d00cabd805c0bd18aaf29485b0c122413c89ad5f7e70684e5f1609dea97f05286919f137baab64d36b&width=374&height=214&cssdiff=https%3a%2f%2fassets.cdnbf.net%2fstatic%2fdatavis%2fbf-css%2fbetfair1.css&flash=y&streamonly=true&partnerId=7&statsswitch=false&lang=en&defaultview=stats&version=1.15").get();

    doc.select("#in-game-stats > div:nth-child(1) > div.si-home-value").text();

Am I doing something wrong? Is there a way around this?

Thanks

Answer 1

The response is in plain text, there is no DOM that you can query. You might want to look at Selenium with PhantomJS driver to emulate user-agent actions. Whether it's legal to scape data from that site or not, is another question.