使用 Python 使用 Javascripts 函数抓取网页

Question

I need to retrieve some information from a webpage. The page is like a powerpoint: several slides are shown one by one. To move from one slide to another you have to press a button that runs a js function " load_image_btn('plus') " which change the image. The URL is exactly the same, and the HTML code only changes de URL of the img " someurl/546 ".

Is there any way to execute that function from python iteratively so I can get all the images?

Answer 1

One generic way to cope with Javascript-induced problems is to use a headless browser to fully execute each page and then scrape from there.

For my last similar project I used a service that provides instances of headless webbrowsers that can be controlled via API, namely https://scrapinghub.com/splash .

But I am sure there are many alternatives.