使用BeautifulSoup并使用不变的网址来抓取网站

Question

I've webscraped before but I'm running into some issues I haven't seen before when trying to scrape from RottenTomatoes/search. The issue is twofold. (I'm waiting for my API request to be 'validated', and Rotten Tomatoes doesn't have a list of all movies unfortunately, ugh)

1. There's a "More Movies" link on the bottom right of the page that has to be "clicked" to bring up the movies. As far as I know, Python doesn't have something like that to interact with that... or does it?

2. Even when the "More Movies" link is clicked, the URL at the top doesn't change when I'm trying to navigate/iterate through all the pages. This seems like a problem for BeautifulSoup.

Any recommendations/tips?

Answer 1

This is indeed not directly possible by using beautifulsoup, since beautifulsoup handles static webpages. The content you want to crawl is being added to the page via JavaScript, rather than baked into the HTML.

The button 'More movies' calls a javascript function that will probably have some AJAX function call for more movies.

There are a few scenario's where you can access the 'more movies' easily:

• sometimes the data is already present in the source, but hidden. The javascript makes this visible
• The javascript uses a api to load its content, this API url can then be found in the source code and you can find what you are looking for if you go directly to that link.

However none of the above seems to be the case for rottentomatoes. I gave it a quick look, perhaps you should investigate it more thorough.

A solution I've used in the past is: Selenium . This has a python library that is easy to use and that allows you to automate browser behaviour. This way you can 'automatically' click the load more button while crawling.

Beware however, this might be slow and cost resources. You can run it headless, which makes it not open a browser and saves some of the resources.