Soup works on one IMBD page but not on another. How to solve?

Question

url1 = "https://www.imdb.com/user/ur34087578/watchlist"
url = "https://www.imdb.com/search/title/?groups=top_1000&ref_=adv_prv"

results1 = requests.get(url1, headers=headers)
results = requests.get(url, headers=headers)
soup1 = BeautifulSoup(results1.text, "html.parser")
soup = BeautifulSoup(results.text, "html.parser")

movie_div1 = soup1.find_all('div', class_='lister-item-content')
movie_div = soup.find_all('div', class_='lister-item mode-advanced')
#using unique tag for each movie in the respective link

print(movie_div1)
#empty list
print(movie_div)
#gives perfect list

Why is movie_div1 giving an empty list? I am not able to identify any difference in the URL structures to indicate the code should be different. All leads appreciated.

Answer 1

Unfortunately the div you want is processed by a javascript code so you can't get by scraping the raw html request.

You can get the movies you want by the request json your browser gets, which you won't need to scrape the code with beautifulsoup, making your script much faster.

2nd option is using Selenium.

Good luck.

Answer 2

As @SakuraFreak mentioned, you could parse the JSON received. However, this JSON response is embedded within the HTML itself which is later converted to HTML by browser JS (this is what you see as <div class="lister-item-content">...</div> .

For example, this is how you would extract the JSON content from the HTML to display movie/show names from the watchlist:

import requests
from bs4 import BeautifulSoup
import json

url = "https://www.imdb.com/user/ur34087578/watchlist"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

details = str(soup.find('span', class_='ab_widget'))

json_initial = "IMDbReactInitialState.push("
json_leftover = ");\n"

json_start = details.find(json_initial) + len(json_initial)
details = details[json_start:]
json_end = details.find(json_leftover)

json_data = json.loads(details[:json_end])

imdb_titles = json_data["titles"]
for item in imdb_titles.values():
    print(item["primary"]["title"])