为什么我在使用 selenium 时只获取首页数据？

Question

I use the python package selenium to click the "load more" button automatically, which is successful. But why do I cannot get data after "load more"?

I want to crawl reviews from imdb using python. It only displays 25 reviews until I click "load more" button. I use the python package selenium to click the "load more" button automatically, which is successful. But why do I cannot get data after "load more" and just get the first 25 reviews data repeatedly?

import requests
from bs4 import BeautifulSoup
from selenium import webdriver      
import time



seed = 'https://www.imdb.com/title/tt4209788/reviews'
movie_review = requests.get(seed)
PATIENCE_TIME = 60
LOAD_MORE_BUTTON_XPATH = '//*[@id="browse-itemsprimary"]/li[2]/button/span/span[2]' 

driver = webdriver.Chrome('D:/chromedriver_win32/chromedriver.exe')
driver.get(seed)

while True:
    try:
        loadMoreButton = driver.find_element_by_xpath("//button[@class='ipl-load-more__button']")

        review_soup = BeautifulSoup(movie_review.text, 'html.parser')
        review_containers = review_soup.find_all('div', class_ ='imdb-user-review')
        print('length: ',len(review_containers))
        for review_container in review_containers:
            review_title = review_container.find('a', class_ = 'title').text
            print(review_title)

        time.sleep(2)
        loadMoreButton.click()
        time.sleep(5)
    except Exception as e:
        print(e)
        break

print("Complete")

I want all the reviews, but now I can only get the first 25.

Answer 1

You have several issues in your script. Hardcoded wait is very inconsistent and certainly the worst option to comply. The way you have written your scraping logic within while True: loop, will slower the parsing process by collecting the same items over and over again. Moreover, every title produces a huge line gap in the output which needs to be properly stripped. I've slightly changed your script to reflect the suggestion I've given above.

Try this to get the required output:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

URL = "https://www.imdb.com/title/tt4209788/reviews"

driver = webdriver.Chrome()
wait = WebDriverWait(driver,10)

driver.get(URL)
soup = BeautifulSoup(driver.page_source, 'lxml')

while True:
    try:
        driver.find_element_by_css_selector("button#load-more-trigger").click()
        wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR,".ipl-load-more__load-indicator")))
        soup = BeautifulSoup(driver.page_source, 'lxml')
    except Exception:break

for elem in soup.find_all(class_='imdb-user-review'):
    name = elem.find(class_='title').get_text(strip=True)
    print(name)

driver.quit()

Answer 2

Your code is fine. Great even. But, you never fetch the 'updated' HTML for the web page after hitting the 'Load More' button. That's why you are getting the same 25 reviews listed all the time.

When you use Selenium to control the web browser, you are clicking the 'Load More' button. This creates an XHR request (or more commonly called AJAX request) that you can see in the 'Network' tab of your web browser's developer tools.

The bottom line is that JavaScript (which is run in the web browser ) updates the page. But in your Python program, you only get the HTML once for the page statically using the Requests library.

seed = 'https://www.imdb.com/title/tt4209788/reviews'
movie_review = requests.get(seed) #<-- SEE HERE? This is always the same HTML. You fetched in once in the beginning.
PATIENCE_TIME = 60

To fix this problem, you need to use Selenium to get the innerHTML of the div box containing the reviews. Then, have BeautifulSoup parse the HTML again. We want to avoid picking up the entire page's HTML again and again because it takes computation resources to have to parse that updated HTML over and over again.

So, find the div on the page that contains the reviews, and parse it again with BeautifulSoup. Something like this should work:

while True:
    try:
        allReviewsDiv = driver.find_element_by_xpath("//div[@class='lister-list']")
        allReviewsHTML = allReviewsDiv.get_attribute('innerHTML')
        loadMoreButton = driver.find_element_by_xpath("//button[@class='ipl-load-more__button']")
        review_soup = BeautifulSoup(allReviewsHTML, 'html.parser')
        review_containers = review_soup.find_all('div', class_ ='imdb-user-review')
        pdb.set_trace()
        print('length: ',len(review_containers))
        for review_container in review_containers:
            review_title = review_container.find('a', class_ = 'title').text
            print(review_title)

        time.sleep(2)
        loadMoreButton.click()
        time.sleep(5)
    except Exception as e:
        print(e)
        break