python使用selenium发送空格键滚动和漂亮的汤解析html

Question

browser.get("https://steamcommunity.com/app/933110/reviews/?browsefilter=toprated&snr=1_5_100010_")
    url = browser.current_url
    page = requests.get(url).text
    soup = BeautifulSoup(page, 'html.parser')
    actions = ActionChains(browser)

    for y in range(50):
        for x in soup.find_all("div", {"class": "apphub_CardTextContent"}):
            print(x.text.strip() + "\n")
            actions.send_keys(Keys.SPACE).perform()
            time.sleep(1)

I am trying to get the text of every review of this game on the Steam store. There are over 2000 reviews. The function I attempted to write will get the same reviews that are within the parsing area of what is loaded but the 1900 other reviews are not being printed. I need the code to scroll down and get every review. I used 50 as a test but unsure how many space bars I will need to get to the bottom of the page. The main problem is that when I print the HTML using prettify() the HTML doesn't show every review.

Answer 1

try this code with requests and BS4

import selenium
import requests
from bs4 import BeautifulSoup
import requests

for i in range (0,10):
    url = "https://steamcommunity.com/app/933110/homecontent/?userreviewscursor=AoIIPwcQI3XbjqgC&userreviewsoffset={0}&p={1}&workshopitemspage={1}&readytouseitemspage={1}&mtxitemspage={1}&itemspage={1}&screenshotspage={1}&videospage={1}&artpage={1}&allguidepage={1}&webguidepage={1}&integratedguidepage={1}&discussionspage={1}&numperpage={1}&browsefilter=toprated&browsefilter=toprated&appid=933110&appHubSubSection=10&appHubSubSection=10&l=english&filterLanguage=default&searchText=&forceanon=1".format(i*10,i+1)
    payload = {}
    headers= {}
    response = requests.request("GET", url, headers=headers, data = payload)
    soup = BeautifulSoup(response.text.encode('utf8'),features="html.parser")
    for x in soup.find_all("div", {"class": "apphub_CardTextContent"}):
        print(x.text.strip() + "\n")

Change range value accordingly, for range (1,10) first 10 pages posts will be displayed.

Answer 2

Re: scrolling to the bottom of the page consistently, I've found it's easiest to use execute_script and let javascript do that for you. That would look like this in your example:

browser.execute_script("scroll(0, document.body.parentNode.scrollHeight)")

Re: not seeing all of the claims, I think the problem is that you define soup before your loop. I believe this means that when you call soup.find_all() inside the loop, you only ever parse that initial page you obtained with page = requests.get(url).text . Since Selenium and Beautiful Soup aren't connected, you'll have to make a new request for every new section of reviews in order to load the response into soup and parse from there.

If you want to go the Selenium route, you could do the scrolling and get the text from the elements using selenium's methods: browser.find_elements_by_class_name("apphub_CardTextContent") , and then get the text when you're looping with element.get_attribute("innerText")

I would lean more towards just using requests to get the html documents and parse with Beautiful Soup, just because that means less DOM interaction, which usually means fewer bugs (in my experience). However, from looking at the site, it seems like you might have to dig in to replicating their requests for the html of the extra reviews.