网页抓取未加载完整页面

Question

我试图从网页中获取前 100 个结果（但只获取前 20 个结果）： https : //www.usnews.com/education/best-high-schools/search?national-rank-range- min=1&national-rank-range-max=100

使用了以下代码：

from selenium import webdriver
from bs4 import BeautifulSoup
import time

url = "https://www.usnews.com/education/best-high-schools/search?national-rank-range-min=1&national-rank-range-max=100"
driver = webdriver.Chrome()
driver.get(url)
time.sleep(10)  
scroll_pause_time = 1 
screen_height = driver.execute_script("return window.screen.height;")  
i = 1
print(screen_height)

while True:
    driver.execute_script("window.scrollTo(0, {screen_height}*{i});".format(screen_height=screen_height, i=i))  
    i += 1
    time.sleep(scroll_pause_time)
    # update scroll height each time after scrolled, as the scroll height can change after we scrolled the page
    scroll_height = driver.execute_script("return document.body.scrollHeight;")  
    # Break the loop when the height we need to scroll to is larger than the total scroll height
    if (screen_height) * i > scroll_height:
        break
    
while True:
    try:
        loadmore = driver.find_element_by_id("pager__ButtonContentContainer-sc-1i8e93j-3 zIUhv")
        loadmore.click()
    except:
        print("Reached bottom of page")
        break


html_source = driver.page_source  


soup = BeautifulSoup(html_source,'html.parser') 

...

我尝试了不同的方法，但没有任何东西通过自动化完全加载页面。 甚至视图源也只显示前 20 个结果。 我希望获得前 100 个结果。

Answer 1

您可以通过以下方式使用其 API：

import requests
for i in range(1,6):
    headers = {
        'authority': 'www.usnews.com',
        'pragma': 'no-cache',
        'cache-control': 'no-cache',
        'sec-ch-ua': '"Chromium";v="94", "Microsoft Edge";v="94", ";Not A Brand";v="99"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'dnt': '1',
        'upgrade-insecure-requests': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36 Edg/94.0.992.50',
        'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
        'sec-fetch-site': 'none',
        'sec-fetch-mode': 'navigate',
        'sec-fetch-user': '?1',
        'sec-fetch-dest': 'document',
        'accept-language': 'en-US,en;q=0.9',
        'cookie': 'usn_visitor_id=754811605664000055a6716138000000471a0000; akacd_www=2177452799~rv=84~id=9abf79d7c5923d3543064d4f3aa45796; _abck=D2BB8DCBA72E28218DD3167A63B0C097~-1~YAAQdUgRYGT7f598AQAAQLzxowY/eA9B9gxVXf1nRYdr1kOohrAwtu9lBLnqBJKZ1nfBDuFcKzUSquGl6S2dTQTdf08PcbRthYlRp6MVt7iAznQsX2cWT88Co+4CepTDBYnYgq13YYaoXBhrBkOiJFfdnYGfhJwnPCRzS25BDBIeU6gIXS9yyhGwYpry6CesOJsfG72jhRThkMzThjiRPXmru0i4OGwK/Mra6bAusUL4QdMT2Hb2uo1xOyhq3kjpdXkzXd/s0lcu0iSjiUSNwP3uN1PnBvOaX9gcruandqXQUhvYjTOAyACoo7vmQa7QUEyAdjb5jRogCdSGzKnPUBfoWIvLSNIdbyAvhEPiObEdl9kZx4mNoIqUeMVu~-1~-1~-1; ak_bmsc=C269F04838BD1A8894743CB8D48975F1~000000000000000000000000000000~YAAQdUgRYGX7f598AQAAQLzxow008kEQ1RsIl5Q1CkSl9jcquYeGilo4jPNgnWNgtjUNFBYDX0X3rC4M3S9D1XNKPhyIWm9nD8vRqGVPGLtFwRFd4HIwu1rUjUBvUEoibbKU9GcOvYQRSUTKoWo2IsE01layvTpK820IyEQuvjBMI74Pc6oxa1BHfXoGsYFmPdkAnz5+7sDPHiEYoUtDrOv1ZTRgadHKeEOtmRPXn5LI/8XBSc2ZuKUkico3mtj8dzukt9JVZWtMJoBwSCQ6ggKptCsuBHwRNl/TpI08Sc/Tm4+e3n41kd3vleJkaU7cmMSTKdivKA0K5I6Uo2EWFimtUP+wImPTwTbVHqTaCLzNuEybsvL/N7F3PMcQ+8tlQEZTJyBgkjcNmyo5; bm_sz=356684A0E8BF30F45CC57FF24454493F~YAAQdUgRYGb7f598AQAAQLzxow2PpE5OzxA/IddRTwC9NeGi/hLUn5mrZMwrvEttWCQOAyeZpGiZd8mRJL2g/Fr3r9XOlUWggRSPXdYXUBg9hnpS0PlG6/ClS6FbsHbKi29g00TgDpQ11ZDFX2OYdxCbYolQ+suZcT1l0UCPwBuF5oOwWd/VZXWpTm64FU95DrOGzLUHvbc+c5uuI1hnAIu5sWUXUp7F6E+AQZv0QX2RoM7p0ZKfrO2Xmq2q7srG8IB/xZ/emJ4O0kiJOM7tjRU7p9nBQuMLY79iYeJrjfQQsEA=~4277043~4408883',
    }

    params = (
        ('format', 'json'),
        ('national-rank-range-min', '1'),
        ('national-rank-range-max', '100'),
        ('page', i),
    )
    response = requests.get('https://www.usnews.com/education/best-high-schools/search', headers=headers, params=params)
    for i in response.json()['matches']:
        print(i['name'])