無法從需要搜索輸入的網站抓取 div 標簽內的數據

Question

我是 web 抓取的新手，我想從此網站搜索結果中獲取相關項目的名稱。 項目名稱在 h4 標簽內

該網站需要登錄和密碼才能查看項目的詳細信息，但我只想獲取所有項目的列表。

環顧四周后，我意識到要抓取結果，我必須從代碼中提供輸入。 我正在使用的代碼如下

import requests
from bs4 import BeautifulSoup as bs

headers = {
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-User': '?1',
    'Referer': 'https://www.devex.com/login?return_to=https%3A%2F%2Fwww.devex.com%2Ffunding%2Fr%3Freport%3Dgrant-21475%26query%255B%255D%3Dbig%2Bdata%26filter%255Bstatuses%255D%255B%255D%3Dforecast%26filter%255Bstatuses%255D%255B%255D%3Dopen%26filter%255Bupdated_since%255D%3D2019-09-02T04%253A57%253A27.714Z%26sorting%255Border%255D%3Ddesc%26sorting%255Bfield%255D%3D_score',
}

params = (
    ('report', 'grant-21475'),
    ('query[]', 'big data'),
    ('filter[statuses][]', ['forecast', 'open']),
    ('filter[updated_since]', '2019-09-02T04:57:27.714Z'),
    ('sorting[order]', 'desc'),
    ('sorting[field]', '_score'),
)

response = requests.get('https://www.devex.com/funding/r', headers=headers, params=params)

result = bs(response.text, 'html.parser')

我得到的結果不包含必需的標簽或信息。 請告訴我。

謝謝，

Answer 1

內容從 API 調用動態返回，您可以在瀏覽器的網絡選項卡中找到

import requests

r = requests.get('https://www.devex.com/api/funding_projects?query[]=big+data&filter[statuses][]=forecast&filter[statuses][]=open&filter[updated_since]=2019-09-03T14:27:15.234Z&page[number]=1&page[size]=1000&sorting[order]=desc&sorting[field]=_score').json()
titles = [project['title'] for project in r['data']]
print(len(titles))

您可以在循環中更改頁面參數和頁面大小。 第一個請求會告訴您總共有多少結果。 在這種情況下，我只是簡單地輸入了一個大於預期結果計數的數字。

示例循環：

import requests
import math

titles = []
page_size = 500

with requests.Session() as s:
    r = s.get(f'https://www.devex.com/api/funding_projects?query[]=big+data&filter[statuses][]=forecast&filter[statuses][]=open&filter[updated_since]=2019-09-03T14:27:15.234Z&page[number]=1&page[size]={page_size}&sorting[order]=desc&sorting[field]=_score').json()
    total = int(r['total'])
    titles += [project['title'] for project in r['data']]
    if total > page_size:
        num_pages = math.ceil(total/page_size)
        for page in range(2, num_pages+1):
            r = s.get(f'https://www.devex.com/api/funding_projects?query[]=big+data&filter[statuses][]=forecast&filter[statuses][]=open&filter[updated_since]=2019-09-03T14:27:15.234Z&page[number]={page}&page[size]={page_size}&sorting[order]=desc&sorting[field]=_score').json()
            titles += [project['title'] for project in r['data']]
print(len(set(titles)))