刷新后如何從網頁中抓取數據

Question

我編寫了一個腳本來獲取在線購物網站中的所有產品。

每當您像往常一樣向下滾動時，網站會為您提供更多項目，因此我無法從網頁上獲得足夠的產品。

我怎樣才能得到我想要的“產品”？

這是我當前的代碼：

from bs4 import BeautifulSoup
import requests 

url = "https://www.trendyol.com/erkek-t-shirt-x-g2-c73"
html_text = requests.get(url).text
main_soup = BeautifulSoup(html_text, 'lxml')
all_items = main_soup.find_all('div', class_="p-card-wrppr with-campaign-view")

for item in all_items:
    title = item.find('div', class_="prdct-desc-cntnr-ttl-w two-line-text").text
    print(title)
    print()

Answer 1

數據通過 Javascript 通過他們的 Rest API 加載，因此您可以發出相同的請求來獲取信息：

import requests

api_url = "https://public.trendyol.com/discovery-web-searchgw-service/v2/api/infinite-scroll/erkek-t-shirt-x-g2-c73"

params = {
    "pi": "1",
    "culture": "tr-TR",
    "userGenderId": "1",
    "pId": "0",
    "scoringAlgorithmId": "2",
    "categoryRelevancyEnabled": "false",
    "isLegalRequirementConfirmed": "false",
    "searchStrategyType": "DEFAULT",
    "productStampType": "A",
    "fixSlotProductAdsIncluded": "false",
    "searchAbDeciderValues": "",
}


page = 1
while True:
    params['pi'] = page
    data = requests.get(api_url, params=params).json()

    if not data.get('result', {}).get('products'):
        break

    for p in data['result']['products']:
        print('{:<15} {}'.format(p['id'], p['name']))
    print()

    page += 1

印刷：

311550273       Erkek Nem Emici Hızlı Kuruma Atletik Teknik Performans T-shirt
114225365       Erkek Sarı Mikro Polyester Performans Antrenman Sporcu Tişört
35907408        Dry Park VII BV6708-010 Erkek Tişört
101771784       Erkek Siyah-Beyaz-Antrasit 3'lü Bisiklet Yaka Düz T-Shirt E001010
39501961        Erkek Siyah Pis Yaka Salaş T-shirt
311550264       Erkek Nem Emici Hızlı Kuruma Atletik Teknik Performans T-shirt
95401172        Erkek Koyu Lacivert Pike Kısa Kollu Basic Tişört
339923963       Fit NBA Brooklyn Nets Regular Fit Bisiklet Yaka Tişört
62247632        Ua Big Logo Ss - 1329583-600
101770796       Erkek Siyah 2'li Bisiklet Yaka %100 Pamuk Basic T-Shirt E001011
270985631       Erkek Beyaz Pis Yaka Salaş T-shirt
96531669        Logo Baskılı Kırmızı Tişört Slim Fit / Dar Kesim 065781-33099
101771002       Erkek Beyaz 2'li Bisiklet Yaka %100 Pamuk Basic T-Shirt E001012
382858755       Sw Tshirt | Bordo
336669786       Fit NBA Brooklyn Nets Oversize Fit Kapüşonlu Tişört
339562039       Fit NBA Golden State Warriors Boxy Fit Bisiklet Yaka Tişört
347257450       Oversize Dragon Vs Phoenix Unisex T-shirt
443287890       Oversize Bisiklet Yaka Les Benjamınsbakılı Tshirt
340645587       TULIO GRİ T-SHIRT S/S TEE
348227065       Fenerbahçe Sk Unisex Mavi Futbol Tişört 77313601
443316559       Oversize Bisiklet Yaka Les Benjamınsbakılı Tshirt
293975462       Chaos Karma Baskı Oversize Siyah Unisex Tshirt
249703589       Unisex Chicago Özel Baskılı Oversize Penye T-shirt Tişört
301833907       Unisex Distraction Siyah Tshirt

302756840       Unisex First Class Beyaz Tshirt
382195053       Wreck The World Oversize | Beige
321828215       Erkek 5'li Paket Dry Fit Siyah Lacivert Beyaz Haki Gri Atletik Nem Emici Günlük Tshirt
90315375        Logo Baskılı Siyah Tişört Slim Fit / Dar Kesim 065781-900

...and so on.