下一頁未加載

Question

我的代碼無法加載下一頁。 此外，當我手動刷新時，網頁顯示“拒絕訪問”。

options = ChromeOptions()
options.add_argument("headless") # to hide window in 'background'
driver = Chrome(executable_path="C:/Users/samira.zade/AppData/Local/Programs/Python/Driver/chromedriver_win32/chromedriver.exe")
driver.get("https://www.connection.com/IPA/Shop/Product/Search?SearchType=1&term=tp- 
link#1st+Matches~12~List")# here change your link
driver.maximize_window()
time.sleep(5)
wait=WebDriverWait(driver,10)
pagenum = 10
data_connection = []
i = 0
for i in range(pagenum):
     driver.get(f"https://www.connection.com/IPA/Shop/Product/Search?SearchType=1&term=tp-link#{i+1}~Best+Matches~12~List")
     time.sleep(5)
     wait=WebDriverWait(driver,10)

Answer 1

我通常盡量避免使用 Selenium（或者換句話說，將其用作最后的手段）來抓取網站。 產品數據來源來自https://www.connection.com/product/searchpage 。 您可以將頁面查詢作為參數傳入。

我只是用pandas解析表快速給大家看。 如果你想從頁面中提取其他/更多內容，你可以使用 BeautifulSoup 來實現。

import pandas as pd
import requests

url = "https://www.connection.com/product/searchpage"
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

page = 1
continueLoop = True
result_df = pd.DataFrame()
while continueLoop == True:
    
    payload = {
        'SearchType': '1',
        'term': 'tp-link',
        '1st Matches~12~List': '',
        'pageNumber': page,
        'pageSize': '36',
        'url': 'https://www.connection.com/IPA/Shop/Product/Search',
        'mode': 'List'}
    
    response = requests.get(url, headers=headers, params=payload)
    if 'Pagination limit reached.' in response.text:
        continueLoop = False
        print('Pagination limit reached.')
        continue
    
    df = pd.read_html(response.text)[0]
    result_df = result_df.append(df).reset_index(drop=True)
    print(f'Collected page: {page}')
    page+=1

Output：

print(result_df)
           Product Image  ...                                        Price
0    Compare  Image Link  ...   $38.70 Qty:  Add To Cart  Add to Quicklist
1    Compare  Image Link  ...   $55.42 Qty:  Add To Cart  Add to Quicklist
2    Compare  Image Link  ...   $19.04 Qty:  Add To Cart  Add to Quicklist
3    Compare  Image Link  ...   $52.03 Qty:  Add To Cart  Add to Quicklist
4    Compare  Image Link  ...  $104.07 Qty:  Add To Cart  Add to Quicklist
..                   ...  ...                                          ...
175  Compare  Image Link  ...   $73.98 Qty:  Add To Cart  Add to Quicklist
176  Compare  Image Link  ...   $99.77 Qty:  Add To Cart  Add to Quicklist
177  Compare  Image Link  ...   $24.99 Qty:  Add To Cart  Add to Quicklist
178  Compare  Image Link  ...   $24.99 Qty:  Add To Cart  Add to Quicklist
179  Compare  Image Link  ...   $44.99 Qty:  Add To Cart  Add to Quicklist

[180 rows x 4 columns]