试图 web 抓取动态网页，没有得到错误代码。 如何验证“它正在工作”？

Question

我正在尝试使用以下代码对动态网站进行网页抓取。 我最初的几次尝试返回了一些错误消息，例如“return self.find_element(by=By.CLASS_NAME, value=name)”和一些“消息：没有这样的元素：无法找到元素”。 但是，现在它一直在工作，但没有给我想要的结果。 我应该定位什么元素？ 我的 selenium 和 chromedriver 的设置是否可能有问题（最初也遇到了问题）。

   from selenium import webdriver
   url = 'https://www.landers.ph/beverages/coffee.html'
   driverlocation = '/Users/username/Downloads/chromedriverdriver = 
   webdriver.Chrome(driverlocation)
   driver.get(url)
   products = driver.find_elements_by_class_name('ld-product-detail-info')

   for product in products:
       name = product.find_element_by_class_name('productTitle').text
       price = product.find_element_by_class_name('priceComponent').text
       print(name,price)

Answer 1

诱导WebDriverWait () 并等待visibility_of_all_elements_located () 并遵循class name

driver.get('https://www.landers.ph/beverages/coffee.html')
products=WebDriverWait(driver,20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME,"productComponent")))
for product in products:
    name = product.find_element_by_class_name('productTitle').text
    price = product.find_element_by_class_name('priceComponent').text
    print(name,price)

Output ：

Cafe Puro 100% Pure Instant Coffee in Mason Jar 90g ₱99.75
Cafe Puro 3-in-1 Instant Coffee 20 x 17g ₱85.75
San Mig Sugar Free White 3 in 1 Coffee 10 x 9g ₱61.75
Nescafe Classic Double Filter Instant Coffee Pack 200g ₱144.75
Nescafe Original Instant Coffee Mix 840g ₱171.75
Nescafe Creamy White 3-in-1 Instant Coffee 870 g ₱175.75
San Mig Super 3-in-1 Sugar Free Coffee 20 pcs x 7 g ₱111.75
Good Day Vanilla Latte 3 in 1 Instant Coffee 30pcs. x 20 g ₱139.75
Good Day Cappuccino with Coklat Granule 30 x 25g ₱199.75
Gold Choice Original Coffee Mix 20x 20g ₱149.75
Kopiko Black 3-in-1 Instant Coffee Mix 30 x 30g ₱209.25
San Mig Original 3-in-1 Coffee Mix 30 x 20g ₱159.75
Nescafe Gold Latte Macchiato 10 x 20g ₱169.75
Nescafe Classic Decaf Resealable Pack Instant Coffee 160 g ₱162.75
Cafe Puro Jade Blend Fresh Ground Coffee in a Tin Can 400g ₱188.75
Kopiko Brown Coffee Mix 30 x 27.5g ₱210.75
Pure Nature 12-in-1 Green Coffee Mix 10 x 21g ₱279.75
Kopiko Blanca Creamy Coffee Mix 30 x 30g ₱210.75
Folgers Classic Medium Roast Ground Coffee 320 g ₱349.75
Kopiko Cappuccino Coffee Mix 30 x 25g ₱210.75
Check Hup 3-in-1 Ipoh White Coffee King 15 x 40 g ₱318.75
Nescafe Taster's Choice French Roast Instant Coffee 198g ₱619.75
Nescafe Taster's Choice House Blend Decaf Coffee 7 oz ₱679.75
Maxwell House Cappuccino 750 g ₱899.75

您需要导入以下库。

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By