简体   繁体   中英

Trying to web scrape a dynamic webpage, didn't get an error code. How can I verify “it's working”?

I'm trying to webscrape a dynamic website with the following code below. My first few attempts returned some error messages like "return self.find_element(by=By.CLASS_NAME, value=name)" and some "Message: no such element: Unable to locate element". However, now it has been working but hasn't given me the results I want. What element should I be targeting? Is it possible something is wrong with my setup of selenium and chromedriver(had trouble with these initially too).

   from selenium import webdriver
   url = 'https://www.landers.ph/beverages/coffee.html'
   driverlocation = '/Users/username/Downloads/chromedriverdriver = 
   webdriver.Chrome(driverlocation)
   driver.get(url)
   products = driver.find_elements_by_class_name('ld-product-detail-info')

   for product in products:
       name = product.find_element_by_class_name('productTitle').text
       price = product.find_element_by_class_name('priceComponent').text
       print(name,price)

Induce WebDriverWait () and wait for visibility_of_all_elements_located () and following class name

driver.get('https://www.landers.ph/beverages/coffee.html')
products=WebDriverWait(driver,20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME,"productComponent")))
for product in products:
    name = product.find_element_by_class_name('productTitle').text
    price = product.find_element_by_class_name('priceComponent').text
    print(name,price)

Output :

Cafe Puro 100% Pure Instant Coffee in Mason Jar 90g ₱99.75
Cafe Puro 3-in-1 Instant Coffee 20 x 17g ₱85.75
San Mig Sugar Free White 3 in 1 Coffee 10 x 9g ₱61.75
Nescafe Classic Double Filter Instant Coffee Pack 200g ₱144.75
Nescafe Original Instant Coffee Mix 840g ₱171.75
Nescafe Creamy White 3-in-1 Instant Coffee 870 g ₱175.75
San Mig Super 3-in-1 Sugar Free Coffee 20 pcs x 7 g ₱111.75
Good Day Vanilla Latte 3 in 1 Instant Coffee 30pcs. x 20 g ₱139.75
Good Day Cappuccino with Coklat Granule 30 x 25g ₱199.75
Gold Choice Original Coffee Mix 20x 20g ₱149.75
Kopiko Black 3-in-1 Instant Coffee Mix 30 x 30g ₱209.25
San Mig Original 3-in-1 Coffee Mix 30 x 20g ₱159.75
Nescafe Gold Latte Macchiato 10 x 20g ₱169.75
Nescafe Classic Decaf Resealable Pack Instant Coffee 160 g ₱162.75
Cafe Puro Jade Blend Fresh Ground Coffee in a Tin Can 400g ₱188.75
Kopiko Brown Coffee Mix 30 x 27.5g ₱210.75
Pure Nature 12-in-1 Green Coffee Mix 10 x 21g ₱279.75
Kopiko Blanca Creamy Coffee Mix 30 x 30g ₱210.75
Folgers Classic Medium Roast Ground Coffee 320 g ₱349.75
Kopiko Cappuccino Coffee Mix 30 x 25g ₱210.75
Check Hup 3-in-1 Ipoh White Coffee King 15 x 40 g ₱318.75
Nescafe Taster's Choice French Roast Instant Coffee 198g ₱619.75
Nescafe Taster's Choice House Blend Decaf Coffee 7 oz ₱679.75
Maxwell House Cappuccino 750 g ₱899.75

You need to import below libraries.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM