我如何使用 python 和 beautifulsoup 进行 webscape？

Question

Im very new to this, but I have an idea for a website and I want to give it a good go, my aim is to scrape the Asda website for prices and products, more specifically in this case whiskey. I want to grab the name and price of all the whiskey on the Asda website and put it into a nice table on my website, however I am having problems doing so, my code so far is getting syntax error, can anyone help? the code so far is..

import requests 
from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://groceries.asda.com/shelf/drinks/spirits-ready-to-drink/spirits/whisky/1579926650')

res = driver.execute_script('return document.documentElement.outerHTML')

html_soup = BeautifulSoup(res, 'html.parser')
type(html_soup)

driver.quit

response = requests.get('https://groceries.asda.com/shelf/drinks/spirits-ready-to-drink/spirits/whisky/1579926650'


whiskey_container = html_soup.find('div', {'class': 'co-product-lazy-container'})

for whiskey in whiskey_container:
    name = whiskey.find('a', {'class': 'co-product__anchor'})
    price = whiskey.find('div', {'class': 'co-product__price'})

    print(name, price)

Answer 1

you have syntax error, you have ")" missing:

response = requests.get('https://groceries.asda.com/shelf/drinks/spirits-ready-to-drink/spirits/whisky/1579926650'

it should be:

response = requests.get('https://groceries.asda.com/shelf/drinks/spirits-ready-to-drink/spirits/whisky/1579926650')

--

btw your code won't work. you have couple of logical errors. and I doubt you can scrape that page with your current code.

Answer 2

Try it:

# for wait time better than time.sleep()
from selenium.webdriver.support.ui import WebDriverWait 
from selenium import webdriver
import time  # or WebDriverWait
import csv # for saving data in table

# save csv file
def save_csv(dct):
    '''
    dct - dictionary with our data:
                                "cap",
                                "title",
                                "price"

    '''
    name = "file.csv"  # file name, it can choice what you want
    print("[INFO] saving...") # for see that function works
    with open(name, 'a', encoding="utf-8") as f: # open file for writing "a"
            # this need for writing data to table 
            writer = csv.writer(f) 
            writer.writerow((dct['cap'],
                            dct['title'],
                            dct['price'],
                        ))

def scroll(driver):
    # for open all interesting us data 
    for i in range(1,6):
        # driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        driver.execute_script("window.scrollTo(0,  1000)") 
        time.sleep(7)



driver = webdriver.Firefox()
driver.get("https://groceries.asda.com/shelf/drinks/spirits-ready-to-drink/spirits/whisky/1579926650?facets=shelf%3A1579926650%3A0000&nutrition=&sortBy=&page=0")

for i in range(2): # 2 because we have only two page with data
    element = WebDriverWait(driver, 30) # or time.sleep(30)
    scroll(driver) # for open all interesting us data

    # get all data to one list in beautifulsoup type
    data = driver.find_elements_by_css_selector(".co-lazy-product-container .co-item")

    # iterating interesting data and create dictionary with data 
    for d in data:
        items = {}
        body = d.text.split("\n")
        items["cap"] = body[0]
        items["title"] = body[1]
        items["price"] = body[-2]
        save_csv(items)

    # pagination
    driver.find_element_by_css_selector(".co-pagination__last-page").click()

# close driver
driver.quit()