用逗号分隔 Python Web 抓取的数据

Question

I am new to web scraping with Python and found a quick tutorial online with some sample code. I adjusted some of the code to add another aspect to the result (output as a csv file). The code is scraping the info about different laptops (name, price, rating, specs).

The issue I am having is separating the specs with a comma in the output.

Here is the code I am using:

    from selenium import webdriver
    from bs4 import BeautifulSoup
    import pandas as pd
    from webdriver_manager.chrome import ChromeDriverManager

    driver = webdriver.Chrome(ChromeDriverManager().install())

    products=[]
    prices=[]
    ratings=[]
    specs=[]
    driver.get('https://www.flipkart.com/laptops/~buyback-guarantee-on-laptops-/pr?sid=6bo%2Cb5g&uniqBStoreParam1=val1&wid=11.productCard.PMU_V2')

    content = driver.page_source
    soup = BeautifulSoup(content)
    for a in soup.find_all('a', href=True, attrs={'class':'_31qSD5'}):
        name = a.find('div', attrs={'class':'_3wU53n'})
        price = a.find('div', attrs={'class':'_1vC4OE _2rQ-NK'})
        rating = a.find('div', attrs={'class':'hGSR34'})
        spec = a.find('div', attrs={'class':'_3ULzGw'})
        products.append(name.text)
        prices.append(price.text)
        ratings.append(rating.text)
        specs.append(spec.text)

    df = pd.DataFrame({'Product name':products, 'Price':prices, 'Rating':ratings, 'Tech Specs':specs})
    df.to_csv('products.csv', index=False, encoding='utf-8')

Here is the current output:

Tech Specs

Intel Core i5 Processor (5th Gen)8 GB DDR3 RAM64 bit Mac OS Operating System128 GB SSD33.78 cm (13.3 inch) Display1 Year Carry In Warranty

Pre-installed Genuine Windows 10 Operating System (Includes Built-in Security, Free Automated Updates, Latest Features)Intel Core i5 Processor (7th Gen)8 GB DDR4 RAM64 bit Windows 10 Operating System1 TB HDD39.62 cm (15.6 inch) Display1 Year Onsite Warranty

Here is how I would like the output to look:

Tech Specs

Intel Core i5 Processor (5th Gen), 8 GB DDR3 RAM, 64 bit Mac OS Operating System, 128 GB SSD, 33.78 cm (13.3 inch) Display, 1 Year Carry In Warranty

Pre-installed Genuine Windows 10 Operating System (Includes Built-in Security, Free Automated Updates, Latest Features), Intel Core i5 Processor (7th Gen), 8 GB DDR4 RAM, 64 bit Windows 10 Operating System, 1 TB HDD, 39.62 cm (15.6 inch) Display, 1 Year Onsite Warranty

Any help is appreciated. Thanks in advance!

Answer 1

You should split tech specs if you need it. Right now you write tech specs as it is. Please take a look of updated code

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(ChromeDriverManager().install())

products=[]
prices=[]
ratings=[]
specs=[]
driver.get('https://www.flipkart.com/laptops/~buyback-guarantee-on-laptops-/pr?sid=6bo%2Cb5g&uniqBStoreParam1=val1&wid=11.productCard.PMU_V2')

content = driver.page_source
soup = BeautifulSoup(content)
for a in soup.find_all('a', href=True, attrs={'class':'_31qSD5'}):
    name = a.find('div', attrs={'class':'_3wU53n'})
    price = a.find('div', attrs={'class':'_1vC4OE _2rQ-NK'})
    rating = a.find('div', attrs={'class':'hGSR34'})
    spec = a.find('div', attrs={'class':'_3ULzGw'})
    products.append(name.text)
    prices.append(price.text)
    ratings.append(rating.text)
    specs.append(", ".join([l.text for l in spec.find_all('li')]))

df = pd.DataFrame({'Product name':products, 'Price':prices, 'Rating':ratings, 'Tech Specs':specs})
df.to_csv('products.csv', index=False, encoding='utf-8', sep=";")

I iterate through all specs and combine them using join and also I changed csv file column separator to ;

Answer 2

selenium is not proffered for such case, as the data is already visible within the HTML script tag, So you can use requests module with bs4 to load the JSON data as below:

import requests
from bs4 import BeautifulSoup
import re
import json


def main(url):
    r = requests.get(url)
    soup = BeautifulSoup(r.content, 'html.parser')
    script = soup.find("script", id="is_script").text
    target = re.search(r"__INITIAL_STATE__ = ({.+});$", script).group(1)
    data = json.loads(target)
    # print(json.dumps(data, indent=4)) to see it in nice view
    print(data.keys()) # it's a dict, so you can deal with it.


main("https://www.flipkart.com/laptops/~buyback-guarantee-on-laptops-/pr?sid=6bo,b5g&uniqBStoreParam1=val1&wid=11.productCard.PMU_V2")