我如何使用Python和BeautifulSoup从一个网络中的多个页面抓取数据

Question

   # -*- coding: utf-8 -*-
"""
Created on Fri Jun 29 10:38:46 2018

@author: Cinthia
"""

from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq
array = ['146-face', '153-palettes-sets', 'https://www.sociolla.com/147-eyes', 'https://www.sociolla.com/150-lips', 'https://www.sociolla.com/149-brows', 'https://www.sociolla.com/148-lashes']
base_url='https://www.sociolla.com/142-face'
uClient = uReq(base_url)
page_html = uClient.read()
uClient.close()

#html parsing
page_soup = soup(page_html, "html.parser")

#grab the product
kosmetik = page_soup.findAll("div", {"class":"col-md-3 col-sm-6 ipad-grid col-xs-12 productitem"})
print(len(kosmetik))

I want to scrape data from that website, that code above just take how much product on the base url. I don't know how that array will work, so it can take data from the product such as description, image, price from all the pages I make in the array.

I'm new to Python and don't know much about loops yet.

Answer 1

You can find the root element of your table/grid which is id=product-list-grid here and extract the attributes which holds all the information you need (brand, link, category) and the first <img> tag.

For pagination, it seems you can get to the next page adding p=<page number> & when the page doesn't exist it redirects to the first one. A workaround here is to check the response url and check if it's the same as the one you requested. If it's the same you can increment the page number otherwise you have scraped all the pages

from bs4 import BeautifulSoup
import urllib.request

count = 1
url = "https://www.sociolla.com/142-nails?p=%d"

def get_url(url):
    req = urllib.request.Request(url)
    return urllib.request.urlopen(req)

expected_url = url % count
response = get_url(expected_url)

results = []

while (response.url == expected_url):
    print("GET {0}".format(expected_url))
    soup = BeautifulSoup(response.read(), "html.parser")

    products = soup.find("div", attrs = {"id" : "product-list-grid"})

    results.append([
        (
            t["data-eec-brand"],    #brand
            t["data-eec-category"], #category
            t["data-eec-href"],     #product link
            t["data-eec-name"],     #product name
            t["data-eec-price"],    #price
            t.find("img")["src"]    #image link
        ) 
        for t in products.find_all("div", attrs = {"class" : "product-item"})
        if t
    ])

    count += 1
    expected_url = url % count
    response = get_url(expected_url)

print(results)

Here the results are stored in results which is an array of tuples