使用 BeautifulSoup 抓取，值不干净

Question

I'm trying to scrape a nutrient tag ( http://smartlabel.generalmills.com/41196891218 ). and I'm having a hard time getting a clean gram value for each category.

For example, this is how it comes out for fat ('fat': '\n 1 g\n ',)\

Any way to get something like this("fat": 1g)?

I just started learning bs4 yesterday, any help will be appreciated..

My code is

def minenutrition1(link):
    driver = webdriver.Chrome()
    driver.get(link)
    # noticed there is an ad here, sleep til page fully loaded.
    time.sleep(1)
    soup = BeautifulSoup(driver.page_source)
    driver.quit()
    calories=soup.find_all("span",{"class":"header2"})[0].text
    fat=soup.find_all("span",{"class":"gram-value"})[0].text
    satfat=soup.find_all("span",{"class":"gram-value"})[1].text
    cholesterol=soup.find_all("span",{"class":"gram-value"})[3].text
    sodium=soup.find_all("span",{"class":"gram-value"})[4].text
    carb=soup.find_all("span",{"class":"gram-value"})[5].text
    Total_sugar=soup.find_all("span",{"class":"gram-value"})[7].text
    protein=soup.find_all("span",{"class":"gram-value"})[9].text
    name = soup.find_all('div',{'class': 'product-header-name header1'})[0].text
    upc=soup.find_all("div",{"class":"upc sub-header"})
    upc=upc[0].text

Answer 1

You get normal string "\n 1 g\n " so you can use string functions to clean/change it.

Using "\n 1 g\n ".strip() you can get "1 g"

So you can add .strip() at the end of this line

fat = soup.find_all("span",{"class":"gram-value"})[0].text.strip()

or do it later

fat = fat.strip()

BS has also function .get_text(strip=True) which you can use instead of .text

fat = soup.find_all("span",{"class":"gram-value"})[0].get_text(strip=True)

---

Minimal working code.

I display fat with > < to see if there are spaces, tabs, enters (new lines).

from selenium import webdriver
from bs4 import BeautifulSoup
import time

url = 'http://smartlabel.generalmills.com/41196891218'
driver = webdriver.Chrome()
#driver = webdriver.Firefox()
driver.get(url)

# noticed there is an ad here, sleep til page fully loaded.
time.sleep(1)

soup = BeautifulSoup(driver.page_source)
driver.quit()

items = soup.find_all("span", {"class": "gram-value"})

fat = items[0].text
print('>{}<'.format(fat))

fat = items[0].text.strip()
print('>{}<'.format(fat))

fat = items[0].get_text(strip=True)
print('>{}<'.format(fat))

Result:

>
                                    1 g
                                <
>1 g<
>1 g<

Answer 2

For this, I would not use Selenium. Not that you can't, but the site is static, and you can get the html source straight away with requests . So this is a little bit of a stretch since you are beginning with BeautifulSoup, but if you open Dev Tools (Ctrl-Shift-I) and reload the page, you will notice the requests made in the right panel under Network -> XHR. There is a requeset to GetNutritionalDetails.

Withibn there, you'll see the request url, and the the requests headers, and at the bottom the payload. You will also see it's a POST request (usually you'll use GET .

The data is within a list ( <li> tags). So it's not just a mater of getting all those tags, then iterate through each of those, to pull out the other data.

You can append that data into a list, and then that list into a table/dataframe with pandas.

Code:

import requests
from bs4 import BeautifulSoup
import pandas as pd


url = 'http://smartlabel.generalmills.com/GTIN/GetNutritionalDetails'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'}

payload = {
'id': '41196891218',
'servingSize': 'AS PACKAGED'}

response = requests.post(url, headers=headers, params=payload)

soup = BeautifulSoup(response.text, 'html.parser')
listItems = soup.find_all('li')

labels = []
gramValues = []
percValues = []

for each in listItems:
    label = each.find('label').text.strip()
    if label == 'Includes':
        label += ' Added Sugar'
    gram = each.find('span', {'class':'gram-value'}).text.strip()
    if each.find('span', {'class':'dv-result'}):
        perc = each.find('span', {'class':'dv-result'}).text.strip()
    else:
        perc = ''

    labels.append(label)
    gramValues.append(gram)
    percValues.append(perc)


df = pd.DataFrame({
        'Label':labels,
        'Grams':gramValues,
        'Percent':percValues})

Output:

print (df)
                   Label   Grams Percent
0              Total Fat     1 g     1 %
1          Saturated Fat     0 g     0 %
2              Trans Fat     0 g        
3            Cholesterol    0 mg     0 %
4                 Sodium  810 mg    35 %
5     Total Carbohydrate    17 g     6 %
6          Dietary Fiber     2 g     6 %
7            Total Sugar     2 g        
8   Includes Added Sugar     2 g     3 %
9                Protein     4 g        
10             Vitamin D    0 ?g     0 %
11               Calcium   60 mg     4 %
12                  Iron  1.2 mg     6 %
13             Potassium    0 mg     0 %