使用 Beautiful soup 进行 Web 抓取时的数据清理

Question

import requests, re
from bs4 import BeautifulSoup

r = requests.get('https://www.nrtcfresh.com/products/whole/vegetables-whole', headers={'User-agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0'})
c = r.content

soup=BeautifulSoup(c,"html.parser")
#print(soup.prettify())

all = soup.find_all("div",{"class":"col-sm-3 nrtc-p-10"})

all[1].find("h4").text

The output is provided below

'\r\n                Tomatoes\t\t\t\t  (Turkey)\n'

To get "Turkey" as the output, I can all[1].find('h4').find("span").text.replace(" ", "").replace("(","").replace(")","") Is there a better way to write this code and more importantly, how do I get just "Tomatoes" as the output?

 <h4> " Tomatoes " <span>(Turkey)</span> </h4>

Answer 1

This is one way:

import requests
from bs4 import BeautifulSoup
countries = []
vegetables = []
remove = ['(', ')']
r = requests.get('https://www.nrtcfresh.com/products/whole/vegetables-whole', headers={'User-agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0'})
c = r.content

soup=BeautifulSoup(c,"html.parser")
text = ''
all = soup.select("div.col-sm-3.nrtc-p-10 h4")

# Vegetables
print('Vegetables:\n')
for vegetable in all:
    print(vegetable.find(text=True, recursive=False).strip())
    vegetables.append(vegetable.find(text=True, recursive=False).strip())


# Countries:
print('\n\nCountries:\n')

for span in all:
    for t in span.find('span').get_text(strip=True):
        if not t in remove:
            text += t
    print(text)
    countries.append(text)
    text= ''


# Vegetables and Countries
for v, c in zip(vegetables, countries):
    print(f'{v} - {c}')

prints:

Vegetables:

Tapioca
Tomatoes
Rosemary
Beef Tomatoes
Red Cherry Tomatoes
Red Cherry Tomatoes (Vine)
Yellow Cherry Tomatoes
Plum Tomatoes
Plum Cherry Tomatoes
Vine Tomatoes
....

Countries:

Srilanka
Turkey
Kenya
Holland
Netherland
Netherland
Netherland
Netherland
Holland
Netherland
....


Tapioca - Srilanka
Tomatoes - Turkey
Rosemary - Kenya
Beef Tomatoes - Holland
Red Cherry Tomatoes - Netherland
Red Cherry Tomatoes (Vine) - Netherland
Yellow Cherry Tomatoes - Netherland
Plum Tomatoes - Netherland
Plum Cherry Tomatoes - Holland
Vine Tomatoes - Netherland
Turnip - Iran
Baby Turnip - South Africa
Yams (Suran) - India
Green Baby Zucchini - South Africa
....

Note: i have shorten the print here..

This method is especially good if there are many different characters that are not accepted

Answer 2

My understanding is that you're looking for the vegetable name only, not the country. If you are happy to dispose of the country name you can do the following:

# Delete the country spans
for span in soup.select("div.col-sm-3.nrtc-p-10 h4 span"):
    span.extract()

# Get a list of all the vegetables
veg_list = [h4.text.strip() for h4 in soup.select("div.col-sm-3.nrtc-p-10 h4")]
print(veg_list)

Tapioca
Tomatoes
Rosemary
Beef Tomatoes
Red Cherry Tomatoes
...