![](/img/trans.png)
[英]How to scrape multiple html page in parallel with beautifulsoup in python?
[英]How to scrape a page with BeautifulSoup and Python?
我正在嘗試從 BBC Good Food 網站提取信息,但在縮小我收集的數據范圍時遇到了一些麻煩。
這是我到目前為止所擁有的:
from bs4 import BeautifulSoup
import requests
webpage = requests.get('http://www.bbcgoodfood.com/search/recipes?query=tomato')
soup = BeautifulSoup(webpage.content)
links = soup.find_all("a")
for anchor in links:
print(anchor.get('href')), anchor.text
這將返回相關頁面的所有鏈接以及鏈接的文本描述,但我想從頁面上的“文章”類型對象中提取鏈接。 這些是特定食譜的鏈接。
通過一些實驗,我設法從文章中返回文本,但我似乎無法提取鏈接。
我看到的與文章標簽相關的唯一兩件事是 href 和 img.src:
from bs4 import BeautifulSoup
import requests
webpage = requests.get('http://www.bbcgoodfood.com/search/recipes?query=tomato')
soup = BeautifulSoup(webpage.content)
links = soup.find_all("article")
for ele in links:
print(ele.a["href"])
print(ele.img["src"])
鏈接在"class=node-title"
from bs4 import BeautifulSoup
import requests
webpage = requests.get('http://www.bbcgoodfood.com/search/recipes?query=tomato')
soup = BeautifulSoup(webpage.content)
links = soup.find("div",{"class":"main row grid-padding"}).find_all("h2",{"class":"node-title"})
for l in links:
print(l.a["href"])
/recipes/681646/tomato-tart
/recipes/4468/stuffed-tomatoes
/recipes/1641/charred-tomatoes
/recipes/tomato-confit
/recipes/1575635/roast-tomatoes
/recipes/2536638/tomato-passata
/recipes/2518/cherry-tomatoes
/recipes/681653/stuffed-tomatoes
/recipes/2852676/tomato-sauce
/recipes/2075/tomato-soup
/recipes/339605/tomato-sauce
/recipes/2130/essence-of-tomatoes-
/recipes/2942/tomato-tarts
/recipes/741638/fried-green-tomatoes-with-ripe-tomato-salsa
/recipes/3509/honey-and-thyme-tomatoes
要訪問您需要預先添加http://www.bbcgoodfood.com
:
for l in links:
print(requests.get("http://www.bbcgoodfood.com{}".format(l.a["href"])).status
200
200
200
200
200
200
200
200
200
200
BBC 美食頁面的結構現在已經改變。
我已經設法調整這樣的代碼,雖然不完美,但可以建立在:
import numpy as np
#Create empty list
listofurls = []
pages = np.arange(1, 10, 1)
ingredientlist = ['milk','eggs','flour']
for ingredient in ingredientlist:
for page in pages:
page = requests.get('https://www.bbcgoodfood.com/search/recipes/page/' + str(page) + '/?q=' + ingredient + '&sort=-relevance')
soup = BeautifulSoup(page.content)
for link in soup.findAll(class_="standard-card-new__article-title"):
listofurls.append("https://www.bbcgoodfood.com" + link.get('href'))
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.