如何使用 BeautifulSoup 和请求从网站获取数据？

Question

I am a beginner in web scraping, and I need help with this problem. The website, allrecipes.com, is a website where you can find recipes based on a search, which in this case is 'pie':

link to the html file: 'view-source: https://www.allrecipes.com/search/results/?wt=pie&sort=re ' (right click-> view page source)

I want to create a program that takes a input, searches it up on allrecipes, and returns a list with tuples of the first five recipes with data such as the time that takes to make, serving yield, ingrediants, and more. This is my program so far:

import requests
from bs4 import BeautifulSoup

def searchdata():
    inp=input('what recipe would you like to search')
    url ='http://www.allrecipes.com/search/results/?wt='+str(inp)+'&sort=re'
    r=requests.get(url)
    soup = BeautifulSoup(r.text, 'html.parser')
    links=[]

    #fill in code for finding top 3 or five links


    for i in range(3)
        a = requests.get(links[i])
        soupa = BeautifulSoup(a.text, 'html.parser')

        #fill in code to find name, ingrediants, time, and serving size with data from soupa



        names=[]
        time=[]
        servings=[]
        ratings=[]
        ingrediants=[]





searchdata()

Yes, i know, my code is very messy but What should I fill in in the two code fill-in areas? Thanks

Answer 1

After searching for the recipe you have to get the links of each recipe and then request again for each of those links, because the information you're looking for is not available on the search page. That would not look clean without OOP so here's the class I wrote that does what you want.

import requests
from time import sleep
from bs4 import BeautifulSoup


class Scraper:
    links = []
    names = []

    def get_url(self, url):
        url = requests.get(url)
        self.soup = BeautifulSoup(url.content, 'html.parser')

    def print_info(self, name):
        self.get_url(f'https://www.allrecipes.com/search/results/?wt={name}&sort=re')
        if self.soup.find('span', class_='subtext').text.strip()[0] == '0':
            print(f'No recipes found for {name}')
            return
        results = self.soup.find('section', id='fixedGridSection')
        articles = results.find_all('article')
        texts = []
        for article in articles:
            txt = article.find('h3', class_='fixed-recipe-card__h3')
            if txt:
                if len(texts) < 5:
                    texts.append(txt)
                else:
                    break
        self.links = [txt.a['href'] for txt in texts]
        self.names = [txt.a.span.text for txt in texts]
        self.get_data()

    def get_data(self):
        for i, link in enumerate(self.links):
            self.get_url(link)
            print('-' * 4 + self.names[i] + '-' * 4)
            info_names = [div.text.strip() for div in self.soup.find_all(
                'div', class_='recipe-meta-item-header')]
            ingredient_spans = self.soup.find_all('span', class_='ingredients-item-name')
            ingredients = [span.text.strip() for span in ingredient_spans]
            for i, div in enumerate(self.soup.find_all('div', class_='recipe-meta-item-body')):
                print(info_names[i].capitalize(), div.text.strip())
            print()
            print('Ingredients'.center(len(ingredients[0]), ' '))
            print('\n'.join(ingredients))
            print()
            print('*' * 50, end='\n\n')


chrome = Scraper()
chrome.print_info(input('What recipe would you like to search: '))