网页抓取 - 我需要一些帮助来理解如何区分页面 BS4 上的项目，请求

Question

I am stuck. I am able to extract product name and prices from amazon, using the following code

import requests
from bs4 import BeautifulSoup
import pandas as pd


headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36"}
#
url = f'https://www.amazon.co.uk/dp/B083PHB6XX'

r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.content, 'lxml')

name = soup.find('span', {'id': 'productTitle'}).text.strip()
price = soup.find('span', {'id': 'priceblock_ourprice'}).text.strip()

print(name)
print(price)

But I am unable to figure out how to extract the sales rank data from within the table, which is lower down on the page, under the additional information section. I'd be most grateful if anyone would be able to assist in helping me figure out how to write the next soup.find line of code, to show '106,505' for the sales rank.

Many thanks in advance.

Answer 1

One solution can be searching for <th> tag that contains string "Best Sellers Rank" and then find next <span> :

import requests
from bs4 import BeautifulSoup


url = "https://www.amazon.co.uk/dp/B083PHB6XX"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36"
}

soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")
ranks = soup.select_one('th:-soup-contains("Best Sellers Rank")').find_next(
    "span"
)

print(ranks.text.split()[0])

Prints:

111,190