我在使用 beautifulsoup 进行 web 抓取时遇到了一些问题

Question

when ever i try to extract text between tags using.text() it gives a blank screen with just [] as output

import requests
from bs4 import BeautifulSoup

page = requests.get("https://www.amazon.in/s?k=ssd&ref=nb_sb_noss")

soup = BeautifulSoup(page.content, "html.parser")

product = soup.find_all("h2",class_="a-link-normal a-text-normal")
results = soup.find_all("span",class_="a-offscreen")

print(product)

this is the output that i got:

C:\Users\Kushal\Desktop\requests-tutorial>C:/Users/Kushal/AppData/Local/Programs/Python/Python37/python.exe c:/Users/Kushal/Desktop/requests-tutorial/scraper.py
[]

when i try listing everything with a for loop then, nothing shows up not even the empty square brackets

Answer 1

Based on your comment below. I've modified the code to fetch all the product title on the said page along with the price details.

Mark as answer if it works, else comment for further analysis.

import requests
from bs4 import BeautifulSoup
import lxml


dataList = list()
headers = {
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5)",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "accept-charset": "cp1254,ISO-8859-9,utf-8;q=0.7,*;q=0.3",
    "accept-encoding": "gzip,deflate,sdch",
    "accept-language": "tr,tr-TR,en-US,en;q=0.8",
} 

url = requests.get('https://www.amazon.in/s?k=ssd&ref=nb_sb_noss'.format(), headers=headers)

soup = BeautifulSoup(url.content, 'lxml')

title = soup.find_all('span', attrs={'class':'a-size-medium a-color-base a-text-normal'})
price = soup.find_all('span', attrs={'class':'a-offscreen'})


for product in zip(title,price):
    title,price=product
    title_proper=title.text.strip()
    price_proper=price.text.strip()
    print(title_proper,'-',price_proper)