[英]nonetype object has no attribute text error while scraping data
when i try to scrap data from this amazon link .当我尝试从这个亚马逊链接中删除数据时。 I got
AttributeError: 'NoneType' object has no attribute 'text'
我得到
AttributeError: 'NoneType' object has no attribute 'text'
My Code:我的代码:
headers = ({'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:103.0) Gecko/20100101 Firefox/103.0',
'Accept-Language' : 'en-US,en;q=0.5'})
lap_site = requests.get('https://www.amazon.in/s?k=laptops&sprefix=%2Caps%2C634&ref=nb_sb_ss_recent_3_0_recent',headers = headers)
lap_soup = bs(lap_site.content,'lxml')
content = lap_soup.find('div',class_ = 's-desktop-width-max s-desktop-content s-opposite-dir sg-row')
lap_detail_block = content.find_all('div',class_ = 'a-section a-spacing-small a-spacing-top-small')
lap_name = lap_price = lap_rating = []
for i in lap_detail_block:
laptop_name = i.find('h2').a.span.text
lap_name.append(laptop_name)
laptop_rating = i.find('span',class_ = 'a-icon-alt').text
lap_rating.append(laptop_rating)
laptop_price = i.find('span',class_ = 'a-price-whole').text
lap_price.append(laptop_price)
laptop_details = {
'Laptop':lap_name,
'Price':lap_price,
'Rating':lap_rating }
print(laptop_details)
I think that the laptop_rating
variable store the content in string format even if we not include .text
.我认为即使我们不包含
.text
, laptop_rating
变量也会以字符串格式存储内容。 I'm thinking that might be the reason for getting NoneType
error, as we are extracting text from text.我认为这可能是出现
NoneType
错误的原因,因为我们正在从文本中提取文本。 Anyway that's not the issue.无论如何,这不是问题。 How to extract the price or rating from that link?
如何从该链接中提取价格或评级?
At least from my tests, that page is recognizing automated access and blocks it.至少从我的测试来看,该页面正在识别自动访问并阻止它。 You need to use something like
cloudscraper
to do it.你需要使用
cloudscraper
之类的东西来做到这一点。 The following code will return the expected results (adapt to your own circumstances):下面的代码会返回预期的结果(适应自己的情况):
import cloudscraper
import pandas as pd
from bs4 import BeautifulSoup
scraper = cloudscraper.create_scraper()
r = scraper.get('https://www.amazon.in/s?k=laptops&sprefix=%2Caps%2C634&ref=nb_sb_ss_recent_3_0_recent')
soup = BeautifulSoup(r.content, 'html.parser')
# print(soup)
content = soup.find('div',class_ = 's-desktop-width-max s-desktop-content s-opposite-dir sg-row')
lap_detail_block = content.find_all('div',class_ = 'a-section a-spacing-small a-spacing-top-small')
lap_name = lap_price = lap_rating = []
for i in lap_detail_block:
try:
laptop_name = i.find('h2').a.span.text
lap_name.append(laptop_name)
laptop_rating = i.find('span',class_ = 'a-icon-alt').text
lap_rating.append(laptop_rating)
laptop_price = i.find('span',class_ = 'a-price-whole').text
lap_price.append(laptop_price)
laptop_details = {
'Laptop':lap_name,
'Price':lap_price,
'Rating':lap_rating
}
print(laptop_name, laptop_rating, laptop_price)
except Exception as e:
print(e)
print('_____________')
This will print out in terminal:这将在终端打印出来:
HP 15s, 12th Gen Intel Core i5 8GB RAM/512GB SSD 15.6-inch(39.6 cm) FHD,Micro-Edge, Anti- Glare Display/Win 11/Intel Iris Xe Graphics/Dual Speakers/Alexa/Backlit KB/MSO/Fast Charge, 15s- fq5111TU 4.2 out of 5 stars 58,699
_____________
Acer Predator Helios 500 Gaming Laptop (11Th Gen Intel Core I9/17.3 Inches 4K Uhd Display/64Gb Ddr4 Ram/2Tb Ssd/1Tb HDD/RTX 3080 Graphics/Windows 10 Home/Per Key RGB Backlit Keyboard) Ph517-52 3.0 out of 5 stars 3,79,990
_____________
ASUS VivoBook 14 (2021), 14-inch (35.56 cm) HD, Intel Core i3-1005G1 10th Gen, Thin and Light Laptop (8GB/1TB HDD/Windows 11/Integrated Graphics/Grey/1.6 kg), X415JA-BV301W 3.8 out of 5 stars 27,990
_____________
[...]
Cloudscraper's details and install instructions: https://pypi.org/project/cloudscraper/ Cloudscraper 的详细信息和安装说明: https://pypi.org/project/cloudscraper/
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.