[英]Getting AttributeError: 'NoneType' object has no attribute 'text' (web-scraping)
[英]Error on .text.strip() using Selenium BeautifulSoup for Web-scraping (AttributeError: 'NoneType' object has no attribute 'text)
我想从网页上抓取价格。 首先,在我将其全部合并为一个代码之前,我已经逐块编写了价格代码。 当我按块编写时,它运行良好。 (特别是对于使用.text.strip()
的价格部分
!pip install selenium
from selenium import webdriver
import time
from bs4 import BeautifulSoup
driver = webdriver.Chrome('D:\chromedriver.exe')
url = "https://www.fashionvalet.com/catalogsearch/result/?q=duck"
driver.get(url)
driver.maximize_window()
time.sleep(3)
btn = driver.find_element_by_xpath('/html/body/main/div/header/div[5]/div[1]/div[1]/div')
btn.click()
time.sleep(5)
soup = BeautifulSoup(driver.page_source, "html.parser")
p_price = card.select_one('.fvPLPProductPrice > strong').text.strip()
#"strong").select_one("strong").text.strip()
print(p_price)
MYR50.00
不幸的是,当我合并所有代码时,错误来自价格部分的.text.strip()
,
!pip install selenium
from selenium import webdriver
import time
import pandas as pd
from bs4 import BeautifulSoup
def get_url(product_name):
product_name = product_name.replace(' ', '+')
url_template = "https://www.fashionvalet.com/catalogsearch/result/?q={}"
url = url_template.format(product_name)
return url
def product_info(card):
# name
p_name = card.find('h3').text.strip()
# price
#p_rice = card.find("p", "fvPLPProductPrice").select("strong")
p_price = card.select_one('.fvPLPProductPrice > strong').text.strip()
# image
p_image = card.find('img')
p_img = p_image['src']
# brand
p_brand = card.find('p', "fvPLPProductBrand").text.strip()
# discount percent
p_dis = card.find('p', "fvPLPProductMeta").text.strip()
info = (p_name, p_price, p_img, p_brand, p_dis)
return info
def main(product):
records = []
url = get_url(product) # 1--generate URL
driver = webdriver.Chrome('D:\chromedriver.exe') # 2--open browser
driver.get(url) # 3--open URL
driver.maximize_window()
time.sleep(5)
# BUTTON
btn = driver.find_element_by_xpath('/html/body/main/div/header/div[5]/div[1]/div[1]/div')
btn.click()
time.sleep(5)
# AUTO-SCROLLING
# -- make the parsing time of python is equivalent to the web
temp_height=0
while True:
driver.execute_script("window.scrollBy(0,1000)")
time.sleep(10)
check_height = driver.execute_script("return document.documentElement.scrollTop || window.pageYOffset || document.body.scrollTop;")
if check_height==temp_height:
break
temp_height=check_height
time.sleep(5)
# AUTO-SCROLL end
soup = BeautifulSoup(driver.page_source, "html.parser")
product_card = soup.select('.fvPLPProducts > li')
for allproduct in product_card:
productDetails = product_info(allproduct)
records.append(productDetails)
col = ['Name', 'Price', 'Image', 'Brand', 'Discount']
all_data = pd.DataFrame(records, columns=col)
all_data.to_csv('D:\\FASHION-{}.csv'.format(product))
这是输出,在我运行main("duck")
,错误是这样出现的,
AttributeError Traceback (most recent call last)
<ipython-input-7-7b75c58eb0da> in <module>
----> 1 main("duck")
<ipython-input-6-7d068e5049f6> in main(product)
70
71 for allproduct in product_card:
---> 72 productDetails = product_info(allproduct)
73 records.append(productDetails)
74
<ipython-input-6-7d068e5049f6> in product_info(card)
20
21 #p_rice = card.find("p", "fvPLPProductPrice").select("strong")
---> 22 p_price = card.select_one('.fvPLPProductPrice > strong').text.strip()
23
24 # image
AttributeError: 'NoneType' object has no attribute 'text
`
我试图删除text.strip()
,它运行良好,但输出包括来自 HTML 代码的标签,这不是我想要的。
作为结论, .text.strip()
在分离代码时起作用,但是当我将它们全部合并时它会变成错误。
任何人都可以帮助我吗? 谢谢你。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.