[英]How do I fix TypeError, Parsing, and all other errors in my python Yahoo Finance Webscrapper
How do I fix TypeError, Parsing, and all other errors in my python Yahoo Finance Webscraper.如何修复我的 python Yahoo Finance Webscraper 中的类型错误、解析和所有其他错误。 I cannot get my code to pull from Yahoo finance.我无法从 Yahoo Finance 获取代码。 Any fixes?任何修复? It looks like span classes are the problem since they were removed and replaced by fin-streamer.看起来 span 类是问题所在,因为它们已被删除并被 fin-streamer 取代。
Code:代码:
import requests
from bs4 import BeautifulSoup
def create_url():
symbol = str(input('Enter Stock Symbol: '))
url = f'https://finance.yahoo.com/quote/{symbol}'
return url
def get_html(url):
header = {"User Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36'}
response = requests.get(url, headers = header)
if response.status_code == 200:
return response.text
else:
return None
def parse_data(html):
soup = BeautifulSoup(html,'html.parser')
name = soup.find('h1', {'class': 'D(ib) Fz(18px)'}).text
price = soup.find('fin-streamer', {'class': 'D(ib) Mend(20px)'}).find_all('fin-streamer')[0].text
change = soup.find('fin-streamer', {'class': 'D(ib) Mend(20px)'}).find_all('fin-streamer')[1].text
previous_close = soup.find('fin-streamer', {'class': 'Trsdu(0.3s)'}).text
open_price = soup.find('td',{'class':'Ta(end) Fw(600) Lh(14px)'}).text
print(f'|Stock Name: {name}|', f'|Stock Price: ${price}|', f'|Change: {change}|', f'|Previous Close: ${previous_close}|', f'|Open Price: ${open_price}|')
# print(f'Stock Price: ${price}')
# print(f'Change: {change}')
# print(f'Previous Close: ${previous_close}')
# print(f'Open Price: ${open_price}')
stock_data = {
'name':name,
'price':price,
'change':change ,
'previous_close': previous_close,
'open_price': open_price
}
return stock_data
def main():
# get users input
url = create_url()
# get html
html = get_html(url)
# while loop
i = True
while i:
# parse data
data = parse_data(html)
if __name__ == '__main__':
main()
def create_url():
symbol = str(input('Enter Stock Symbol: '))
url = f'https://finance.yahoo.com/quote/%7Bsymbol%7D'
return url
This function contains an error: in order to interpolate your symbol
into the url
, you need to do something like this:这个 function 包含一个错误:为了将你的symbol
插入到url
中,你需要做这样的事情:
url = f'https://finance.yahoo.com/quote/{symbol}'
As a result, you where GET'ing the wrong URL, but your function get_html()
returns None
if you fail to get a status 200. That None
gets passed to BeautifulSoup as HTML, generating your error.结果,你得到了错误的 URL,但你的 function get_html()
如果你未能获得状态 200,则返回None
将作为None
传递给 BeautifulSoup,从而生成你的错误。
It's good that your get_html()
function checks the status, but it should be failing if status indicates failure.你的get_html()
function 检查状态很好,但如果状态指示失败,它应该失败。
Update: that was a copy and paste error.更新:那是一个复制和粘贴错误。
Your error is caused by passing None
to BeautifulSoup - you can confirm this by running:您的错误是由于将None
传递给 BeautifulSoup 引起的 - 您可以通过运行来确认这一点:
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(None, 'html.parser')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/private/tmp/venv/lib/python3.9/site-packages/bs4/__init__.py", line 312, in __init__
elif len(markup) <= 256 and (
TypeError: object of type 'NoneType' has no len()
You are passing None
because that's what your get_html()
func might be returning:您正在传递None
因为那是您的get_html()
函数可能返回的内容:
if response.status_code == 200:
return response.text
else:
return None
If your function fails to GET a 200 status code, you need to fail, not return None
.如果您的 function 未能获得 200 状态代码,则您需要失败,而不是返回None
。
Try replacing the entire get_html()
func with this:尝试用这个替换整个get_html()
函数:
def get_html(url):
header = {"User Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36'}
response = requests.get(url, headers = header)
response.raise_for_status()
return response.text
This function will raise an exception if the http request failed - otherwise, it will return the html, which you can then feed to BeautifulSoup如果 http 请求失败,此 function 将引发异常 - 否则,它将返回 html,然后您可以将其提供给 BeautifulSoup
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.