使用BeautifulSoup抓取网站

Question

Im getting attribute error while scraping

import urllib2
from bs4 import BeautifulSoup

quote_page ='https://www.bloomberg.com/quote/SPX:IND'
page = urllib2.urlopen(quote_page)

soup = BeautifulSoup(page,'html.parser')

name_box = soup.find('h1', attires ={'class': 'name'})

name = name_box.text.strip()
print name

Traceback (most recent call last):

File "word1.py", line 11, in

 name = name_box.text.strip() 

AttributeError: 'NoneType' object has no attribute 'text'

Viveks-MacBook-Pro:py vivek$

Answer 1

when you do this

print(name_box)

you will get

 None
Traceback (most recent call last):
  File "C:/Users/devsurya/python/demo programs/b4s.py", line 13, in <module>
    name = name_box.text.strip()
AttributeError: 'NoneType' object has no attribute 'text'

and when you do this -

print(soup)    ## it says following message with weird html and css

We've detected unusual activity from your computer network

and soup.find('h1', attires ={'class': 'name'}) should be soup.find('h1', {'class': 'companyName__99a4824b'})

Answer 2

Assuming you want the company name I would go with requests and a couple of headers are required (you will need to test to see if this performs consistently over time). I use a css attribute = value selector to get the appropriate element and use starts with operator ^ in case the value is dynamic ie I assume a constant start string of companyName . This makes it more versatile for other requests.

import requests
from bs4 import BeautifulSoup as bs

quote_page ='https://www.bloomberg.com/quote/SPX:IND'
page = requests.get(quote_page, headers = {'User-Agent':'Mozilla/5.0', 'accept-language':'en-US,en;q=0.9'})
soup = bs(page.content,'lxml')
name_box = soup.select_one('[class^=companyName]')
name = name_box.text.strip()
print(name)