Error: AttributeError: 'NoneType' object has no attribute 'find'

Question

I am writing a webscraping code and I am getting the above error.

import requests
import lxml
import bs4

title = ''
date = ''
text = ''
top = []
link = []  


web_link = 'https://timesofindia.indiatimes.com/{}/'
web_link = web_link.format('india')
req = requests.get(web_link)
soup = bs4.BeautifulSoup(req.text, 'lxml')
topi = soup.find('div', {'class':'main-content'})
topi = topi.find_all('span', {'class':'w_tle'})
for i in range(len(topi)):
   top = topi[i].find('a').get('href')
   link.append('https://timesofindia.indiatimes.com' + top)
for i in range(len(link)):
   rq = requests.get(link[i])
   sp = bs4.BeautifulSoup(rq.text, 'lxml')
   title = sp.find('div', {'class':'_2NFXP'})
   title = title.find('h1',{'class':'_23498'})

Traceback:

Traceback (most recent call last):
  File "C:\Users\xxx\xxx\py\so65702068.py", line 26, in <module>
    title=title.find('h1',{'class':'_23498'})
AttributeError: 'NoneType' object has no attribute 'find'

I am new to web scraping and I am not understanding why is it showing this error.

Answer 1

You should try learning from your errors yourself. Python errors also specify the error's location(line).

Anyway, your last line is causing the problem. You basically applied sp.find() function on another sp.find() function. As sp.find('div',{'class':'_2NFXP'}) returns None , it justifies the error you got.