美丽的汤findAll不计算所有div

Question

from bs4 import BeautifulSoup

html = 'index.html'
soup = BeautifulSoup(open(html))
print len(soup.findAll('div'))

where the file index.html is the source code of this shopping webpage .

My code shows that only 1 div tag was found. But what's weirder is findAll('a') returns a huge (so probably correct) number. span works etc, but not div .

Answer 1

You are experiencing the differences between parsers that BeautifulSoup uses under-the-hood.

Choose either html.parser , or html5lib :

>>> from bs4 import BeautifulSoup
>>> html = 'index.html'
>>> soup = BeautifulSoup(open(html), 'html')
>>> len(soup.findAll('div'))
0
>>> soup = BeautifulSoup(open(html), 'lxml')
>>> len(soup.findAll('div'))
0
>>> soup = BeautifulSoup(open(html), 'html.parser')
>>> len(soup.findAll('div'))
774
>>> soup = BeautifulSoup(open(html), 'html5lib')
>>> Alen(soup.findAll('div'))
774

Note that if you don't specify a parser , BeautifulSoup would pick it up automatically:

If you don't specify anything, you'll get the best HTML parser that's installed. Beautiful Soup ranks lxml's parser as being the best, then html5lib's, then Python's built-in parser.