使用Python从网站中收集HTML数据

Question

I'm trying to scrape certain pieces of HTML data from certain websites, but I can't seem to scrape the parts I want. For instance I set myself the challenge of scraping the number of followers from this blog , but I can't seem to do so.

I've tried using urllib, request, beautifulsoup as well as Jam API .

Here's what my code looks like at the moment:

from bs4 import BeautifulSoup
from urllib import urlopen
import json
import urllib2

html = urlopen('http://freelegalconsultancy.blogspot.co.uk/')
soup = BeautifulSoup(html, "lxml")
print soup

How would I go about pulling the number of followers in this instace?

Answer 1

You can't grab the followers as it's a widget loaded by javascript. You need to grab parts of the html by css class or id or by the element.

Eg:

from bs4 import BeautifulSoup
from urllib import urlopen

html = urlopen('http://freelegalconsultancy.blogspot.co.uk/')
soup = BeautifulSoup(html)

assert soup.h1.string == '\nLAW FOR ALL-M.MURALI MOHAN\n'