在一个混乱的网站上使用美丽汤进行Python Web抓取

Question

I want to scrape the following three data points from this site : %verified, the numerical value for FAR, and the numerical value for POD. I'm trying to do this in BeautifulSoup, but I'm not practiced in site traversing, so I can't describe the location of those elements.

What is the easiest way to go about doing this?

Answer 1

If you haven't yet, install Firebug for Firefox and use it to inspect the html source of the page.

Use a combination of urllib and BeautifulSoup to handle html retrieval and parsing. Here is a short example:

import urllib
from BeautifulSoup import BeautifulSoup

url = 'http://mesonet.agron.iastate.edu/cow/?syear=2009&smonth=9&sday=12&shour=12&eyear=2012&emonth=9&eday=12&ehour=12&wfo=ABQ&wtype[]=TO&hail=1.00&lsrbuffer=15&ltype[]=T&wind=58'
fp = urllib.urlopen(url).read()
soup = BeautifulSoup(fp)

print soup

From here, the links I provided should give you a good start into how to retrieve the elements you're interested in.

Answer 2

Like That1Guy's says, you need to analyse the source page structure. In this case, you're lucky... the numbers you are looking a specifically highlighted in red using <span> .

This will do this:

>>> import urllib2
>>> import lxml.html
>>> url = ... # put your URL here
>>> html = urllib2.urlopen(url)
>>> soup = lxml.html.soupparser.fromstring(html)
>>> elements = soup.xpath('//th/span')
>>> print float(elements[0].text) # FAR
0.67
>>> print float(elements[1].text) # POD
0.58

Note lxml.html.soupparser is pretty much equivalent to the BeautifulSoup parser (which I don't have to hand at the moment).

Answer 3

I ended up solving it myself-- I was utilizing a strategy similar to isedev, but I was hoping I could find a better way of getting the 'Verified' Data:

import urllib2
from bs4 import BeautifulSoup

wfo = list()

def main():
    wfo = [i.strip() for i in open('C:\Python27\wfo.txt') if i[:-1]]
    soup = BeautifulSoup(urllib2.urlopen('http://mesonet.agron.iastate.edu/cow/?syear=2009&smonth=9&sday=12&shour=12&eyear=2012&emonth=9&eday=12&ehour=12&wfo=ABQ&wtype%5B%5D=TO&hail=1.00&lsrbuffer=15&ltype%5B%5D=T&wind=58').read())
    elements = soup.find_all("span")
    find_verify = soup.find_all('th')

    far= float(elements[1].text)
    pod= float(elements[2].text)
    verified = (find_verify[13].text[:-1])