使用BeautifulSoup 4从网页中提取表格

Question

Be forgiving, only started using beautifulSoup today to deal with this problem.

I've managed to get it working by dragging in the URL's on the website, Each of the product pages on this website has a table that looks like the following:

<table width="100%" class="product-feature-table">
  <tbody>
    <tr>
      <td align="center"><table cellspacing="0" class="stats2">
        <tbody>
          <tr>
          <td class="hed" colspan="2">YYF Shutter Stats:</td>
          </tr>
          <tr>
          <td>Diameter:</td>
          <td>56 mm / 2.20 inches</td>
          </tr>
          <tr>
            <td>Width:</td>
            <td>44.40 mm / 1.74 inches</td>
          </tr>
          <tr>
            <td>Gap Width:</td>
            <td>4.75 mm / .18 inches</td>
          </tr>
          <tr>
            <td>Weight:</td>
            <td>67.8 grams</td>
          </tr>
          <tr>
            <td>Bearing Size:</td>
            <td>Size C (.250 x .500 x .187)<br>CBC SPEC Bearing</td>
          </tr>
          <tr>
            <td>Response:</td>
            <td>CBC Silicone Slim Pad (19mm)</td>
          </tr>
        </tbody>
        </table>
      <br>
      <br>
      </td>
    </tr>
  </tbody>
</table>

I'm trying to pull this table into some form of data that I could work with within a webapp.

How would I go about extracting this from each webpage, the website has around 400 product pages that include this table, I'd preferably like to get each of the tables from the page and put it into a database entry or text file with the name of the product.

As you can see the table isn't exactly formatted well, but it is the only table on the page labeled with

class="product-feature-table"

I have just been trying to edit a URL scraping script but I'm starting to get the feeling I'm going about it all wrong trying to do that.

My url script is as follows:

import urllib2
from bs4 import BeautifulSoup

url = raw_input('Web-Address: ')

html = urllib2.urlopen('http://' +url).read()
soup = BeautifulSoup(html)
soup.prettify()
for anchor in soup.findAll('a', href=True):
    print anchor['href']

I can get all these URL's into a text file but would much prefer to use Sqlite or Postgresql, are there any articles online that would help me understand these concepts better, that don't drown the newbie?

Answer 1

You have probably already been here, but when I used BS (no pun intended) a while back, it's doc page was where I started: http://www.crummy.com/software/BeautifulSoup/bs4/doc/

Personally, I found this official documentation could have been better, and the Beautiful Soup resources from the online community also seemed lacking at the time - this was about 3 or 4 years ago though.

I hope both have come farther since.

Another resource perhaps worth looking into is Mechanize: http://wwwsearch.sourceforge.net/mechanize/

Answer 2

First of all, if you want to extract all the tables inside a site using BeautifulSoup you could do it in the following way :

import urllib2
from bs4 import BeautifulSoup

url = raw_input('Web-Address: ')

html = urllib2.urlopen('http://' +url).read()
soup = BeautifulSoup(html)
soup.prettify()

# extract all the tables in the HTML 
tables = soup.find_all('table')

#get the class name for each
for table in tables:
  class_name = table['class']

Once you have all the tables in the page you could do anything you want with its data moving for the tags tr and td in the following way :

for table in tables:
  tr_tags = table.find_all('tr')

Remember that the tr tags are rows inside the table. Then to obtain the data inside the tags td you could use something like this :

for table in tables:
  tr_tags = table.find_all('tr')

  for tr in tr_tags:
    td_tags = tr.find_all('td')

    for td in td_tags:
      text = td.string  

If you want to surf in all the links inside the table and then find the tables the code explained above would work for you, making first the retrieve of all the urls inside an then moving between them. For example :

initial_url = 'URL'
list_of_urls = []

list_of_url.append(initial_url)

while len(list_of_urls) > 0:

  html = urllib2.urlopen('http://' + list_of_url.pop()).read()
  soup = BeautifulSoup(html)
  soup.prettify()

  for anchor in soup.find_all('a', href=True):
     list_of_urls.append(anchor['href'])

  #here put the code explained above, for example

  for table in tables:
    class_name = table['class']

    # continue with the above code..

To insert the data to a database in SQLite I recommend you read the following tutorial Python: A Simple Step-by-Step SQLite Tutorial