Python Web爬网； 美丽的汤

Question

这篇文章涵盖了这一点： Python Web抓取涉及具有属性的HTML标签

但我无法对此网页执行类似操作： http : //www.expatistan.com/cost-of-living/comparison/melbourne/auckland ？

我正在尝试刮除以下内容的价值：

  <td class="price city-2">
                                                      NZ$15.62
                                      <span style="white-space:nowrap;">(AU$12.10)</span>
                                                  </td>
  <td class="price city-1">
                                                      AU$15.82
                              </td>

基本上价格城市2和价格城市1（NZ $ 15.62和AU $ 15.82）

目前拥有：

import urllib2

from BeautifulSoup import BeautifulSoup

url = "http://www.expatistan.com/cost-of-living/comparison/melbourne/auckland?"
page = urllib2.urlopen(url)

soup = BeautifulSoup(page)

price2 = soup.findAll('td', attrs = {'class':'price city-2'})
price1 = soup.findAll('td', attrs = {'class':'price city-1'})

for price in price2:
    print price

for price in price1:
    print price

理想情况下，我还希望将逗号分隔的值用于：

<th colspan="3" class="clickable">Food</th>, 

提取“食物”，

<td class="item-name">Daily menu in the business district</td>

提取“商务区的每日菜单”

然后是price city-2和price-city1的值

因此打印输出为：

美食，商业区的每日菜单，NZ $ 15.62，AU $ 15.82

谢谢！

Answer 1

我发现BeautifulSoup很难使用。 这是基于webscraping模块的版本：

from webscraping import common, download, xpath

# download html
D = download.Download()
html = D.get('http://www.expatistan.com/cost-of-living/comparison/melbourne/auckland')

# extract data
items = xpath.search(html, '//td[@class="item-name"]')
city1_prices = xpath.search(html, '//td[@class="price city-1"]')
city2_prices = xpath.search(html, '//td[@class="price city-2"]')

# display and format
for item, city1_price, city2_price in zip(items, city1_prices, city2_prices):
    print item.strip(), city1_price.strip(), common.remove_tags(city2_price, False).strip()

输出：

商务区的每日菜单AU $ 15.82 NZ $ 15.62

快餐店的套餐（巨人套餐或类似餐饮）AU $ 7.40 NZ $ 8.16

1/2公斤（1磅）鸡胸肉AU $ 6.07 NZ $ 10.25

...

Answer 2

这是预期的吗？

import requests
from bs4 import BeautifulSoup as bs
import re

r = requests.get('https://www.expatistan.com/cost-of-living/comparison/melbourne/auckland')
soup = bs(r.content, 'lxml')
table = soup.select_one('table.comparison')

for row in table.select('tr:has(.clickable, .item-name, .price)'):
    if row.select_one('.clickable'):
        header = row.select_one('th').text
    if row.select_one('.item-name, td.price'):
        r = [re.sub(r'\s+',' ',re.sub(r'\n','',item.text.strip())) for item in row.select('.item-name, td.price') ]
        r.insert(0, header)
        print(r)

Answer 3

如果将目标页面的HTML加载到变量htmlsource中，则此pyparsing网络爬虫将以给定的CSV格式设置数据格式：

from pyparsing import *

th,thEnd = makeHTMLTags("th")
thCategory = th.setParseAction(withAttribute(**{'class':'clickable', 'colspan':'3'}))
category = thCategory.suppress() + SkipTo(thEnd)('category') + thEnd

# set up tag recognizers, with specialized patterns based on class attribute
td, tdEnd = makeHTMLTags("td")
tdWithClass = lambda cls : td.copy().setParseAction(withAttribute(**{'class':cls}))
itemTd = tdWithClass('item-name')
price1Td = tdWithClass('price city-1')
price2Td = tdWithClass('price city-2')

# define some currencies
currency = oneOf("NZ$ AU$ US$ SG$").setName("currency")

# define a currency amount as a real number
amount = Regex(r'\d+,\d{3}|\d+(\.\d+)?').setParseAction(lambda t:float(t[0].replace(',','')))

# define the format of a city value
cityval = Group((price1Td | price2Td) + currency("currency") + amount("amt") + SkipTo(tdEnd) + tdEnd)

# define a comparison item, including item name and item cost in city1 and city2
comparison = Group(itemTd + SkipTo(tdEnd)("item") + tdEnd + (cityval*2)("valuedata"))

# attach a parse action to clean up automated token naming
def assignPriceTags(t):
    for v in t[0].valuedata:
        if v['class'] == 'price city-1':
            t[0]['price1'] = v
        else:
            t[0]['price2'] = v

    # remove extraneous results names created by makeHTMLTags
    for tg in 'class tag startTd endTd empty'.split():
        del t[0][tg]
        for v in t[0].valuedata:
            del v[tg]
    del t[0]['valuedata']
comparison.setParseAction(assignPriceTags)


currentcategory = ''
for compdata in (category|comparison).searchString(htmlsource):
    if 'category' in compdata:
        currentcategory = compdata.category
        continue
    compdata = compdata[0]
    #~ print compdata.dump()
    print "%s, %s, %s%s, %s%s" % (currentcategory, compdata.item,
                 compdata.price1.currency, compdata.price1.amt,
                 compdata.price2.currency, compdata.price2.amt)

打印：

Food, Daily menu in the business district, AU$15.82, NZ$15.62
Food, Combo meal in fast food restaurant (Big Mac Meal or similar), AU$7.4, NZ$7.91
Food, 1/2 Kg (1 lb.) of chicken breast, AU$6.07, NZ$10.25
Food, 1 liter (1 qt.) of whole fat milk, AU$1.8, NZ$2.65
Food, 500 gr (16 oz.) of local cheese, AU$5.99, NZ$7.2
Food, 1 kg (2 lb.) of apples, AU$4.29, NZ$3.46
Food, 2 kg (4,5 lb.) of potatoes, AU$4.31, NZ$5.29
Food, 0.5 l (16 oz) beer in the supermarket, AU$4.12, NZ$4.36
Food, 2 liters of Coca-Cola, AU$3.07, NZ$2.64
Food, bread for 2 people for 1 day, AU$2.32, NZ$1.93
Housing, monthly rent for a 85 m2 (900 Sqft) furnished apartment in expensive area of the city, AU$1766.0, NZ$2034.0
Housing, Internet 8MB (1 month), AU$49.0, NZ$61.0
Housing, 40” flat screen TV, AU$865.0, NZ$1041.0
Housing, utilities 1 month (heating, electricity, gas ...), AU$211.0, NZ$170.0
Clothes, 1 pair of Levis 501, AU$119.0, NZ$123.0
Clothes, 1 summer dress in a chain store (Zara, H&M, ...), AU$63.0, NZ$50.0
Clothes, 1 pair of Adidas trainers, AU$142.0, NZ$166.0
Clothes, 1 pair of average business shoes, AU$130.0, NZ$133.0
Transportation, Volkswagen Golf 2.0 TDI 140 CV 6 vel. (or equivalent), with no extras, new, AU$28321.0, NZ$45574.0
Transportation, 1 liter (1/4 gallon) of gas, AU$1.43, NZ$2.13
Transportation, monthly ticket public transport, AU$110.0, NZ$138.0
Personal Care, medicine against cold for 6 days (Frenadol, Coldrex, ...), AU$14.27, NZ$17.85
Personal Care, 1 box of 32 tampons (Tampax, OB, ...), AU$5.51, NZ$7.71
Personal Care, 4 rolls of toilet paper, AU$3.57, NZ$3.07
Personal Care, Tube of toothpaste, AU$3.37, NZ$3.39
Personal Care, Standard men's haircut in expat area of the city, AU$27.0, NZ$27.0
Entertainment, 2 tickets to the movies, AU$33.0, NZ$30.0
Entertainment, 2 tickets to the theater (best available seats), AU$163.0, NZ$139.0
Entertainment, dinner out for two in Italian restaurant with wine and dessert, AU$100.0, NZ$100.0
Entertainment, basic dinner out for two in neighborhood pub, AU$46.0, NZ$46.0
Entertainment, 1 cocktail drink in downtown club, AU$14.31, NZ$14.38
Entertainment, 1 beer in neighbourhood pub, AU$4.69, NZ$6.72
Entertainment, iPod nano 8GB (6th generation), AU$176.0, NZ$252.0
Entertainment, 1 min. of prepaid mobile tariff (no discounts or plans), AU$1.14, NZ$0.84
Entertainment, 1 month of gym in business district, AU$90.0, NZ$91.0
Entertainment, 1 package of Marlboro cigarretes, AU$15.97, NZ$14.47