Python-使用BeautifulSoup在页面内抓取多个类

Question

I am trying to crawl Agoda's daily hotel price of multiple room types along with additional information such as the promotion information, breakfast condition, and book-now-pay-later regulation.

The codes I have are as below:

import requests
import math
from bs4 import BeautifulSoup

url = "http://www.agoda.com/ambassador-hotel-taipei/hotel/taipei-tw.html?asq=8m91A1C3D%252bTr%252bvRSmuClW5dm5vJXWO5dlQmHx%252fdU9qxilNob5hJg0b218wml6rCgncYsXBK0nWktmYtQJCEMu0P07Y3BjaTYhdrZvavpUnmfy3moWn%252bv8f2Lfx7HovrV95j6mrlCfGou99kE%252bA0aX0aof09AStNs69qUxvAVo53D4ZTrmAxm3bVkqZJr62cU&tyra=1%257c2&searchrequestid=2e2b0e8c-cadb-465b-8dea-2222e24a1678&pingnumber=1&checkin=2015-10-01&los=1"
res = requests.get(url)
soup = BeautifulSoup(res.text, 'html.parser')
n = len(soup.select('.room-name'))

for i in range(0, n):
    en_room = soup.select('.room-name')[i].text.strip()
    currency = soup.select('.currency')[i].text
    price = soup.select('.sellprice')[i].text

    try:
        sp_info = soup.select('.left-room-info')[i].text.strip()
    except Exception as e:
        sp_info = "N/A"

    try:
        pay_later = soup.select('.book-now-paylater')[i].text.strip()
    except Exception as e:
        pay_later = "N/A"


    print en_room, i+1, currency, price, en_room, sp_info, pay_later
    time.sleep(1)

I have two questions:

(1) The "left-room-info" class seems to contain two sub-classes "breakfast" and "room-promo". These sub-classes only show up when the particular room type provides such services.

When there is only one of the sub-classes shows up, the output works out well. However, when none of the sub-classes shows up, the output is empty when I expect to show "N/A". Also when both of the sub-classes show up, the output format has unnecessary empty lines which cannot be removed by .strip().

Is there any way to solve these problems?

(2) When I tried to extract information from the class '.book-now-paylater', the extracted data does not match each room type. For example, assuming there are 10 room types and only room 2, 4, 6, 8 allow travelers to book now pay later, the codes can extract exactly 4 pieces of book-now-pay-later information but these 4 pieces of information are then assigned inappropriately to room type 1, 2, 3, 4.

Is there any way to fix this problem?

Thank you for your help!

Gary

Answer 1

(1) This is happening because even if there is no text in the '.left-room-info' selection, it won't throw an exception, and your except will never run. You should be checking to see if the value is an empty string ( '' ). You can do this with a simple if not string_var like this

sp_info = soup.select('.left-room-info')[i].text.strip()
if not sp_info:
    sp_info = "N/A"

When both subclasses show up, you should split the string on the carriage return ( '\\r' ) and then strip each of the resulting pieces. The code would look something like this: (note that now sp_info is a list, not just a string)

sp_info = soup.select('.left-room-info')[i].text.strip().split('\r')
if len(sp_info) > 1:
    sp_info = [ info.strip() for info in sp_info ]

Putting these pieces together, we'll get something like this

sp_info = soup.select('.left-room-info')[i].text.strip().split('\r')
if len(sp_info) > 1:
    sp_info = [ info.strip() for info in sp_info ]
elif not sp_info[0]: # check for empty string
    sp_info = ["N/A"] # keep sp_info a list for consistancy 

(2) is a little more complicated. You're going to have to change how you parse the page. Namely, you're probably going to have to select on .room-type . The way you're selecting the book now pay laters, it doesn't associate them with any other elements, it just selects the 8 instances of that class. Here is how I would go about doing it:

import requests
import math
from bs4 import BeautifulSoup

url = "http://www.agoda.com/ambassador-hotel-taipei/hotel/taipei-tw.html?asq=8m91A1C3D%252bTr%252bvRSmuClW5dm5vJXWO5dlQmHx%252fdU9qxilNob5hJg0b218wml6rCgncYsXBK0nWktmYtQJCEMu0P07Y3BjaTYhdrZvavpUnmfy3moWn%252bv8f2Lfx7HovrV95j6mrlCfGou99kE%252bA0aX0aof09AStNs69qUxvAVo53D4ZTrmAxm3bVkqZJr62cU&tyra=1%257c2&searchrequestid=2e2b0e8c-cadb-465b-8dea-2222e24a1678&pingnumber=1&checkin=2015-10-01&los=1"
res = requests.get(url)
soup = BeautifulSoup(res.text)

rooms = soup.select('.room-type')[1:] # the first instance of the class isn't a room

room_list = []

for room in rooms:
    room_info = {}

    room_info['en_room'] = room.select('.room-name')[0].text.strip()
    room_info['currency'] = room.select('.currency')[0].text.strip()
    room_info['price'] = room.select('.sellprice')[0].text.strip()

    sp_info = room.select('.left-room-info')[0].text.strip().split('\r')
    if len(sp_info) > 1:
        sp_info = ", ".join([ info.strip() for info in sp_info ])
    elif not sp_info[0]: # check for empty string
        sp_info = "N/A"
    room_info['sp_info'] = sp_info

    pay_later = room.select('.book-now-paylater')
    room_info['pay_later'] = pay_later[0].text.strip() if pay_later else "N/A"

    room_list.append(room_info)

Answer 2

In your code, you are not traversing the dom correctly. This will cause problems in scraping. (eg second problem). I shall give suggestive code snippet(not exact solution) hopeing you could solve the first problem by yourself.

# select all room types by tables tr tag
room_types = soup.find_all('tr', class_="room-type")

# iterate over the list to scrape data form each td or div inside tr
for room in room_types:
    en_room = room.find('div', class_='room-name').text.strip()