我尝试了很多次从预订中获取数据。com。但我不能

Question

I want to scrape the data from the booking.com but got some errors and couldn't find any similar codes. I want to scrape the name of the hotel,price and etc.

i have tried beautifulSoup 4 and tried to get data to a csv file.

import requests
from bs4 import BeautifulSoup
import pandas

# Replace search_url with a valid one byb visiting and searching booking.com
search_url = 'https://www.booking.com/searchresults.....'
page = requests.get(search_url)
soup = BeautifulSoup(page.content, 'html.parser')

week = soup.find(id = 'search_results_table'  )
#print(week)

items = week.find_all(class_='sr-hotel__name')
print(items[0])
print(items[0].find(class_ = 'sr-hotel__name').get_text())
print(items[0].find(class_ = 'short-desc').get_text())

Here is a sample URL that can be used in place of search_url .

This is the error msg...

<span class="sr-hotel__name " data-et-click="
">
The Fort Printers
</span>
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-44-77b38c8546bb> in <module>
     11 items = week.find_all(class_='sr-hotel__name')
     12 print(items[0])
---> 13 print(items[0].find(class_ = 'sr-hotel__name').get_text())
     14 print(items[0].find(class_ = 'short-desc').get_text())
     15 

AttributeError: 'NoneType' object has no attribute 'get_text'

Answer 1

first of all, buddy, using requests might be really hard since you have to completely imitate the request your browser will send. You'll have to use some sniffing tool (burp, fiddler, wireshark) or in some cases look at the network in the developer mode on your browser which is relatively hard...

I'd suggest you to use "selenium" which is a web driver that makes your life easy when trying to scrape sites... read more about it here- https://medium.com/the-andela-way/introduction-to-web-scraping-using-selenium-7ec377a8cf72

And as for your error, I think you should use only.text instead of.get_text()

Answer 2

Instead of using find() method multiple times, if you consider using getText() method directly it can help.

import requests
from bs4 import BeautifulSoup
import pandas

# Replace search_url with a valid one byb visiting and searching booking.com
search_url = 'https://www.booking.com/searchresults.....'
page = requests.get(search_url)
soup = BeautifulSoup(page.content, 'html.parser')

week = soup.find(id = 'search_results_table'  )
#print(week)

items = week.find_all(class_='sr-hotel__name')
# print the whole thing
print(items[0])
hotel_name = items[0].getText()

# print hotel name
print(hotel_name)

# print without newlines
print(hotel_name[1:-1])

Hope this helps. I would suggest reading more of BeautifulSoup documentation.