网络抓取：使用python从air的xpath中提取url：airbnb列表

Question

I am trying to extract urls for listings from a city page in AirBnb, using python 3 libraries. I am familiar with how to scrape simpler websites with Beautifulsoup and requests libraries.

url: ' https://www.airbnb.com/s/Denver--CO--United-States/homes '

element in the html

If I inspect the element of a link on the page (in Chrome), I get:

xpath: "//*[@id="listing-9770909"]/div[2]/a"
selector: "listing-9770909 > div._v72lrv > a"

My attempts:

import requests
from bs4 import BeautifulSoup

url = 'https://www.airbnb.com/s/Denver--CO--United-States/homes'
html = requests.get(url)
soup = BeautifulSoup(html.text, 'html.parser')
divs = soup.find_all('div', attrs={'id': 'listing'})

attempt 2:

import requests
from lxml import html

page = requests.get(url)
root = html.fromstring(page.content)
tree = root.getroottree()
result = root.xpath('//div[@id="listing-9770909"]/div[2]/a')
for r in result:
    print(r)

Neither of these returns anything. What I need to be able to extract is the url for the page link. Any ideas?

Answer 1

To extract the links, first you have to make sure that the urls to the links exists in the page source. For this you can search with any of the listing ids in the page source(ctrl+u if you are using google chrome,mozilla firefox). If the urls exist in the page source you can directly scrape them using xpath in the response text of the listing page. Here the above listing page of Airbnb is not having the links in the page source, so the page might be sending requests to some other pages(usually json requests). You can find out those requests and send requests to those pages and get the required data. Please comment if you have any doubt regarding this.