没有[href]的多层网站上的Python网络抓取

Question

I am looking for a way to scrape data from the student-accomodation website uniplaces: https://www.uniplaces.com/en/accommodation/berlin .

In the end, I would like to scrape particular information for each property, such as bedroom size, number of roommates, location. In order to do this, I will first have to scrape all property links and then scrape the individual links afterwards.

However, even after going through the console and using BeautifulSoup for the extraction of urls, I was not able to extract the urls leading to the separate listings. They don't seem to be included as a [href] and I wasn't able to identify the links in any other format within the html code.

This is the python code I used but it also didn't return anything: from bs4 import BeautifulSoup import urllib.request

resp = urllib.request.urlopen("https://www.uniplaces.com/accommodation/lisbon")
soup = BeautifulSoup(resp, from_encoding=resp.info().get_param('charset'))

for link in soup.find_all('a', href=True):
    print(link['href'])

So my question is: If links are not included in http:// format or referenced as [href]: is there any way to extract the listings urls?

I would really highly appreciate any support on this!

All the best, Hannah

Answer 1

If you look at the network tab, you find some API call specifically to this url : https://www.uniplaces.com/api/search/offers?city=PT-lisbon&limit=24&locale=en_GB&ne=38.79507211908374%2C-9.046124472314432&page=1&sw=38.68769060641113%2C-9.327992453271463

which specifies the location PT-lisbon and northest(ne) and southwest(sw) direction. From this file, you can get the id for each offers and append it to the current url, you can also get all info you get from the webpage (price, description etc...)

For instance :

import requests

resp = requests.get(
    url = 'https://www.uniplaces.com/api/search/offers', 
    params = {
        "city":'PT-lisbon',
        "limit":'24',
        "locale":'en_GB',
        "ne":'38.79507211908374%2C-9.046124472314432',
        "page":'1',
        "sw":'38.68769060641113%2C-9.327992453271463'
    })
body = resp.json()

base_url = 'https://www.uniplaces.com/accommodation/lisbon'

data = [
    (
        t['id'],                  #offer id
        base_url + '/' + t['id'], #this is the offer page
        t['attributes']['accommodation_offer']['title'], 
        t['attributes']['accommodation_offer']['price']['amount'],
        t['attributes']['accommodation_offer']['available_from']
    )
    for t in body['data']
]

print(data)