抓取时字典循环调用同一网站

Question

I'm very new to Python so this is probably straightforward, and might be an indentation issue. I'm trying to scrape over several webpages using beautiful soup, creating a list of dictionaries that I can use afterwards to manipulate the data.

The code seems to work fine, but the list I end up with (liste_flat) is just a list of the same two dictionaries. I want a list of different dictionaries.

def scrap_post(url):
    url = "https://www.findproperly.co.uk/property-to-rent-london/commute/W3siaWQiOjkxMDYsImZyZXEiOjUsIm1ldGgiOiJwdWJ0cmFucyIsImxuZyI6LTAuMTI0Nzg5LCJsYXQiOjUxLjUwODR9XQ==/max-time/90/page/".format(i)
    dictionary = {}
    response = requests.get(url)
    soup = bs(response.text,"lxml")
    taille = len(soup.find_all("div", class_="col-sm-6 col-md-4 col-lg-3 pl-grid-prop not-viewed ")) #48 entries
    for num_ville in range(0,taille):
        print(num_ville)
        apt_id = soup.find_all("div", class_="col-sm-6 col-md-4 col-lg-3 pl-grid-prop not-viewed ")[num_ville]['data-id']
        entry = soup.find_all("div", class_="col-sm-6 col-md-4 col-lg-3 pl-grid-prop not-viewed ")[num_ville]
        pricepw = soup.find_all('div', class_='col-xs-5 col-sm-4 price')[num_ville].find('h3').text.encode('utf-8').replace('\xc2\xa3','',).replace('pw','',).strip()
        rooms = soup.find_all('div', class_='col-xs-6 type')[num_ville].find('p').text.encode('utf-8').strip()
        lat = soup.find_all('div', {"itemprop":"geo"})[num_ville].find('meta', {'itemprop':'latitude'})['content']
        lon = soup.find_all('div', {"itemprop":"geo"})[num_ville].find('meta', {'itemprop':'longitude'})['content']
        dictionary[num_ville]={'Price per week':pricepw,'Rooms':rooms,'Latitude':lat,'Longitude':lon}
    return dictionary

#get all URLs
liste_url = []
liste_url = ['https://www.findproperly.co.uk/property-to-rent-london/commute/W3siaWQiOjkxMDYsImZyZXEiOjUsIm1ldGgiOiJwdWJ0cmFucyIsImxuZyI6LTAuMTI0Nzg5LCJsYXQiOjUxLjUwODR9XQ==/max-time/90/page/''%i' %i for i in range(1,3)]

#get flats
liste_flat = [scrap_post(i) for i in liste_url] 

I must somehow be looping over the same website twice. Any advice on how to make sure I'm looping over different websites?

Thanks!

Answer 1

Yes, you are looping over the same website, because you have hardcoded the url variable in your function.

url = "https://www.findproperly.co.uk/property-to-rent-london/commute/W3siaWQiOjkxMDYsImZyZXEiOjUsIm1ldGgiOiJwdWJ0cmFucyIsImxuZyI6LTAuMTI0Nzg5LCJsYXQiOjUxLjUwODR9XQ==/max-time/90/page/".format(i)

Meaning regardless of what you send to the function, it will always use this url. You might want to remove that. You also haven't placed a placeholder in your string and the .format(i) essentially does nothing.