使用 BeautifulSoup 从网站上抓取每个元素

Question

I wrote a code for scraping one real estate website. This is the link:

https://www.nekretnine.rs/stambeni-objekti/stanovi/lista/po-stranici/10/

From this page I can get only location, size and price of the apartment, but Is it possible to write a code that will go on page of each appartment and scrape values from it, because it contains much more info. Check this link:

https://www.nekretnine.rs/stambeni-objekti/stanovi/arena-bulevar-arsenija-carnojevica-97m-2-lode-energoprojekt/NkvJK0Ou5tV/

I have posted a code. I noticed that my url changes when I click on specific real estate. For example:

arena-bulevar-arsenija-carnojevica-97m-2-lode-energoprojekt/NkvJK0Ou5tV/

I taught about creating for loop, but there is no way to know how it changes because it has some id number at the end:

NkvJK0Ou5tV

This is the code that I have:

from bs4 import BeautifulSoup
import requests

website = "https://www.nekretnine.rs/stambeni-objekti/stanovi/lista/po-stranici/10/"

soup = requests.get(website).text
my_html = BeautifulSoup(soup, 'lxml')

lokacija = my_html.find_all('p', class_='offer-location text-truncate')
ukupna_kvadratura = my_html.find_all('p', class_='offer-price offer-price--invert')
ukupna_cena = my_html.find_all('div', class_='d-flex justify-content-between w-100')
ukupni_opis = my_html.find_all('div', class_='mt-1 mb-1 mt-lg-0 mb-lg-0 d-md-block offer-meta-info offer-adress')


for lok, kvadratura, cena_stana, sumarno in zip(lokacija, ukupna_kvadratura, ukupna_cena, ukupni_opis):

    lok = lok.text.split(',')[0] #lokacija

    kv = kvadratura.span.text.split(' ')[0] #kvadratura
    jed = kvadratura.span.text.split(' ')[1] #jedinica mere

    cena = cena_stana.span.text #cena

    sumarno = sumarno.text

    datum = sumarno.split('|')[0].strip()
    status = sumarno.split('|')[1].strip()
    opis = sumarno.split('|')[2].strip()

    print(lok, kv, jed, cena, datum, status, opis)

Answer 1

You can get href from div class="placeholder-preview-box ratio-4-3". From here you can find the URL.

Answer 2

You can iterate over the links provided by the pagination at the bottom of the page:

from bs4 import BeautifulSoup as soup
import requests
d = soup(requests.get('https://www.nekretnine.rs/stambeni-objekti/stanovi/lista/po-stranici/10/').text, 'html.parser')
def scrape_page(page):
   return [{'title':i.h2.get_text(strip=True), 'loc':i.p.get_text(strip=True), 'price':i.find('p', {'class':'offer-price'}).get_text(strip=True)} for i in page.find_all('div', {'class':'row offer'})]

result = [scrape_page(d)]
while d.find('a', {'class':'pagination-arrow arrow-right'}):
   d = soup(requests.get(f'https://www.nekretnine.rs{d.find("a", {"class":"pagination-arrow arrow-right"})["href"]}').text, 'html.parser')
   result.append(scrape_page(d))