Python BeautifulSoup 解析/爬表

Question

For my own interst I want to crawl the table of properties from ```"https://thinkimmo.com/search?noReset=true". After having clicked on "TABELLE" (TABLE) you can see all properties listed in a table.

Whit the following code I am able to see the table:
driver.get("https://thinkimmo.com/search?noReset=true")
driver.find_element_by_xpath('/html/body/div[1]/div[2]/div[2]/div/div[2]/div/div[2]/div/div/div/div[1]/div/div/button[2]/span[1]').click()

Now I am able to crawl some parts of the table with the following code:
soup = BeautifulSoup(driver.page_source, 'html.parser')
htmltable = soup.find('table', { 'class' : 'MuiTable-root' })
def tableDataText(table):       
    rows = []
    trs = table.find_all('tr')
    headerow = [td.get_text(strip=True) for td in trs[0].find_all('th')] # header row
    if headerow: # if there is a header row include first
        rows.append(headerow)
        trs = trs[1:]
    for tr in trs: # for every table row
        rows.append([td.get_text(strip=True) for td in tr.find_all('td')]) # data row
    return rows
list_table = tableDataText(htmltable)
list_table

The result however is not what I expect. I only get the first 7 headings, but all other headings are not returned.

After I had a closer look in the HTMl of the webpage I am not sure how to get all headings and results of the table.

I am looking forward to solve the problem of getting only some parts of the heading. And more closely I am interested in why I am failing.


What I see in the result of table = soup.find("table") is that after the 7th heading title the table closes.  

Thanks in advance.

Steffen

Answer 1

The site uses a backend api you can edit to bulk download data:

import requests
import pandas as pd

results = 1000

url = f'https://api.thinkimmo.com/immo?active=true&type=APARTMENTBUY&sortBy=publishDate,desc&from=0&size={str(results)}&grossReturnAnd=false&allowUnknown=false&excludePlatforms=ebk,immowelt&favorite=false&noReset=true&excludedFields=true&geoSearches=[]&averageAggregation=buyingPrice%3BpricePerSqm%3BsquareMeter%3BconstructionYear%3BrentPrice%3BrentPricePerSqm%3BrentPricePerSqm%3BrunningTime&termsAggregation=platforms.name.keyword,60'

resp = requests.get(url).json()
df = pd.DataFrame(resp['results'])
df.to_csv('thinkimmo.csv',index=False)
print('Saved to thinkimmo.csv')

This is alot of unstructured data but should help. If you want to inspect what is in this api call and only get certain parts of the returned JSON then you can open your browser's Developer Tools - Network - fetch/XHR and reload the page to see all the backend requests fire. You are looking for one that starts "immo?" take a look at the Payload and Preview to see all the data. That's what we are scraping above.