Crawling IDs and getting TypeError: list indices must be integers or slices, not str

Question

I try to get ids from a real estate website. In my first attempt i just got always the first real estate in one building project. Now i tried to go futher down the html-tree to get the rest of the building project. But im getting this error:

"TypeError: list indices must be integers or slices, not str"

The HTML looks like this:

  "resultlist.resultlist": { "paging": { "next": { "@xlink.href": "\\/Suche\\/ST\\/P-2\\/Wohnung-Kauf\\/Nordrhein-Westfalen\\/Duesseldorf\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/true" }, "current": { "@xlink.href": "\\/Suche\\/ST\\/Wohnung-Kauf\\/Nordrhein-Westfalen\\/Duesseldorf\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/-\\/true" }, "pageNumber": 1, "pageSize": 20, "numberOfPages": 3, "numberOfHits": 140, "numberOfListings": 50 }, "matchCountList": "", "resultlistEntries": [{ "@numberOfHits": "140", "@realEstateType": "2", "resultlistEntry": [{ "@id": "111337199", "@modification": "2019-06-09T13:36:23.513+02:00", "@creation": "2019-05-05T14:10:47.000+02:00", "@publishDate": "2019-05-05T14:10:47.000+02:00", "similarObjects": [{ "similarObject": [{ "@id": "105147583", "@modification": "2019-05-05T10:37:59.830+02:00", "@creation": "2018-05-30T11:44:29.000+02:00", "@publishDate": "2018-05-30T11:44:29.000+02:00", "realEstateId": 105147583, 

My First attempt looked like this:

page1 = ('https://www.immobilienscout24.de/Suche/S-1/Wohnung-Kauf/Nordrhein-Westfalen/Duesseldorf/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/true')
res_page1 = requests.get(page1)
soup_page1 = bs(res_page1.content, 'lxml')
r_page1 = re.compile(r'resultListModel:(.*)')
data_page1 = soup_page1.find('script', text=r_page1).text
script_page1 = r_page1.findall(data_page1)[0].rstrip(',')
results_page1 = json.loads(script_page1)
ids_page1 = [item["@id"] for item in results_page1['searchResponseModel']['resultlist.resultlist']['resultlistEntries'][0]['resultlistEntry']]

And got this output:

['111353960', '110253440', '111994208', '110517626', '109984070', '109855231', '108761945', '108639776', '106997219', '106492496', '111604737', '111357085', '92741038', '112031279', '111988597', '111876292', '111870285', '111798416', '110742328', '110742299']

Now i went down to get the rest as well like this:

ids_page1 = [item["@id"] for item in results_page1['searchResponseModel']['resultlist.resultlist']['resultlistEntries']['resultlistEntry']['similarObjects'][0]['similarObject']]

And now Im getting the Type Error.

Could someome please explain what i am doing wrong.

Answer 1

Try the following which goes through the levels

from bs4 import BeautifulSoup as bs #4.7.1
import requests

page1 = 'https://www.immobilienscout24.de/Suche/S-1/Wohnung-Kauf/Nordrhein-Westfalen/Duesseldorf/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/true'
res_page1 = requests.get(page1)
soup_page1 = bs(res_page1.content, 'lxml')
r_page1 = re.compile(r'resultListModel:(.*)')
data_page1 = soup_page1.find('script', text=r_page1).text
script_page1 = r_page1.findall(data_page1)[0].rstrip(',')
results_page1 = json.loads(script_page1)
ids = []

for item in results_page1['searchResponseModel']['resultlist.resultlist']['resultlistEntries'][0]['resultlistEntry']:
    ids.append(item['@id'])
    if 'similarObjects' in item:
        for i in item['similarObjects'][0]['similarObject']:
            if isinstance(i,dict):
                ids.append(i['@id'])
            elif i == '@id':
                ids.append(item['similarObjects'][0]['similarObject'][i])

Answer 2

In the output, you see a list of strings. Note the ' surrounding the values. Call int(s) on a string of digits to turn it into a number.