I want to scrape information from a real estate website for market research purpose. My problem right now is that if some house doesn't have all the information I want (eg the estate agent didn't put the size of house in his exposé) then the script stops and I get an error.
I already tried it with if
/ else
statements but I made some mistake so that it doesn't work like this. I get the error:
"UnboundLocalError: local variable 'Flattyp' referenced before assignment"
def get_house_info (House):
for id in ids:
sourceCode = urllib.request.urlopen('https://www.immobilienscout24.de/expose/' + str(id)).read()
purchasePrice = str(sourceCode).split('"purchasePrice":')[1].split(',"geoCode"')[0]
Spacesize = str(sourceCode).split('"area":')[1].split('},"details"')[0]
District = str(sourceCode).split('"quarter":')[1].split('},')[0]
if Flattyp != str(sourceCode).split('"is24qa-typ grid-item three-fifths">')[1].split('</dd> </dl> <dl class')[0]:
Flattyp = "nicht vorhanden"
else:
Flattyp = str(sourceCode).split('"is24qa-typ grid-item three-fifths">')[1].split('</dd> </dl> <dl class')[0]
Rooms = str(sourceCode).split('is24qa-zimmer grid-item three-fifths"> ')[1].split(' </dd> </dl> <dl class=')[0]
parking_space = str(sourceCode).split('<dd class="is24qa-garage-stellplatz grid-item three-fifths">')[1].split('</dd> </dl>')[0]
if parking_space != str(sourceCode).split('<dd class="is24qa-garage-stellplatz grid-item three-fifths">')[1].split('</dd> </dl>')[0]:
parking_space = "nicht vorhanden"
else:
parking_space = str(sourceCode).split('<dd class="is24qa-garage-stellplatz grid-item three-fifths">')[1].split('</dd> </dl>')[0]
with open('fooneu23.txt', 'a') as csvfile:
cols = ['id', 'price', 'size', 'district', 'flattyp', 'rooms', 'parking_space','Flattypp']
dict_result = {'id': id, 'price': purchasePrice, 'size': Spacesize, 'district': District, 'flattyp': Flattyp, 'rooms': Rooms, 'parking_space':parking_space, 'Flattypp':Flattypp}
writer = csv.DictWriter(csvfile, fieldnames=cols)
writer.writeheader()
writer.writerow(dict_result)
csvfile.close()
Error is here :
if Flattyp != str(sourceCode).split('"is24qa-typ grid-item three-fifths">')[1].split('</dd> </dl> <dl class')[0]:
Flattyp = "nicht vorhanden"
else:
Flattyp = str(sourceCode).split('"is24qa-typ grid-item three-fifths">')[1].split('</dd> </dl> <dl class')[0]
At first iteration, you compare Flattyp
with something and Flattyp
has not been set yet.
For your initial issue, have you heard of try/catch statements ?
You should decode once and for all the response of you request instead of casting it to string multiple times :
sourceCode = urllib.request.urlopen(
'https://www.immobilienscout24.de/expose/' + str(id)
).read().decode('utf8')
Finally, you should never use str functions (regexp, split) for parsing html. Use appropriate html parsers like beautifulsoup .
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.