[英]Scraping dynamic website by Python Selenium
我嘗試通過 BS4 python 抓取動態網站:
https://www.nadlan.gov.il/?search=%D7%AA%D7%9C%20%D7%90%D7%91%D7%99%D7%91%20%20%D7%99% D7%A4%D7%95
我試過:
from urllib.request import urlopen
from bs4 import BeautifulSoup
page = urlopen(wiki)
soup = BeautifulSoup("https://www.nadlan.gov.il/?search=תל אביב יפו")
我有兩個問題:
該網站是動態的,當我查看頁面源代碼時,我沒有看到頁面內容只有 JavaScript 腳本:
<script> document.write("<script src='scripts/dis/bundleJS.js?v=" + globalAppVersion + "'><\/script>") document.write("<script id='srcGovmap' src='https://new.govmap.gov.il/govmap/api/govmap.api.js?v='" + globalAppVersion + "'><\/script>") document.write("<script src='MainLoader.js?v=" + globalAppVersion + "'><\/script>") document.write("<script id='tld-search-srcipt' src='https://www.nadlan.gov.il/TldSearch/Scripts/ac.js?v=" + globalAppVersion + "'><\/script>"); </script> <script src="scripts/dis/accessibility/b1.js?v=3" type="text/javascript"></script> <script type="text/javascript"> accessibility_rtl = true; pixel_from_side = 20; pixel_from_start = 15; $(document).ready(function () { $('#accessibility_icon').attr('src', 'images/accessibility_icon.png') $('.accessibility_div_wrap>.btn_accessibility > span.accessibility_component').html('') });
當我打開網站時,加載數據需要幾秒鍾:
Selenium如何解決這些問題?
數據通過 JavaScript 動態加載。您可以使用requests
/ json
模塊模擬 Ajax 調用。 例如:
import json
import requests
url = 'https://www.nadlan.gov.il/Nadlan.REST/Main/GetAssestAndDeals'
data = {"MoreAssestsType":0,"FillterRoomNum":0,"GridDisplayType":0,"ResultLable":"תל אביב -יפו","ResultType":1,"ObjectID":"5000","ObjectIDType":"text","ObjectKey":"UNIQ_ID","DescLayerID":"SETL_MID_POINT","Alert":None,"X":180428.31832654,"Y":665726.5550939,"Gush":"","Parcel":"","showLotParcel":False,"showLotAddress":False,"OriginalSearchString":"תל אביב יפו","MutipuleResults":False,"ResultsOptions":None,"CurrentLavel":2,"Navs":[{"text":"מחוז תל אביב - יפו","url":None,"order":1}],"QueryMapParams":{"QueryToRun":None,"QueryObjectID":"5000","QueryObjectType":"number","QueryObjectKey":"SETL_CODE","QueryDescLayerID":"KSHTANN_SETL_AREA","SpacialWhereClause":None},"isHistorical":False,"PageNo":1,"OrderByFilled":"DEALDATETIME","OrderByDescending":True,"Distance":0}
result = requests.post(url, json=data).json()
# uncomment this to print all data:
# print(json.dumps(result, indent=4))
# print all results to screen:
for r in result['AllResults']:
for k, v in r.items():
print('{:<30} {}'.format(k, v))
print('-' * 80)
印刷:
DEALDATE 12.12.2020
DEALDATETIME 2020-12-12T00:00:00
FULLADRESS
DISPLAYADRESS
GUSH 7104-289-264
DEALNATUREDESCRIPTION דירה
ASSETROOMNUM 3
FLOORNO None
DEALNATURE 90
DEALAMOUNT 3,650,000
NEWPROJECTTEXT 1
PROJECTNAME מגדלי גינדי תל אביב
BUILDINGYEAR None
YEARBUILT
BUILDINGFLOORS None
KEYVALUE 10812534855
TYPE 2
POLYGON_ID 7104-289
TREND_IS_NEGATIVE False
TREND_FORMAT
--------------------------------------------------------------------------------
DEALDATE 31.07.2020
DEALDATETIME 2020-07-31T00:00:00
FULLADRESS עגנון ש"י 28, תל אביב -יפו
DISPLAYADRESS עגנון ש"י 28
GUSH 6634-336-33
DEALNATUREDESCRIPTION דירה
ASSETROOMNUM 5
FLOORNO None
DEALNATURE 130
DEALAMOUNT 6,363,000
NEWPROJECTTEXT 1
PROJECTNAME הפילהרמונית
BUILDINGYEAR 2020
YEARBUILT
BUILDINGFLOORS 9
KEYVALUE 10812534851
TYPE 1
POLYGON_ID 6634-336
TREND_IS_NEGATIVE False
TREND_FORMAT
--------------------------------------------------------------------------------
...and so on.
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.