Suppose I want to download data here: http://www.dce.com.cn/publicweb/quotesdata/memberDealPosiQuotes.html
When click the button shown below, I got a .csv
file:
I want to do this automatically using python where I can specify the date etc.
I find here that one can use pandas pd.read_csv
to read data from webpage, but first one need to get the right url. However in my case I don't know what the url is.
Besides, I also want to specify the date and the contract etc. myself.
Before asking, I actually tried to the dev tool, I still can't see the url, and I don't know how to make it programatic.
The javascript exportData('excel')
results in a form that is submitted. By using Chrome devtools and the Network panel, you can figure out the headers and the post data used, and then write a python script to submit an identical http request.
import requests
url = 'http://www.dce.com.cn/publicweb/quotesdata/exportMemberDealPosiQuotesData.html'
formdata = {
'memberDealPosiQuotes.variety':'a',
'memberDealPosiQuotes.trade_type':0,
'contract.contract_id':'all',
'contract.variety_id':'a',
'exportFlag':'excel',
}
response = requests.post(url, data=formdata)
filename = response.headers.get('Content-Disposition').split('=')[-1]
with open(filename, 'wb') as fp:
fp.write(response.content)
It's probably possible to find ways to modify the post data to fetch different data. Either by reverse engineering, by trial and error or by finding some documentation.
For example, you can include fields for year and date:
'year':2017,
'month':3,
'day':20
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.