如何使用 Python 从 .aspx 页面检索数据？

Question

I am looking to access data from a .aspx website with a number of fields where parameters need to be entered. The data will be further analyzed in Pandas. I'm obviously missing some steps here, so any help would be appreciated. The website is https://www.cocorahs.org/ViewData/StationPrecipSummary.aspx

I'm trying a simple method using the Python library Requests, getting the json, and converting to a DataFrame.

parameters = {'Station 1':'MD-BL-13','Start Date':'8/01/2019','End Date':'08/10/2017'}
response = requests.get('https://www.cocorahs.org/ViewData/StationPrecipSummary.aspx', params=parameters)
data = response.json()
pd.read_json(data)

I would like to get a DataFrame with columns 'Date' and 'Precip mm' with data from the time period requested. A check of response.content shows that the parameters are not correctly taken, as only the content of the web page before a query has been entered appears.

Answer 1

I find ASP.NET sites to be a pain in the ass to deal with, but here's a solution with pandas and requests-html.

from requests_html import HTMLSession
import pandas as pd


with HTMLSession() as s:

    r = s.get('https://www.cocorahs.org/ViewData/StationPrecipSummary.aspx')
    hiddens = r.html.find('input[name=__VIEWSTATE]', first=True).attrs.get('value')

    payload = {
        '__EVENTTARGET': '',
        '_VIEWSTATE': hiddens,
        'obsSwitcher:ddlObsUnits': 'usunits',
        'tbStation1': 'MD-BL-13',
        'ucDateRangeFilter:dcStartDate': '8/1/2019',
        'ucDateRangeFilter_dcStartDate_p': '2019-8-1-0-0-0-0',
        'ucDateRangeFilter:dcEndDate': '8/10/2019',
        'ucDateRangeFilter_dcEndDate_p': '2019-8-10-0-0-0-0',
        'btnSubmit': 'Get Summary'
        }


    r = s.post('https://www.cocorahs.org/ViewData/StationPrecipSummary.aspx', data=payload)
    table = r.html.find('table.Grid', first=True)
    df = pd.read_html(table.html, header=0)[0]
    print(df)


          Date Precip in.
0   08/01/2019       0.00
1   08/02/2019       0.00
2   08/03/2019       0.00
3   08/04/2019       0.00
4   08/05/2019       0.00
5   08/06/2019       0.00
6   08/07/2019          T
7   08/08/2019       1.73
8   08/09/2019         --
9   08/10/2019         --
10    Totals :   1.73 in.