使用Beautiful Soup和Python的Web Scraping Asp.NET网站

Question

I have the following code, but its gives be 200 OK with first page (state of default drop down) response. Please note that the Drop Down lists are dymanic and progressive until final search button appears , Can someone correct me as to what is wrong with my code?

def process(ghatno):
    home_url = 'http://igrmaharashtra.gov.in/eASR/eASRCommon.aspx?hDistName=Nashik'
    post_url = 'http://igrmaharashtra.gov.in/eASR/eASRCommon.aspx?hDistName=Nashik'

    print "Please wait...getting details of :" + ghatno

    with requests.Session() as session:
        r = session.get(url=post_url)
        cookies = r.cookies
        pprint.pprint(r.headers)
        gethead = r.headers
        soup = BeautifulSoup(r.text, 'html.parser')
        viewstate = soup.select('input[name="__VIEWSTATE"]')[0]['value']
        csrftoken = soup.select('input[name="__CSRFTOKEN"]')[0]['value']
        eventvalidation = soup.select('input[name="__EVENTVALIDATION"]')[0]['value']
        viewgen = soup.select('input[name="__VIEWSTATEGENERATOR"]')[0]['value']

        data = {
            '__CSRFTOKEN':csrftoken,
            '__EVENTARGUMENT':'',
            '__EVENTTARGET':'',
            '__LASTFOCUS':'',
            '__SCROLLPOSITION':'0',
            '__SCROLLPOSITIONY':'0',
            '__EVENTVALIDATION': eventvalidation,
            '__VIEWSTATE':viewstate,
            '__VIEWSTATEGENERATOR': viewgen,
            'ctl00$ContentPlaceHolder5$ddlLanguage' : 'en-US',
            'ctl00$ContentPlaceHolder5$btnSearchCommonSr':'Search',
            'ctl00$ContentPlaceHolder5$ddlTaluka': '2',
            'ctl00$ContentPlaceHolder5$ddlVillage': '25',
            'ctl00$ContentPlaceHolder5$ddlYear': '20192020',
            'ctl00$ContentPlaceHolder5$grpSurveyLocation': 'rdbSurveyNo',
            'ctl00$ContentPlaceHolder5$txtCommonSurvey': 363
        }


        headers = {
        'Host': 'igrmaharashtra.gov.in',
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0',
        'Referer': 'http://igrmaharashtra.gov.in/eASR/eASRCommon.aspx?hDistName=Nashik',
        'Host': 'igrmaharashtra.gov.in',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
        'Accept-Language': 'en-US,en;q=0.5',

        }


        r = requests.post(url=post_url, data=json.dumps(data), cookies=cookies, headers = headers)
        soup = BeautifulSoup(r.text, 'html.parser')
        table = SoupStrainer('tr')
        soup = BeautifulSoup(soup.get_text(), 'html.parser', parse_only=table)
        print(soup.get_text())
        pprint.pprint(r.headers)
        print r.text
        getpost = r.headers
        getpostrequest = r.request.headers
        getresponsebody = r.request.body

        f = open('/var/www/html/nashik/hiren.txt', 'w')
        f.write(str(gethead))
        f.write(str(getpostrequest))
        f.write(str(getresponsebody))
        f.write(str(getpost))

My response is as below :

Response header - (GET Request)

{'Content-Length': '5994', 'X-AspNet-Version': '4.0.30319', 'Set-Cookie': 'ASP.NET_SessionId=24wwh11lwvzy5gf0xlzi1we4; path=/; HttpOnly, __CSRFCOOKIE=d7b10286-fc9f-4ed2-863d-304737df8758; path=/; HttpOnly', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding', 'X-Powered-By': 'ASP.NET', 'Server': 'Microsoft-IIS/8.0', 'Cache-Control': 'private', 'Date': 'Thu, 02 May 2019 08:21:48 GMT', 'Content-Type': 'text/html; charset=utf-8'}

Response header - (GET Request)

{'Content-Length': '3726', 'Accept-Language': 'en-US,en;q=0.5', 'Accept-Encoding': 'gzip, deflate', 'Host': 'igrmaharashtra.gov.in', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0', 'Connection': 'keep-alive', 'Referer': 'http://igrmaharashtra.gov.in/eASR/eASRCommon.aspx?hDistName=Nashik', 'Cookie': '__CSRFCOOKIE=d7b10286-fc9f-4ed2-863d-304737df8758; ASP.NET_SessionId=24wwh11lwvzy5gf0xlzi1we4', 'Content-Type': 'application/x-www-form-urlencoded'}

Response header - (POST Request)

{'Content-Length': '7834', 'X-AspNet-Version': '4.0.30319', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding', 'X-Powered-By': 'ASP.NET', 'Server': 'Microsoft-IIS/8.0', 'Cache-Control': 'private', 'Date': 'Fri, 03 May 2019 10:21:45 GMT', 'Content-Type': 'text/html; charset=utf-8'}

**Default Page Selected Drop Down is returned **

नाशिक and - - Select Taluka - - INSTEAD of option value "2" ie इगतपुरी once option "2" is selected I want value "25" in next drop down before I put my final survey "363" for results.

Please note I tried Mechanize browser too, but no luck !!

Answer 1

Finally the solution is to do post requests multiple times in same "session" with same "cookie" and iterate through them. It works now !