python requests cannot download a zip file while browser/selenium can

Question

I tried to use requests module to download a bunch of zip files with the code below:

s = requests.Session()
url='http://data.theice.com/MyAccount/Login.aspx'
z=s.get(url)
soup=BeautifulSoup(z.content,'html.parser')
hidden=soup.find_all('input',attrs={'type':'hidden'})
values={'ctl00$ContentPlaceHolder1$LoginControl$m_userName':'Acorn437',
    'ctl00$ContentPlaceHolder1$LoginControl$m_password':'*******',
    '__EVENTTARGET':'ctl00$ContentPlaceHolder1$LoginControl$LoginButton',
    '__EVENTARGUMENT':'',
    '__LASTFOCUS':''}
values=dict(values,**{i['id']:i['value'] for i in hidden})
z=s.post(url,data=values,allow_redirects=True)

After here, I verified that I have successfully loginned into the website by checking the response. Now I would like to download the zip file from a link on the website

link='http://data.theice.com/MyAccount/Download.aspx?PUID=69590&PDS=0&PRODID=580&TS=2018'
resp=s.get(link,allow_redirects=True)   
path=os.getcwd()+'\\data\\ice_zip\\'
fname='test.zip'
zfile=open(path+fname,'wb')
zfile.write(resp.content)
zfile.close()

However, it turned out that what I downloaded is acutally a html file intead of the zip file I need. I have no idea why the requests module does not work for this website. I think after I login in with requests.session, I should be able to download it because I can do it with a browser or the selenium module.

Clearly, I have no problem logining into the

Answer 1

This works for me - given of course you provide your own credentials and download path... I think your main problem might be that your login URL was wrong. When I ran your code I could NOT login to the site. The intial URL and the login URL are different ones.

import requests
from bs4 import BeautifulSoup

# define variables
username = ""
password = ""
path_to_store_output = ""

session = requests.Session()
r = session.get('http://data.theice.com/MyAccount/Login.aspx'')
soup=BeautifulSoup(r.text,'html.parser')

vs_generator = soup.find('input', attrs={'id': '__VIEWSTATEGENERATOR'}).get('value')
vs = soup.find('input', attrs={'id': '__VIEWSTATE'}).get('value')
event_validation = soup.find('input', attrs={'id': '__EVENTVALIDATION'}).get('value')


payload = {
    "__EVENTTARGET": "ctl00$ContentPlaceHolder1$LoginControl$LoginButton",
    "__EVENTARGUMENT":"", 
    "__LASTFOCUS": "", 
    "__VIEWSTATE": vs,
    "__VIEWSTATEGENERATOR": vs_generator,
    "__EVENTVALIDATION": event_validation,
    "ctl00$ContentPlaceHolder1$LoginControl$m_userName": username,
    "ctl00$ContentPlaceHolder1$LoginControl$m_password": password  
}
# doing a POST to login
r = session.post("http://www.ice.if5.com/MyAccount/Login.aspx", data=payload, headers={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'})

# check if we're logged in
if not username in r.text:
    print("[!] Bommer, dude! We're not logged in...")

else:
    print("[*] Score, we're in. Let's download stuff...")
    r = session.get("http://www.ice.if5.com/MyAccount/Download.aspx?PUID=70116&PDS=2&PRODID=4133", headers={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'})
    with open(path_to_store_output, 'wb') as f:
        f.write(r.content)

There's actually not much to this. Login and grab the stuff. Replace the url, I tested with whatever you're interested in. The one you provided gave me a 404. Cheers.