Using requests and HTTPBasicAuth to retrieve a zip file from a password-protected site
Question
I am trying to use requests and HTTPBasicAuth to retrieve data from a website that requires a username/password, and while I have tried to follow the usual advice in getting this to work, it doesn't seem to be functioning correctly.
Specifically, I am trying to download data from Columbia's SEDAC website as follows.
url = "http://sedac.ciesin.columbia.edu/downloads/data/sdei/sdei-global-nnual-avg-pm2-5-modis-misr-seawifs-aod-1998-2012/global-annual-avg-pm2-5-modis-misr-seawifs-aod-2001-2010-geotiff.zip"
username = '<name>'
password = '<password>'
r = requests.get(url, auth=HttpBasicAuth(username, password))
Now, as it happens, if I look at the content of the request, I find that it gives me an html file, not the .zip file that I desire. Something like:
r.content
b'<!DOCTYPE html>\n<!--[if lt IE 7]><html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->\n<!--[if IE 7]><html class="no-js lt-ie9 lt-ie8"> <![endif]-->\n<!--[if IE 8]><html class="no-js lt-ie9"> <![endif]-->\n<!--[if gt IE 8]><!--><html lang="en" class="no-js"><!--<![endif]-->\n <head>\n <meta charset="utf-8">\n <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">\n <title>Earthdata Login</title>\n
and so on. It seems that I can't actually automate the getting of the actual zip file I desire. I am using Python 3.5.2 and would be much obliged for any help!
Best,
Matt
1 answers
solution1
0 2017-01-11 16:10:13
The url you are trying to access is not protected by http basic auth. You need to submit the form with the credentials.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.