requests: cannot download zip file properly with verify is False

Question

I am using Python 2.7.3 and requests(requests==2.10.0). I am trying to get some zipfile from some link. The website's certificate is not verified but I just want to download that zip so I used verfiy=False .

link = 'https://webapi.yanoshin.jp/rde.php?https%3A%2F%2Fdisclosure.edinet-fsa.go.jp%2FE01EW%2Fdownload%3Fuji.verb%3DW0EZA104CXP001006BLogic%26uji.bean%3Dee.bean.parent.EECommonSearchBean%26lgKbn%3D2%26no%3DS1007NMV'
r = requests.get(link, timeout=10, verify=False)
print r.content
# 'GIF89a\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\xff\xff\xff!\xf9\x04\x01\x00\x00\x01\x00,\x00\x00\x00\x00\x01\x00\x01\x00@\x02\x02L\x01\x00;'
print r.headers
# {'Content-Length': '43', 'Via': '1.0 localhost (squid/3.1.19)', 'X-Cache': 'MISS from localhost', 'X-Cache-Lookup': 'MISS from localhost:3128', 'Server': 'Apache', 'Connection': 'keep-alive', 'Date': 'Mon, 06 Jun 2016 07:59:52 GMT', 'Content-Type': 'image/gif'}

However, I tried with Firefox & Chromium, if I choose to trust that cert, I will be able to download zip file. wget --no-check-certificate [that link] results in a zip file with correct size as well.

(I wrote that gif to disk and checked, no content, just too small in terms of file size)

Maybe it is a header issue? I do not know. I can use wget of course. Just want to figure out the reason behind this and make this work.

(Browser will download some zip, 23.4KB)( wget [link] -O test.zip will download the zip file as well)

Answer 1

The server is trying to block scripts from downloading ZIP files; you'll see the same issue when using curl :

$ curl -sD - -o /dev/null "https://webapi.yanoshin.jp/rde.php?https%3A%2F%2Fdisclosure.edinet-fsa.go.jp%2FE01EW%2Fdownload%3Fuji.verb%3DW0EZA104CXP001006BLogic%26uji.bean%3Dee.bean.parent.EECommonSearchBean%26lgKbn%3D2%26no%3DS1007NUS"
HTTP/1.1 302 Found
Server: nginx
Date: Mon, 06 Jun 2016 08:56:20 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/7.0.7
Location: https://disclosure.edinet-fsa.go.jp/E01EW/download?uji.verb=W0EZA104CXP001006BLogic&uji.bean=ee.bean.parent.EECommonSearchBean&lgKbn=2&no=S1007NUS

Notice the text/html response.

The server seems to be looking for browser-specific Accept and User-Agent headers; copying the Accept header Chrome sends, plus adding a minimal User-Agent string, seems to be enough to fool the server:

>>> r = requests.get(link, timeout=10, headers={'User-Agent': 'Mozilla/5.0', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'}, verify=False)
# ... two warnings about ignoring the certificate ...
>>> r.headers
{'Content-Length': '14078', 'Content-Disposition': 'inline;filename="Xbrl_Search_20160606_175759.zip"', 'Set-Cookie': 'FJNADDSPID=3XWzlS; expires=Mon, 05-Sep-2016 08:57:59 GMT; path=/; secure, JSESSIONID=6HIMAP1I60PJ2P9HC5H3AC1N68PJAOR568RJIEB5CGS3I0UITOI5A08000P00000.E01EW_001; Path=/E01EW; secure', 'Connection': 'close', 'X-UA-Compatible': 'IE=EmulateIE9', 'Date': 'Mon, 06 Jun 2016 08:57:59 GMT', 'Content-Type': 'application/octet-stream'}