下载仅允许使用Python从内部服务器访问的文件

Question

I was trying to download some files from a sever but it came back with an error page saying only links from internal server is allowed. I was able to download the file with any browser by clicking the link and I have verified the link I captured in Python was correct. Is there any way this can be done by using python? I tried urllib, urllib2 and requests, but none works. I could use selenium but the solution is not elegent

Answer 1

When you use your browser, it sends a header known as a User-Agent that identifies it. You need to 'spoof' the user agent from your python script to make it think a human is browsing the website. Set the User-Agent header to that of a common browser, this makes it difficult for the server to detect that you are using a script.

Answer 2

This should work if the webserver is checking on the referer (the address that contains the link):

import urllib2
req = urllib2.Request('http://www.example.com/linked_resource')
req.add_header('Referer', 'http://www.example.com/page_with_link')
r = urllib2.urlopen(req)