在python中下载文件时出现问题

Question

I am trying to download a file using requests. I am running it on python 3.6.5. Below is my code:

import requests 
file_url = "http://codex.cs.yale.edu/avi/db-book/db4/slide-dir/ch1-2.pdf"

r = requests.get(file_url, stream = True) 

with open("python.pdf","wb") as pdf: 
    for chunk in r.iter_content(chunk_size=1024): 
        if chunk: 
            pdf.write(chunk)

Getting the below error:

ConnectionError: HTTPConnectionPool(host='codex.cs.yale.edu', port=80): Max retries exceeded with url: /avi/db-book/db4/slide-dir/ch1-2.pdf (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000001421CF5080>: Failed to establish a new connection: [Errno 11002] getaddrinfo failed',))

I have tried a number of suggested methods on the same issues like increasing the timeout but it doesn't help. Also, that link is working perfectly fine.

Any idea on whats wrong here?

Answer 1

I would suggest using looking into fake user agents eg https://pypi.org/project/fake-useragent/ and using proxy rotation to hit the endpoint you are trying to hit. A good example on how to achieve those is https://www.scrapehero.com/how-to-rotate-proxies-and-ip-addresses-using-python-3/

Answer 2

The problem was in the remote terminal. Somehow the remote terminal won't execute the connections and it might throw an error. It worked fine on my personal machine.

Thanks all for your suggestions.