请求无法获取 pdf URL 并下载它

Question

For my job, we get a lot of product pdfs that we need to download. This leads to long lists of urls that I'd rather not click on over and over again. For some, I'm able to use the code below to download a pdf, but for others (like the one included) it seems like requests gets stuck in an endless loop of some sort when I ask it to get the url.

I've tried different parameters and different tips that I've seen elsewhere and nothing has worked. I'm new to code and to python so I'm probably missing something obvious here. Any help and explanation would be greatly appreciated. Thank you!

import requests # to get image from the web
import shutil # to save it locally

url = "https://www.us.kohler.com/webassets/kpna/catalog/pdf/en/K-10411_spec_US-CA_Kohler_en.pdf"
filename = 'TEST-Image.pdf'

r = requests.get(url, stream = True)

if r.status_code == 200:

    r.raw.decode_content = True

with open(filename,'wb') as f:
    shutil.copyfileobj(r.raw, f)
    
    print('PDF sucessfully Downloaded: ',filename)
else:
    print('PDF Couldn\'t be retrieved')

Answer 1

The issue here, at least with the specific link provided, is that something on Kohler's side does not appreciate requests without a user-agent set in the headers. This is either a bug, or intentional. It may actually be an attempt to prevent people from doing exactly what you're doing - mass downloading their manuals. Regardless, the solution is simple.

Modify your requests call to look like this:

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'}
r = requests.get(url, stream = True, headers = headers)

Note that the actual user-agent string provided is just the standard string for Chrome on Windows 10. You could probably use any user-agent string you wanted.