Cap download size with Python requests library

Question

I'm crawling a bunch of web pages using Python's request library, but occasionally the crawler will stumble upon an absolutely mammoth page, be it a PDF or video or otherwise gargantuan file. Is there a good way to limit the maximum size of file it will download?

Answer 1

The urlopen object has a method info() which gives all kinds of useful header information, including Content-Length

Occassionally this is not correctly set but should be in most cases and will help