Python3，Urllib.request，urlopen（）超时

Question

I'm using urlopen() to open a website and pull (financial) data from it. Here is my line:

sourceCode = urlopen('xxxxxxxx').read()

After this, I then pull the data I need out. I loop through different pages on the same domain to pull data (stock info). I end the body of the loop with:

time.sleep(1)

as I'm told that keeps the site from blocking me. My program will run for a few minutes, but at some point, it stalls and quits pulling data. I can rerun it and it'll run another arbitrary amount of time and then stall.

Is there something I can do to prevent this?

Answer 1

This worked (for most websites) for me:

If you're using the urllib.request library, you can create a Request and spoof the user agent. This might mean that they stop blocking you.

from urllib.request import Request, urlopen
req = Request(path, headers={'User-Agent': 'Mozilla/5.0})
data = urlopen(req).read()

Hope this helps