从Google App Engine调用Reddit api时出现错误429

Question

I have been running a cron job on Google App Engine for over a month now without any issues. The job does a variety of things, one being that it uses urllib2 to make a call to retrieve a json response from Reddit as well as a few other sites. About two weeks ago I started seeing errors when invoking Reddit, but no errors when invoking the other sites. The error I am receiving is HTTP error 429.

I have tried executing the same code outside of Google App Engine and do not have any issues. I tried using urlFetch, but receive the same error.

You can see the error when using the app engine's interactive shell with the following code.

import urllib2
data = urllib2.urlopen('http://www.reddit.com/r/Music/.json', timeout=60)

Edit: Not sure why it always fails for me and not someone else. This is the error that I receive:

>>> import urllib2
>>> data = urllib2.urlopen('http://www.reddit.com/r/Music/.json', timeout=60)
Traceback (most recent call last):
  File "/base/data/home/apps/s~shell-27/1.356011914885973647/shell.py", line 267, in get
    exec compiled in statement_module.__dict__
  File "<string>", line 1, in <module>
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 400, in open
    response = meth(req, response)
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 513, in http_response
    'http', request, response, code, msg, hdrs)
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 438, in error
    return self._call_chain(*args)
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 521, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 429: Unknown

similar code running outside of app engine with no problem:

print urllib2.urlopen('http://www.reddit.com/r/Music/.json').read()

At first I thought it had to do with a timeout problem since it was originally working, but since there is not a timeout error but a the strange HttpError code, I'm not sure. Any ideas?

Answer 1

Reddit rate limits the api pretty severely for the default user agent for the python shell. You need to set a unique user agent with your reddit username in it, like this:

User-Agent: super happy flair bot by /u/spladug

More info about the reddit api here https://github.com/reddit/reddit/wiki/API .

Answer 2

It's possible that Reddit is counting calls based on IP - which means that other applications on GAE which share your IP might already be exhausting the quota.

This might get better if you use Reddit API keys (I don't know if they issue them) or if they agree to rate limit API calls based on the app header.