Cron works on local host but not when deployed Appengine

Question

I have a cron job in python that works on my localhost but when it is deployed to appengine, it no longer works.

  pl = db.Query(Venue).order("id")
    list = pl.fetch(limit=0)
    for p in pl:
        base_url = 'http://search.twitter.com/search.json?rpp=100&q=4sq.com/'
        query = p.twitter_ID
        url_string = base_url + query
        json_text = fetch(url_string)
        json_response = simplejson.loads(json_text.content) 
        result = json_response['results']
        for f in result:
            user = f['from_user'] 
            print user   

This works fine locally but on the server I get the following error:

'results' Traceback (most recent call last): File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/ init .py", line 515, in call handler.get(*groups) File "/base/data/home/apps/hoosheer/4.347697940058059704/hoosheer_main.py", line 199, in get result = json_response['results'] KeyError: 'results'

This did work until I deployed my second version. Is there any way I can fix this?

Answer 1

The JSON 'results' field is not returned because the requests per Ip that Twitter allows has reached the max quota of available requests; this explains why from your domestic IP you don't have any problem and you don't get an HTTP 420 response code.

Unluckily Google App Engine uses a shared pool of IP addresses ^* for outgoing urlfetch requests and Twitter search APIs does not support authentication.

Search API Rate Limiting

Requests to the Search API, hosted on search.twitter.com, do not count towards the REST API limit. However, all requests coming from an IP address are applied to a Search Rate Limit. The Search Rate Limit isn't made public to discourage unnecessary search usage and abuse, but it is higher than the REST Rate Limit. We feel the Search Rate Limit is both liberal and sufficient for most applications and know that many application vendors have found it suitable for their needs.

This would force you to seriously think if Google App Engine is a correct choice for your application.

^{*I had the same problem here but luckily the API's developer has enabled an authentication mechanism that allows authenticated requests from the same IP.}