Twitter Search API Rate Limit Zend

Question

We are trying to create a app which allows the user to display the latest twitter news on 20 football teams. We are using Zend_Service_Twitter_Search with the football team query eg "Arsenal".

The problem we are having is at any 1 point the website could be taking requests to view all 20 teams news so 20 calls to the API if many users are doing this then the hourly limit will be hit.

We want the latest news on all 20 teams so we dont want to cache the data longer than 1 mintue. Does any one have some advise on this I have been through the documentation but no suggestions there.

Cheers

J

Answer 1

This seems to me as more of a backend design issue. The problems you want to deal with are:

1. Provide current data for users of any scale
2. not exceed api limits, that are unpublished Twitter API limits

Seems like the solution is a database/datastore of some kind to persist the data.
You could make your 20 api calls every minute store the info and then make it available to users as long as you like.

After a brief look at the Twitter API docs it doesn't look like there are any restrictions on storing the data. Although you may want to reconsider how often to use the search API, as the limit in not posted.

Answer 2

I implemented something similar recently. The simple answer is work out how many queries you need to make. Let's say 1 for each team so 20 at a time. Divide 60 seconds (the period of the rate limit) by the (rate limit / number of queries) and simply cache your results in something like memcache for that number of seconds. Then when a user visits the site, pull the results from memcache if they are there, otherwise get them fresh. Memcache is great in that it automatically clears out data that expires so you won't go over the rate limit and will always have the freshest data possible (without breaking your rate limit).

So, if the rate limit is 200 per minute (I know it isn't but it makes the maths easy), you'd cache your results for 60 / (200/20) = 6 seconds. So at max rate, you would make 10 queries for each of the 20 teams in one minute, so 200 queries a minute, which is bang on the rate limit.