我可以为我的应用程序使用cron作业（需要具有高度可伸缩性）吗？

Question

I'm about to undertake a large project, where I'll need scheduled tasks (cron jobs) to run a script that will loop through my entire database of entities and make calls to multiple API's such as Facebook, Twitter & Foursquare every 10 minutes. I need this application to be scalable.

I can already foresee a few potential pitfalls...

1. Fetching data from API's is slow..
2. With thousands of records (constantly increasing) in my database, it's going to take too much time to process every record within 10 minutes.
3. Some shared servers only stop scripts running after 30 seconds.
4. Server issues due to constant intensive scripts running.

My question is how to structure my application...?

1. Could I create multiple cron jobs to handle small segments of my database (this will have to be automated)?
2. This will require potentially thousands of cron jobs.. Is that sustainable?
3. How to bypass the 30 sec issue with some servers?
4. Is there a better way to go about this?

Thanks!

Answer 1

I'm about to undertake a large project, where I'll need scheduled tasks (cron jobs) to run a script that will loop through my entire database of entities and make calls to multiple API's such as Facebook, Twitter & Foursquare every 10 minutes. I need this application to be scalable.

Your best option is to design the application to make use of a distributed database, and deploy it on multiple servers.

You can design it to work in two "ranks" of servers, not unlike the map-reduce approach: lightweight servers that only perform queries and "pre-digest" some data ("map"), and servers that aggregate the data ("reduce").

Once you do that, you can establish a performance baseline and calculate that, say, if you can generate 2000 queries per minute and you can handle as many responses, then you need a new server every 20,000 users. In that "generate 2000 queries per minute" you need to factor in:

• data retrieval from the database
• traffic bandwidth from and to the control servers
• traffic bandwidth to Facebook, Foursquare, Twitter etc.
• necessity to log locally (and maybe distill and upload log digests to Command and Control)

An advantage of this architecture is that you can start small - a testbed can be built with a single machine running both Connector, Mapper, Reducer, Command and Control and Persistence. When you grow, you just outsource different services to different servers.

On several distributed computing platforms, this also allows you to run queries faster by judiciously allocating Mappers geographically or connectivity-wise, and reduce the traffic costs between your various platforms by playing with, eg Amazon "zones" (Amazon has also a message service that you might find valuable for communicating between the tasks)

One note: I'm not sure that PHP is the right tool for this whole thing. I'd rather think Python.

At the 20,000 users-per-instance traffic level, though, I think that you'd better take this up with the guys at Facebook, Foursquare etc. . At a minimum you might glean some strategies such as running the connector scripts as independent tasks, each connector sorting its queue based on that service's user IDs , to leverage what little data locality there might be, and taking advantage of pipelining to squeeze more bandwidth with less server load. At the most, they might point you to bulk APIs or different protocols, or buy you for one trillion bucks :-)

Answer 2

See http://php.net/manual/en/function.set-time-limit.php to bypass the 30 second limit.

For scheduling jobs in PHP look at:

1. http://www.phpjobscheduler.co.uk/
2. http://www.zend.com/en/products/server/zend-server-job-queue

I personally would look at a more robust framework that handles job scheduling (see Grails with Quartz) instead of reinventing the wheel and writing your own job scheduler. Don't forget that you are probably going to need to be checking on the status of tasks from time to time so you will need a logging solution around the tasks.