Run cron job every 10 seconds n times then stop

Question

I have a special situation where I need to query a service to get information for n users at a specific time each day. The problem is that if I do this all at once it will put the service offline / crash it.

So to overcome this it would be better to run this for x number of users every 10 seconds or so until x = n and then stop.

I could setup 1 cron script that runs daily and another that runs every 10 seconds. The daily script would set a value in the DB to 1 ('start query') for example (default would be 0 for off), where then the second script (run every 10 seconds) checks this database value for 1. Upon finding the setting set to true it then iterates through the users querying the service x users at a time and incrementing another column in the same DB table to keep track of where in the list it is at.

The problem with this solution (well according to me) is that the 2nd script that runs every 10secs has to query the DB each time to find out if the 'start query' setting is set to 1. This can be quite processor heavy. Does anyone have a better solution?

NB: Code is written in PHP - cannot use sleep due to max execution time of php scripts on server

I could equally do this in python, is there a max execution for limit on cgi scripts?

Answer 1

Why not just have one cron script that runs daily, and just calls the other script every ten seconds until the other script says it's done (eg returns a value that says "nothing more to process")?

Or one script that runs daily and just runs through the chunks of users in a loop, sleep()ing every X users...

Bear in mind that when using the CLI version of PHP you're not constrained by execution time limits, and you can interact with cron, and other shell scripts quite nicely, using standard error and output streams, etc.

For example, for the first approach, you don't even have to use PHP for the "controlling" script -- a very simple shell script that just loops, calling your PHP script and then sleeping, and checking for a "success" return value (as returned by the PHP script's exit () function) would be fine.

Answer 2

You could use 'at' in combination with 'cron'. Set up a cron job that runs a single task once a day, and that task is (I don't have a UNIX box in front of me at the moment, so there may be the odd syntax error).

0 9 * * * at now + 10 secs /path/to/shellscript 1

The argument being the run count. When the shellscript runs, it reschedules itself to run, incrementing the run count, so you could have:

#!/bin/ksh

integer i
let i=1+$1
if [[ $i -lt 11 ]]
then
    at now + 10 secs /path/to/shellscript $i

    # do everything else the script needs to do
fi

NOTE: This solution also assumes 'at' can go down to per-second times (not sure if that's possible or not; may depend on the implementation). If it doesn't this solution's probably null and void from the off :-) In which case you're better off having one script that does stuff and sleeps (as per Matt Gibson's answer).

Answer 3

multiple ways to do this:

• Have a script that runs part of the job, putting all data in a variable that you serialize the data and the state. I have implemented a scheduler controller for code igniter to fetch metro routes one at the time http://www.younelan.com/index.php?&repository=DCExplorer . It may be more complex than what you are looking but may give you an idea on how to have multiple tasks run at various intervals, breaking them up into smaller parts
• run the script from the command line either through cron or manually - no suffering of timeouts