每x分钟运行python作业

Question

I have a small python script that basically connects to a SQL Server (Micrsoft) database and gets users from there, and then syncs them to another mysql database, basically im just running queries to check if the user exists, if not, then add that user to the mysql database.

The script usually would take around 1 min to sync. I require the script to do its work every 5 mins (for example) exactly once (one sync per 5 mins).

How would be the best way to go about building this?

I have some test data for the users but on the real site, theres a lot more users so I can't guarantee the script takes 1 min to execute, it might even take 20 mins. However having an interval of say 15 mins everytime the script executes would be ideal for the problem...

Update: I have the connection params for the sql server windows db, so I'm using a small ubuntu server to sync between the two databases located on different servers. So lets say db1 (windows) and db2 (linux) are the database servers, I'm using s1 (python server) and pymssql and mysql python modules to sync.

Regards

Answer 1

I am not sure cron is right for the job. It seems to me that if you have it run every 15 minutes but sometimes a synch takes 20 minutes you could have multiple processes running at once and possibly collide.

If the driving force is a constant wait time between the variable execution times then you might need a continuously running process with a wait.

def main():
    loopInt = 0
    while(loopInt < 10000):
        synchDatabase() 

        loopInt += 1
        print("call #" + str(loopInt))
        time.sleep(300)  #sleep 5 minutes

main()

(obviously not continuous, but long running) You can set the result of while to true and it will be continuous. (comment out loopInt += 1 )

Edited to add: Please see note in comments about monitoring the process as you don't want the script to hang or crash and you not be aware of it.

Answer 2

You might want to use a system that handles queues, for example RabbitMQ, and use Celery as the python interface to implement it. With Celery, you can add tasks (like execution of a script) to a queue or run a schedule that'll perform a task after a given interval (just like cron ).

Get started http://celery.readthedocs.org/en/latest/