运行相同的Django管理命令的重叠cron作业：有问题吗？

Question

I have a recurring cron job that runs a Django management command. The command interacts with the ORM, sends email with sendmail, and sends SMS with Twilio. It's possible that the cron jobs will begin to overlap. In other words, the job (that runs this command) might still be executing when the next job starts to run. Will this cause any issues? (I don't want to wait for the management command to finish executing before running the management command again with cron).

EDIT:

The very beginning of the management command gets a timestamp of when the command was run. At a minimum, this timestamp needs to be accurate. It would be nice if the rest of the command didn't wait for the previous cron job to finish running, but that's non-critical.

EDIT 2:

The cron job only reads from the DB, it doesn't write to it. The application has to continue to work while the cron job is running. The application reads and writes from the DB.

Answer 1

My understanding of cron is that it will fork off a job as a background process, allowing multiple jobs to run at the same time. This can be problematic if the second job depends on the first job to be done (if the second is running a daily report of aggregated data provided by the first job etc...). If you don't want them to run concurrently, there are workarounds to that:

How to prevent the cron job execution, if it is already running .

Will Cron start a new job if the current job is not complete?

Answer 2

Yes. This could definitely cause issues. You have a race condition . If you wish, you could acquire a lock somehow on a critical section which would prevent the next invocation from entering a section of code until the first invocation of the command finished. You may be able to do a row lock or a table lock for the underlying data.

Let's presume you're using MySQL which has specific lock syntax (DB dependent) and you have this model:

class Email(models.Model):
    sent = models.BooleanField(default=False)
    subj = models.CharField(max_length=140)
    msg = models.TextField()

You can create a lock object like this:

from django.db import connection
[...]
class EmailLocks(object):
    def __init__(self):
        self.c = connection.cursor()
    def __enter__(self):
        self.c.execute('''lock tables my_app_email write''')
    def __exit__(self, *err):
        self.c.execute('unlock tables')

Then lock all of your critical sections like:

with EmailLocks():
    # read the email table and decide if you need to process it
    for e in Email.objects.filter(sent=False):
        # send the email
        # mark the email as sent
        e.sent = True
        e.save()

The lock object will automatically unlock the table on exit. Also, if you throw an exception in your code, the table will still be unlocked.

Answer 3

So you have a cron that runs django management command and you dont want them to overlap.

You can use flock , Which generates a lockfile and deletes it after executing the cron.If the second cron starts before the first one has ended it will see that there a lockfile already created and thus not execute the second one.

Below is the cron i used:

* * * * * /usr/bin/flock -n /tmp/fcj.lockfile /usr/bin/python /home/txuser/dev/Project1/projectnew/manage.py flocktest

There is lot more you can do with this. more on this