Maximally simple django timed/scheduled tasks (e.g.: reminders)?

Question

Question is relevant to this and this ;

the difference is, I'd prefer something with possibly more precision and low load (per-minute cron job isn't preferable for those) and with minimal overhead (ie installing celery with rabbitmq seems like a big overkill).

An example task for such is personal reminders server (with reminders that could be edited over web and sent out through e-mail or XMPP).

I'm probably looking for something more like node.js's setTimeout but for django (and though I might prefer to implement reminders in node.js anyway, it's still a possibly interesting question).

For example, it's possible to start new threads in django app (with functions consisting of sleep() and send()); in what ways this can be bad?

Answer 1

The problem with using threads for this solution are the typical problems with Python threads that always drive people towards multi-process solutions instead. The problem is compounded here by the fact your thread isn't driven by the normal request-response cycle. This is summarized nicely by Malcolm Tredinnick here :

Have to disagree. Threads are not a good solution to this problem. The issue is process management. As written, your threads will never be rejoined. Webserver processes have a lifecycle uncontrollable by you (the MaxRequestsPerChild Apache parameter and similar things in other servers) and you are messing with that by using threads.

If you need a process with a lifecycle that is not matched by the request-response path — something long running and independent of the response — a completely separate process is definitely the right model to use. Using a thread is tying it to the response lifecycle, which wil have unintended side-effects.

A possible solution for you might be to have a long running process performing your tasks which gets a wake-up signal from a light cron process.

Another possibility would be build something using 0mq , which is much lighter than AMQP style queues (at the cost of some features of course). Tarek Ziade is working on a Mozilla project called powerhose that uses 0mq, looks super simple, and has a heartbeat capability with resolution to the second.