Celery task timeout/time limit for windows?

Question

I have a web app written in Flask that is currently running on IIS on Windows (don't ask...).

I'm using Celery to handle some asynchronous processing (accessing a slow database and generating a report).

However, when trying to set up some behavior for error handling, I came across this in the docs :

"Time limits do not currently work on Windows and other platforms that do not support the SIGUSR1 signal."

Since the DB can get really slow, I would really like to be able to specify a timeout behavior for my tasks, and have them retry later when the DB might not be so tasked. Given that the app, for various reasons, has to be served from Windows, is there any workaround for this?

Thanks so much for your help.

Answer 1

If you really need to set the task timeout, you can use the child process to achieve, the code as follows

import json
from multiprocessing import Process
from celery import current_app
from celery.exceptions import SoftTimeLimitExceeded

soft_time_limit = 60

@current_app.task(name="task_name")
def task_worker(self, *args, **kwargs):
    def on_failure():
        pass
    worker = Process(target=do_working, args=args, kwargs=kwargs, name='worker')
    worker.daemon = True
    worker.start()
    worker.join(soft_time_limit)
    while worker.is_alive():
        worker.terminate()
        raise SoftTimeLimitExceeded  
    return json.dumps(dict(message="ok"))

def do_working(*args, **kwargs):
    pass # do something

Answer 2

It doesn't look like there is any built in workaround for this in Celery. Could you perhaps code this into your task directly? In other words, in your python code, start a timer when you begin the task, if the task takes too long to complete, raise an exception, and resubmit the job to the queue again.