检查Celery队列中来自Python的任务

Question

I have a Django+Celery application, where Celery is used to push (and pull) Django model instances to a third party SOAP service.

My Django models have dependencies between them and also a simple hash like this:

class MyModel(Models):
    def get_dependencies(self):
        # ...
        return [...]

    def __hash__(self):
        return hash(self.__class__.__name__+str(self.pk))

This hash came handy in my own implementation which I had to drop due to stability issues. Celery is a much sounder ground.

When I push an instance over to the SOAP service, I must make sure that its dependencies have been pushed. This is done by checking all related instances for a pushed_ok timestamp fields.

The difficult part is when an instance a which depends on list of instances deps (all are instances of MyModel subclasses) is being pushed. I cannot push a unless all instances in deps have been processed by Celery. In other words I need to serialize tasks so that the dependecies order is respected.

Celery is run like this:

celery -A server worker -P eventlet -c 100

How can I prevent one of the the eventlets (/process/thread) from running a before its dependencies, if any, have finished being run by other eventlets?

Thank you for your help.

Answer 1

I went for a pragmatic solution of moving all dependency checking of a resource (which includes pushing out-of-sync dependencies to the soap server) inside the celery task. Instead of trying to serialize tasks depending on the resources' dependencies.

The up side is that it keeps it simple and I could implement it rapidly.

The down side is that I'm locking a worker for a moment and potentially many synchronous SOAP operations instead of dispatching these operations accross workers.