How to make some parts of code queue-specific in celery?

Question

I have two files containing celery task definitions. Each of them contains code for a specific queue. One of them imports scikit-learn and therefore is a little memory-consuming for the limited memory the VPS has. When celery initializes it executes both files to look for tasks and each celery worker imports scikit-learn. Is there a way to prevent this?

I have tried using inspect to get the current active queue and continue if this worker consumes this queue, but I think it doesn't work when initializing:

i = inspect(['celery@hostname'])
print i.active_queues()    # None

Answer 1

I think the best way to go is to start two workers, let them load 2 different apps and create 2 different queues.

example worker start cmd from top of my head:

celery -A scikit -Q learning worker

celery -A default -Q default worker

That of course requires you to add task routing (so that scikit tasks goes into the learning queue and the others go to the default queue).

Answer 2

I was able to solve it by emptying the CELERY_IMPORTS list and then including via cmd

celery -A proj worker -l info -Q first_queue -I proj.file1

which only looks for tasks in proj.file1 .