根据可用RAM使用python / celery /任何其他任务队列调度任务

Question

Is there a way to get Celery to allocate tasks to workers based on available RAM? I am developing a web app which applies memory hungry algorithms to large in-memory data, is there a way to make this more robust using Celery to respect available RAM?

Answer 1

Allocating tasks to workers based on available RAM is not possible. But, depending on your setup and exact requirements there are ways to work address this problem. Off the top of my head I can think of:

• you run one worker (and nothing else) on a machine with loads of RAM and have this worker process these memory-hungry tasks only
• if you know how much memory you need for processing a task, you could check if that memory is available as a first step in your task (assuming that not much else runs on that box which could use up that memory in between the check and the actual running of your memory-hungry stuff) and if not, you can retry the task after a certain time