我们如何分析每个 Celery 任务的 memory 使用情况？

Question

I run tests using pytest which in turn runs a couple of celery tasks. But, when I use

mprof run --include-children py.test test.py

I am not able to track the memory consumed by each celery task that is running inside a celery worker. All I could see 50 MB of RAM being consumed by the py.test process (which I am able see in the mprof plot) whereas the celery worker consumes around 600-700 MB of RAM which does not get reflected in the mprof plot. How can I plot the memory consumption of each Celery task triggered by the py.test?

PS I use memory profiler package to profile the memory consumption of a python script.

Answer 1

What about --multiprocess flag? https://github.com/pythonprofilers/memory_profiler#tracking-forked-child-processes