Profile Memory Usage of a running Python Process

Question

I have a Python process that starts leaking memory after a long time (at least 10 hours, sometimes more). The issue is difficult to reproduce, therefore I would like to attach to the running Python Interpreter when the problem comes up and inspect memory usage somehow, eg by getting a list of objects that currently allocate the most memory.

This is difficult with the usual profiling tools like tracemalloc or memory-profiler , because they need to be part of the code or started together with the process, and they have a significant impact on runtime performance.

What I would like to have is a sampling profiler that I can simply attach to an existing Python process like py-spy , but py-spy only gives me insights on CPU time spent in functions, not memory usage.

Is there another tool or a different approach that would help me to get insights into the memory usage of an existing Python process?

edit : I just found pyrasite , which provides the pyrasite-memory-viewer command, which is exactly what I'm looking for, but unfortunately the project seems to be abandoned and I can't get it to work on Python 3.8.

Answer 1

https://github.com/vmware/chap (open source code) does what you are requesting here, as long as you can run your application on Linux.

1. Wait until you believe you are interested in what allocations the application has at that point in time.

2. Gather a live core (for example using gcore) for your process, but make sure the coredump filter is set first as in:

   echo 0x37 >/proc/ pid-of-your-python-program /coredump_filter

   gcore pid-of-your-python-program

3. Open the resulting core in chap.

4. Do the following from the chap prompt:

   redirect on

   describe used

5. Edit the resulting file or post-process it using the tool of choice.

There are other commands you can use from chap to understand why any allocations of interest to you are anchored, basically because they allow you to traverse backwards by incoming references, but the above should suffice to allow you to figure out which kinds of allocations have high counts.