如何在Python中使用psutil获取程序的最大内存使用量

Question

I am using the following code to get the max memory usage of the program.

    import os, subprocess , psutil
    def mem(cmd):
        try:
            with open('in.txt','r') as infile, open('out.txt', 'w') as outfile:
                p=psutil.Popen("./"+cmd,shell=False,stdin=infile,stdout = outfile)
            print p.memory_info()
        except Exception:
             print "Error"
    cmd=raw_input()
    mem(cmd)

The problem is sometimes for initial runs of the program the memory usage output is (0,0) but subsequently it displays the correct output. I dont why this happening. For some programs such as the hello world program in c++ the output is pmem(rss=4096, vms=315392) which is about 0.3M( I think the output is in bytes ) but running the hello world program in ideone.com gives the output as ~3M. Why is there this disperency?

cmd is the name of the executable.

The output of command print subprocess.check_output(['ps', 'v', '-p', str(p.pid)])

PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 16150 pts/16 Z+ 0:00 0 0 0 0 0.0 [a.out] <defunct>

One of my sample C++ program:

`int a[1000000];
int main()
{
    return 0;
}`

returns pmem(rss=4096, vms=4313088) sometimes and pmem(rss=0,vms=0) sometimes

Answer 1

The problem here is that psutils takes a quick snapshot from the /proc filesystem, as you can see in the source .

When you run your hello world example, in some cases it finishes before python gets a chance to read the values from /proc.

Once the process is finished, it effectively no longer consumes any ram. You can confirm this with an strace.

open("/proc/13420/statm", O_RDONLY)     = 3
read(3, "0 0 0 0 0 0 0\n", 1024)        = 14

If you modify your example to use something like sleep, you'll notice that psutils consistently gives memory usage back.

#include <iostream>
#include <unistd.h>

int main()
{
  std::cout << "Hello World.. sleeping!";
  sleep(3);
}

Output of your python script...

a.out
meminfo(rss=286720, vms=12931072)

One simple way to accomplish what you are trying to do, is by using the /usr/bin/time command, which on most platforms will give you the average total memory usage of the process you launch OR use valgrind as JF Sebastian suggests... whom posted as I was researching and testing my answer ;)

Hello World.. sleeping!0.00user 0.00system 0:03.00elapsed 0%CPU     
(0avgtext+0avgdata 1144maxresident)k
0inputs+0outputs (0major+348minor)pagefaults 0swaps

Answer 2

<defunct> means that the subprocess is a zombie process (it is dead but its status has not been read yet by the parent ( p.poll() or p.wait() )). It seems both psutil and ps shows RSS to be zero for such processes.

The result depends on whether the subprocess will exit sooner than p.memory_info() is called. It is a race. If you add a delay at the exit in the C++ program then p.memory_info() may be called before the subprocess exits and you should get non-zero results.

The problem is that I can get arbitrary programs to evaluate . The language is also not fixed. Isn't there an elegant solution to this problem?

You might need OS support to save info about the memory usage by a subprocess even after it exits. Or you could run the program using a memory profiler such as valgrind and read its results. To collect results:

$ valgrind --tool=massif cmd arg1 arg2 

To see the results, you could use ms_print :

$ ms_print massif.out.* | less

Or GUI Massif-Visualizer

@mdadm suggested a simpler solution: time command :

from subprocess import Popen, PIPE

p = Popen(['time', '-f', '%M'] + args, stderr=PIPE)
ru_maxrss = int(p.communicate()[1])
print("Maximum rss %d KB" % ru_maxrss)

GNU time uses wait3() to populate resource usage info if it is available. It can be called in Python:

import os
from subprocess import Popen

p = Popen(args)
ru = os.wait4(p.pid, 0)[2]
print("Maximum rss %d KB" % ru.ru_maxrss)

I've compared the maximum value returned by psutil.Process.memory_info (rss) with ru_maxrss value returned by os.wait4 and with the maximum total memory reported by valgrind --tool=massif : they are similar .

See also:

• A way to determine a process's “real” memory usage, ie private dirty RSS?
• How is memory usage reported in Linux?
• Find maximum memory consumed by a process specified at the command-line.