I have a python
script that use mpi4py
called main_parallel.py. I can measure the time using time
form the cli
but, how I can make a profile similar to cProfile? I like to see the number of call for each part of the code. I can't use cProfile because is only for serial code.
Thanks!
Why can't you use cprofile? Have you tried?
For MPICH, I ran like this:
$ mpiexec -l -np 4 python -m cProfile ./simple-io.py doodad
This gives me 4 sets of output, but the '-l' argument lists the MPI rank in front of each bit of output. Note: that '-l' argument is MPICH specific. OpenMPI uses --tag-output
. Other implementations might use something else.
I see cprofile can take a file name argument. make a per-rank output file and then process it with the Stats
% python
Python 2.7.10 (default, Oct 14 2015, 16:09:02)
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pstats
>>> pstats.Stats("simple-io.cprofile").sort_stats('cumulative').print_stats()
gives me lots of cprofile information... but my toy program was too tiny to give me anything useful.
As Rob Latham said you could use cProfile. You can save the output from each process in a different file. If you want to profile a function you could use a decorator like this:
from mpi4py import MPI
import cProfile
def profile(filename=None, comm=MPI.COMM_WORLD):
def prof_decorator(f):
def wrap_f(*args, **kwargs):
pr = cProfile.Profile()
pr.enable()
result = f(*args, **kwargs)
pr.disable()
if filename is None:
pr.print_stats()
else:
filename_r = filename + ".{}".format(comm.rank)
pr.dump_stats(filename_r)
return result
return wrap_f
return prof_decorator
@profile(filename="profile_out")
def my_function():
# do something
The output of each process can be visualized using snakeviz
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.