profiling python multiprocessing pool

Question

I've got some code that is using a Pool from Python's multiprocessing module. Performance isn't what I expect and wanted to profile the code to figure out what's happening. The problem I'm having is that the profiling output gets overwritten for each job and I can't accumulate a sensible amount of stats.

For example, with:

import multiprocessing as mp
import cProfile
import time
import random

def work(i):
    x = random.random()
    time.sleep(x)
    return (i,x)

def work_(args):
    out = [None]
    cProfile.runctx('out[0] = work(args)', globals(), locals(),
                    'profile-%s.out' % mp.current_process().name)
    return out[0]

pool = mp.Pool(10)

for i in pool.imap_unordered(work_, range(100)):
    print(i)

I only get stats on the "last" job, which may not be the most computationally demanding one. I presume I need to store the stats somewhere and then only write them out when the pool is being cleaned up.

Answer 1

My solution involves holding onto a profile object for longer and only writing it out at the "end". Hooking into the Pool teardown is described better elsewhere , but involves using a Finalize object to execute dump_stats() explicitly at the appropriate time.

This also allows me to tidy up the awkward work_ trampoline needed with the runctx I was using before.

import multiprocessing as mp
import cProfile
import time
import random

def work(i):
    # enable profiling (refers to the global object below)
    prof.enable()
    x = random.random()
    time.sleep(x)
    # disable so we don't profile the Pool
    prof.disable()
    return (i,x)

# Initialise a good profile object and make sure it gets written during Pool teardown
def _poolinit():
    global prof
    prof = cProfile.Profile()
    def fin():
        prof.dump_stats('profile-%s.out' % mp.current_process().pid)

    mp.util.Finalize(None, fin, exitpriority=1)

# create our pool
pool = mp.Pool(10, _poolinit)

for i in pool.imap_unordered(work, range(100)):
    print(i)

Loading the output shows that multiple invocations were indeed recorded:

> p = pstats.Stats("profile-ForkPoolWorker-5.out")
> p.sort_stats("time").print_stats(10)
Fri Sep 11 12:11:58 2015    profile-ForkPoolWorker-5.out

         30 function calls in 4.684 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       10    4.684    0.468    4.684    0.468 {built-in method sleep}
       10    0.000    0.000    0.000    0.000 {method 'random' of '_random.Random' objects}
       10    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Answer 2

according to the docs you can use the pid attribute to get a unique name for each output file

cProfile.runctx('out[0] = work(args)', globals(), locals(),
                'profile-%s-%s.out' % (mp.current_process().pid, datetime.now().isoformat()))