mpi4py包含进程和线程

Question

Hi This is a pretty specific question, so I hope StackOverflow is meant for all programming languages and not just javascript/html

I am writing a multi program in MPICH2 (popular message passing interface). My program is written in Python so I use the MPI4Py Python bindings. MPI is best for situations with no shared memory, therefore, it is not ideal for multicore programming. To use the full 4 cores of my 5 node cluster I am further using threads. However, I have noticed that using threads actually slows my simulation down. My program is several tens of thousands of lines of code, so I can not put it all up, but here is the snippet which is causing problems

from threading import Thread
...
threadIndeces=[[0,10],[11,20],[21,30],[31,40]] #subset for each thread
for indeces in treadIndeces:
  t=Thread(target=foo,args=(indeces,))
  t.start()

Also, I make sure to join the threads later. If I run it with no threads, and just call foo with all the indeces, it is about 10-15x times faster. When I record the times of the multithreaded version, the creation of the threads in the call t=Thread(target=foo,args=(indeces,)) takes around 0.05 seconds, the join similarly takes 0.05 seconds but the t.start() calls takes a whopping 0.2 seconds.

Is start() an expensive call? Should I be changing my approach? I thought about keeping a pool of threads rather than creating new ones every iteration, but it does not seem like the t=Thread(target=foo,args=(indeces,)) is what's causing the slow down.

Also, incase anyone wants to know the complexity of the foo , here is one of the functions which gets called i times for the indeces every iteration (non discrete time):

def HD_training_firing_rate(HD_cell):
    """During training, the firing rate is governed by the difference between the 
       current heading direction and the preferred heading direction. This is made
       to resemble a Gaussian distribution
    """
    global fabs
    global exp
    global direction

    #loop over twice due to concurrent CW and CCW HD training
    for c in [0,1]:
        d=direction[c]
        dp=HD_cell.dp  #directional preferance
        s_d=20.0  #standard deviation
        s_i=min(fabs(dp-d),360-fabs(dp-d)) #circular deviation from preferred dir.

        HD_cell.r[c]=exp(-s_i*s_i/(2*s_d*s_d))  #normal distribution

Answer 1

If you need threads, python may not be your best option due to the Global Interpreter Lock which prevents true concurrency. See also Dave Beazly's disturbing talk .

You might be better off just running 20 processes to keep your 4 cores and 5 nodes busy, and just use MPI for all communication.

Python incurs a lot of overhead on the big iron--you may want to think about C or C++ (or dare I say Fortran?) if you're really committed to a joint threads/message passing approach.