multiprocessing simultaneous python scripts efficiently

Question

I am trying to run a Python simulation several times simultaneously, but with slightly different parameters in each run. I am trying to use the multiprocessing module to do this. I begin my code like this, where I have the basic simulation defined as a function, with the parameters as arguments:

import multiprocessing
from math import *

def sim_seq(output_name,input_name,s_val...#more arguments):
   #do work here
   output.write(#data)
   output.close()
   return

I have also created a text file with the parameters to be used for each run of the simulation, which I read in and use as the arguments in the following loop, where I am trying to use multiprocessing:

input_batch=('batch_file.txt')
if __name__ == '__main__':
   jobs=[]
   with open(input_batch) as f:
       for line in f:
           line=line.split(' ')
           for i in line:
              if i[0]=='o':
                 output_name=str(i[2:])
              #read in more parameters from batch_file.txt
       p = multiprocessing.Process(
           target=sim_seq,
           args=(output_name,input_name,s_val...#more arguments))
       jobs.append(p)
   for i in jobs:
      i.start()

This essentially accomplishes what I want it to do, it runs three simulations at once, each with different parameters. The machine I am using, however, has 16 compute nodes available with 32 processors per node. I want to know how I can control where each simulation is being run. For instance, can I tell each processor to run a separate simulation? I am new to using multiprocessing, and I want to know how I can tell what processor or what node to do what. Can I have 32 separate parameter settings, and run each 32 instances of the simulation on its own processor, yet all running at the same time? Using multiprocessing, what would be the fastest computationally way to run the same python function multiple times simultaneously, but with different arguments for each run? Thanks in advance for any input/advice.

Answer 1

(I'm assuming that each of your compute nodes is a separate machine, with its own set of cores. If your compute cluster has some sort of OS that virtualizes the cores so they all appear to be local, then you can ignore the "Multiple nodes" bit below.)

On one node

The multiprocessing module natively handles multiple processes within a single instance of an operating system. If you start up top or a similar process list on one node and it shows N cores, then that's the number of cores that are available to your Python simulation.

Within that constraint, however, you can spawn and manage as many processes as you want, and the operating system will arrange them on the available cores using its normal process scheduler. So, in your situation, it sounds to me like you should be able to run 32 separate simulations in parallel on a single node . All you need to do is set up your loop to create 32 processes, give them the parameters to run, and wait until they all finish.

If you have more than 32 simulations to run, you could set up a multiprocessing.Pool containing 32 workers, and then use pool.map over a list of simulation parameters to distribute the work to each of your cores.

Multiple nodes

If you do have more than 32 simulations, and you want to start taking advantage of cores on separate nodes (where you might need to log in to the separate nodes using ssh or the like), then in theory you could use a "Remote Manager" from the multiprocessing module to handle this.

However, I would recommend taking a look at the awesome capabilities of IPython.parallel -- it allows you to start up "processing engines" on multiple nodes, and then you can distribute work to the nodes using an IPython shell. This would end up being quite similar to the process pool described above, only it would take advantage of all cores on all compute nodes in your cluster.

Alternatively, you could set up or take advantage of any of a number of existing cluster schedulers (Condor, Sun GridEngine, etc.) to start up your simulation once (or even 32 times) on each processing node.