Run a python program on multiple cores

Question

I am new to multi-core programming. The following program using only one core. How can I make it to run on multiple cores (I have 4 cores).

simDict={}

def sim(outer,inner, ...):
    val= /*do some math*/
    simDict[...]=val

def foo():
   for outer in xrange(0, limit):
      for inner in xrange(outer, limit):
          sim(outer,inner, ...)
foo()

Answer 1

Easy:

from multiprocessing import Pool

p = Pool()

def do_inner(outer, limit):
    for inner in xrange(outer, limit):
        sim(outer, inner, ...)

def foo():
    p.map(do_inner, xrange(limit))

foo()

This employs multiprocessing.Pool to create a pool of worker processes.

Answer 2

For your problem I would use a queue then I would stick with the consumer/producer problem and go from there. Unless there is some data dependency between your threads then you will need to get into some more advanced locking mechanisms and have to protect against far more threading pitfalls.

Producer/Consumer concept: http://en.wikipedia.org/wiki/Producer-consumer_problem And just for reference: http://docs.python.org/library/multiprocessing.html

multiprocessing may be helpful to use as well as the global interpreter lock can slow things down in some specific cases if just threads are used.