Efficient Python programming for 4-core PC with threading?

Question

I'm looking to write a script to explore the parameter space in some simulation software. Initially, I just want to run the simulation script, say, 100 times, each time incrementing one parameter.

This is to run on a machine not being used for anything else, so ideally I'd make use of all four of its cores to speed up processing.

Could anyone recommend an efficient way of doing this please?

I considered generating 100 python scripts and then queuing them in a bash script. Alternatively, I could write a single python script to perform everything in a FOR loop. I thought I could use threading to allow 4 operations to be performed at once. IF this seems sensible, is the best implementation method to hard-code into the software to make and handle 4 threads, or is there a clever way for Python to handle CPU usage efficiently?

Many thanks,

Answer 1

You should use multiprocessing module instead of threading .

http://docs.python.org/2/library/multiprocessing.html

You can create multiple processes, it is up to the system to optimize CPU usage.