Best allocation of resources for multiprocessed command running within python

Question

I've developed a tool that requires the user to provide the number of CPUs available to run it.

As part of the program, the tool calls HMMER (hmmer - http://eddylab.org/software/hmmer3/3.1b2/Userguide.pdf ) which itself is quite slow and needs multiple CPUs to run.

I'm confused about the most efficient way to spread the CPUs considering how many CPUs the user has specified.

For instance, assuming the user gave N cpus, I could run

• N HMMER jobs with 1 CPU each

• N/2 jobs with 2 CPUs each

etc..

My current solution is arbitrary opening pool size of N/5 and open a pool, then to call HMMER with 5 CPUs in each process in the pool.:

pool = multiprocessing.Pool(processes = N/5)

pool.map_async(run_scan,tuple(jobs))

pool.close()

pool.join()

where run_scan calls HMMER and jobs holds all the command line arguments for each HMMER job as dictionaries.

The program is very slow and I was wondering if there was a better way to do this.

Thanks

Answer 1

Almost always, parallelization comes at some cost in efficieny, but the cost depends strongly on the specifics of the computation, so I think the only way to answer this question is a series of experiments.

(I'm assuming memory or disk I/O isn't an issue here; don't know much about HMMER, but the user's guide doesn't mention memory at all in the requirements section.)

• Run the same job on one core ( --cpu 1 ), then two cores, four, six, ..., and see how long it takes. That will give you an idea of how well the jobs get parallelized. Used CPU time = runtime * number of cores should remain constant.
• Once you notice a below-linear correlation between runtime and number of cores devoted to the job, that's when you start to run multiple jobs in parallel. Say you have 24 cores, and a job takes 240 seconds on a single core, 118 seconds on two cores, 81 seconds on three cores, 62 seconds on four, but a hardly faster 59 seconds for five cores (instead of the expected 48 secs), you should run 6 jobs in parallel with 4 cores each.
• You might see a sharp decline at about n_cores/2: some computations don't work well with Hyperthreading , and the number of cores is effectively half of what the CPU manufacturer claims.