Python multiprocessing is sharing cores

Question

I'm using miltiprocessing to read a lot of chunks of a huge file and then process it, but I'm loosing something because sometimes when I launch some process, some of them share a core. Example: system with 12 cores, I run 10 process and 6 are en 6 cores (one with each) and other 4 are en 2 cores (2 in each core)... but other times, it works fine using 10 cores for 10 process...

Code:

from multiprocessing import Process
[...]
process1=Process(target=run_thread, args=("./splited/"+file,result,))
process2=Process(target=run_thread, args=("./splited/"+file,result,))
process3=Process(target=run_thread, args=("./splited/"+file,result,))
process1.start() 
process2.start()
process3.start()
[...]
process1.join()
process2.join()
process3.join()

This is an example of my code, I was trying with 10 when I saw the issue.

Thanks.

EDITED:

• This machine works without multithread, so it has 12 cores with a maximum of 12 parallel threads.
• Sometimes there are more than 2 sharing a core.
• The neck of bottle is not the IO on this program it's process each line of the file.

Answer 1

I am certainly not an expert on this question, but it appears that most modern processsors actually are structured with physical cores and virtual cores. For example I have an Intel Core I7, which means I have 4 physical cores, but each of them actually contain 2 virtual cores, meaning that my processor can run 8 distinct processes in parallel.

You mention sometimes you have 2 processes per core, so that would be due to core virtualization.

By the way, if you aim at processing large amounts of data in smaller chunks, there is a specific module in python dedicated to such tasks: Pool , that is imported using

from multiprocessing import Pool

Hope this is useful to you, Cheers