Python multiprocessing: dealing with 2000 processes

Question

Following is my multi processing code. regressTuple has around 2000 items. So, the following code creates around 2000 parallel processes. My Dell xps 15 laptop crashes when this is run.

1. Can't python multi processing library handle the queue according to hardware availability and run the program without crashing in minimal time? Am I not doing this correctly?
2. Is there a API call in python to get the possible hardware process count?
3. How can I refactor the code to use an input variable to get the parallel thread count(hard coded) and loop through threading several times till completion - In this way, after few experiments, I will be able to get the optimal thread count.
4. What is the best way to run this code in minimal time without crashing. (I cannot use multi-threading in my implementation)

Hereby my code:

regressTuple = [(x,) for x in regressList]
processes = []

for i in range(len(regressList)):                  
    processes.append(Process(target=runRegressWriteStatus,args=regressTuple[i]))

for process in processes: 
    process.start() 

for process in processes:
    process.join()

Answer 1

1. Can't python multi processing library handle the queue according to hardware availability and run the program without crashing in minimal time? Am I not doing this correctly?

I don't think it's python's responsibility to manage the queue length. When people reach out for multiprocessing they tend to want efficiency, adding system performance tests to the run queue would be an overhead.

2. Is there a API call in python to get the possible hardware process count?

If there were, would it know ahead of time how much memory your task will need?

3. How can I refactor the code to use an input variable to get the parallel thread count(hard coded) and loop through threading several times till completion - In this way, after few experiments, I will be able to get the optimal thread count.

As balderman pointed out, a pool is a good way forward with this.

4. What is the best way to run this code in minimal time without crashing. (I cannot use multi-threading in my implementation)

Use a pool, or take the available system memory, divide by ~3MB and see how many tasks you can run at once.

This is probably more of a sysadmin task to balance the bottlenecks against the queue length, but generally, if your tasks are IO bound, then there isn't much point in having a long task queue if all the tasks are waiting at a the same T-junction to turn into the road. The tasks will then fight with each other for the next block of IO.

Answer 2

There are multiple things that we need to keep in mind

1. Spinning the number of processes are not limited by number of cores on your system but the ulimit for your user id on your system that controls total number of processes that be launched by your user id.

2. The number of cores determine how many of those launched processes can actually be running in parallel at one time.

3. Crashing of your system can be due to the fact your target function that these processes are running is doing something heavy and resource intensive, which system is not able to handle when multiple processes run simultaneously or nprocs limit on the system has exhausted and now kernel is not able to spin new system processes.

That being said it is not a good idea to spawn as many as 2000 processes, no matter even if you have a 16 core Intel Skylake machine, because creating a new process on the system is not a light weight task because there are number of things like generating the pid, allocating memory, address space generation, scheduling the process, context switching and managing the entire life cycle of it that happen in the background. So it is a heavy operation for the kernel to generate a new process,

Unfortunately I guess what you are trying to do is a CPU bound task and hence limited by the hardware you have on the machine. Spinning more number of processes than the number of cores on your system is not going to help at all, but creating a process pool might. So basically you want to create a pool with as many number of processes as you have cores on the system and then pass the input to the pool. Something like this

def target_func(data):
    # process the input data

with multiprocessing.pool(processes=multiprocessing.cpu_count()) as po:
    res = po.map(f, regressionTuple)