python multiprocessing. use pool or process

Question

I have a scenario where i have a loop in the main process that constantly creates new objects. There are some processing to be done on each created object that i want to do in a different process.

I have tried creating a shared dictionary that i constantly fill with new objects and i created a process to loop over objects in the dictionary.

is there a better way to do this? as i have limited resources, and the above method is kind of slow.

perhaps it is better to create a separate worker each time an object is created? if so how can i achieve this?

thanks

Answer 1

A process is a more "heavy" entity than a thread. A process contains one or more threads.

As a rule, I would choose processes over threads only if I foresee the possibility of processing the data on a different processor than the one that produces the data (a distributed system). If the data generation and the data processing (always!) takes place on the same processor, then using threads is more efficient (the data sharing and the synchronization primitives specific to threads is simpler, as the operating system does not need to take into account other existing processes).

The number of threads is dependent on the hardware. You can use the a thread-pool to have some flexibility here.

Have a look atproducer-consumer , thread pool , IPC and map these concepts on the python language capabilities ( multiprocessing , ThreadPoolExecutor ).