Python: multithreading in infinite loop

Question

I have a code which is basically running an infinite loop, and in each iteration of the loop I run some instructions. Some of these instructions have to run in "parallel", which I do by using multiprocessing. Here is an example of my code structure:

from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool 

def buy_fruit(fruit, number):
    print('I bought '+str(number)+' times the following fruit:'+fruit)
    return 'ok'

def func1(parameter1, parameter2):
    myParameters=(parameter1,parameter2)
    pool= Threadpool(2)
    data = pool.starmap(func2,zip(myParameters))
    return 'ok'

def func2(parameter1):
    print(parameter1)
    return 'ok'



while true:
    myFruits=('apple','pear','orange')
    myQuantities=(5,10,2)
    pool= Threadpool(2)
    data = pool.starmap(buy_fruit,zip(myFruits,myQuantities))
    func1('hello', 'hola')

I agree it's a bit messy, because I have multi-processes within the main loop, but also within functions.

So everything works well, until the loop runs a few minutes and I get an error: "RuntimeError: can't start new thread"

I saw online that this is due to the fact that I have opened too many threads.

What is the simplest way to close all my Threads by the end of each loop iteration, so I can restart "fresh" at the start of the new loop iteration?

Thank you in advance for your time and help!

Best, Julia

PS: The example code is just an example, my real function opens many threads in each loop and each function takes a few seconds to execute.

Answer 1

You are creating a new ThreadPool object inside the endless loop, which is a likely cause to your problem, because you are not terminating the threads at the end of the loop. Have you tried creating the object outside of the endless loop?

pool = ThreadPool(2)
while True:
    myFruits = ('apple','pear','orange')
    myQuantities = (5,10,2)
    data = pool.starmap(buy_fruit, zip(myFruits,myQuantities))

Alternatively, and to answer your question, if your use case for some reason requires creating a new ThreadPool Object in each loop iteration, use a ContextManager ( with Notation) to make sure all threads are closed upon leaving the ContextManager.

while True:
    myFruits = ('apple','pear','orange')
    myQuantities = (5,10,2)
    with ThreadPool(2) as pool:
        data = pool.starmap(buy_fruit, zip(myFruits,myQuantities))

Notice however the noticable performance difference this has compared to the above code. Creating and terminating Threads is expensive, which is why the example above will run much faster, and is probably what you'll want to use.

Regarding your edit involving "nested ThreadPools": I would suggest to maintain one single instance of your ThreadPool, and pass references to your nested functions as required.

def func1(pool, parameter1, parameter2):
    ...
...

pool = ThreadPool(2)
while True:
    myFruits=('apple','pear','orange')
    myQuantities=(5,10,2)
    data = pool.starmap(buy_fruit, zip(myFruits,myQuantities))
    func1(pool, 'hello', 'hola')