[英]Python concurrent.futures starvation
I'm using Python concurrent.futures, executes parent multi threads and each parent thread execute child threads.我正在使用 Python concurrent.futures,执行父多线程,每个父线程执行子线程。 When ThreadPoolExecutor is less than number of required parent threads I got starvation and program stuck.
当 ThreadPoolExecutor 少于所需的父线程数时,我会饿死并且程序卡住。
What is the best approach to:什么是最好的方法:
1. Use const ThreadPoolExecutor 1. 使用 const ThreadPoolExecutor
2. Do not get into starvation 2.不要挨饿
Please find below example code:请在下面找到示例代码:
import time
import sys
import concurrent.futures
MAX_THREAD_EXECUTORS = 5
threadPool = concurrent.futures.ThreadPoolExecutor(MAX_THREAD_EXECUTORS)
threads = []
command_threads = []
def main():
start_tests()
join_threads()
def start_tests():
for i in range(1,14):
threads.append(threadPool.submit(start_test_flow, i))
def start_test_flow(test):
print(f"Start test flow for: {test}")
execute_commands()
join_command_threads()
def execute_commands():
for i in range(1,5):
command_threads.append(threadPool.submit(start_command, i))
def start_command(command):
print(f"Start command for: {command}")
time.sleep(120)
def join_threads():
for thread in threads:
result = thread.result()
print(f"test result={result}")
def join_command_threads():
for thread in command_threads:
result = thread.result()
print(f"command result={result}")
if __name__ == '__main__':
main()
sys.exit(0)
Best Regards, Moshe最好的问候, Moshe
The minimum number of threads you actually need is non-deterministic and depends on timing, although there is a number (13 + 1, ie one thread for each of the parent threads and at least one thread to run a child thread) that will guarantee that you will never stall.您实际需要的最小线程数是不确定的,并且取决于时间,尽管有一个数字(13 + 1,即每个父线程一个线程和至少一个运行子线程的线程)可以保证你永远不会停滞不前。 What is most likely happening is that you are quickly creating 5 parent threads and then waiting to create further parent threads and child threads because you only have 5 worker threads.
最有可能发生的是您正在快速创建 5 个父线程,然后等待创建更多的父线程和子线程,因为您只有 5 个工作线程。 But until you are able to create 4 child threads (in
execute_commands
) and run them to completion, a parent thread will not complete and thus you are stuck.但是在您能够创建 4 个子线程(在
execute_commands
中)并运行它们完成之前,父线程将不会完成,因此您会被卡住。
Now, for example, insert a call to time.sleep(1)
in function start_tests
as follows:现在,例如,在 function
start_tests
中插入对 time.sleep time.sleep(1)
的调用,如下所示:
def start_tests():
for i in range(1,14):
threads.append(threadPool.submit(start_test_flow, i))
time.sleep(1)
This will allow the 4 child threads to be created and there will be some progress.这将允许创建 4 个子线程并且会有一些进展。 But depending on timing, you may eventually stall.
但根据时间的不同,您最终可能会停滞不前。 To guarantee that you never stall, you would have to sleep long enough to allow all 4 child threads to complete before attempting to start the next parent thread.
为了保证您永远不会停顿,您必须睡眠足够长的时间以允许所有 4 个子线程完成,然后再尝试启动下一个父线程。
The bottom line is that you just don't have enough worker threads (13 + 1) to guarantee that you won't stall.底线是您没有足够的工作线程 (13 + 1) 来保证您不会停顿。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.