[英]Python multithreading: Obtain worker thread results in main thread
On Python 3.8, I have implemented multithreading for a network I/O task in which a bunch of worker threads download some data off of the network, process it and create their individual list of results.在 Python 3.8 上,我为网络 I/O 任务实现了多线程,其中一堆工作线程从网络下载一些数据,处理它并创建他们各自的结果列表。 Now, when all threads finish, I want the main thread to obtain all the worker threads' result lists and process further.现在,当所有线程完成时,我希望主线程获取所有工作线程的结果列表并进一步处理。
For this discussion, I've eliminated the network I/O calls and introduced some dummy code.在本次讨论中,我消除了网络 I/O 调用并引入了一些虚拟代码。 This is how it looks:这是它的外观:
from queue import Queue
from threading import Thread
from random import randint as ri
class DownloadWorker(Thread):
def __init__(self, queue, result_q):
Thread.__init__(self)
self.queue = queue
self.result_q = result_q
def run(self):
while True:
start_val = self.queue.get()
try:
# dummy code. Real code has network calls here
thread_output = [ri(0, 10) + start_val, ri(0, 10) + start_val, ri(0, 10) + start_val]
self.result_q.put(thread_output)
finally:
self.queue.task_done()
def main():
queue = Queue() # Communication between main thread and its workers
result_q = Queue() # Result queue so workers results can finally be pooled together by main thread
# Create 2 worker threads
for x in range(2):
worker = DownloadWorker(queue, result_q)
# Setting daemon to True will let the main thread exit even if worker threads block
worker.daemon = True
worker.start()
start_values = [10, 100] # pass start value to differentiate between thread outputs
for start_val in start_values:
queue.put(start_val)
queue.join()
# Both workers tasks done. Now let's pool the results(just printing here for simiplification..)
while not result_q.empty():
print(result_q.get())
if __name__ == '__main__':
main()
This code works well so far but I want to know if there is a better way to pool results in main thread using multithreading in Python 3.8.到目前为止,这段代码运行良好,但我想知道是否有更好的方法在 Python 3.8 中使用多线程在主线程中合并结果。 I looked at this old thread but it throws errors when I change it as per my requirement(frankly I don't understand that solution there very well).我查看了这个旧线程,但是当我按照我的要求更改它时它会引发错误(坦率地说,我不太了解那里的解决方案)。
Appreciate some pointers on this!欣赏这方面的一些指示!
You have invented your own thread pooling which has has already been provided by the ThreadPoolExecutor
class in the concurrent.futures
module:您已经发明了自己的线程池,该线程池已经由concurrent.futures
模块中的ThreadPoolExecutor
类提供:
import concurrent.futures
from random import randint as ri
def worker(start_val):
# dummy code. Real code has network calls here
return [ri(0, 10) + start_val, ri(0, 10) + start_val, ri(0, 10) + start_val]
def main():
NUMBER_THREADS = 2
with concurrent.futures.ThreadPoolExecutor(max_workers=NUMBER_THREADS) as executor:
start_values = [10, 100] # pass start value to differentiate between thread outputs
# method submit returns a Future instance, which encapsulates the asynchronous execution of a callable:
futures = [executor.submit(worker, start_val) for start_val in start_values]
for future in futures:
result = future.result() # block until a result is returned
print(result)
# or you can do: results = executor.map(worker, start_values)
if __name__ == '__main__':
main()
Prints:印刷:
[20, 14, 11]
[104, 104, 108]
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.