[英]How to use queue with concurrent future ThreadPoolExecutor in python 3?
I am using simple threading modules to do concurrent jobs.我正在使用简单的线程模块来执行并发作业。 Now I would like to take advantages of concurrent futures modules.现在我想利用并发期货模块的优势。 Can some put me a example of using a queue with concurrent library?有人能给我举个使用并发库队列的例子吗?
I am getting TypeError: 'Queue' object is not iterable I dont know how to iterate queues我收到 TypeError: 'Queue' object is not iterable 我不知道如何迭代队列
code snippet:代码片段:
def run(item):
self.__log.info(str(item))
return True
<queue filled here>
with concurrent.futures.ThreadPoolExecutor(max_workers = 100) as executor:
furtureIteams = { executor.submit(run, item): item for item in list(queue)}
for future in concurrent.futures.as_completed(furtureIteams):
f = furtureIteams[future]
print(f)
I would suggest something like this:我会建议这样的事情:
def run(queue):
item = queue.get()
self.__log.info(str(item))
return True
<queue filled here>
workerThreadsToStart = 10
with concurrent.futures.ThreadPoolExecutor(max_workers = 100) as executor:
furtureIteams = { executor.submit(run, queue): index for intex in range(workerThreadsToStart)}
for future in concurrent.futures.as_completed(furtureIteams):
f = furtureIteams[future]
print(f)
The problem you will run in is that a queue is thought to be endless and as a medium to decouple the threads that put something into the queue and threads that get items out of the queue.您将遇到的问题是,队列被认为是无止境的,并且可以作为一种媒介来解耦将某些内容放入队列的线程和将项目从队列中取出的线程。
When什么时候
and afterwards process them in parallel, a queue makes no sense.然后并行处理它们,队列没有意义。 A ThreadPoolExecutor makes a queue obsolete in these cases.在这些情况下, ThreadPoolExecutor 会使队列过时。
I had a look at the ThreadPoolExecutor source:我查看了 ThreadPoolExecutor 源代码:
def submit(self, fn, *args, **kwargs): # line 94
self._work_queue.put(w) # line 102
A Queue is used inside.里面使用了一个Queue。
As commented above, you can use the iter()
function to execute a ThreadPool on a queue object.如上所述,您可以使用iter()
函数在队列对象上执行 ThreadPool。 A very general code for this would look something like this:一个非常通用的代码如下所示:
with concurrent.futures.ThreadPoolExecutor() as executor:
executor.map(run, iter(queue.get, None))
Where the run method executes the aspired work on the items of the queue. run 方法在队列的项目上执行期望的工作。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.