[英]Knowing how many task have been completed in Pool
I'm using Python's multiprocessing with map to process larges amount of list Here are the overview 我正在使用带有地图的Python多处理来处理大量列表这是概述
pool = ThreadPool(4) someList #About 300k elements here results = pool.map(someMethod,someList)
Is it possible for python to print out reports say for 10k elements processed? python是否可以打印出报告的数据,说明已处理了1万个元素?
The point of mapping as I understand it is that the function called for each element is the same for each element. 据我了解,映射的要点是,每个元素调用的函数对于每个元素都是相同的。 However you can hack it a little - just pass the index along with your element: 但是,您可以稍微修改一下-只需将索引和元素一起传递即可:
someListWithIndices = [{'index': i, 'data': x} for i,x in enumerate(someList)]
Now you need to tweak someMethod to use the data field from given dictionary and do something about the new index provided. 现在,您需要调整someMethod以使用给定字典中的数据字段,并对提供的新索引进行一些操作。
You probably won't get exactly your desired result (print exactly every 10k elements), but if you want to roughly know the progress you are making this could be the thing you are looking for. 您可能不会完全得到所需的结果(每10k个元素仅打印一次),但是如果您想大致了解要取得的进展,这可能就是您想要的。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.