Knowing how many task have been completed in Pool

Question

I'm using Python's multiprocessing with map to process larges amount of list Here are the overview

pool = ThreadPool(4) someList #About 300k elements here results = pool.map(someMethod,someList)

Is it possible for python to print out reports say for 10k elements processed?

Answer 1

The point of mapping as I understand it is that the function called for each element is the same for each element. However you can hack it a little - just pass the index along with your element:

someListWithIndices = [{'index': i, 'data': x} for i,x in enumerate(someList)]

Now you need to tweak someMethod to use the data field from given dictionary and do something about the new index provided.

You probably won't get exactly your desired result (print exactly every 10k elements), but if you want to roughly know the progress you are making this could be the thing you are looking for.