python multiprocessing pool.starmap does not work

Question

I am using multiprocessing pool.starmap function. I discover a strange issue.

from multiprocessing import Pool
p = multiprocessing.Pool()

NODE = [1,2,3,4];
PageRank = [0.25,0.25,0.25,0.25];
Destination = [[2,3,4],[3,4],[1,4],[2]];

Data = zip(NODE,PageRank,Destination)

So I use zip function to create a data set Data , which is a list with each entry being a tuple of length 3. Then I call the function

p.starmap(MyFunction, zip(NODE,PageRank,Destination))

It works great.

However, when I type

p.starmap(MyFunction, Data))

It output empty list [] !!!! I really have no clue what is going on. I literally just replaced zip(NODE,PageRank,Destination) by Data , which should be the same thing, right?

Is that because I am using Jupyter notebook that causes this?

Answer 1

This answer is only valid if

• you are using Python 3, and
• you are doing things with the zip object (eg debug printing) that do not appear in your post

In Python 2, zip(...) returns a list ; however in Python 3, it returns a zip object (which is not the same as a list like you say in your post).

A zip object is an iterator , and so can only be iterated over once. After you reach the end of the iterator, any attempt to iterate over it again will yield nothing. For example,

>>> z = zip([1, 2], [3, 4])
>>> for x in z:
...     x
... 
(1, 3)
(2, 4)
>>> for x in z:
...     x
... 
>>> list(z)
[]

To speak on my second bullet point, I suspect you are doing something seemingly innocuous like printing all the elements of Data before you pass it as an argument to pool.starmap . If that is the case, you are exhausting the iterator, then effectively telling pool.starmap to apply MyFunction to absolutely nothing.

To fix this, you have three options.

1. Do it the first way you mentioned in which the zip object is created inside the call to pool.starmap .
2. Do not loop over Data prior to passing it to pool.starmap .
3. Cast the zip object as a list ( Data = list(zip(NODE,PageRank,Destination)) ). Then it is an iterable and you can iterate over it as many times as you would like.

In my humble opinion, this issue is just a rite of passage for newcomers to Python. If it applies to you and you want to learn more, you should read up on the differences between an iterator and an iterable , perhaps starting with this SO post .