How to use mapPartitions in pyspark

Question

After following the Apache Spark documentation, I tried to experiment with the mapPartition module. In the following code, I expected to see initial RDD as in the function myfunc I am just returning back the iterator after printing the values. But when I do collect on the RDD it is empty.

from pyspark import SparkConf
from pyspark import SparkContext                                                          

def myfunc(it):
    print(it.next())
    return it

def fun1(sc):
    n = 5
    rdd = sc.parallelize([x for x in range(n+1)], n)
    print(rdd.mapPartitions(myfunc).collect())


if __name__ == "__main__":                                                                
    conf = SparkConf().setMaster("local[*]")                                              
    conf = conf.setAppName("TEST2")                                                       
    sc = SparkContext(conf = conf)                                                        
    fun1(sc)

Answer 1

mapPartitions is not relevant here. Iterators (here itertools.chain ) are stateful and can be traversed only once. When you call it.next() you read and discard the first element and what you return is a tail of the sequence.

When partition has only one item (it should be the case for all but one) you effectively discard the whole partition.

A few notes:

• Putting anything to stdout inside a task is typically useless.
• The way you use next is not portable and cannot be used in Python 3.