Spark：为什么在rdd映射中未调用我的UDF？

Question

I have the following code

def get_general_popularity_count():
    def test(t):
        a = 1 # this is just a random variable for testing
        print "a"
        pickle.dump(a, open("a.p", "wb"))
    count_dict = pickle.load(open("list.p","rb"))
    rdd = session.sparkContext.parallelize(count_dict)
    rdd.map(lambda x:test(x))

However, nothing is printed, and pickle didn't save a file either. In fact, I know that the UDF was never called because once I had a syntax error in test(x) , but the program never caught it.
So why is my UDF never called? Any help is appreciated

Answer 1

It is not called because map is a transformation. Unless it is followed by an action Spark has no reason to execute it at all.

Furthermore your code is not a good choice for Apache Spark:

• print outputs data to the standard output of the worker.
• pickle.dump will write to a local file system of the worker and, when execute like this in map , overwrite output all over again.

You could try RDD.foreach or RDD.saveAsPickleFile