pyspark中的内部定义函数

Question

Initially I put date_compare_(date1, date2) as a method of the whole class, but it is keeping to report error. Does that mean we could not call function outside the function itself in the map or filter? like method in class? Specifically, initially I put date_compare_(date1, date2) as a class method but it does not work. It seems that now it does not work neither, only when I put all things into one function

def extract_neighbors_from_users_network(self):
    end_date = self.arguments_dict['end_day']
    print("===================================")
    print(end_date)
    print("===================================")
    print("===================================")

    def date_compare_(date1, date2):
        date1_arr = date1.split("-")
        date2_arr = date2.split("-")
        for i in range(len(date1_arr)):
            if date1_arr[i] < date2_arr[i]:
                return True
            elif date1_arr[i] > date2_arr[i]:
                return False
        return True

    def date_filter_(date, start_date, end_date):
        return date_compare_(start_date, date) and self.date_compare_(date, end_date)

    def date_filter1(x):
        return date_filter_(x[0], "0000-00-00", end_date)

    rdd = self.sc.textFile(action_file).map(lambda x: x.split(',')).filter(lambda x: date_filter1(x)).filter(lambda x: x[4] == 'F')

Answer 1

From rdd = self.sc.textFile I guess your initially class is something like :

class YourClass():
    def __init__(self):
        self.sc = SparkContext()

    def date_compare_(self, date1, date2):
        pass

    def extract_neighbors_from_users_network(self):
        rdd = self.sc.textFile().map(self.date_compare_())

If so, you should change date_compare_ to

@staticmethod
def date_compare_(date1, date2):
    pass

And:

def extract_neighbors_from_users_network(self):
    rdd = self.sc.textFile().map(YourClass.date_compare_())

UPDATE:

If you reference self.date_compare_ inside rdd.map() , spark will send the whole instance of YourClass to executors as tasks. That is ok.

But! Instance of YourClass contain SparkContext() which can not be referenced on executors.That is why the error come out.