pyspark 数据帧重新分区后无法执行连接

Question

We are joining two huge files.

So we are trying to repartition on key column and then we are trying to join on the key column.

Code snippet

def repartition_df(df,primary_key,partition_value):
    df = df.repartition(partition_value,primary_key)


df_1 = repartition_df(df1,'pk1', 4 )
df_2 = repartition_df(df2,'pk1', 4 )

df3 = df_1.join(df_2,on =  ['pk1'] , how = 'left')

Error message

An error was encountered:
'NoneType' object has no attribute 'join'
Traceback (most recent call last):
AttributeError: 'NoneType' object has no attribute 'join'

When it works:

Now, if i dont repartion and go ahead with the join, it works fine.

But from performance perspective, we would like to join after repartition

Can you please let me know, how do I proceed.

Answer 1

Just add a return statement and your solution will work just fine.

def repartition_df(df, primary_key, partition_value):
    df = df.repartition(partition_value, primary_key)
    return df

df_1 = repartition_df(df1, 'pk1', 4)
df_2 = repartition_df(df2, 'pk1', 4)

df3 = df_1.join(df_2, on=['pk1'], how='left')