如何将第二个数据帧的列传递到PySpark 1.6.1中的UDF

Question

Here's what I'm trying to do. I want to perform a comparison between each entry of two columns in two different dataframes. The dataframes are shown below:

>>> subject_df.show()
+------+-------------+
|USERID|     FULLNAME|
+------+-------------+
| 12345|  steve james|
| 12346| steven smith|
| 43212|bill dunnigan|
+------+-------------+

>>> target_df.show()
+------+-------------+
|USERID|     FULLNAME|
+------+-------------+
|111123|  steve tyler|
|422226|  linda smith|
|123333|bill dunnigan|
| 56453|  steve smith|
+------+-------------+

Here is the logic I tried using:

# CREATE FUNCTION    
def string_match(subject, targets):
    for target in targets:
        <logic>
    return logic_result

# CREATE UDF
string_match_udf = udf(string_match, IntegerType())

# APPLY UDF
subject_df.select(subject_df.FULLNAME, string_match_udf(subject_df.FULLNAME, target_df.FULLNAME).alias("score"))

This is the error I get when running the code in a pyspark shell:

py4j.protocol.Py4JJavaError: An error occurred while calling o45.select.
: java.lang.RuntimeException: Invalid PythonUDF PythonUDF#string_match(FULLNAME#2,FULLNAME#5), requires attributes from more than one child.

I think the root of my problem is trying to pass in the second column to the function. Should I be using RDDs? Keep in mind that the actual subject_df and target_df are both over 100,000 rows. I'm open to any advice.

Answer 1

It looks like you have a wrong idea how user defined functions works:

• Function receives values from only one row at the time
• You cannot use a data from unrelated DataFame .

The only way to do what you want is to take a cartesian product.

subject_df.join(target_df).select(
 f(subject_df.FULLNAME, target_df.FULLNAME)
)

where f is a function that compares two elements at the time.