在创建 UDF Pyspark 时，何时需要使用 lambda（何时不需要）？

Question

I do not completely understand when I need to use a lambda function in the definition of a UDF.

My prior understanding was that I needed lambda in order for the DataFrame to recognize that it has to iterate over each row but I have seen many applications of UDFs without a lambda expression.

For example:

I have a silly function that works well like this without using lambda:

@udf("string")
def unknown_city(s, city):
  if s == 'KS' and 'MI':
    return 'Unknown'
  else: 
    return city


display(df2.
        withColumn("new_city", unknown_city(col('geo.state'), col('geo.city')))
       )

How can I make it work with lambda? Is it necessary?

Answer 1

Python lambda is just a way to write your functions. See the example code below and you will see they're pretty much the same, except that the lambda function is only for one-line code.

With lambda function

from pyspark.sql import functions as F
from pyspark.sql import types as T

df.withColumn('num+1', F.udf(lambda num: num + 1, T.IntegerType())('num')).show()

# +---+-----+
# |num|num+1|
# +---+-----+
# | 10|   11|
# | 20|   21|
# +---+-----+

With normal function

from pyspark.sql import functions as F
from pyspark.sql import types as T

def numplus2(num):
    return num + 2

df.withColumn('num+2', F.udf(numplus2, T.IntegerType())('num')).show()

# +---+-----+
# |num|num+2|
# +---+-----+
# | 10|   12|
# | 20|   22|
# +---+-----+