pyspark 的用户定义函数 (UDF) 是否需要单元测试？

Question

I'm new to pysaprk, so I have a function and I've written unit test for it, and I have defined a UDF function by using this function for pyspark, something like:

udf_my_function = udf(lambda s: my_function(s), StringType())

My question is if I already have the unit test for my_function() , do I need a unit test for udf_my_function ? If so, how can I write it? Any relevant articles or links will also be appreciated? Many thanks.

Answer 1

from my personal opinion, it's not strictly necessary. But sometimes it's still desirable to have the test as part of the testing suite that is doing data transformations. Usually it will have form of:

sourceDf = .... # read data from somewhere, or define in test
resultDf = sourceDf.withColumn("result", udf_my_function(col("some_column")))
assertEqual(resultDf, expectedDf)

There are several libraries available for writing unit tests for PySpark:

• spark-testing-base - supports both Scala & Python
• chispa - simple, and easy to use

you can also use pytest-spark to simplify the maintenance of the Spark parameters, include 3rd-party packages, etc.