udf（用户定义函数）如何在 pyspark 中工作？

Question

I want to understand the working of udf in pyspark. Does a python shell opens up everytime we use a udf on top of a dataframe?

Answer 1

UDF can be given to PySpark in 2 ways.

1. UDF can be defined in Scala and run using PySpark
2. UDF can be defined in Python and run by PySpark

In first case UDF will run as part of Executor JVM itself, since UDF itself is defined in Scala. There is no need to create python process.

In second case for each executor a python process will be started. data will be serialised and deserialised between executor and python to process. This leads to lots of performance impact and overhead on spark job.

usually it is preferred to use Scala based UDF, since they will give you better performance.

in spark 2.x There is something called Pandas Udf(vectorised UDF). They are implemented using Pandas(Converting Dataframe using apache arrow). Basically you write code in python but that code is converted and run using pandas. which drastically improves performance of UDF in python. I hope this helps.

Answer 2

Spark runs a python process parallel to each executor and passes data back and forth between the Scala part (the executor) and python.

This has a lot of implications for performance and memory consumption (and management of them when you tune a spark job)