如何在 Spark DataFrame 中使用已注册的 Hive UDF？

Question

I have registered my hive UDF using the following code:

hiveContext.udf().register("MyUDF",new UDF1(String,String)) {
public String call(String o) throws Execption {
//bla bla
}
},DataTypes.String);

Now I want to use above MyUDF in DataFrame . How do we use it? I know how to use it in a SQL and it works fine

hiveContext.sql(select MyUDF("test") from myTable);

My hiveContext.sql() query involves group by on multiple columns so for scaling purpose I am trying to convert this query into DataFrame APIs

dataframe.select("col1","col2","coln").groupby(""col1","col2","coln").count();

Can we do the following: dataframe.select(MyUDF("col1")) ?

Answer 1

I tested the following with pyspark 3.x running on top of yarn and it works

from pyspark.sql.functions import expr

df1 = df.withColumn("result", expr("MyUDF('test')"))
df1.show()
df2 = df.selectExpr("MyUDF('test') as result").show()
df2.show()

In case you come across Class not found error. Then you might want to add the jar using spark.sql("ADD JAR hdfs://...")