如何注册我的自定义UDF，以便可以在spark-shell中运行它

Question

I defined the following custom UDF:

def stddev1 (columnName: Column): Column = {
    sqrt(avg(columnName * columnName) - avg(columnName) * avg(columnName))
}

I want to run this function in spark-shell and test it with some example data but I keep running into errors: "Schema for type org.apache.spark.sql.Column is not supported."

I might have to register it but I'm unsure how to do this

Answer 1

It depends on how you want to use it. For example, this works fine:

val df = sc.parallelize(Seq(1,2,3,4)).toDF("myCol")
df.show

>+-----+
>|myCol|
>+-----+
>|    1|
>|    2|
>|    3|
>|    4|
>+-----+

def stddev(col: Column): Column = sqrt(avg(col * col) - avg(col) * avg(col))
df.agg(stddev($"myCol")).first

> [1.118033988749895]

However if you want to use it within a Spark SQL statement, you will need something like this:

val squared = (s: Int) => {
  s * s
}
sqlContext.udf.register("square", squared)

%sql select id, square(id) as id_squared from test

Check this out for more info.