简体   繁体   English

如何在Spark中将non dataframe参数传递给udf

[英]How to pass the non dataframe argument to udf in spark

Can any please help to know how to pass the non dataframe value as an argument to udf. 有什么可以帮助您知道如何将非数据帧值作为参数传递给udf的。

val df2 = df1.withColumn("newcol", udffunc(df1("col1"), x)).

The udf function i defined as below. 我定义的udf函数如下。

udffunc = udf(method _)

Method i defined as below. 方法我定义如下。

def method(inputvar1: String, inputvar2: String): Option[Long] = {
  ...
  ...
  return Longvariable
}

I am able to fix the issue by just writing the below code for UDF. 我可以通过为UDF编写以下代码来解决此问题。

def udffunc = udf((y:String) => {
    val format:String = "x"
    method(y, "x")
  })

So with this, I am able to pass one argument(dataframe column) from .withColumn to the UDF function udffunc and then am able to pass external argument which is "x" in the above example to the function named as "method". 因此,我可以将一个参数(数据框列)从.withColumn传递给UDF函数udffunc ,然后可以将上述示例中的外部参数“ x”传递给名为“ method”的函数。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM