[英]Passing a list of tuples as a parameter to a spark udf in scala
I am trying to pass a list of tuples to a udf in scala. 我试图将一个元组列表传递给scala中的udf。 I am not sure how to exactly define the datatype for this.
我不确定如何为此精确定义数据类型。 I tried to pass it as a whole row but it can't really resolve it.
我试图将它作为整行传递,但它无法真正解决它。 I need to sort the list based on the first element of the tuple and then send n number of elements back.
我需要根据元组的第一个元素对列表进行排序,然后再发送n个元素。 I have tried the following definitions for the udf
我为udf尝试了以下定义
def udfFilterPath = udf((id: Long, idList: Array[structType[Long, String]] )
def udfFilterPath = udf((id: Long, idList: Array[Tuple2[Long, String]] )
def udfFilterPath = udf((id: Long, idList: Row)
This is what the idList looks like: 这就是idList的样子:
[[1234,"Tony"], [2345, "Angela"]]
[[1234,"Tony"], [234545, "Ruby"], [353445, "Ria"]]
This is a dataframe with a 100 rows like the above. 这是一个如上所述的100行数据帧。 I call the udf as follows:
我把udf称为如下:
testSet.select("id", "idList").withColumn("result", udfFilterPath($"id", $"idList")).show
When I print the schema for the dataframe it reads it as a array of structs. 当我打印数据帧的架构时,它将其作为结构数组读取。 The idList itself is generated by doing a collect list over a column of tuples grouped by a key and stored in the dataframe.
idList本身是通过在由密钥分组并存储在数据帧中的元组列上执行收集列表来生成的。 Any ideas on what I am doing wrong?
关于我做错了什么的任何想法? Thanks!
谢谢!
When defining a UDF, you should use plain Scala types (eg Tuples, Primitives...) and not the Spark SQL types (eg StructType
) as the output types . 定义UDF时,应使用普通的Scala类型(例如Tuples,Primitives ...)而不是 Spark SQL类型(例如
StructType
)作为输出类型 。
As for the input types - this is where it gets tricky (and not too well documented) - an array of tuples would actually be a mutable.WrappedArray[Row]
. 至于输入类型 - 这是它变得棘手(并且没有太多文档记录) - 一个元组数组实际上是一个
mutable.WrappedArray[Row]
。 So - you'll have to "convert" each row into a tuple first, then you can do the sorting and return the result. 所以 - 你必须先将每一行“转换”成一个元组,然后你可以进行排序并返回结果。
Lastly, by your description it seems that id
column isn't used at all, so I removed it from the UDF definition, but it can easily be added back. 最后,根据您的描述,似乎根本没有使用
id
列,因此我将其从UDF定义中删除,但可以轻松地将其添加回来。
val udfFilterPath = udf { idList: mutable.WrappedArray[Row] =>
// converts the array items into tuples, sorts by first item and returns first two tuples:
idList.map(r => (r.getAs[Long](0), r.getAs[String](1))).sortBy(_._1).take(2)
}
df.withColumn("result", udfFilterPath($"idList")).show(false)
+------+-------------------------------------------+----------------------------+
|id |idList |result |
+------+-------------------------------------------+----------------------------+
|1234 |[[1234,Tony], [2345,Angela]] |[[1234,Tony], [2345,Angela]]|
|234545|[[1234,Tony], [2345454,Ruby], [353445,Ria]]|[[1234,Tony], [353445,Ria]] |
+------+-------------------------------------------+----------------------------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.