[英]Get last element of list in Spark Dataframe column
I have a DataFrame
that has the following schema. 我有一个具有以下架构的
DataFrame
。
root
|-- memberId: long (nullable = true)
|-- items: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- timestamp: long (nullable = true)
| | |-- itemId: integer (nullable = true)
| | |-- weight: double (nullable = true)
Say, the DataFrame (called df
) looks like this. 说,DataFrame(称为
df
)看起来像这样。
+-----------+------------------------------------------------------------------------+
|memberId |items |
+-----------+------------------------------------------------------------------------+
|10000000001|[[1234567891, 104, 1.0], [1234567892, 103, 3.0]] |
|10000000002|[[1234567891, 103, 1.0], [1234567893, 102, 1.0], [1234567894, 101, 2.0]]|
+-----------+------------------------------------------------------------------------+
As can be seen, the df
is a map of memberId
to a list
of struct
s. 可以看出,
df
是memberId
到struct
list
的映射。 I want to transform it such that I retrieve the last element in the list of struct
s corresponding to each member. 我想对其进行转换,以便检索对应于每个成员的
struct
列表中的最后一个元素。 So, the resulting DataFrame
should look like 因此,产生的
DataFrame
应该看起来像
+-----------+----------------------+
|memberId |lastItem |
+-----------+----------------------+
|10000000001|[1234567892, 103, 3.0]|
|10000000002|[1234567894, 101, 2.0]|
+-----------+----------------------+
I tried this 我试过了
val newDf = df
.withColumn("lastItem", last($"items"))
.drop("items")
But this just throws an exception of the form: 但这仅引发以下形式的异常:
grouping expressions sequence is empty,
and '`memberId`' is not an aggregate function.
Wrap '(last(`items`, false) AS `item`)' in
windowing function(s) or wrap '`memberId`' in
first() (or first_value) if you don't care which value you get
I believe this happens because last
is an aggregation
function and requires me to .groupBy("memberId")
before I invoke last
. 我相信发生这种情况是因为
last
是一个aggregation
函数,并且要求我在调用last
之前使用.groupBy("memberId")
。
How can I do this? 我怎样才能做到这一点? Use of
UDF
s is not encouraged when working with DataFrame
s but I can't find a native function that can do what I intend to do. 在使用
DataFrame
时,不鼓励使用UDF
,但是我找不到能完成我打算做的事情的本机函数。
You can do this using apply
method on Column
of type array, with which you can access array elements: 您可以使用array类型的
Column
的apply
方法来执行此操作,通过它可以访问数组元素:
val newDf = df
.withColumn("lastItem", $"items"(size($"items")-1))
.drop("items")
Edit: 编辑:
to get the first n-1 items, I would use an UDF : 要获得前n-1个项目,我将使用UDF:
val sliceUDF = udf((arr:Seq[Row],from:Int,to:Int) => arr.slice(from,to).map{case Row(ts:Long,Id:Int,w:Double) => (ts,Id,w)})
val newDf = df
.withColumn("subItems", sliceUDF($"items",lit(0),size($"items")-1))
.drop("items")
Maybe it could also be done using pure DataFrame API, but I think it would be rather complicated (eg using a combination of posexplode
, window-function and collect_list
) 也许也可以使用纯DataFrame API来完成,但是我认为这会相当复杂(例如,结合使用
posexplode
,window-function和collect_list
)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.