[英]Spark DF pivot error: Method pivot([class java.lang.String, class java.lang.String]) does not exist
I am a newbie at using Spark dataframes.我是使用 Spark 数据帧的新手。 I am trying to use the pivot
method with Spark (Spark version 2.x) and running into the following error:我正在尝试对 Spark(Spark 版本 2.x)使用pivot
方法并遇到以下错误:
Py4JError: An error occurred while calling o387.pivot. Py4JError:调用 o387.pivot 时出错。 Trace: py4j.Py4JException: Method pivot([class java.lang.String, class java.lang.String]) does not exist跟踪:py4j.Py4JException: Method pivot([class java.lang.String, class java.lang.String]) 不存在
Even though I have the agg
function as first
here, I really do not need to apply any aggregation.尽管我在这里first
使用agg
函数,但我真的不需要应用任何聚合。
My dataframe looks like this:我的数据框如下所示:
+-----+-----+----------+-----+
| name|value| date| time|
+-----+-----+----------+-----+
|name1|100.0|2017-12-01|00:00|
|name1|255.5|2017-12-01|00:15|
|name1|333.3|2017-12-01|00:30|
Expected:预期的:
+-----+----------+-----+-----+-----+
| name| date|00:00|00:15|00:30|
+-----+----------+-----+-----+-----+
|name1|2017-12-01|100.0|255.5|333.3|
The way I am trying:我正在尝试的方式:
df = df.groupBy(["name","date"]).pivot(pivot_col="time",values="value").agg(first("value")).show
What is my mistake here?我在这里有什么错误?
The problem is the values="value"
parameter in the pivot
function.问题在于pivot
函数中的values="value"
参数。 This should be used for a list of actual values to pivot on, not a column name.这应该用于要透视的实际值列表,而不是列名。 From the documentation :从文档:
values – List of values that will be translated to columns in the output DataFrame. values – 将被转换为输出 DataFrame 列的值列表。
and an example:和一个例子:
df4.groupBy("year").pivot("course", ["dotNET", "Java"]).sum("earnings").collect() [Row(year=2012, dotNET=15000, Java=20000), Row(year=2013, dotNET=48000, Java=30000)]
For the example in the question values
should be set to ["00:00","00:15", "00:30"]
.对于问题中的示例, values
应设置为["00:00","00:15", "00:30"]
。 However, the values
argument is often not necessary (but will make the pivot more efficient), so you can simply change to:但是, values
参数通常不是必需的(但会使数据透视更有效),因此您可以简单地更改为:
df = df.groupBy(["name","date"]).pivot("time").agg(first("value"))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.