Spark数据帧转换异常

Question

Sorry for the terrible title. I don't know how else to describe it.

what I'm looking for is how to transform a dataframe that looks like

+---+----+---+
| id|name|age|
+---+----+---+
|  1| Tim|123|
|  1| Bob|456|
|  2| Amy|789|
|  3| Tom|101|
+---+----+---+

into a data frame that looks like

+------+-----+------+-----+------+-----+
|1-name|1-age|2-name|2-age|3-name|3-age|
+------+-----+------+-----+------+-----+
|   Tim|  123|   Amy|  789|   Tom|  101|
|   Bob|  456|  null| null|  null| null|
+------+-----+------+-----+------+-----+

Answer 1

This "abnormal" transformation is called a pivot , and it was implemented in Spark 1.6. Link to Python docs

More reading material:

• https://databricks.com/blog/2016/02/09/reshaping-data-with-pivot-in-apache-spark.html
• https://www.svds.com/pivoting-data-in-sparksql/