如何将 spark DataFrame 行值映射到列？

Question

I am trying to map values in rows to columns in another dataframe.

I have the following DataFrame, the values in "id" are known to be unique:

sqlContext.createDataFrame(Seq(("a", 1),("b",2))).toDF("id","number")

And:

sqlContext.createDataFrame(Seq(("jane",10),("John",12))).toDF("mcid", "age")

And I wish to produce a DataFrame with the schema:

| mcid | age | a | b |

Answer 1

I have no idea what you are try to do, but assuming you have this:

val df1 = sqlContext.createDataFrame(Seq(("a", 1),("b",2))).toDF("id","number")
val df2 = sqlContext.createDataFrame(Seq(("jane",10),("John",12))).toDF("mcid", "age")

This will get you a DataFrame with the schema you are looking for:

df2.join(df1).groupBy($"mcid", $"age").pivot("id").sum("number")