简体   繁体   English

在 PySpark 中将 Spark DataFrame 从行移到列,并附加另一个 DataFrame

[英]Transposing a Spark DataFrame from row to column in PySpark and appending it with another DataFrame

I have a Spark DataFrame in PySpark avg_length_df that looks like -我在 PySpark avg_length_df中有一个 Spark DataFrame 看起来像 -

+----------------+---------+----------+-----------+---------+-------------+----------+
|       id       |        x|         a|          b|        c|      country|     param|
+----------------+---------+----------+-----------+---------+-------------+----------+
|            40.0|      9.0|     5.284|      5.047|    6.405|         13.0|avg_length|
+----------------+---------+----------+-----------+---------+-------------+----------+

I want to transpose it from row to column so that it becomes -我想将它从行移到列,使其变为 -

+----------+
|avg_length|
+----------+
|      40.0|
|       9.0|
|     5.284|
|     5.047|
|     6.405|
|      13.0|
+----------+

Next, I have a second DataFrame df2 :接下来,我有第二个 DataFrame df2

+----------------+------+
|       col_names|dtypes|
+----------------+------+
|              id|string|
|               x|   int|
|               a|string|
|               b|string|
|               c|string|
|         country|string|
+----------------+------+

I want to create a column avg_length in df2 the equals to the transposed DataFrame above.我想在df2中创建一个列avg_length等于上面的转置 DataFrame 。 So the expected output would look like:所以预期的 output 看起来像:

+----------------+------+----------+
|       col_names|dtypes|avg_length|
+----------------+------+----------+
|              id|string|      40.0|
|               x|   int|       9.0|
|               a|string|     5.284|
|               b|string|     5.047|
|               c|string|     6.405|
|         country|string|      13.0|
+----------------+------+----------+

How do I complete the 2 operations?如何完成这2个操作?

>>> from pyspark.sql import *
#Input DataFrame
>>> df.show()
+----+---+-----+-----+-----+-------+----------+
|  id|  x|    a|    b|    c|country|     param|
+----+---+-----+-----+-----+-------+----------+
|40.0|9.0|5.284|5.047|6.405|   13.0|avg_length|
+----+---+-----+-----+-----+-------+----------+

>>> avgDF  = df.groupBy(df["id"],df["x"],df["a"],df["b"],df["c"],df["country"]).pivot("param").agg(concat_ws("",collect_list(to_json(struct("id","x","a","b","c","country"))))).drop("id","x","a","b","c","country")
>>> avgDF.show(2,False)
+----------------------------------------------------------------------------+
|avg_length                                                                  |
+----------------------------------------------------------------------------+
|{"id":"40.0","x":"9.0","a":"5.284","b":"5.047","c":"6.405","country":"13.0"}|
+----------------------------------------------------------------------------+

>>> finalDF = avgDF.withColumn("value", explode(split(regexp_replace(col("avg_length"),"""[\\{ " \\}]""",""),","))).withColumn("avg_length", split(col("value"), ":")[1]).withColumn("col_names", split(col("value"), ":")[0]).drop("value")
>>> finalDF.show(10,False)
+----------+---------+
|avg_length|col_names|
+----------+---------+
|40.0      |id       |
|9.0       |x        |
|5.284     |a        |
|5.047     |b        |
|6.405     |c        |
|13.0      |country  |
+----------+---------+

#other dataframe
>>> df2.show()
+---------+------+
|col_names|dtypes|
+---------+------+
|       id|string|
|        x|   int|
|        a|string|
|        b|string|
|        c|string|
|  country|string|
+---------+------+

>>> df2.join(finalDF,"col_names").show(10,False)
+---------+------+----------+
|col_names|dtypes|avg_length|
+---------+------+----------+
|id       |string|40.0      |
|x        |int   |9.0       |
|a        |string|5.284     |
|b        |string|5.047     |
|c        |string|6.405     |
|country  |string|13.0      |
+---------+------+----------+

Below is the code to transpose a data frame (RDD) in pyspark.下面是在 pyspark 中转置数据帧 (RDD) 的代码。

import numpy as np
from pyspark.sql import SQLContext
from pyspark.sql.functions import lit

dt1 = {'avg_length':[40.0, 9.0, 5.284, 5.047, 6.405, 13.0]}
dt = sc.parallelize([ (k,) + tuple(v[0:]) for k,v in dt1.items()]).toDF()
dt.show()


#--- Transpose Code ---


# Grad data from first columns, since it will be transposed to new column headers
new_header = [i[0] for i in dt.select("_1").rdd.map(tuple).collect()]

# Remove first column from dataframe
dt2 = dt.select([c for c in dt.columns if c not in ['_1']])

# Convert DataFrame to RDD
rdd = dt2.rdd.map(tuple)

# Transpose Data
rddT1 = rdd.zipWithIndex().flatMap(lambda (x,i): [(i,j,e) for (j,e) in enumerate(x)])
rddT2 = rddT1.map(lambda (i,j,e): (j, (i,e))).groupByKey().sortByKey()
rddT3 = rddT2.map(lambda (i, x): sorted(list(x), cmp=lambda (i1,e1),(i2,e2) : cmp(i1, i2)))
rddT4 = rddT3.map(lambda x: map(lambda (i, y): y , x))

# Convert back to DataFrame (along with header)
df = rddT4.toDF(new_header)

df.show()

After transposing you can simply merge the two data frames.转置后,您可以简单地合并两个数据帧。 I hope this helps.我希望这有帮助。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM