如何从另一个数据框中的dict创建数据框？

Question

I'm have a column of spark-dataframe
Output from df.select('parsed').show() :

+--------------------+
|              parsed|
+--------------------+
|{Action Flags=I, ...|
|{Action Flags=I, ...|
|{Action Flags=I, ...|
|{Action Flags=I, ...|
+--------------------+

All elements of this column is dict.
How I can made new spark-dataframe from dicts using keys as column names?

Answer 1

Before converting columns from a column having dict as values, you must know about its keys. So can label columns. Below i creating sample dataframe and then converting dict keys to column.

df = sqlContext.createDataFrame([
     [{'a':1,'b':2, 'c': 3}],
     [{'a':1,'b':2, 'c': 3}],
     [{'a':1,'b':2, 'c': 3}]], ["col"]
)
df.show(truncate=False)
+---------------------------+
|col                        |
+---------------------------+
|Map(b -> 2, c -> 3, a -> 1)|
|Map(b -> 2, c -> 3, a -> 1)|
|Map(b -> 2, c -> 3, a -> 1)|
+---------------------------+

After creating sample dataframe lets get first row from it -

first_row = df.first()['col'] #select column which have dict as values 
print (first_row)
{u'a': 1, u'b': 2, u'c': 3}

Now we have values from first row and also dict column values, extract keys from it so we can create column from it -

columns = first_row.keys()
print (columns)
[u'a', u'c', u'b']

After this loop over column list and select these as column from dict column -

from pyspark.sql import functions as F
col_list = [F.col("col").getItem(col).alias(col) for col in columns]
df.select(col_list).show()
+---+---+---+
|  a|  c|  b|
+---+---+---+
|  1|  3|  2|
|  1|  3|  2|
|  1|  3|  2|
+---+---+---+

There are others ways to do this also. Above i have mentioned one way, below is second by creating new column with withColumn -

for cl in columns: #already created columns variable
    df = df.withColumn(cl, F.col("col").getItem(cl))
df.show(truncate=False)

+---------------------------+---+---+---+
|col                        |a  |c  |b  |
+---------------------------+---+---+---+
|Map(b -> 2, c -> 3, a -> 1)|1  |3  |2  |
|Map(b -> 2, c -> 3, a -> 1)|1  |3  |2  |
|Map(b -> 2, c -> 3, a -> 1)|1  |3  |2  |
+---------------------------+---+---+---+