合并列是可变结构的数据帧 - Pyspark

Question

I have a bunch of dataframes that I need to merge, the 4 columns that they have are same but one column out of those (params) has variable fields in it depending on the dataframe, I've displayed examples below:

+---------+-----------+--------------------------------------------------------------------------------------------------------------------------+-------------------+
|attribute|operation  |params                                                                                                                    |timestamp          |
+---------+-----------+--------------------------------------------------------------------------------------------------------------------------+-------------------+
|profile  |CREATE_CARD|[50d966f2-2820-441a-afbe-851e45eeb13e, 1s9miu7t6an50fplvvhybow6edx9_STG, 993270335, CREATED_CARD, 8236961209881953, kobo] |2020-02-24 03:07:04|
+---------+-----------+--------------------------------------------------------------------------------------------------------------------------+-------------------+

+---------+---------+--------------------------------------------------------------------------------------------------+-------------------+

|attribute|operation|params                                                                                            |timestamp          |
+---------+---------+--------------------------------------------------------------------------------------------------+-------------------+
|profile  |UPDATE   |[0792b8d1-7ad9-43fc-9e75-9b1f2612834c, rkm9a7mescuwp0s4i01zlwi2ftu9_STG, 993270329, primary_email]|2020-02-12 18:13:08|
+---------+---------+--------------------------------------------------------------------------------------------------+-------------------+

+---------+---------+-----------------------------------------------------------------------------------+-------------------+
|attribute|operation|params                                                                             |timestamp          |
+---------+---------+-----------------------------------------------------------------------------------+-------------------+
|member   |CREATE   |[ea8e7e39-4a0a-4d41-b47e-70c8e56a2bca, h4m015wf1qxwrogj6d9l2uc5bsa9_STG, 993270331]|2020-01-02 09:51:32|
+---------+---------+-----------------------------------------------------------------------------------+-------------------+

How do I get all the rows in these dataframes into a single dataframe without adding null values to the missing fields? I have to merge dataframes to store the final dataframe sorted on the field timestamp. I don't want to save params as a string as I need to store the final merged dataframe as a JSON in text and saving it as a string will add escaped characters to the final file, which I'm trying to avoid.

I tried converting the Dataframes to JSON object using toJSON() and then merging it, but toJSON() gave me a RDD with elements of string type, which I can't sort on. I also tried union , but that didn't work as the column 'params' is a different struct in each of the dataframes shown above. What is the most efficient way to do this?

Final Output should look like this:

+---------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------+
|attribute|operation  |timestamp           |params                                                    
+---------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------+
|profile  |CREATE_CARD|2020-02-24 03:07:04 |[50d966f2-2820-441a-afbe-851e45eeb13e, 1s9miu7t6an50fplvvhybow6edx9_STG, 993270335, CREATED_CARD, 8236961209881953, kobo]|
|profile  |UPDATE     |2020-02-12 18:13:08 |[0792b8d1-7ad9-43fc-9e75-9b1f2612834c, rkm9a7mescuwp0s4i01zlwi2ftu9_STG, 993270329, primary_email]|
|member   |CREATE     |2020-01-02 09:51:32 |[ea8e7e39-4a0a-4d41-b47e-70c8e56a2bca, h4m015wf1qxwrogj6d9l2uc5bsa9_STG, 993270331]

Answer 1

You can use "unionByName" (from 2.3) https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/dataframe.html

like:

from functools import reduce
from pyspark.sql import DataFrame

dfs = [df1, df2, df3]
df = reduce(DataFrame.unionByName, dfs)

If you have a Spark version lower than 2.3 you can use "union", but beware of column order.