从Spark中具有不同标头的多个csv文件创建一个数据帧

Question

In Spark, with Pyspark, i want to create one dataframe (where the path is actually a folder in S3), which contains multi csv files with common columns and different columns. To say it more easily, i want only one dataframe from multiple csv files with different headers.

I can have a file with this header "raw_id, title, civility", and another file with this header "raw_id, first_name, civility".

This is my code in python 3 :

df = spark.read.load(
    s3_bucket + 'data/contacts/normalized' + '/*/*/*/*',
    format = 'csv',
    delimiter = '|',
    encoding = 'utf-8',
    header = 'true',
    quote = ''
)

This is an example of file_1.csv :

|raw_id|title|civility|
|1     |M    |male    |

And an example of file2.csv :

|raw_id|first_name|civility|
|2     |Tom       |male    |

The result i expect in my dataframe is :

|raw_id|first_name|title|civility|
|1     |          |M    |male    |
|2     |Tom       |     |male    |

But, what is happening is that i have all united columns but the data is not in the right place after the first file. Do you know how to do this ?

Thank you very much by advance.

Answer 1

您需要将它们中的每一个加载到不同的数据框中，并将它们在raw_id列上连接在一起。