如何将3个DataFrame合并到一个DataFrame中？

Question

I am trying to join three different DataFrames together in to one but I am having trouble in joining all three. I have been able to join two.

How can I join three DataFrames correctly?

Pyspark: 1.6.0

Below is my working so far:

# EXPECTED OUTPUT:

# -------file1.csv---------|---file2.csv--|---file3.csv------------|
# |col1|col2|col3|col4|col5|col1|col2|col3|col1|col2|col3|col4|col5|

# Loading in all the files
file1_rdd = sc.textFile("file1.csv").map(lambda line: line.split(","))
file2_rdd = sc.textFile("file2.csv").map(lambda line: line.split(","))
file3_rdd = sc.textFile("file3.csv").map(lambda line: line.split(","))

# Capturing the header
file1_header = file1_rdd.first()
file2_header = file2_rdd.first()
file3_header = file3_rdd.first()

# Removing the header from the table rows
df_file1 = file1_rdd.filter(lambda row : row != file1_header).toDF(file1_header)
df_file2 = file1_rdd.filter(lambda row : row != file2_header).toDF(file2_header)
df_file3 = file1_rdd.filter(lambda row : row != file3_header).toDF(file3_header)

# WORKS: df_file1.join(df_file2, df_file1.col1 == df_file2.col2)

# OUTPUT:
# -------file1.csv---------|---file2.csv--|
# |col1|col2|col3|col4|col5|col1|col2|col3|

# DOES NOT WORK: df_file1.join(df_file2, df_file1.col1 == df_file2.col2).join(df_file3, df_file2.col2 == df_file3.col2)

# OUTPUT:
# Caused by: java.lang.IllegalStateException: Input row doesn't have expected number of values required by the schema. 4 fields are required while 5 values are provided.

Why is there an error that 4 fields are required when I can join the first two without that error even though the fields are not of the same length?

Answer 1

The issue was that file3.csv contained data which was not properly sanitised. In order to fix it I simply enforced a maximum split like so:

file3_rdd = sc.textFile("file3.csv").map(lambda line: line.split(",", 3))

To anybody reading who may have faced a similar issue: check that you can first view the tables independently without errors. Doing df_file3.show() returned the same error and would have helped me see the issue much faster.