Pyspark-将数据帧写入2个不同的csv文件

Question

I want to save a single DataFrame into 2 different csv files (splitting the DataFrame) - one would include just the header and another would include the rest of the rows.

I want to save the 2 files under the same directory so Spark handling all the logic would be the best option if possible instead of splitting the csv file using pandas.

what would be the most efficient way to do this?

Thanks for your help!

Answer 1

Let's assume you've got Dataset called "df".

You can: Option one: write twice:

df.write.(...).option("header", "false").csv(....)
df.take(1).option("header", "true").csv() // as far as I remember, someone had problems with saving DataFrame without rows -> you must write at least one row and then manually cut this row using normal Java or Python file API

Or you can write once with header = true and then manually cut the header and place it in new file using normal Java API

Answer 2

Data, without header:

df.to_csv("filename.csv", header=False)

Header, without data:

df_new = pd.DataFrame(data=None, columns=df_old.columns) # data=None makes sure no rows are copied to the new dataframe
df_new.to_csv("filename.csv")