SPARK SCALA将DF导出到TextFile

Question

I want to export DF to text file.

There are two problems to export it as text file.

1. It has 46 columns, and error log says text only support 1 column.

2. I want the file is rolling fie means I want that same file store the result of next run.

Can you suggest me the way?

Answer 1

There are two methods of writing out the dataframe. The easiest method is to write out the file using the Spark SQL API, but you can also use the RDD API (keep in mind it will be written out as a single column with the RDD API).

Regarding writing out as a rolling file, I'm not sure that's possible with Spark. It would require opening the previous file in Spark, unioning the data, then writing it back to disk. You would probably be better off using incremental file names, which can be glob loaded as a single file.

Example Code for saving the DF as a csv:

val exampleDF = spark.read.option("header","true").option("inferSchema","true").format("csv").load("example.csv")

exampleDF.write.csv("example.out")

Spark SQL DF Example Results:

1,0,3,Braund, Mr. Owen Harris,male,22.0,1,0,A/5 21171,7.25,null,S
2,1,1,Cumings, Mrs. John Bradley (Florence Briggs Thayer),female,38.0,1,0,PC 17599,71.2833,C85,C
3,1,3,Heikkinen, Miss. Laina,female,26.0,0,0,STON/O2. 3101282,7.925,null,S
4,1,1,Futrelle, Mrs. Jacques Heath (Lily May Peel),female,35.0,1,0,113803,53.1,C123,S

Example Code for saving the RDD as text:

val exampleDF = spark.read.option("header","true").option("inferSchema","true").format("csv").load("example.csv")

exampleDF.rdd.saveAsTextFile("example.out")

RDD Example Results:

[1,0,3,Braund, Mr. Owen Harris,male,22.0,1,0,A/5 21171,7.25,null,S]
[2,1,1,Cumings, Mrs. John Bradley (Florence Briggs Thayer),female,38.0,1,0,PC 17599,71.2833,C85,C]
[3,1,3,Heikkinen, Miss. Laina,female,26.0,0,0,STON/O2. 3101282,7.925,null,S]
[4,1,1,Futrelle, Mrs. Jacques Heath (Lily May Peel),female,35.0,1,0,113803,53.1,C123,S]