Scala-将数据帧作为二进制文件写入文件

Question

I have a hive table of type parquet, with column Content storing various documents as base64 encoded.

Now, I need to read that column and write into a file in HDFS, so that the base64 column will be converted back to a document for each row.

val profileDF = sqlContext.read.parquet("/hdfspath/profiles/");
profileDF.registerTempTable("profiles")
val contentsDF = sqlContext.sql(" select unbase64(contents) as contents from profiles where file_name'file1'")

Now that contentDF is storing the binary format of a document as a row, which I need to write to a file. Tried different options but couldn't get back the dataframe content to a file.

Appreciate any help regarding this.

Answer 1

I would suggest save as parquet:

https://spark.apache.org/docs/1.6.3/api/java/org/apache/spark/sql/DataFrameWriter.html#parquet(java.lang.String)

Or convert to RDD and do save as object:

https://spark.apache.org/docs/1.6.3/api/java/org/apache/spark/rdd/RDD.html#saveAsObjectFile(java.lang.String)