如何使用Scala将Map保存到Spark中的Json？

Question

I need to save a Map(key-value pairs) in one column using Spark. The requirement is that other people may use the data with other tools like PIG, so it is better to save the Map with a general format rather than a special formatted string. I create the column using this code:

StructField("cMap", DataTypes.createMapType(StringType, StringType), true) ::

Then after I create the dataframe, I got the schema:

|-- cMap: map (nullable = true)
|    |-- key: string
|    |-- value: string (valueContainsNull = true)

Then I save the dataframe to Json:

df.write.json(path)

I found the Json output is:

"cMap":{"1":"a","2":"b","3":"c"}

So once I read it from the file next time:

val new_df = sqlContext.read.json(path)

I got the schema:

|-- cMap: struct (nullable = true)
|    |-- 1: string
|    |-- 2: string
|    |-- 3: string

Is there any efficient way to save and read the map in Json without extra processing( I could save the map into a special string and decoded it, but I think it should not be that complex). Thanks.

Answer 1

You can save the table as parquet file

• Write:

  df.write.parquet("mydf.parquet")

• Read

  val new_df = spark.read.parquet("mydf.parquet")

spark guide save-modes

 // Encoders for most common types are automatically provided by importing spark.implicits._ import spark.implicits._ val peopleDF = spark.read.json("examples/src/main/resources/people.json") // DataFrames can be saved as Parquet files, maintaining the schema information peopleDF.write.parquet("people.parquet") // Read in the parquet file created above // Parquet files are self-describing so the schema is preserved // The result of loading a Parquet file is also a DataFrame val parquetFileDF = spark.read.parquet("people.parquet") // Parquet files can also be used to create a temporary view and then used in SQL statements parquetFileDF.createOrReplaceTempView("parquetFile") val namesDF = spark.sql("SELECT name FROM parquetFile WHERE age BETWEEN 13 AND 19") namesDF.map(attributes => "Name: " + attributes(0)).show() 

Answer 2

Parquet format should solve the issue you are having. Parquet stores binary data in a column-oriented way, where the values of each column are organized so that they are all adjacent, enabling better compression

Just save it to Parquet as below

df.write.mode(SaveMode.Overwrite).parquet("path to the output")

And read it as below

val new_df = sqlContext.read.parquet("path to the above output")

I hope this helps