无法将Spark JSON数据框加载到配置单元表中

Question

I want to convert a dataframe into a json object and load it into a json table.

Below is the code

Create the table

spark.sql("""create table IF NOT EXISTS user_tech.tests (
Z struct<A:string, 
 B:string,
 C:string>
)
stored as orc """)

import org.apache.spark.sql._

Initial data frame

val df = Seq((1,2,3),(2,3,4)).toDF("A", "B", "C")    


val jsonColumns = df.select("A", "B", "C")

Converting it into json

import org.apache.spark.sql.functions._
val finalDF = jsonColumns.select(to_json(struct(col("A"), col("B"), col("C")))).as("Z")

Insert the rows into the table

finalDF.registerTempTable("test")

spark.sql(""" select * from test """).show()

spark.sql("""Insert into  user_tech.tests select * from test""")

I am getting the following error:

org.apache.spark.sql.AnalysisException: cannot resolve 'test.`structstojson(named_struct(NamePlaceholder(), A, NamePlaceholder(), B, NamePlaceholder(), C))`' due to data type mismatch: cannot cast StringType to StructType(StructField(guid,StringType,true), StructField(sessionid,StringType,true));;

Answer 1

The problem is with the following statement.

val finalDF = jsonColumns.select(to_json(struct(col("A"), col("B"), col("C")))).as("Z")

A quick validation of the above DataFrame will make you understand that you are creating a single column of String type.

scala> finalDF.show
+--------------------------------------------------------------------------------------------+
|structtojson(named_struct(NamePlaceholder(), A, NamePlaceholder(), B, NamePlaceholder(), C))|
+--------------------------------------------------------------------------------------------+
|                                                                         {"A":1,"B":2,"C":3}|
|                                                                         {"A":2,"B":3,"C":4}|
+--------------------------------------------------------------------------------------------+


scala> finalDF.printSchema
root
 |-- structtojson(named_struct(NamePlaceholder(), A, NamePlaceholder(), B, NamePlaceholder(), C)): string (nullable = true)

And when you try to insert from Temp table registered on finalDF, the schema doesn't match and you got the exception.

Following should work for you.

spark.sql("""create table IF NOT EXISTS tests (
Z struct<A:string, 
 B:string,
 C:string>
)
stored as orc """)


import org.apache.spark.sql._

val df = Seq((1,2,3),(2,3,4)).toDF("A", "B", "C")    

val jsonColumns = df.select("A", "B", "C")

jsonColumns.registerTempTable("tmp")

spark.sql("""Insert into tests select struct(*) from tmp""")

You can see the data using the following statement.

spark.sql("select * from tests").show


+-------+
|      Z|
+-------+
|[1,2,3]|
|[1,2,3]|
|[2,3,4]|
|[2,3,4]|
+-------+

Hope that helps!