在Spark中使用dataFrame将NULL值插入Hive

Question

I'm trying insert values into a Hive table, and if every column has a value there are no problems, but I need to insert NULL value in one of that columns. I'm doing it in that way:

val errorsToAlert = List(("source1", "table1","27-01-2002", null))
val data = sqlContext.createDataFrame(errorsToAlert).toDF("source", 
"table_name", "open_date", "close_date")
data.write.mode("append").saveAsTable("management.alerts")

I've tried with NULL and None, but both represent this error:

17/06/26 11:59:38 ERROR yarn.ApplicationMaster: User class threw exception:
scala.MatchError: scala.None.type (of class scala.reflect.internal.Types$UniqueSingleType)
scala.MatchError: scala.None.type (of class scala.reflect.internal.Types$UniqueSingleType)

Answer 1

The problem is completely unrelated to Hive. If you check the type of errorsToAlert you'll see it is:

List[(String, String, String, Null)]

and scala.Null is not an acceptable input for Dataset .

If required type itself is nullable you can specify it explicitly:

sqlContext.createDataFrame(Seq(
  ("source1", "table1","27-01-2002", null: String)
))

otherwise use scala.Option :

sqlContext.createDataFrame(Seq(
  ("source1", "table1","27-01-2002", None: Option[Int])
))