How to filter nullable Array-Elements in Spark 1.6 UDF

Question

Consider the following DataFrame

root
 |-- values: array (nullable = true)
 |    |-- element: double (containsNull = true)

with content:

+-----------+
|     values|
+-----------+
|[1.0, null]|
+-----------+

Now I want to pass thie value column to an UDF:

val inspect = udf((data:Seq[Double]) => {
  data.foreach(println)
  println()
  data.foreach(d => println(d))
  println()
  data.foreach(d => println(d==null))
  ""
})

df.withColumn("dummy",inspect($"values"))

I'm really confused from the output of the above println statements:

1.0
null

1.0
0.0

false
false

My questions:

1. Why is foreach(println) not giving the same output as foreach(d=>println(d)) ?
2. How can the Double be null in the first println-statement, I thought scala's Double cannot be null?
3. How can I filter null values in my Seq other han filtering 0.0 which isnt really safe? Should I use Seq[java.lang.Double] as type for my input in the UDF and then filter nulls? (this works, but I'm unsure if that is the way to go)

Note that I'm aware of this Question , but my question is specific to array-type columns.

Answer 1

Why is foreach(println) not giving the same output as foreach(d=>println(d))?

In the context where Any is expected data cast is skipped completely. This is explained in detail in If an Int can't be null, what does null.asInstanceOf[Int] mean?

How can the Double be null in the first println-statement, I thought scala's Double cannot be null?

Internal binary representation doesn't use Scala types at all. Once array data is decoded it is represented as an Array[Any] and elements are coerced to a declared type with simple asInstanceOf .

Should I use Seq[java.lang.Double] as type for my input in the UDF and then filter nulls?

In general if values are nullable then you should use external type which is nullable as well or Option . Unfortunately only the first option is applicable for UDFs.