How can i use nvl function in scala

Question

I am trying to code the following:

df.select(nvl(col("id"),0))

When I execute this, I get an error value nvl not found.

Please help me to solve this issue.

Answer 1

In Spark its called coalesce, you can check this article for more details

# create new column with non Null values
tmp = testDF.withColumn('newColumn', coalesce(testDF['id'], testDF['number']))

# Check the content of new df
tmp.show()

+----+------+---------+
|  id|number|newColumn|
+----+------+---------+
|   1|     1|        1|
|   2|     2|        2|
|null|     3|        3|
|   4|  null|        4|
+----+------+---------+

In your case it may look like this:

df.select(coalesce(col("id"),lit(0)))

Answer 2

You can use a when-otherwise construct as well - see the snippet below:

df = spark.createDataFrame([(1, 2), (2, None), (None, 3)], "id: int, value: int")
df.withColumn("non_null_value", when(col("value").isNull(), 0).otherwise(col("value"))).show()

+----+-----+--------------+
|  id|value|non_null_value|
+----+-----+--------------+
|   1|    2|             2|
|   2| null|             0|
|null|    3|             3|
+----+-----+--------------+