Apache Spark function to_timestamp() 无法在 Databricks 上使用 PySpark

Question

I'm getting NULL output when I execute the code to_timestamp()

The code that I'm executing is as follows:

.withColumn("LAST_MODIFICATION_DT", to_timestamp(concat(col('LAST_MOD_DATE'), lit(' '), col('LAST_MOD_TIME')), 'yyy-MM-dd HH:mm:ss'))

The schema for the fields LAST_MOD_DATE & LAST_MOD_TIME is as follows:

I'm getting the output 'NULL' for the column 'LAST_MODIFICATION_DT'

Any thoughts?

Answer 1

In Spark SQL concat doesn't convert null to '' ; any null argument will cascade into a null result. It's often easier to write these kind of expressions in python and register them as UDFs, eg

from pyspark.sql.types import StringType

def concat2_(s1, s2) -> str:
  return str(s1) + ' ' + str(s2)

concat2 = spark.udf.register("concat2", concat2_, StringType())

Then you can use it in Spark queries in built in python,

from pyspark.sql.functions import col
df = spark.sql('select 1 a, 2 b').withColumn("c",concat2(col('a'),col('b')))
display(df)

or SQL

%sql
with q as
(select 1 a, 2 b)
select a,b,concat2(a,b) c 
from q