Scala - 如何在 Spark SQL 查询中将日期字符串转换为时间戳？

Question

I have a formattedDataInputDateTime String that I want to insert into a table as a Timestamp type as a second field.

// Returns 2019-10-30T13:00Z
val localDateTimeZoned = OffsetDateTime.of(java.time.LocalDate.parse(currentDate), java.time.LocalTime.now, ZoneOffset.UTC).truncatedTo(ChronoUnit.HOURS)

// Returns 2019-10-30T13:00:00.000+0000
val formattedDataInputDateTime: String = localDateTimeZoned.format(DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSxx")).toString

So I wrote the following query but can't figure out how to insert the formattedDataInputDateTime as a timestamp here?

spark.sql(
  s"""INSERT INTO main.basic_metrics
     |VALUES ('metric_name', ???,
     |'metric_type', current_timestamp, false)""".stripMargin)

I've tried to test this approach but it resulted in the following error:

val ts = cast(unix_timestamp("$formattedDataInputDateTime", "yyyy-MM-dd'T'HH:mm:ss.SSSxx") as timestamp)

type mismatch;
 found   : String("$formattedDataInputDateTime")
 required: org.apache.spark.sql.Column

Answer 1

val ts = cast(unix_timestamp("$formattedDataInputDateTime", "yyyy-MM-dd'T'HH:mm:ss.SSSxx") as timestamp)

type mismatch;
 found   : String("$formattedDataInputDateTime")
 required: org.apache.spark.sql.Column

This basically means the $ is inside the quoted string. It should be outside like $"formattedDataInputDateTime"

Answer 2

You are passing String instead of Column , you can wrap it using lit :

cast(unix_timestamp(lit(formattedDataInputDateTime), "yyyy-MM-dd'T'HH:mm:ss.SSSxx")

However you can get current date and format it using spark functions current_date and date_format .