[英]Spark Scala: Add 10 days to a date string (not a column)
I have a date and want to add and subtract 10 days to it. 我有一个日期,想要加减10天。 Start_date and end_date are dynamic variables from one table and will be used to filter another table.
Start_date和end_date是一个表中的动态变量,将用于过滤另一个表。
eg. 例如。
val start_date = "2018-09-08"
val end_date = "2018-09-15"
I want to use the two dates above in a filter shown below; 我想在下面显示的过滤器中使用上面的两个日期;
myDF.filter($"timestamp".between(date_sub(start_date, 10),date_add(end_date, 10)))
The functions date_add and date_sub only take in columns as an input. 函数date_add和date_sub仅将列作为输入。 How can I add/subtract 10 (this is an arbitrary number) from my dates?
如何从我的日期中加减10(这是一个任意数字)?
Thanks 谢谢
Thank you Luis! 谢谢路易斯! Your solution worked, for anyone interested the solution looks like;
您的解决方案有效,对于感兴趣的任何人来说,解决方案都是这样
val start_date = lit("2018-09-08")
val end_date = lit("2018-09-15")
myDF.filter($"timestamp".between(date_sub(start_date, 10),date_add(end_date, 10)))
Another way...If you can create a temp view, then you can access the vals
using $ interpolation. 另一种方式......如果你可以创建一个临时的观点,那么你可以访问
vals
使用$插值。 You should make sure the format is of default ones for date/timestamp. 您应该确保该格式是日期/时间戳的默认格式。
Check this out: 看一下这个:
scala> val start_date = "2018-09-08"
start_date: String = 2018-09-08
scala> val end_date = "2018-09-15"
end_date: String = 2018-09-15
scala> val myDF=Seq(("2018-09-08"),("2018-09-15")).toDF("timestamp").withColumn("timestamp",to_timestamp('timestamp))
myDF: org.apache.spark.sql.DataFrame = [timestamp: timestamp]
scala> myDF.show(false)
+-------------------+
|timestamp |
+-------------------+
|2018-09-08 00:00:00|
|2018-09-15 00:00:00|
+-------------------+
scala> myDF.createOrReplaceTempView("ts_table")
scala> spark.sql(s""" select timestamp, date_sub('$start_date',10) as d_sub, date_add('$end_date',10) d_add from ts_table """).show(false)
+-------------------+----------+----------+
|timestamp |d_sub |d_add |
+-------------------+----------+----------+
|2018-09-08 00:00:00|2018-08-29|2018-09-25|
|2018-09-15 00:00:00|2018-08-29|2018-09-25|
+-------------------+----------+----------+
scala>
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.