简体   繁体   English

Spark Scala:将10天添加到日期字符串(而不是列)

[英]Spark Scala: Add 10 days to a date string (not a column)

I have a date and want to add and subtract 10 days to it. 我有一个日期,想要加减10天。 Start_date and end_date are dynamic variables from one table and will be used to filter another table. Start_date和end_date是一个表中的动态变量,将用于过滤另一个表。

eg. 例如。

val start_date = "2018-09-08"
val end_date   = "2018-09-15"

I want to use the two dates above in a filter shown below; 我想在下面显示的过滤器中使用上面的两个日期;

myDF.filter($"timestamp".between(date_sub(start_date, 10),date_add(end_date, 10)))

The functions date_add and date_sub only take in columns as an input. 函数date_add和date_sub仅将列作为输入。 How can I add/subtract 10 (this is an arbitrary number) from my dates? 如何从我的日期中加减10(这是一个任意数字)?

Thanks 谢谢

Thank you Luis! 谢谢路易斯! Your solution worked, for anyone interested the solution looks like; 您的解决方案有效,对于感兴趣的任何人来说,解决方案都是这样

val start_date = lit("2018-09-08")
val end_date   = lit("2018-09-15")
myDF.filter($"timestamp".between(date_sub(start_date, 10),date_add(end_date, 10)))

Another way...If you can create a temp view, then you can access the vals using $ interpolation. 另一种方式......如果你可以创建一个临时的观点,那么你可以访问vals使用$插值。 You should make sure the format is of default ones for date/timestamp. 您应该确保该格式是日期/时间戳的默认格式。

Check this out: 看一下这个:

scala> val start_date = "2018-09-08"
start_date: String = 2018-09-08

scala> val end_date   = "2018-09-15"
end_date: String = 2018-09-15

scala> val myDF=Seq(("2018-09-08"),("2018-09-15")).toDF("timestamp").withColumn("timestamp",to_timestamp('timestamp))
myDF: org.apache.spark.sql.DataFrame = [timestamp: timestamp]

scala> myDF.show(false)
+-------------------+
|timestamp          |
+-------------------+
|2018-09-08 00:00:00|
|2018-09-15 00:00:00|
+-------------------+


scala> myDF.createOrReplaceTempView("ts_table")


scala> spark.sql(s""" select timestamp, date_sub('$start_date',10) as d_sub, date_add('$end_date',10) d_add from ts_table """).show(false)
+-------------------+----------+----------+
|timestamp          |d_sub     |d_add     |
+-------------------+----------+----------+
|2018-09-08 00:00:00|2018-08-29|2018-09-25|
|2018-09-15 00:00:00|2018-08-29|2018-09-25|
+-------------------+----------+----------+


scala>

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 将天数列添加到 Spark Scala 应用程序的同一数据框中的日期列 - Add Number of days column to Date Column in same dataframe for Spark Scala App 将带有时区列的日期字符串转换为 spark scala 中的时间戳 - Convert the date string with timezone column to timestamp in spark scala 使用 Spark Scala 解析字符串列以获取日期格式的数据 - Parse the String column to get the data in date format using Spark Scala 在火花数据框中,如何使用 scala 将字符串类型的日期列转换为日期类型的日期列 - In spark Data frame how to convert Date column of type string to Date column of type Date using scala 计算scala spark中的列字符串 - Count column string in scala spark 如何在 spark-scala 中将字符串列(仅包含时间而不包含日期的列)转换为 time_stamp? - How to convert a string column (column which contains only time and not date ) to time_stamp in spark-scala? 日期时间列中的Spark(scala)更改日期 - Spark (scala) change date in datetime column 使用 Scala 和 Spark 将日期列转换为年龄 - Convert Date Column to Age with Scala and Spark Spark Scala 从日期创建时间戳列 - Spark Scala creating timestamp column from date Array[Array[String]] to String in a column with Scala 和 Spark - Array[Array[String]] to String in a column with Scala and Spark
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM