[英]Filter only particular format of date in Scala
I've a dataframe where some of the fields are having the date format of D.HH:mm:ss, D.HH:mm:ss.SSSSSSS & HH:mm:ss.SSSSSSS.我有一个数据框,其中一些字段的日期格式为 D.HH:mm:ss、D.HH:mm:ss.SSSSSSS & HH:mm:ss.SSSSSSS。 I'll need to filter only the date of type HH:mm:ss.SSSSSSS and convert this date to seconds(integer).我只需要过滤类型为 HH:mm:ss.SSSSSSS 的日期并将此日期转换为秒(整数)。
I've written below scala code that converts the date to seconds.我在下面写了将日期转换为秒的 Scala 代码。 I need help in filtering the date type of a particular format(HH:mm:ss.SSSSSSS) only and skip other formats of date in a dataframe.我需要帮助来过滤特定格式的日期类型(HH:mm:ss.SSSSSSS)并跳过数据框中的其他日期格式。 Any help would be appreciated.任何帮助,将不胜感激。
def hoursToSeconds(a: Any): Int = {
val sec = a.toString.split('.')
val fields = sec(0).split(':')
val creationSeconds = fields(0).toInt*3600 + fields(1).toInt*60 + fields(2).toInt
return creationSeconds
}
The task can be split up into two parts:任务可以分为两部分:
Create some test data:创建一些测试数据:
val df = Seq(
("one", "1.09:39:26"),
("two", "1.09:39:26.1234567"),
("three", "09:39:26.1234567")
).toDF("info", "time")
Definition of regexp and udf: regexp 和 udf 的定义:
val pattern = "\\A(\\d{1,2}):(\\d{2}):(\\d{2})\\.\\d{7}\\z".r
val toSeconds = udf{in: String => {
val pattern(hour, minute, second) = in
hour.toInt * 60 * 60 + minute.toInt * 60 + second.toInt
}}
The actual code:实际代码:
df
.filter('time rlike pattern.regex)
.select('info, 'time, toSeconds('time).as("seconds"))
.show
prints印刷
+-----+----------------+-------+
| info| time|seconds|
+-----+----------------+-------+
|three|09:39:26.1234567| 34766|
+-----+----------------+-------+
If the lines that do not have the correct format should be kept, the udf can be changed slightly and the filter has to be removed:如果应保留格式不正确的行,则可以稍微更改 udf 并删除过滤器:
val pattern = "\\A(\\d{1,2}):(\\d{2}):(\\d{2})\\.\\d{7}\\z".r
val toSeconds = udf{in: String => {
in match {
case pattern(hour, minute, second)=> hour.toInt * 60 * 60 + minute.toInt * 60 + second.toInt
case _ => 0
}
}}
df
.select('info, 'time, toSeconds('time).as("seconds"))
.show
prints印刷
+-----+------------------+-------+
| info| time|seconds|
+-----+------------------+-------+
| one| 1.09:39:26| 0|
| two|1.09:39:26.1234567| 0|
|three| 09:39:26.1234567| 34766|
+-----+------------------+-------+
You can try matching using a regex with extractors like so:您可以尝试使用正则表达式与提取器进行匹配,如下所示:
val dateRegex = """(\d{2}):(\d{2}):(\d{2}).(\d{7})""".r
val D_HH_mm_ss = "1.12:12:12"
val D_HH_mm_ss_SSSSSSS = "1.12:12:12.1234567"
val HH_mm_ss_SSSSSSS = "12:12:12.1234567"
val dates = List(HH_mm_ss_SSSSSSS, D_HH_mm_ss_SSSSSSS, D_HH_mm_ss)
dates.foreach {
_ match {
case dateRegex(hh, mm, ss, sssssssss) => println(s"Yay! $hh-$mm-$ss")
case _ => println("Nay :(")
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.