Spark Scala - 過濾時間戳

Question

我正在嘗試使用 Spark Scala（忽略日期）過濾 2 個值之間的時間戳。 我試圖只選擇晚上 9:00:00 和晚上 11:00:00（包括 9:00:00 和 11:00:00）之間的所有記錄。 下面列出了我當前的輸入、輸出和代碼。

我的思考過程是能夠使用我的pickupWindow 列大於或小於我的值進行過濾。

有什么想法嗎？

輸入：

+----------------------+----------------------+----------+------------+------------+
|tpep_pickup_datetime  |tpep_dropoff_datetime |total_amount|pickupWindow|
+----------------------+----------------------+----------+------------+------------+
|05/18/2018 09:09:29 PM|05/18/2018 09:52:53 PM|42.8        |09:09:29    |
|05/18/2018 11:00:00 PM|05/18/2018 11:09:13 PM|23.5        |11:00:00    |
|05/18/2018 02:47:21 PM|05/18/2018 03:30:00 PM|46.62       |02:47:21    |

電流輸出：

+--------------------+---------------------+---------+------------+------------+
|tpep_pickup_datetime|tpep_dropoff_datetime|timestamp|total_amount|pickupWindow|
+--------------------+---------------------+---------+------------+------------+
+--------------------+---------------------+---------+------------+------------+

當前代碼：

stamp.withColumn("pickupWindow",date_format(to_timestamp(col("tpep_pickup_datetime"),"MM/dd/yyyy hh:mm:ss a"),"hh:mm:ss")).select("tpep_pickup_datetime","tpep_dropoff_datetime","timestamp","total_amount","pickupWindow").filter(col("pickupWindow")>="9:00:00").filter(col("pickupWindow")<="11:00:00").where($"tpep_pickup_datetime".contains("PM")).show(false)

Answer 1

嘗試使用.geq和.leq如下所示 -

scala> df.withColumn("pickupWindow",date_format(to_timestamp(col("tpep_pickup_datetime"),"MM/dd/yyyy hh:mm:ss a"),"hh:mm:ss")).filter(col("pickupWindow").geq("09:00:00") && col("pickupWindow").leq("11:00:00")).
    show()

    +----------------------+----------------------+---------+------------+------------+
    |tpep_pickup_datetime  |tpep_dropoff_datetime |timestamp|total_amount|pickupWindow|
    +----------------------+----------------------+---------+------------+------------+
    |05/18/2018 09:56:20 PM|05/18/2018 10:50:38 PM|35780    |52.87       |09:56:20    |
    |05/18/2018 10:52:49 PM|05/18/2018 11:08:47 PM|39169    |14.76       |10:52:49    |
    |05/18/2018 09:01:22 PM|05/18/2018 09:05:36 PM|32482    |6.3         |09:01:22    |
    |05/18/2018 09:00:29 PM|05/18/2018 09:05:31 PM|32429    |7.56        |09:00:29    |
    +----------------------+----------------------+---------+------------+------------+

如果你想要 AM & PM 轉換

scala> df.withColumn("pickupWindow",date_format(to_timestamp(col("tpep_pickup_datetime"),"MM/dd/yyyy hh:mm:ss a"),"HH:mm:ss")).filter(col("pickupWindow").geq("21:00:00") && col("pickupWindow").leq("23:00:00")).
    show(false)
    +----------------------+----------------------+---------+------------+------------+
    |tpep_pickup_datetime  |tpep_dropoff_datetime |timestamp|total_amount|pickupWindow|
    +----------------------+----------------------+---------+------------+------------+
    |05/18/2018 09:56:20 PM|05/18/2018 10:50:38 PM|35780    |52.87       |21:56:20    |
    |05/18/2018 10:52:49 PM|05/18/2018 11:08:47 PM|39169    |14.76       |22:52:49    |
    |05/18/2018 09:01:22 PM|05/18/2018 09:05:36 PM|32482    |6.3         |21:01:22    |
    |05/18/2018 09:00:29 PM|05/18/2018 09:05:31 PM|32429    |7.56        |21:00:29    |
    +----------------------+----------------------+---------+------------+------------+