在 pyspark 中将文件时间转换为本地时间
[英]Convert filetime to localtime in pyspark
问题描述
I have a parquet file which contains a column as given below
我有一个镶木地板文件，其中包含如下所示的列
+--------------------+
| Start_Time |
+--------------------+
| 131696565231590000 |
| 131696565251590000 |
| 131726072567140000 |
| 131726073867090000 |
| 131726076586810000 |
| 131726079656940000 |
| 131726081667130000 |
| 131726085077090000 |
| 131726085207190000 |
+--------------------+
I need to read this file and conver this column to a datetime column of the localtime.我需要阅读此文件并将此列转换为本地时间的日期时间列。 The expected output is as below (my local time is IST):预期的 output 如下（我的当地时间是 IST）：
+-------------------------+
| Start_Time |
+-------------------------+
| 2018-05-01 13:55:23.160 |
| 2018-05-01 13:55:25.160 |
| 2018-06-04 17:34:16.713 |
| 2018-06-04 17:36:26.710 |
| 2018-06-04 17:40:58.680 |
| 2018-06-04 17:46:05.693 |
| 2018-06-04 17:49:26.713 |
| 2018-06-04 17:55:07.710 |
| 2018-06-04 17:55:20.720 |
+-------------------------+
I tried converting timestamps individually as shown below and adding +5.30 to it, but is there a better solution that I can use while I read the parquet file itself or after?我尝试单独转换时间戳，如下所示并为其添加 +5.30，但是在我读取镶木地板文件本身或之后可以使用更好的解决方案吗？
datetime.fromtimestamp((131696565231590000 - 116444736000000000) // 10000000)
1 个解决方案
解决方案1
1 已采纳 2021-01-25 20:26:19
You can use from_utc_timestamp and specify the IST timezone:您可以使用from_utc_timestamp并指定IST时区：
df2 = df.selectExpr("""
from_utc_timestamp(
timestamp((Start_Time - 116444736000000000) / 10000000), 'IST'
) as Start_Time
""")
df2.show(truncate=False)
+-----------------------+
|Start_Time |
+-----------------------+
|2018-05-01 19:25:23.159|
|2018-05-01 19:25:25.159|
|2018-06-04 23:04:16.714|
|2018-06-04 23:06:26.709|
|2018-06-04 23:10:58.681|
|2018-06-04 23:16:05.694|
|2018-06-04 23:19:26.713|
|2018-06-04 23:25:07.709|
|2018-06-04 23:25:20.719|
+-----------------------+
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.