I have sample records as looks below and I want to convert the Time numeric column (Which represents milliseconds value) into the timestamp column.
Value Time
0.00346 0
0.01584 0.002
0.00621 0.004
.....
.....
.....
0.00204 0.998
0.00617 1
0.00204 1.002
Let's take HH:mm:ss.SSS is the timestamp format and the output should be as below
Value Time timestamp
0.00346 0 00:00:00.000
0.01584 0.002 00:00:00.002
0.00621 0.004 00:00:00.004
..... ........
..... ........
..... ........
0.00204 0.998 00:00:00.998
0.00617 1 00:00:01.000
0.00204 1.002 00:00:01.002
You can cast the column to timestamp type and then use date_format
to convert to the desired format:
import pyspark.sql.functions as F
df2 = df.withColumn('timestamp', F.date_format(F.col('Time').cast('timestamp'), 'HH:mm:ss.SSS'))
df2.show()
+-------+-----+------------+
| Value| Time| timestamp|
+-------+-----+------------+
|0.00346| 0.0|00:00:00.000|
|0.01584|0.002|00:00:00.002|
|0.00621|0.004|00:00:00.004|
|0.00204|0.998|00:00:00.998|
|0.00617| 1.0|00:00:01.000|
|0.00204|1.002|00:00:01.002|
+-------+-----+------------+
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.