如何将 pyspark 数据帧中的时间戳列值减少 1 毫秒

Question

I have pyspark data-frame which has timestamp column, I want to reduce timestamp by 1 ms. Is there some in-built function available in spark for handling such scenario?

for example value for timestamp column: timestamp value: 2020-07-13 17:29:36

Answer 1

By using double type, you can do this.

import pyspark.sql.functions as f

df = spark.createDataFrame([(1, '2020-07-13 17:29:36')], ['id', 'time'])

df.withColumn('time', f.to_timestamp('time', 'yyyy-MM-dd HH:mm:ss')) \
  .withColumn('timediff', (f.col('time').cast('double') - f.lit(0.001)).cast('timestamp')) \
  .show(10, False)

+---+-------------------+-----------------------+
|id |time               |timediff               |
+---+-------------------+-----------------------+
|1  |2020-07-13 17:29:36|2020-07-13 17:29:35.999|
+---+-------------------+-----------------------+

Answer 2

You can use pyspark.sql.functions.expr to subtract INTERVAL 1 milliseconds

from pyspark.sql.functions import expr

df = spark.createDataFrame([('2020-07-13 17:29:36',)], ['time'])
df = df.withColumn('time2', expr("time - INTERVAL 1 milliseconds"))
df.show(truncate=False)
#+-------------------+-----------------------+
#|time               |time2                  |
#+-------------------+-----------------------+
#|2020-07-13 17:29:36|2020-07-13 17:29:35.999|
#+-------------------+-----------------------+

Even if time is a string of this format, Spark will make an implicit conversion for you.

df.printSchema()
#root
# |-- time: string (nullable = true)
# |-- time2: string (nullable = true)

Answer 3

You could also use INTERVAL with expr .

import pyspark.sql.functions as F

df = spark.createDataFrame(
    [
        (1, '2020-07-13 17:29:36')
    ], 
    [
        'id', 'time'
    ]
)

df.withColumn(
    'time', 
    F.col('time').cast('timestamp')
).withColumn(
    'timediff', 
    (
        F.col('time') -  F.expr('INTERVAL 1 milliseconds')
    ).cast('timestamp') 
).show(truncate=False)