pyspark向前填充具有特定值的时间戳列（1秒）

Question

in the past I've asked this question regarding python pandas library: pandas forward fill Time Stamp columns with specific value (1 second)

But now I will be doing huge data processing in pyspark so would ask for another solution in pyspark:

I have a spark DataFrame:

df = spark.createDataFrame([Row(a=1, b='2018-09-26 04:38:32.544', c='11', d='foo'),
                            Row(a=2, b='', c='22', d='bar'),
                            Row(a=3, b='', c='33', d='foo'),
                            Row(a=4, b='', c='44', d='bar'),
                            Row(a=5, b='2018-09-26 04:58:32.544', c='55', d='foo'),
                            Row(a=6, b='', c='66', d='bar')])
df.show(truncate=False)

|a  |b                      |c  |d  |
+---+-----------------------+---+---+
|1  |2018-09-26 04:38:32.544|11 |foo|
|2  |                       |22 |bar|
|3  |                       |33 |foo|
|4  |                       |44 |bar|
|5  |2018-09-26 04:58:32.544|55 |foo|
|6  |                       |66 |bar|
+---+-----------------------+---+---+

And I would like to add consecutively 1 second to each NaT from the previous available:

|a  |b                      |c  |d  |
+---+-----------------------+---+---+
|1  |2018-09-26 04:38:32.544|11 |foo|
|2  |2018-09-26 04:39:32.544|22 |bar|
|3  |2018-09-26 04:40:32.544|33 |foo|
|4  |2018-09-26 04:41:32.544|44 |bar|
|5  |2018-09-26 04:58:32.544|55 |foo|
|6  |2018-09-26 04:59:32.544|66 |bar|
+---+-----------------------+---+---+

I've read that udf's should be avoided as they will slow down the processing on millions of rows. Thanks for help!

UPDATE 2019/09/09

After talking with @cronoik below there is a case study where one column d is for partitioning the dataset:

df2 = spark.createDataFrame([Row(a=1, b='2018-09-26 04:38:32.544', c='11', d='foo'),
                             Row(a=2, b='', c='22', d='foo'),
                             Row(a=3, b='', c='33', d='foo'),
                             Row(a=4, b='', c='44', d='foo'),
                             Row(a=5, b='2018-09-26 04:58:32.544', c='55', d='foo'),
                             Row(a=6, b='', c='66', d='foo'),
                             Row(a=1, b='2018-09-28 05:40:32.544', c='111', d='bar'),
                             Row(a=2, b='', c='222', d='bar'),
                             Row(a=3, b='2018-09-28 05:49:32.544', c='333', d='bar'),
                             Row(a=4, b='', c='444', d='bar'),
                             Row(a=5, b='2018-09-28 05:55:32.544', c='555', d='bar'),
                             Row(a=6, b='', c='666', d='bar')]

|a  |b                      |c  |d  |
+---+-----------------------+---+---+
|1  |2018-09-26 04:38:32.544|11 |foo|
|2  |                       |22 |foo|
|3  |                       |33 |foo|
|4  |                       |44 |foo|
|5  |2018-09-26 04:58:32.544|55 |foo|
|6  |                       |66 |foo|
|1  |2018-09-28 05:40:32.544|111|bar|
|2  |                       |222|bar|
|3  |2018-09-28 05:49:32.544|333|bar|
|4  |                       |444|bar|
|5  |2018-09-28 05:55:32.544|555|bar|
|6  |                       |666|bar|
+---+-----------------------+---+---+

Answer 1

This is probably not the most efficient solution as we can't partition the dataframe according to your requirements. That means that all the data is loaded to a single partition and ordered there. Maybe someone can come up with a better solution.

The code below use a lag window function which returns the value of the previous row. We apply this only when the current value for b is null otherwise we keep the current value. When the current value is null, we add one second to the value of the previous row. We have to do this several times as a row which contains null in the b column and the previous row which also contains null in the 'b' column will get null returned from lag (ie lag is not applied consecutively and therefore we have to do this by ourself).

import pyspark.sql.functions as F
from pyspark.sql import Row
from pyspark.sql import Window

df = spark.createDataFrame([Row(a=1, b='2018-09-26 04:38:32.544', c='11', d='foo'),
                            Row(a=2, b='', c='22', d='bar'),
                            Row(a=3, b='', c='33', d='foo'),
                            Row(a=4, b='', c='44', d='bar'),
                            Row(a=5, b='2018-09-26 04:58:32.544', c='55', d='foo'),
                            Row(a=6, b='', c='66', d='bar')])

df = df.withColumn('a',  df.a.cast("int"))
df = df.withColumn('b',  df.b.cast("timestamp"))

w = Window.orderBy('a')

while df.filter(df.b.isNull()).count() != 0:
    df = df.withColumn('b', F.when(df.b.isNotNull(), df.b).otherwise(F.lag('b').over(w)  + F.expr('INTERVAL 1 SECONDS')))

df.show(truncate=False)

Output:

+---+-----------------------+---+---+ 
| a |                     b | c | d | 
+---+-----------------------+---+---+ 
| 1 |2018-09-26 04:38:32.544|11 |foo| 
| 2 |2018-09-26 04:38:33.544|22 |bar| 
| 3 |2018-09-26 04:38:34.544|33 |foo| 
| 4 |2018-09-26 04:38:35.544|44 |bar| 
| 5 |2018-09-26 04:58:32.544|55 |foo| 
| 6 |2018-09-26 04:58:33.544|66 |bar| 
+---+-----------------------+---+---+

UPDATE 2019/09/09
In your edit you said that the column d can be used as partition key. There is not much you have to change for partitioning. Just replace w = Window.orderBy('a') with w = Window.partitionBy('d').orderBy('a') and spark will generate a partition for each distinct value of d and execute the code in parallel for each partition.