尝试使用具有多种条件的 pyspark 创建新列

Question

I have a dataset and in that dataset i have X and Y column

• X.dtype : timestamp

• Y.dtype: string

i want to make a Z column which has count value of unique Y column values. But it has to count before than the X time column for each row.

example dataset :

X	Y
2021-09-08	number1
2021-09-09	number2
2021-09-10	number2
2021-09-11	number3
2021-09-12	number2
2021-09-13	number2
2021-09-14	number3

example result dataset :

X	Y	Z
2021-09-08	number1	0
2021-09-09	number2	0
2021-09-10	number2	1
2021-09-11	number3	0
2021-09-12	number2	2
2021-09-13	number2	3
2021-09-14	number3	1

Note : The X column is not sorted on original dataset and i dont want to sort X column.

Answer 1

you can do that with a row_number :

from pyspark.sql import functions as F, Window

df.withColumn(
    "z",
    F.row_number().over(Window.partitionBy("y").orderBy("x")) - 1
).show()