如何使用条件更新 pyspark 中的 Dataframe 列？

Question

I am trying to update one dataframe column based on some other column conditions. I have two columns in my dataframe DATE_JOINING AND BONUS

df=spark.sparkContext.parallelize([(23-08-2021,''),(12-11-2009,''),(24-09-2013,'')

df = spark.createDataFrame(rdd, schema=['DATE_JOINING', 'BONUS'])

Basically IN SQL we can write query as

UPDATE EMPLOYEE.SALARY
SET BONUS= 'GIVE BONUS'
where DATE_JOINING < 09-01-2015

In Pyspark i am trying below code

df=df.withColumn('SALARY',when(df.DATE_JOINING <'01-09-2018',"GIVE BONUS").otherwise(''))

But it is giving me data more year than 2018 year also which i dont want and not giving any result for Column less than 2018. How can i correct this code.

Answer 1

This is because your date column is not actually of date type, but string type.
To perform operations on dates, you need to convert the column with to_date ; furthermore, you must explicit dates through the format yyyy-mm-dd .

import pyspark.sql.functions as F

df = (df
 .withColumn('DATE_JOINING', F.to_date('DATE_JOINING', format='dd-mm-yyyy'))
 .withColumn('SALARY', F.when(F.col('DATE_JOINING') < '2018-09-01', 'GIVE BONUS').otherwise(''))
)

df.show()
+------------+-----+----------+
|DATE_JOINING|BONUS|    SALARY|
+------------+-----+----------+
|  2021-01-23|     |          |
|  2009-01-12|     |GIVE BONUS|
|  2013-01-24|     |GIVE BONUS|
+------------+-----+----------+