在 pyspark 中使用独立条件的时间和其他方式

Question

I am trying to make a new column based on list of items. cond is a separate independent list outside dataframe "ABC". A new column needs to be created based on few conditions, please see a dummy condition below:-

Rules = ['2'. '5', '18', '33', '47']

ABC = ABC.withColumn("XYZ", when ('2' in cond), col("sqr_4231"))\
                                   .when(('3' in cond), col("sqr_4999"))

but this is giving error - have tried lit('2') and lit('3') but still giving error.

Answer 1

To verify if a value contains in an array you need to use:

lit('value').isin(array_value)

aggregate = aggregate.withColumn("ENR", when(lit('2').isin(Rules), col("sqr_4231"))\
                                   .when(lit('3').isin(Rules), col("sqr_4999"))