Pyspark - withColumn + when with variable give "Method or([class java.lang.Boolean]) does not exist"

Question

我需要根据其他列之一和变量值（此处表示为otherThing ）向数据框添加一列，见下文：

otherThing = "test"
dataDF = spark.createDataFrame([(66, "a", "4"), 
                                (67, "a", "0"), 
                                (70, "b", "4"), 
                                (71, "d", "4")],
                                ("id", "code", "amt"))
#this works fine
dataDF.withColumn("new_column", when((dataDF["id"] <= 70), "A").otherwise("B")).display() 
#this gives me error
dataDF.withColumn("new_column", when((dataDF["id"] <= 70) | (otherThing == ""), "A").otherwise("B")).display()

这将返回以下错误：Method or([class java.lang.Boolean]) does not exist 在示例otherThing是常量，但在实际场景中它可以有不同的值

Answer 1

问题是由于变量缺少lit

https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.functions.lit.html https://sparkbyexamples.com/pysplit-add-park/pyspark -不变/

工作代码：

import pyspark.sql.functions as F
otherThing = ""
dataDF = spark.createDataFrame([(66, "a", "4"), 
                                (67, "a", "0"), 
                                (70, "b", "4"), 
                                (71, "d", "4")],
                                ("id", "code", "amt"))
dataDF.withColumn("new_column", when((dataDF["id"] <= 70) | F.lit(otherThing == ""), "A").otherwise("B")).display()