pyspark 中的动态条件生成器

Question

I have developed modules according to the business requirements. Now, what I need is a dynamic condition generator or query generator. So, for example, consider the below case:

B1 = spark.sql("select * from xyz where ABC <> DEF and CONDITIONS1 or CONDITIONS2 or CONDITIONS3")  

I have many and different business logic as above. So, in this case, I identified a common pattern like "select *" and I created a property file with .properties extension and I read that variable inside the .py file:

Key-Value pair in properties file:

selectVar = "Select * from "   

But, now what demand is needed to create a way or interface where user can modify the conditions as per their need. Like they can add multiple conditions or remove a condition. In the above example they can remove CONDITIONS2 or add CONDITIONS4 or they can change the CONDITIONS3 as per their need. It should be dynamic. No, coding part should be required on client-side. They just want to pass conditions and it should be substituted in the query and it should execute accordingly. So, how can I do it in pyspark. I tried searching for available tools in this case but, no luck. Can anyone help me with the approach?

Answer 1

It is non-trivial to write a generic interface that parses every type of expression. However for the specific case of multiple filter expressions you can do something like this:

def customExprEval(df: DataFrame, expr: String*): DataFrame = {
    expr.foldLeft(df){(d, i) => d.where(i)}
}

You can now call this function with variable number of conditional expressions:

val B1 = spark.sql("select * from xyz")
val B2 = customExprEval(B1,  "ABC <> DEF", "CONDITIONS1 or CONDITIONS2 or CONDITIONS3")  
val B3 = customExprEval(B1, "ABC <> DEF")