[英]Apply multiple LIKE filters for a Dataframe
Is there a better implementation in Spark SQL by regexp_like for the following regexp_like 在 Spark SQL 中是否有更好的实现如下
SELECT col1,
col2
FROM fact_table
WHERE UPPER((TRIM(NAME))) LIKE 'VAL1 %'
OR UPPER((TRIM(NAME))) LIKE '% VAL1 %'
OR UPPER((TRIM(NAME))) ='VAL1'
OR UPPER((TRIM(NAME))) LIKE 'VAL1-%'
OR UPPER((TRIM(NAME))) LIKE 'VAL2 %'
OR UPPER((TRIM(NAME))) LIKE '% VAL2 %'
OR UPPER((TRIM(NAME)))='VAL2'
OR UPPER((TRIM(NAME))) LIKE 'VAL2-%'
In raw SQL no.在原始 SQL 编号。 With Dataframe API you can use higher-level functions to combine the filter expressions:使用 Dataframe API 您可以使用更高级别的函数来组合过滤器表达式:
val filterExpr = Seq("%VAL1 %", "% VAL1%", "VAL1" /* ... */)
.map(x => upper(trim($"name")).like(x))
.reduce(_ || _)
val df = Seq("MYVAL1 X", "FOO", "val1", "AVAL1").toDF("name").filter(filterExpr).show
+--------+
| name|
+--------+
|MYVAL1 X|
| val1|
+--------+
Alternatively you could also write a UDF.或者,您也可以编写 UDF。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.