正则表达式模式从 pyspark 中的单词中删除数值

Question

I am working on pyspark dataframe and I have a column of words (array<string> type) . What should be the regex pattern to remove numeric values and numeric values from words?

+---+----------------------------------------------+
|id |    words                                     |
+---+----------------------------------------------+
|564|[fhbgtrj5, 345gjhg, ghth578ghu, 5897, fhrfu44]|
+---+----------------------------------------------+

expected output:

+---+----------------------------------------------+
|id |words                                         |
+---+----------------------------------------------+
|564|               [fhbgtrj, gjhg, ghthghu, fhrfu]|
+---+----------------------------------------------+

Please help.

Answer 1

You can use transform together with regexp_replace to remove the numbers, and use array_remove to remove the empty entries (which comes from those entries which only consist of numbers).

df2 = df.withColumn(
    'words', 
    F.expr("array_remove(transform(words, x -> regexp_replace(x, '[0-9]', '')), '') as words")
)

df2.show(truncate=False)
+---+-------------------------------+
|id |words                          |
+---+-------------------------------+
|564|[fhbgtrj, gjhg, ghthghu, fhrfu]|
+---+-------------------------------+