从 pyspark 数据框中删除重复的标点符号

Question

I need to remove the repeated punctuations and keep the last occurrence only.

For example: !!!! -> !
             !!$$ -> !$

I have a dataset that looks like below

temp = spark.createDataFrame([
    (0, "This is Spark!!!!"),
    (1, "I wish Java could use case classes!!##"),
    (2, "Data science is  cool#$@!"),
    (3, "Machine!!$$")
], ["id", "words"])

+---+--------------------------------------+
|id |words                                 |
+---+--------------------------------------+
|0  |This is Spark!!!!                     |
|1  |I wish Java could use case classes!!##|
|2  |Data science is  cool#$@!             |
|3  |Machine!!$$                             |
+---+--------------------------------------+

I tried regex to remove specific punctuations and that is below

df2 = temp.select(
    [F.regexp_replace(col, r',|\.|&|\\|\||-|_', '').alias(col) for col in temp.columns]
)

but the above is not working. Can anyone tell how to achieve this in pyspark?

Below is the desired output.

    id  words
0   0   This is Spark!
1   1   I wish Java could use case classes!#
2   2   Data science is cool#$@!
3   3   Machine!$

Answer 1

I am doing the solution in Scala since I do not have the environment for Python, but it is almost the same;

First, I declared an array of symbols you want to replace:

val puncs = Array("!", "@", "#", "$", "%", "^", "&")

then, I loop through these and modify the dataset, as:

puncs.foreach(punc => {
  ds1 = ds1
    .withColumn("words",regexp_replace(col("words"), s"(\\${punc}+)", punc))
})

The final result:

+---+------------------------------------+
|id |words                               |
+---+------------------------------------+
|0  |This is Spark!                      |
|1  |I wish Java could use case classes!#|
+---+------------------------------------+

I can not think of a better solution for now because the replacing has to be the same character.

Answer 2

You can use this regex.

df2 = temp.select('id',
    F.regexp_replace('words', r'([!$#])\1+', '$1').alias('words'))

Add more characters between [] for your needs.