如何从 python 的 spark df 中的字符串中删除特定字符（“）？

Question

I have the following spark data frame

   description

"""Piotr is ""running"
"""Leo is ""running"
"""Marta is ""running"

I want the following sparkdf

   description

"Piotr is ""running
"""Leo is ""running"
"""Marta is ""running"

Logic is if Piotr is in the string the first two " and last " will be removed.

Answer 1

You can apply a conditional substr , for the conditional clause you can use when .

from pyspark.sql import functions as F

data = [('"""Piotr is ""running"', ), ('"""Leo is ""running"',), ('"""Marta is ""running"', )]


df = spark.createDataFrame(data, ("description", ))

df.withColumn("description", \
              F.when(F.col("description").contains("Piotr"), \
                     F.col("description").substr(F.lit(3), (F.length(F.col("description")) - F.lit(3))))\
              .otherwise(F.col("description")))\
  .show(200, False)

Output

+----------------------+
|description           |
+----------------------+
|"Piotr is ""running   |
|"""Leo is ""running"  |
|"""Marta is ""running"|
+----------------------+