在 pyspark dataframe 中用双引号替换单引号

Question

from the below code I am writing a dataframe to csv file.

As my dataframe contains "" for None , I have added replace("", None) because Null values are supposed to be represented as None instead of "" (double quotes)

newDf.coalesce(1).replace("", None).replace("'", "\"").write.format('csv').option('nullValue', None).option('header', 'true').option('delimiter', '|').mode('overwrite').save(destination_csv)

I tried adding .replace("'", "\""). but it doesn't work

the data also contains data with single quotes

eg:

Survey No. 123, 'Anjanadhri Godowns', CityName

I need to replace the single quotes from the dataframe and replace it with double-quotes.

How can it be achieved?

Answer 1

You can use regexp_replace to replace single quotes with double quotes in all columns before writing the output:

import pyspark.sql.functions as F

df2 = df.select([F.regexp_replace(c, "'", '"').alias(c) for c in df.columns])

# then write output
# df2.coalesce(1).write(...)

Answer 2

Using translate

from pyspark.sql.functions import *

data_list = [(1, "'Name 1'"), (2, "'Name 2' and 'Something'")]
df = spark.createDataFrame(data = data_list, schema = ["ID", "my_col"])
# +---+--------------------+
# | ID|              my_col|
# +---+--------------------+
# |  1|            'Name 1'|
# |  2|'Name 2' and 'Som...|
# +---+--------------------+

df.withColumn('my_col', translate('my_col', "'", '"')).show()
# +---+--------------------+
# | ID|              my_col|
# +---+--------------------+
# |  1|            "Name 1"|
# |  2|"Name 2" and "Som...|
# +---+--------------------+

This will replace all occurrences of the single quote character with a double quotation mark in the column my_col .