如何从pyspark中的字符串中删除特定字符？

Question

I am trying to remove specific character from a string but not able to get any proper solution. Could you please help me how to do this?

I am loading the data into dataframe using pyspark. One of the column having the extra character which i want to remove.

Example:

|"\""warfarin was discontinued 3 days ago and xarelto was started when the INR was 2.7, and now the INR is 5.8, should Xarelto be continued or stopped?"|

But in result i want only :

|"warfarin was discontinued 3 days ago and xarelto was started when the INR was 2.7, and now the INR is 5.8, should Xarelto be continued or stopped?"|

I am using below code to write dataframe into file:

df.repartition(1).write.format('com.databricks.spark.csv').mode('overwrite').save(output_path, escape='\"', sep='|',header='True',nullValue=None)

Answer 1

Try the below to remove punctuation from the starting of your string

from string import punctuation
mystring = """\""warfarin was discontinued 3 days ago and xarelto was started when the INR was 2.7, and now the INR is 5.8, should Xarelto be continued or stopped?"""
print(mystring.lstrip(punctuation))

output:

'warfarin was discontinued 3 days ago and xarelto was started when the INR was 2.7, and now the INR is 5.8, should Xarelto be continued or stopped?'

Answer 2

You can use some other escape characters instead of '\\' you can change this to anything else. If you have option to save file to any other format prefer parquet (or orc) over csv.