sparksql没有读取多分隔符csv文件

Question

I'm trying to read multidelimter (|,||) csv file by using pyspark sql, am not able read any data from dataframe its giving 0 records

sample data of csv file

Newyork|234567|company Ltd||PIN

---

df = spark.read.option.("sep","|").option("header","true").load(csv)

I need to read the data, is there any other way to handle this?

Answer 1

Try this-

spark.read
      .option("sep", "|")
      .option("header", "true")
      .csv(spark.read.text("<path>").as(Encoders.STRING).map(_.replaceAll("\\|\\|", "|")))