sparksql沒有讀取多分隔符csv文件

Question

I'm trying to read multidelimter (|,||) csv file by using pyspark sql, am not able read any data from dataframe its giving 0 records

csv 文件的樣本數據

Newyork|234567|company Ltd||PIN

---

df = spark.read.option.("sep","|").option("header","true").load(csv)

我需要讀取數據，還有其他方法可以處理嗎？

Answer 1

嘗試這個-

spark.read
      .option("sep", "|")
      .option("header", "true")
      .csv(spark.read.text("<path>").as(Encoders.STRING).map(_.replaceAll("\\|\\|", "|")))