Spark 与异常编码的 CSV 文件不一致

Question

语境：

• 作为数据管道的一部分，我正在处理一些平面 CSV 文件
• 这些文件具有不寻常的编码和转义规则
• 我的意图是对这些进行预处理并转换为镶木地板以进行后续管道步骤

MCVE：

spark = SparkSession.builder.appName("...").getOrCreate()

min_schema = StructType(
    [
        StructField("dummy_col", StringType(), True),
        StructField("record_id", IntegerType(), nullable=False),
        StructField("dummy_after", StringType(), nullable=False),
    ]
)


df = (
    spark.read.option("mode", "FAILFAST")
    .option("quote", '"')
    .option("escape", '"')
    .option("inferSchema", "false")
    .option("multiline", "true")
    .option("ignoreLeadingWhiteSpace", "true")
    .option("ignoreTrailingWhiteSpace", "true")
    .schema(min_schema)
    .csv(f'min_repro.csv', header=True)
)

dummy_col,record_id,dummy_after
"",1,", Unusual value with comma included"
B,2,"Unusual value with escaped quote and comma ""like, this"

CSV 解析良好：

df.collect()

[Row(dummy_col=None, record_id=1, dummy_after=', Unusual value with comma included'),
Row(dummy_col='B', record_id=2, dummy_after='Unusual value with escaped quote and comma "like, this')]

然而，同一 DF 上的微不足道的 Spark 代码失败并出现模糊错误：

if df.count() != df.select('record_id').distinct().count():
    pass

Py4JJavaError: An error occurred while calling o357.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 17.0 failed 1 times, most recent failure: Lost task 0.0 in stage 17.0 (TID 13, localhost, executor driver): org.apache.spark.SparkException: Malformed records are detected in record parsing. Parse Mode: FAILFAST.
...
Caused by: java.lang.NumberFormatException: For input string: "Unusual value with comma included""
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

我不明白同一个 DF 上的.collect()如何提供正确的行，但是同一个 DF 上的任何查询都失败了。

Answer 1

正确的忽略方式，在 Data 是

1. 用双引号将数据括起来。
2. 使用选项“escapeQuotes”、“true”

df = ( spark.read.option("mode", "FAILFAST") .option("escapeQuotes", "true") .option("inferSchema", "false") .option("multiline", "true") .option("ignoreLeadingWhiteSpace", "true") .option("ignoreTrailingWhiteSpace", "true") .schema(min_schema) .csv(f'C:/Users/pc/Desktop/sample2.csv', header=True) )
------------------------------------------------------------------------

    >>> df.select('dummy_after').show(truncate=False)
    +-----------------------------------+
    |dummy_after                        |
    +-----------------------------------+
    |, Unusual value with comma included|
    +-----------------------------------+

    >>> if df.count() != df.select('record_id').distinct().count():
    ...    pass