解析存储为字符串 Spark 的结构列数组

Question

I have the below Array of Structs stored as a string column in my input file. The input file cannot be modified because it is sent from an upstream team.

[{"col1":"B078Z655KG","col2":2,"col3":351,"col4":"kindle_edition","col6":1,"transaction_info":[]},{"col1":"0736973540","col2":1,"col3":14,"col4":"paperback","col5":1,"col6":[]}]

My Code:

val ds = df.select(col("faceout_features"))

val schema = StructType(Seq(StructField(“col1”, CatalystSqlParser.parseDataType("string")), StructField(“col2”, CatalystSqlParser.parseDataType("string"))))

val dfFromCSVJSON = df.select(col(“col1”), from_json(col("faceout_features"),schema).as("jsonData")) .select(“col1”,”jsonData.*")

I am getting the output columns as null when trying to parse as json and the input record is processed as a corrupt record instead. Any help in debugging the issues with the file format is appreciated.

Error:

Found at least one malformed records (sample: "[{""col1"":""B078Z655KG"")

Answer 1

The error message shows you one of the records that cannot be parsed. The general approach to debugging this would be to look at that record and try to parse it outside for spark. You could paste it into a text editor that supports JSON and see if it highlights any issues. Once you find the issue, be sure to check if any other rows have the same issue.

For the specific error message shown in the question, we can already tell that it isn't valid JSON because of the double quotes. In other words, ""B078Z655KG"" should be "B078Z655KG" .