如何使用 python/pyspark 在嵌套的 JSON 中添加具有空值/空白值的缺失键，其中嵌套了 dict 列表

Question

I have a JSON as below :

{"id": 1, "type": "int", "data": {"key0": "val1", "key2": "val2"}}
{"id": 2, "type": "int", "data": {"key2": "val3", "key3": "val4"}}
{"id": 3, "type": "int", "data": {"key1": "val5", "key3": "val6"}}

Now when flattening using pyspark I need to have all the columns as key0,key1,key2,key3 but when selecting columns data.key3 or any other key which is not present in either record the job fails with the error "pyspark.sql.utils.AnalysisException: 'No such struct field" Tried passing schema but the issue still persisted and tried with the withColumn approach using when but that is also failing. Have someone faced similar type of issue and fixed kindly help.

Below is the way I am reading schema:

df_landing = spark.read.format("json").option("multiline", "true").load(input_file)
print(df_landing.printSchema())

below is the result :

root 
|-- data: struct (nullable = true) 
| |-- key0: string (nullable = true) 
| |-- key2: string (nullable = true) 
|-- id: long (nullable = true) 
|-- type: string (nullable = true)

Answer 1

You should remove the .option("multiline", "true") , this is when 1 JSON record is go across multiple lines.

Your data is JSONL, each line is valid JSON and each JSON doesn't go across multiple lines.

If you remove the option, you should see the schema like this.

root
 |-- data: struct (nullable = true)
 |    |-- key0: string (nullable = true)
 |    |-- key1: string (nullable = true)
 |    |-- key2: string (nullable = true)
 |    |-- key3: string (nullable = true)
 |-- id: long (nullable = true)
 |-- type: string (nullable = true)

Then, you can use this code to expand the struct.

df_landing = df_landing.select('id', 'type', 'data.*')

# df_landing.show()
+---+----+----+----+----+----+
| id|type|key0|key1|key2|key3|
+---+----+----+----+----+----+
|  1| int|val1|null|val2|null|
|  2| int|null|null|val3|val4|
|  3| int|null|val5|null|val6|
+---+----+----+----+----+----+