Json使用Spark分析包含包含Struct的Struct的数组

Question

my json: (stored in minijson.json)

{
  "arr": [
    {
      "st1": {},
      "st2": {
        "a": {}
      },
      "val": 0.0,
      "x": "1"
    }
  ]
}

i am using spark version 2.1.0 to read json.

read code:

minidf = spark.read.json("minijson.json")
minidf.printSchema()

output:

root
 |-- arr: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- val: double (nullable = true)
 |    |    |-- x: string (nullable = true)

I do not understand why it is not able to detect st1 , st2 and a fields which are present in json.

Please help to solve this problem.

Answer 1

Spark does not creates the Column if the key has empty key as "abc": {} It must have at-least a key and value to create a column.

Below is a simple example by adding a key and value

{
"arr": [
  {
    "st1": {
    "name": ""
  },
    "st2": {
      "a": {
        "abc": ""
      }
    },
    "val": 0.0,
    "x": "1"}
  ]
}

Wich will have a schema as

root
 |-- st1: struct (nullable = true)
 |    |-- name: string (nullable = true)
 |-- st2: struct (nullable = true)
 |    |-- a: struct (nullable = true)
 |    |    |-- abc: string (nullable = true)
 |-- val: double (nullable = true)
 |-- x: string (nullable = true)