如何在 PySpark 中為嵌套的 JSON 列創建模式？

Question

我有一個包含多列的鑲木地板文件，其中我有 2 列是 JSON/Struct，但它們的類型是字符串。 可以存在任意數量的 array_elements。

{
  "addressline": [

    {
      "array_element": "F748DK’8U1P9’2ZLKXE"
    },
    {
      "array_element": "’O’P0BQ04M-"
    },
    {
      "array_element": "’fvrvrWEM-"
    }

  ],
  "telephone": [
    {
      "array_element": {
        "locationtype": "8.PLT",
        "countrycode": null,
        "phonenumber": "000000000",
        "phonetechtype": "1.PTT",
        "countryaccesscode": null,
        "phoneremark": null
      }
    }
  ]
}

如何創建一個模式來處理 PySpark 中的這些列？

Answer 1

將您提供的示例視為字符串，我創建了這個數據框：

from pyspark.sql import functions as F, types as T
df = spark.createDataFrame([('{"addressline":[{"array_element":"F748DK’8U1P9’2ZLKXE"},{"array_element":"’O’P0BQ04M-"},{"array_element":"’fvrvrWEM-"}],"telephone":[{"array_element":{"locationtype":"8.PLT","countrycode":null,"phonenumber":"000000000","phonetechtype":"1.PTT","countryaccesscode":null,"phoneremark":null}}]}',)], ['c1'])

這是要應用於此列的架構：

schema = T.StructType([
    T.StructField('addressline', T.ArrayType(T.StructType([
        T.StructField('array_element', T.StringType())
    ]))),
    T.StructField('telephone', T.ArrayType(T.StructType([
        T.StructField('array_element', T.StructType([
            T.StructField('locationtype', T.StringType()),
            T.StructField('countrycode', T.StringType()),
            T.StructField('phonenumber', T.StringType()),
            T.StructField('phonetechtype', T.StringType()),
            T.StructField('countryaccesscode', T.StringType()),
            T.StructField('phoneremark', T.StringType()),
        ]))
    ])))
])

將架構提供給from_json函數的結果：

df = df.withColumn('c1', F.from_json('c1', schema))

df.show()
# +-------------------------------------------------------------------------------------------------------+
# |c1                                                                                                     |
# +-------------------------------------------------------------------------------------------------------+
# |{[{F748DK’8U1P9’2ZLKXE}, {’O’P0BQ04M-}, {’fvrvrWEM-}], [{{8.PLT, null, 000000000, 1.PTT, null, null}}]}|
# +-------------------------------------------------------------------------------------------------------+

df.printSchema()
# root
#  |-- c1: struct (nullable = true)
#  |    |-- addressline: array (nullable = true)
#  |    |    |-- element: struct (containsNull = true)
#  |    |    |    |-- array_element: string (nullable = true)
#  |    |-- telephone: array (nullable = true)
#  |    |    |-- element: struct (containsNull = true)
#  |    |    |    |-- array_element: struct (nullable = true)
#  |    |    |    |    |-- locationtype: string (nullable = true)
#  |    |    |    |    |-- countrycode: string (nullable = true)
#  |    |    |    |    |-- phonenumber: string (nullable = true)
#  |    |    |    |    |-- phonetechtype: string (nullable = true)
#  |    |    |    |    |-- countryaccesscode: string (nullable = true)
#  |    |    |    |    |-- phoneremark: string (nullable = true)