[英]Pyspark: Create Schema from Json Schema involving Array columns
我已經在json文件中為df定義了我的架構,如下所示:
{
"table1":{
"fields":[
{"metadata":{}, "name":"first_name", "type":"string", "nullable":false},
{"metadata":{}, "name":"last_name", "type":"string", "nullable":false},
{"metadata":{}, "name":"subjects", "type":"array","items":{"type":["string", "string"]}, "nullable":false},
{"metadata":{}, "name":"marks", "type":"array","items":{"type":["integer", "integer"]}, "nullable":false},
{"metadata":{}, "name":"dept", "type":"string", "nullable":false}
]
}
}
EG JSON數據:
{
"table1": [
{
"first_name":"john",
"last_name":"doe",
"subjects":["maths","science"],
"marks":[90,67],
"dept":"abc"
},
{
"first_name":"dan",
"last_name":"steyn",
"subjects":["maths","science"],
"marks":[90,67],
"dept":"abc"
},
{
"first_name":"rose",
"last_name":"wayne",
"subjects":["maths","science"],
"marks":[90,67],
"dept":"abc"
},
{
"first_name":"nat",
"last_name":"lee",
"subjects":["maths","science"],
"marks":[90,67],
"dept":"abc"
},
{
"first_name":"jim",
"last_name":"lim",
"subjects":["maths","science"],
"marks":[90,67],
"dept":"abc"
}
]
}
我想從這個json文件創建等效的spark模式。 下面是我的代碼:( 參考: 從json模式表示形式創建spark數據框架模式 )
with open(schemaFile) as s:
schema = json.load(s)["table1"]
source_schema = StructType.fromJson(schema)
如果我沒有任何數組列,則上面的代碼可以正常工作。 但是如果我的架構中有數組列,則會引發以下錯誤。
“無法解析數據類型:數組”(“無法解析數據類型:%s” json_value)
在您的情況下,數組的表示存在問題。 正確的語法是:
{ "metadata": {}, "name": "marks", "nullable": true, "type": {"containsNull": true, "elementType": "long", "type": "array" } }
。
為了從json檢索模式,您可以編寫下一個pyspark代碼段:
jsonData = """{
"table1": [{
"first_name": "john",
"last_name": "doe",
"subjects": ["maths", "science"],
"marks": [90, 67],
"dept": "abc"
},
{
"first_name": "dan",
"last_name": "steyn",
"subjects": ["maths", "science"],
"marks": [90, 67],
"dept": "abc"
},
{
"first_name": "rose",
"last_name": "wayne",
"subjects": ["maths", "science"],
"marks": [90, 67],
"dept": "abc"
},
{
"first_name": "nat",
"last_name": "lee",
"subjects": ["maths", "science"],
"marks": [90, 67],
"dept": "abc"
},
{
"first_name": "jim",
"last_name": "lim",
"subjects": ["maths", "science"],
"marks": [90, 67],
"dept": "abc"
}
]
}"""
df = spark.read.json(sc.parallelize([jsonData]))
df.schema.json()
這應該輸出:
{
"fields": [{
"metadata": {},
"name": "table1",
"nullable": true,
"type": {
"containsNull": true,
"elementType": {
"fields": [{
"metadata": {},
"name": "dept",
"nullable": true,
"type": "string"
}, {
"metadata": {},
"name": "first_name",
"nullable": true,
"type": "string"
}, {
"metadata": {},
"name": "last_name",
"nullable": true,
"type": "string"
}, {
"metadata": {},
"name": "marks",
"nullable": true,
"type": {
"containsNull": true,
"elementType": "long",
"type": "array"
}
}, {
"metadata": {},
"name": "subjects",
"nullable": true,
"type": {
"containsNull": true,
"elementType": "string",
"type": "array"
}
}],
"type": "struct"
},
"type": "array"
}
}],
"type": "struct"
}
另外,您可以使用df.schema.simpleString()
這將返回相對簡單的架構格式:
struct<table1:array<struct<dept:string,first_name:string,last_name:string,marks:array<bigint>,subjects:array<string>>>>
最后,您可以將上面的架構存儲到文件中,並在以后使用以下方式加載它:
import json
new_schema = StructType.fromJson(json.loads(schema_json))
如您所願。 請記住 ,您可以為任何json數據動態地實現所描述的過程。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.