解析大型JSON文件（Python）
[英]Parse large JSON files (Python)
問題描述
我是JSON的新手，但是在進行了一些有關讀取JSON並進行解析的研究之后，我發現所有操作都是手動完成的：
"image":{
"full":"fullArt.png",
"sprite":"sprite4.png",
"group":"Art",
"x":192,
"y":0,
"w":48,
"h":48
}
這意味着，如果我有這個小的JSON塊要讀取，我將使用類似：
jfile="image.json"
file=open(jfile, 'r')
jdata=json.load(file)
data=jdata["image"]
full=data["full"]
sprite=data["sprite"]
....
但這是假設我知道文件的內部位置，並且所有圖像都是以這種方式寫入的，這可能是錯誤的。
另一個問題是，我旨在讀取具有200,000行以上JSON的文件，以便將從文件中提取的數據放入數據庫中。
所以我的問題是：如何讀取大型JSON文件並將結果存儲在數據庫中？
感謝您的閱讀。
編輯：添加了一些我想閱讀的東西
{
"type":"champion",
"format":"standAloneComplex",
"version":"6.4.2",
"data":{
"Ekko":{
"id":"Ekko",
"key":"245",
"name":"Ekko",
"title":"the Boy Who Shattered Time",
"image":{
"full":"Ekko.png",
"sprite":"champion4.png",
"group":"champion",
"x":192,
"y":0,
"w":48,
"h":48
},
"skins":[
{
"id":"245000",
"num":0,
"name":"default",
"chromas":false
},
{
"id":"245001",
"num":1,
"name":"Sandstorm Ekko",
"chromas":false
},
{
"id":"245002",
"num":2,
"name":"Academy Ekko",
"chromas":false
}
],
"lore":"A prodigy from the rough streets of Zaun, Ekko manipulates time to spin any situation to his advantage. Using his own invention, the Zero-Drive, he explores the branching possibilities of reality. As well as experimenting with multi-dimensional probability, Ekko spends his days running wild with the other lost children of Zaun. Though he revels in this freedom, when there's a threat to his friends, he'll endure anything to defend them. To the unknowing observer, Ekko accomplishes the impossible with ease, time and time again.",
"blurb":"A prodigy from the rough streets of Zaun, Ekko manipulates time to spin any situation to his advantage. Using his own invention, the Zero-Drive, he explores the branching possibilities of reality. As well as experimenting with multi-dimensional ...",
"allytips":[
"Chronobreak is a potent escape tool, but it can also be quite powerful when used offensively. Don't underestimate its damage potential.",
"If you can proc Z-Drive Resonance on an enemy champion, it's worth taking risks to do so. The bonus movement speed makes it easy to escape.",
"Phase Dive's dash is a great tool for setting up Ekko's other abilities. Use it to get double hits with Timewinder or get into position to detonate Parallel Convergence."
],
"enemytips":[
"Ekko is significantly weaker when his ultimate is down. Watch for the trail he leaves behind to determine if Chronobreak is available.",
"Ekko's stun zone takes 3 seconds to arm. Watch for the image he creates on cast and try to guess where the zone was placed.",
"The second hit of Timewinder does more damage than the first; try to avoid it."
],
"tags":[
"Assassin",
"Fighter"
],
"partype":"MP",
"info":{
"attack":5,
"defense":3,
"magic":7,
"difficulty":8
},
"stats":{
"hp":580.0,
"hpperlevel":80.0,
"mp":280.0,
"mpperlevel":50.0,
"movespeed":340.0,
"armor":27.0,
"armorperlevel":3.0,
"spellblock":32.0,
"spellblockperlevel":1.25,
"attackrange":125.0,
"hpregen":9.0,
"hpregenperlevel":0.9,
"mpregen":6.0,
"mpregenperlevel":0.8,
"crit":0.0,
"critperlevel":0.0,
"attackdamage":59.0,
"attackdamageperlevel":3.0,
"attackspeedoffset":0.0,
"attackspeedperlevel":3.3
},
"spells":[
{
"id":"EkkoQ",
"name":"Timewinder",
"description":"Ekko throws a temporal grenade that expands into a time-distortion field upon hitting an enemy champion, slowing and damaging anyone caught inside. After a delay, the grenade rewinds back to Ekko, dealing damage on its return.",
"tooltip":"Ekko throws a device that deals {{ e1 }} <span class=\"color99FF99\">(+{{ a1 }})</span> magic damage to enemies it passes through. It expands into a slowing field on the first champion hit, slowing everything inside by {{ e2 }}%. It then returns to him after a delay, dealing {{ e3 }} <span class=\"color99FF99\">(+{{ a2 }})</span> magic damage to all targets hit upon return.",
"leveltip":{
"label":[
"Outgoing Damage",
"Return Damage",
"Slow",
"Cooldown",
"Mana Cost"
],
"effect":[
"{{ e1 }} -> {{ e1NL }}",
"{{ e3 }} -> {{ e3NL }}",
"{{ e2 }}% -> {{ e2NL }}%",
"{{ cooldown }} -> {{ cooldownnNL }}",
"{{ cost }} -> {{ costnNL }}"
]
},
"maxrank":5,
"cooldown":[
9,
8.5,
8,
7.5,
7
],
"cooldownBurn":"9/8.5/8/7.5/7",
"cost":[
50,
60,
70,
80,
90
],
"costBurn":"50/60/70/80/90",
"effect":[
null,
[
60,
75,
90,
105,
120
],
[
32,
39,
46,
53,
60
],
[
60,
85,
110,
135,
160
],
[
100,
100,
100,
100,
100
]
],
"effectBurn":[
null,
"60/75/90/105/120",
"32/39/46/53/60",
"60/85/110/135/160",
"100"
],
"vars":[
{
"link":"spelldamage",
"coeff":0.1,
"key":"a1"
},
{
"link":"spelldamage",
"coeff":0.6,
"key":"a2"
}
],
"costType":"Mana",
"maxammo":"-1",
"range":[
1075,
1075,
1075,
1075,
1075
],
"rangeBurn":"1075",
"image":{
"full":"EkkoQ.png",
"sprite":"spell13.png",
"group":"spell",
"x":192,
"y":144,
"w":48,
"h":48
},
"resource":"{{ cost }} Mana"
},
{
"id":"EkkoW",
"name":"Parallel Convergence",
"description":"Ekko splits the timeline, creating an anomaly after a few seconds that slows enemies caught inside. If Ekko enters the anomaly, he gains shielding and triggers a detonation, stunning enemies by suspending them in time.",
"tooltip":"<span class=\"colorFF9900\">Passive:</span> Ekko's basic attacks deal bonus magic damage to enemies under 30% health equal to {{ e3 }}% <span class=\"color99FF99\">(+{{ a2 }}%)</span> of their missing health (max {{ e5 }} damage vs. minions and monsters).<br><br><span class=\"colorFF9900\">Active:</span> After a 3 second delay, Ekko creates a short-lived chronosphere at the target location that slows enemies who enter by 40%. If Ekko enters the sphere, he will detonate it, gaining a shield that absorbs up to {{ e4 }} <span class=\"color99FF99\">(+{{ a1 }})</span> damage for 2 seconds. Enemies caught inside are stunned for {{ e2 }} seconds.",
"leveltip":{
"label":[
"Shield Amount",
"Cooldown",
"Mana Cost"
],
"effect":[
"{{ e4 }} -> {{ e4NL }}",
"{{ cooldown }} -> {{ cooldownnNL }}",
"{{ cost }} -> {{ costnNL }}<"
]
},
"maxrank":5,
"cooldown":[
22,
20,
18,
16,
14
],
"cooldownBurn":"22/20/18/16/14",
"cost":[
50,
55,
60,
65,
70
],
"costBurn":"50/55/60/65/70",
"effect":[
null,
[
375,
375,
375,
375,
375
],
[
2.25,
2.25,
2.25,
2.25,
2.25
],
[
5,
5,
5,
5,
5
],
[
80,
100,
120,
140,
160
],
[
150,
150,
150,
150,
150
]
],
"effectBurn":[
null,
"375",
"2.25",
"5",
"80/100/120/140/160",
"150"
],
"vars":[
{
"link":"spelldamage",
"coeff":0.018,
"key":"a2"
},
{
"link":"spelldamage",
"coeff":1.5,
"key":"a1"
}
],
"costType":"Mana",
"maxammo":"-1",
"range":[
1600,
1600,
1600,
1600,
1600
],
"rangeBurn":"1600",
"image":{
"full":"EkkoW.png",
"sprite":"spell13.png",
"group":"spell",
"x":240,
"y":144,
"w":48,
"h":48
},
"resource":"{{ cost }} Mana"
},
{
"id":"EkkoE",
"name":"Phase Dive",
"description":"Ekko rolls evasively while charging up his Z-Drive. His next attack deals bonus damage and warps reality, teleporting him to his target.",
"tooltip":" Ekko dashes a short distance in the targeted direction. His next attack will deal {{ e1 }} <span class=\"color99FF99\">(+{{ a1 }})</span> bonus magic damage and teleport him to his target.",
"leveltip":{
"label":[
"Damage",
"Cooldown",
"Mana Cost"
],
"effect":[
"{{ e1 }} -> {{ e1NL }}",
"{{ cooldown }} -> {{ cooldownnNL }}",
"{{ cost }} -> {{ costnNL }}"
]
},
"maxrank":5,
"cooldown":[
13,
11.5,
10,
8.5,
7
],
"cooldownBurn":"13/11.5/10/8.5/7",
"cost":[
40,
50,
60,
70,
80
],
"costBurn":"40/50/60/70/80",
"effect":[
null,
[
50,
80,
110,
140,
170
]
],
"effectBurn":[
null,
"50/80/110/140/170"
],
"vars":[
{
"link":"spelldamage",
"coeff":0.2,
"key":"a1"
}
],
"costType":"Mana",
"maxammo":"-1",
"range":[
325,
325,
325,
325,
325
],
"rangeBurn":"325",
"image":{
"full":"EkkoE.png",
"sprite":"spell13.png",
"group":"spell",
"x":288,
"y":144,
"w":48,
"h":48
},
"resource":"{{ cost }} Mana"
},
{
"id":"EkkoR",
"name":"Chronobreak",
"description":"Ekko shatters his timeline, becoming untargetable and rewinding to a more favorable point in time. He returns to whenever he was a few seconds ago, and heals for a percentage of the damage received in that duration. Enemies near his arrival zone take massive damage.",
"tooltip":"Makes Ekko invulnerable and untargetable, and teleports him back to wherever he was 4 seconds ago. Ekko heals himself for {{ e3 }} + {{ e2 }}% <span class=\"color99FF99\">(+{{ a2 }}%)</span> of the damage taken over the last 4 seconds, and deals {{ e1 }} <span class=\"color99FF99\">(+{{ a1 }})</span> magic damage to all nearby enemies on arrival.",
"leveltip":{
"label":[
"Damage",
"Flat Heal",
"Cooldown"
],
"effect":[
"{{ e1 }} -> {{ e1NL }}",
"{{ e3 }} -> {{ e3NL }}",
"{{ cooldown }} -> {{ cooldownnNL }}"
]
},
"maxrank":3,
"cooldown":[
110,
90,
70
],
"cooldownBurn":"110/90/70",
"cost":[
100,
100,
100
],
"costBurn":"100",
"effect":[
null,
[
200,
350,
500
],
[
20,
20,
20
],
[
100,
150,
200
]
],
"effectBurn":[
null,
"200/350/500",
"20",
"100/150/200"
],
"vars":[
{
"link":"spelldamage",
"coeff":0.06667,
"key":"a2"
},
{
"link":"spelldamage",
"coeff":1.3,
"key":"a1"
}
],
"costType":"Mana",
"maxammo":"-1",
"range":[
850,
850,
850
],
"rangeBurn":"850",
"image":{
"full":"EkkoR.png",
"sprite":"spell13.png",
"group":"spell",
"x":336,
"y":144,
"w":48,
"h":48
},
"resource":"{{ cost }} Mana"
}
],
"passive":{
"name":"Z-Drive Resonance",
"description":"Ekko's Zero-Drive charges his spells and attacks with temporal energy. The third hit deals bonus damage and saps the target's movement speed. If the target is a champion, Ekko gains accelerated movement speed.",
"image":{
"full":"Ekko_P.png",
"sprite":"passive4.png",
"group":"passive",
"x":384,
"y":0,
"w":48,
"h":48
}
},
"recommended":[
" ANOTHER 1k lines here "
]
}
}
}
2 個解決方案
解決方案1
2 已采納 2016-03-11 08:50:37
正如@Hugo已經提到的那樣，使用Pandas可以簡化很多事情。 除此之外，使用json_normalize()函數將有助於您展平嵌套的數據，因此將它們加載到數據庫中會更加容易。
我也強烈建議您仔細看看ujson （Ultra Fast JSON）模塊，它比標准模塊快
並且使用許多大的JSON文件，幾乎不可避免地會在JSON文件/數據中出現錯誤-為此，我建議使用以下技巧：
python -m json.tool file.json
它應該告訴您JSON數據出了什么問題
附言：如果您需要進一步的幫助，請更新您的問題或打開一個新問題。
更新：
為了將解析的JSON數據存儲在DB中，您想使用pandas.DataFrame.to_sql（）函數
解決方案2
1 2016-03-11 08:37:19
您可以使用pandas.io.json中的pandas中的pandas.io.json import json_normalize來從json中提取數據。 給定您的json“圖片”，它將在規范化的數據庫中返回
import json
from pandas.io.json import json_normalize
image = json.dumps({"full":"fullArt.png","sprite":"sprite4.png","group":"Art","x":192,"y":0,"w":48,"h":48})
data1 = json.loads(image)
data2 = json_normalize(data1)
print data2
輸出：
full group h sprite w x y
0 fullArt.png Art 48 sprite4.png 48 192 0
使用JSON對象解析和處理大型文件的更有效方法
[英]Much more efficient way to parse and process large files with JSON Objects
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.