解析大型JSON文件（Python）

Question

I'm new with JSON but after some research about reading JSON and parse it, all I found was to do it manually:

 "image":{
        "full":"fullArt.png",
        "sprite":"sprite4.png",
        "group":"Art",
        "x":192,
        "y":0,
        "w":48,
        "h":48
   }

It means that if I have this little JSON block to read , I'll use something like :

jfile="image.json"
file=open(jfile, 'r')
jdata=json.load(file)
data=jdata["image"]
full=data["full"]
sprite=data["sprite"]
....

but this is assuming that I know how the file is inside, and that all of the images are written that way which is maybe false.

The other problem is that I aim to read files with more than 200k lines of JSON in order to put the datas extracted from them into a database.

So my question is : How can I read large JSON files and store results in a database ??

Thanks for reading me.

EDIT: Added of piece of something I want to read

    {
   "type":"champion",
   "format":"standAloneComplex",
   "version":"6.4.2",
   "data":{
      "Ekko":{
         "id":"Ekko",
         "key":"245",
         "name":"Ekko",
         "title":"the Boy Who Shattered Time",
         "image":{
            "full":"Ekko.png",
            "sprite":"champion4.png",
            "group":"champion",
            "x":192,
            "y":0,
            "w":48,
            "h":48
         },
         "skins":[
            {
               "id":"245000",
               "num":0,
               "name":"default",
               "chromas":false
            },
            {
               "id":"245001",
               "num":1,
               "name":"Sandstorm Ekko",
               "chromas":false
            },
            {
               "id":"245002",
               "num":2,
               "name":"Academy Ekko",
               "chromas":false
            }
         ],
         "lore":"A prodigy from the rough streets of Zaun, Ekko manipulates time to spin any situation to his advantage. Using his own invention, the Zero-Drive, he explores the branching possibilities of reality. As well as experimenting with multi-dimensional probability, Ekko spends his days running wild with the other lost children of Zaun. Though he revels in this freedom, when there's a threat to his friends, he'll endure anything to defend them. To the unknowing observer, Ekko accomplishes the impossible with ease, time and time again.",
         "blurb":"A prodigy from the rough streets of Zaun, Ekko manipulates time to spin any situation to his advantage. Using his own invention, the Zero-Drive, he explores the branching possibilities of reality. As well as experimenting with multi-dimensional ...",
         "allytips":[
            "Chronobreak is a potent escape tool, but it can also be quite powerful when used offensively. Don't underestimate its damage potential.",
            "If you can proc Z-Drive Resonance on an enemy champion, it's worth taking risks to do so. The bonus movement speed makes it easy to escape.",
            "Phase Dive's dash is a great tool for setting up Ekko's other abilities. Use it to get double hits with Timewinder or get into position to detonate Parallel Convergence."
         ],
         "enemytips":[
            "Ekko is significantly weaker when his ultimate is down. Watch for the trail he leaves behind to determine if Chronobreak is available.",
            "Ekko's stun zone takes 3 seconds to arm. Watch for the image he creates on cast and try to guess where the zone was placed.",
            "The second hit of Timewinder does more damage than the first; try to avoid it."
         ],
         "tags":[
            "Assassin",
            "Fighter"
         ],
         "partype":"MP",
         "info":{
            "attack":5,
            "defense":3,
            "magic":7,
            "difficulty":8
         },
         "stats":{
            "hp":580.0,
            "hpperlevel":80.0,
            "mp":280.0,
            "mpperlevel":50.0,
            "movespeed":340.0,
            "armor":27.0,
            "armorperlevel":3.0,
            "spellblock":32.0,
            "spellblockperlevel":1.25,
            "attackrange":125.0,
            "hpregen":9.0,
            "hpregenperlevel":0.9,
            "mpregen":6.0,
            "mpregenperlevel":0.8,
            "crit":0.0,
            "critperlevel":0.0,
            "attackdamage":59.0,
            "attackdamageperlevel":3.0,
            "attackspeedoffset":0.0,
            "attackspeedperlevel":3.3
         },
         "spells":[
            {
               "id":"EkkoQ",
               "name":"Timewinder",
               "description":"Ekko throws a temporal grenade that expands into a time-distortion field upon hitting an enemy champion, slowing and damaging anyone caught inside. After a delay, the grenade rewinds back to Ekko, dealing damage on its return.",
               "tooltip":"Ekko throws a device that deals {{ e1 }} <span class=\"color99FF99\">(+{{ a1 }})</span> magic damage to enemies it passes through. It expands into a slowing field on the first champion hit, slowing everything inside by {{ e2 }}%. It then returns to him after a delay, dealing {{ e3 }} <span class=\"color99FF99\">(+{{ a2 }})</span> magic damage to all targets hit upon return.",
               "leveltip":{
                  "label":[
                     "Outgoing Damage",
                     "Return Damage",
                     "Slow",
                     "Cooldown",
                     "Mana Cost"
                  ],
                  "effect":[
                     "{{ e1 }} -> {{ e1NL }}",
                     "{{ e3 }} -> {{ e3NL }}",
                     "{{ e2 }}% -> {{ e2NL }}%",
                     "{{ cooldown }} -> {{ cooldownnNL }}",
                     "{{ cost }} -> {{ costnNL }}"
                  ]
               },
               "maxrank":5,
               "cooldown":[
                  9,
                  8.5,
                  8,
                  7.5,
                  7
               ],
               "cooldownBurn":"9/8.5/8/7.5/7",
               "cost":[
                  50,
                  60,
                  70,
                  80,
                  90
               ],
               "costBurn":"50/60/70/80/90",
               "effect":[
                  null,
                  [
                     60,
                     75,
                     90,
                     105,
                     120
                  ],
                  [
                     32,
                     39,
                     46,
                     53,
                     60
                  ],
                  [
                     60,
                     85,
                     110,
                     135,
                     160
                  ],
                  [
                     100,
                     100,
                     100,
                     100,
                     100
                  ]
               ],
               "effectBurn":[
                  null,
                  "60/75/90/105/120",
                  "32/39/46/53/60",
                  "60/85/110/135/160",
                  "100"
               ],
               "vars":[
                  {
                     "link":"spelldamage",
                     "coeff":0.1,
                     "key":"a1"
                  },
                  {
                     "link":"spelldamage",
                     "coeff":0.6,
                     "key":"a2"
                  }
               ],
               "costType":"Mana",
               "maxammo":"-1",
               "range":[
                  1075,
                  1075,
                  1075,
                  1075,
                  1075
               ],
               "rangeBurn":"1075",
               "image":{
                  "full":"EkkoQ.png",
                  "sprite":"spell13.png",
                  "group":"spell",
                  "x":192,
                  "y":144,
                  "w":48,
                  "h":48
               },
               "resource":"{{ cost }} Mana"
            },
            {
               "id":"EkkoW",
               "name":"Parallel Convergence",
               "description":"Ekko splits the timeline, creating an anomaly after a few seconds that slows enemies caught inside. If Ekko enters the anomaly, he gains shielding and triggers a detonation, stunning enemies by suspending them in time.",
               "tooltip":"<span class=\"colorFF9900\">Passive:</span> Ekko's basic attacks deal bonus magic damage to enemies under 30% health equal to {{ e3 }}% <span class=\"color99FF99\">(+{{ a2 }}%)</span> of their missing health (max {{ e5 }} damage vs. minions and monsters).<br><br><span class=\"colorFF9900\">Active:</span> After a 3 second delay, Ekko creates a short-lived chronosphere at the target location that slows enemies who enter by 40%. If Ekko enters the sphere, he will detonate it, gaining a shield that absorbs up to {{ e4 }} <span class=\"color99FF99\">(+{{ a1 }})</span> damage for 2 seconds. Enemies caught inside are stunned for {{ e2 }} seconds.",
               "leveltip":{
                  "label":[
                     "Shield Amount",
                     "Cooldown",
                     "Mana Cost"
                  ],
                  "effect":[
                     "{{ e4 }} -> {{ e4NL }}",
                     "{{ cooldown }} -> {{ cooldownnNL }}",
                     "{{ cost }} -> {{ costnNL }}<"
                  ]
               },
               "maxrank":5,
               "cooldown":[
                  22,
                  20,
                  18,
                  16,
                  14
               ],
               "cooldownBurn":"22/20/18/16/14",
               "cost":[
                  50,
                  55,
                  60,
                  65,
                  70
               ],
               "costBurn":"50/55/60/65/70",
               "effect":[
                  null,
                  [
                     375,
                     375,
                     375,
                     375,
                     375
                  ],
                  [
                     2.25,
                     2.25,
                     2.25,
                     2.25,
                     2.25
                  ],
                  [
                     5,
                     5,
                     5,
                     5,
                     5
                  ],
                  [
                     80,
                     100,
                     120,
                     140,
                     160
                  ],
                  [
                     150,
                     150,
                     150,
                     150,
                     150
                  ]
               ],
               "effectBurn":[
                  null,
                  "375",
                  "2.25",
                  "5",
                  "80/100/120/140/160",
                  "150"
               ],
               "vars":[
                  {
                     "link":"spelldamage",
                     "coeff":0.018,
                     "key":"a2"
                  },
                  {
                     "link":"spelldamage",
                     "coeff":1.5,
                     "key":"a1"
                  }
               ],
               "costType":"Mana",
               "maxammo":"-1",
               "range":[
                  1600,
                  1600,
                  1600,
                  1600,
                  1600
               ],
               "rangeBurn":"1600",
               "image":{
                  "full":"EkkoW.png",
                  "sprite":"spell13.png",
                  "group":"spell",
                  "x":240,
                  "y":144,
                  "w":48,
                  "h":48
               },
               "resource":"{{ cost }} Mana"
            },
            {
               "id":"EkkoE",
               "name":"Phase Dive",
               "description":"Ekko rolls evasively while charging up his Z-Drive. His next attack deals bonus damage and warps reality, teleporting him to his target.",
               "tooltip":" Ekko dashes a short distance in the targeted direction. His next attack will deal {{ e1 }} <span class=\"color99FF99\">(+{{ a1 }})</span> bonus magic damage and teleport him to his target.",
               "leveltip":{
                  "label":[
                     "Damage",
                     "Cooldown",
                     "Mana Cost"
                  ],
                  "effect":[
                     "{{ e1 }} -> {{ e1NL }}",
                     "{{ cooldown }} -> {{ cooldownnNL }}",
                     "{{ cost }} -> {{ costnNL }}"
                  ]
               },
               "maxrank":5,
               "cooldown":[
                  13,
                  11.5,
                  10,
                  8.5,
                  7
               ],
               "cooldownBurn":"13/11.5/10/8.5/7",
               "cost":[
                  40,
                  50,
                  60,
                  70,
                  80
               ],
               "costBurn":"40/50/60/70/80",
               "effect":[
                  null,
                  [
                     50,
                     80,
                     110,
                     140,
                     170
                  ]
               ],
               "effectBurn":[
                  null,
                  "50/80/110/140/170"
               ],
               "vars":[
                  {
                     "link":"spelldamage",
                     "coeff":0.2,
                     "key":"a1"
                  }
               ],
               "costType":"Mana",
               "maxammo":"-1",
               "range":[
                  325,
                  325,
                  325,
                  325,
                  325
               ],
               "rangeBurn":"325",
               "image":{
                  "full":"EkkoE.png",
                  "sprite":"spell13.png",
                  "group":"spell",
                  "x":288,
                  "y":144,
                  "w":48,
                  "h":48
               },
               "resource":"{{ cost }} Mana"
            },
            {
               "id":"EkkoR",
               "name":"Chronobreak",
               "description":"Ekko shatters his timeline, becoming untargetable and rewinding to a more favorable point in time. He returns to whenever he was a few seconds ago, and heals for a percentage of the damage received in that duration. Enemies near his arrival zone take massive damage.",
               "tooltip":"Makes Ekko invulnerable and untargetable, and teleports him back to wherever he was 4 seconds ago. Ekko heals himself for {{ e3 }} + {{ e2 }}% <span class=\"color99FF99\">(+{{ a2 }}%)</span> of the damage taken over the last 4 seconds, and deals {{ e1 }} <span class=\"color99FF99\">(+{{ a1 }})</span> magic damage to all nearby enemies on arrival.",
               "leveltip":{
                  "label":[
                     "Damage",
                     "Flat Heal",
                     "Cooldown"
                  ],
                  "effect":[
                     "{{ e1 }} -> {{ e1NL }}",
                     "{{ e3 }} -> {{ e3NL }}",
                     "{{ cooldown }} -> {{ cooldownnNL }}"
                  ]
               },
               "maxrank":3,
               "cooldown":[
                  110,
                  90,
                  70
               ],
               "cooldownBurn":"110/90/70",
               "cost":[
                  100,
                  100,
                  100
               ],
               "costBurn":"100",
               "effect":[
                  null,
                  [
                     200,
                     350,
                     500
                  ],
                  [
                     20,
                     20,
                     20
                  ],
                  [
                     100,
                     150,
                     200
                  ]
               ],
               "effectBurn":[
                  null,
                  "200/350/500",
                  "20",
                  "100/150/200"
               ],
               "vars":[
                  {
                     "link":"spelldamage",
                     "coeff":0.06667,
                     "key":"a2"
                  },
                  {
                     "link":"spelldamage",
                     "coeff":1.3,
                     "key":"a1"
                  }
               ],
               "costType":"Mana",
               "maxammo":"-1",
               "range":[
                  850,
                  850,
                  850
               ],
               "rangeBurn":"850",
               "image":{
                  "full":"EkkoR.png",
                  "sprite":"spell13.png",
                  "group":"spell",
                  "x":336,
                  "y":144,
                  "w":48,
                  "h":48
               },
               "resource":"{{ cost }} Mana"
            }
         ],
         "passive":{
            "name":"Z-Drive Resonance",
            "description":"Ekko's Zero-Drive charges his spells and attacks with temporal energy. The third hit deals bonus damage and saps the target's movement speed. If the target is a champion, Ekko gains accelerated movement speed.",
            "image":{
               "full":"Ekko_P.png",
               "sprite":"passive4.png",
               "group":"passive",
               "x":384,
               "y":0,
               "w":48,
               "h":48
            }
         },
         "recommended":[
            " ANOTHER 1k lines here "
         ]
      }
   }
}

Answer 1

As @Hugo has already mentioned, using Pandas will simplify a lot of things. Beside that using json_normalize() function will help you to flatten your nested data so it will be much easier to load them into your DB.

Also i would strongly recommend you to take a closer look at the ujson (Ultra Fast JSON) module, which is faster compared to the standard one

And working with many big JSON files it's almost inevitable to hit errors in your JSON files/data - for that i would recommend the following trick:

python -m json.tool file.json

It should show you where is the problem with your JSON data

PS if you need further help please update your question or open a new one.

UPDATE:

in order to store your parsed JSON data in DB you want to use pandas.DataFrame.to_sql() function

Answer 2

You could use the json_normalize from pandas from pandas.io.json import json_normalize to extract data from json. Given your json "image" this will return it in a normalized database

import json
from pandas.io.json import json_normalize

image = json.dumps({"full":"fullArt.png","sprite":"sprite4.png","group":"Art","x":192,"y":0,"w":48,"h":48})
data1 = json.loads(image)
data2 = json_normalize(data1)
print data2

with the output:

         full group   h       sprite   w    x  y
0  fullArt.png   Art  48  sprite4.png  48  192  0