Python 解析 JSON 并将 output 写入 ZCC8D68C551C4A9A9A6D5313DZDE7

Question

I am trying to parse a json file which contains the following.

{"short_desc":
{"3641":[{"when":1002747507,"what":"DCR: Cant compare from outliner (1GDHJKK)","who":14},{"when":1002771621,"what":"DCR: Can't compare from outliner (1GDHJKK)","who":21}],
"3470":[{"when":1002747341,"what":"Can't compare code editions in type hierarchy view (1GGNI4W)","who":24},{"when":1002771649,"what":"DCR: Can't compare code editions in type hierarchy view (1GGNI4W)","who":21}]

I try to take the text that has the what header and every number in front of each array and save it into csv. The expected results are as follows.

id  | description
3641 | DCR: Cant compare from outliner (1GDHJKK)
3641 | DCR: Can't compare from outliner (1GDHJKK)
3470 | Can't compare code editions in type hierarchy view (1GGNI4W)
3470 | DCR: Can't compare code editions in type hierarchy view (1GGNI4W)

I tried the following code and tried only to get the value of what , but got an error KeyError: 'what'

import csv
import json
from glob import glob

# Open CSV output files for reading and writing
input_dir = ""
output_dir = ""

# Open main twitter data CSV file and write header row
output_file = output_dir + "coba.csv"
f_out = open(output_file, 'w', encoding='utf-8')
rowwriter = csv.writer(f_out, delimiter=',', lineterminator='\n')
outputrow = ['description']
rowwriter.writerow(outputrow)

# Define variables
inc = 0

with open('msr2013/data/v02/eclipse/short_desc.json', 'r', encoding='utf-8') as f:
    for line in f:
        bug = json.loads(line)

        # Set standard variables equal to tweet data
        bug = bug['short_desc']['what']          

        # Write to main output file
        outputrow = [bug] 
        rowwriter.writerow(outputrow)

        inc += 1
        # Optional counter increments variables to track progress, useful for very large files.
        if inc%10000 == 0:
            print(inc)

# Close the output file
f_out.close()

Can anyone give me a solution?

Answer 1

Based on the sample data, you should do like the following.

        bug = json.loads(line)

        # Set standard variables equal to tweet data
        for _id in bug['short_desc']:
            for i in range(len(bug['short_desc'][_id])):
                bug_what = bug['short_desc'][_id][i]['what']

                # Write to main output file
                outputrow = [bug_what]

bug['short_desc'] is an object mapping id to an array.

Answer 2

you can use pandas to parse json, apply changes and write output to csv.

import pandas as pd
data = pd.read_json('test.json')
data['id'] = data.index
filter_data = data[data['short_desc'].notnull()].to_dict(orient='records')
inner_data = pd.json_normalize(filter_data, record_path='short_desc', meta=['id'],errors='ignore')
inner_data = inner_data[['what','id']]
inner_data = inner_data.rename(columns={'what':'description'})
inner_data.to_csv('test.csv')

output:

Answer 3

Based on your data sample you don't have " what " key in " short_description ".

You should do something like this:

...
for line in f:
    bug = json.loads(line)
    for bug_id, bug_data in bug.get('short_desc', {}).items():
        #For first sample: bug_id='3470' and bug_data=[{...}, {...}] 
        for row in bug_data:
            rowwriter.writerow([bug_id, row['what']])
...