根据csv文件的内容创建JSON /文本文件

Question

I am trying to loop through a csv file (approximately 91 million records) and create a new json/text file using a Python dict based on the sample records below (File is sorted on id,type).

id,type,value
4678367,1,1001
4678367,2,1007
4678367,2,1008
5678945,1,9000
5678945,2,8000

The code should append values when it matches id and type else create a new record as below. I would like to write this to a target file

How can I do this in Python?

{'id':4678367,
 'id_1':[1001],
 'id_2':[1007,1008]
},
{'id':5678945,
 'id_1':[9000],
 'id_2':[8000]
}

Answer 1

Here is one way to collect the items. I have left the writing to a file as an exercise:

Code:

with open('test.csv') as f:
    reader = csv.reader(f)
    columns = next(reader)
    results = []
    record = {}
    current_type = 0
    items = []
    for id_, type, value in reader:
        if current_type != type:
            if current_type:
                record['id_{}'.format(current_type)] = items
                items = []
            current_type = type

        if id_ != record.get('id'):
            if record:
                results.append(record)
            record = dict(id=id_)

        items.append(value)

    if record:
        record['id_{}'.format(current_type)] = items
        results.append(record)

print(results)

Results:

[
    {'id': '4678367', 'id_1': ['1001'], 'id_2': ['1007', '1008']}, 
    {'id': '5678945', 'id_1': ['9000'], 'id_2': ['8000']}
]

Answer 2

import csv
from collections import namedtuple

with open("data.csv","r") as f:
    read = csv.reader(f)
    header = next(read)
    col = namedtuple('col',header)
    dictionary = {}
    for values in read:
        data = col(*values)
        type_ = 'id_' + str(data.type)
        if data.id in dictionary:
            local_dict = dictionary[data.id]                
            if type_ in local_dict:
                local_dict[type_].append(data.value)
            else:
                local_dict[type_] = [data.value]
        else:
            dictionary.setdefault(data.id,{'id':data.id,type_:[data.value]})
print(*dictionary.values(),sep="\n")
>>>{'id': '4678367', 'id_1': ['1001'], 'id_2': ['1007', '1008']}
   {'id': '5678945', 'id_1': ['9000'], 'id_2': ['8000']}