解析日志文件并将其有效地写入csv文件

Question

I have a log file which I am parsing using regex. It returns me 3 elements

1) timestamp

2) numberid

3) objectvalue

I intend to write this in a CSV file efficiently (As the log file size could be huge).
I have tried this

def read_logs(input_file):
    data = defaultdict()
    for each in input_file:
        regex_match = re(r'',each)
        data['timestamp'].append(regex_match.group(1))
        data['numberid'].append(regex_match.group(2))
        data['objectvalue'].append(regex_match.group(3))
    return data

def main(inputname,outputname):
    with open(inputname) as input_file:
        data = read_logs(input_file)
    with open(outputname,'w') as out_file:
        write_file(out_file,data)

def write_file(out_file):
    out = csv.writer(out_file)
    out.writerow(['timestamp_val','numberid','objectvalue'])

1) I thought using defaultdict would be the most efficient way of writing such data it in a file. Here defaultdict keys are timestamp numberid and obejctvalue with list as its value. How do I write this in a CSV file?

Sample data value is
data = ('timestamp_val':['10:10:54','13:02:07','03:02:10'],'numberid':[AA10,BB18,FF34],'objectvalue':['NHAG','ABCD','YTAB'])

2) If this is not an efficient way, what could be a better way to accomplish this?

Other way, I could think of is reading each line using regex and writing simultaneously in CSV file. Is this a good approach?

Answer 1

I think you don't need to read all the file in dic of list: written as soon as read

def main(inputname,outputname):
    with open(inputname) as input_file, open(outputname,'w') as out_file:
        out = csv.writer(out_file)
        out.writerow(['timestamp_val','numberid','objectvalue'])
        for each in input_file:
            regex_match = re(r'',each)
            out.writerow([regex_match.group(1), regex_match.group(2), regex_match.group(3)])