Converting Dictionary log file into csv using python

Question

I want to convert log file, Here the log file:

{'convolution': 2, 'cov2d_layers': [256, 256], 'fc_layers': 3, 'neurons': [256, 128, 1024], 'optimizer': 'rmsprop'}
Accuracy: 46.63%
{'convolution': 3, 'cov2d_layers': [128, 32, 128], 'fc_layers': 3, 'neurons': [1024, 1024, 1024], 'optimizer': 'adam'}
Accuracy: 39.57%
...

I need a complete csv file like so:

convolution, conv2d_layers, fc_layers, neurons, optimizer, accuracy
2, [256,256], 3, [256, 128,1024], 46,63%

How can I do this using a python script?

Answer 1

Just to be clear: the example output you requested does not comply with the typical 'csv'-style (comma separated values). Similarly, the input log file only partly conforms the json-style.

read log-file :

import itertools
import json

data = [];
with open('in.txt', 'r') as file:
    for line1,line2 in itertools.zip_longest(*[file]*2):
        line1 = line1.replace('\'','"')      #convert to json
        line2 = line2.split(':')[-1].strip() #extract number

        d = json.loads(line1) #create dictionary
        d['accuracy'] = line2 #add manually

        data.append(d)

create csv :

In your case I'd suggest you transform the data contained in the dictionaries to the string format according to your custom style.

dlm = ';'
with open('out.txt','w') as file:
    bheader = True
    for d in data:
        #header
        shead = list( d.keys() )
        if bheader: 
            file.write( dlm.join( shead ) + '\n' )
            bheader=False
        #data
        sdata = ['%s'%e for e in d.values() ]
        file.write( dlm.join(sdata) + '\n' ) 

excel :

As implied by your comment you want to import the data afterwards to excel. In order to achieve this you could use the excel import dialogue and tell it to use the delimiter sign ";" (as used above). In you example you also converted the decimal separator; you could either specify the '.' separator during the import or transform entire column afterwards in excel.

Note that it might be hard to access the array elements of 'neurons' within the excel.

files :

'in.txt'

{'convolution': 2, 'cov2d_layers': [256, 256], 'fc_layers': 3, 'neurons': [256, 128, 1024], 'optimizer': 'rmsprop'}
Accuracy: 46.63%
{'convolution': 3, 'cov2d_layers': [128, 32, 128], 'fc_layers': 3, 'neurons': [1024, 1024, 1024], 'optimizer': 'adam'}
Accuracy: 39.57%

'out.txt'

convolution;cov2d_layers;fc_layers;neurons;optimizer;accuracy
2;[256, 256];3;[256, 128, 1024];rmsprop;46.63%
3;[128, 32, 128];3;[1024, 1024, 1024];adam;39.57%

Answer 2

Below

import json
import ast

log_lines = []
headers = None
with open('data.txt') as f:
    lines = [l.strip() for l in f.readlines()]
    for idx, l in enumerate(lines, 1):
        if idx % 2 != 0:
            d = ast.literal_eval(l)
            if not headers:
                headers = list(d.keys())
                headers.append('accuracy')
        else:
            d['accuracy'] = l[l.rfind(' '):]
            log_lines.append(list(d.values()))

with open('log.txt', 'w') as f:
    f.write(';'.join(headers) + '\n')
    for line in log_lines:
        line = [str(x) for x in line]
        f.write(';'.join(line) + '\n')

data.txt

{'convolution': 2, 'cov2d_layers': [256, 256], 'fc_layers': 3, 'neurons': [256, 128, 1024], 'optimizer': 'rmsprop'}
Accuracy: 46.63%
{'convolution': 3, 'cov2d_layers': [128, 32, 128], 'fc_layers': 3, 'neurons': [1024, 1024, 1024], 'optimizer': 'adam'}
Accuracy: 39.57%

log.txt

convolution;cov2d_layers;fc_layers;neurons;optimizer;accuracy
2;[256, 256];3;[256, 128, 1024];rmsprop; 46.63%
3;[128, 32, 128];3;[1024, 1024, 1024];adam; 39.57%