Converting txt file into a list of dictionaries

Question

I am trying to convert a file with string dictionaries into a list of dictionaries. The file can be found here: https://github.com/szczor/sharing/blob/master/data.txt . Here is how I try it:

lista = []
with open(os.path.join(path_to_data, 'data.txt')) as filehandle:
    for line in filehandle:
        # remove linebreak which is the last character of the string
        currentPlace = line[:-1]

        # add item to the list
        lista.append(currentPlace)

So I have a list, now with elements which are string.

print(type(lista[0]))

Now trying to convert those strings to dictionaries with

lista = [json.loads(str(string.replace('\'','\"'))) for string in lista]

But get an error

    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I tried the same method with manually created list and everything worked, but have no idea why it does not work with my txt file.

Answer 1

You can use ast.literal_eval for this:

import ast
with open(os.path.join(path_to_data, 'data.txt')) as fp:

    lines = fp.readlines()


# if there are irregularities, use a try/except to pass these (note, you may wish to use a logger in practice)    
output = []
for line in lines:
    try:
        output.append(ast.literal_eval(line))
    except ValueError:
        print("malformed string; skipping this line")
    except SyntaxError:
        print("looks like some encoding errors with this file...")

output:

[{'Lines': '130',
  'Lon': 21.0082356,
  'VehicleNumber': '1000',
  'Time': '2020-12-23 14:19:31',
  'Lat': 52.207217,
  'Brigade': '2'},
 {'Lines': '213',
  'Lon': 21.1013111,
  'VehicleNumber': '1001',
  'Time': '2020-12-23 15:40:37',
  'Lat': 52.2230756,
  'Brigade': '2'},
 {'Lines': '130',
  'Lon': 21.003573,
  'VehicleNumber': '1002',
  'Time': '2020-12-23 15:40:39',
  'Lat': 52.2060591,
  'Brigade': '1'},
 {'Lines': '311',
  'Lon': 21.0807406,
  'VehicleNumber': '1003',
  'Time': '2020-12-23 15:40:35',
  'Lat': 52.2414656,
  'Brigade': '2'},
 {'Lines': '213',
  'Lon': 21.1746171,
  'VehicleNumber': '1004',
  'Time': '2020-12-23 15:40:24',
  'Lat': 52.186474,
  'Brigade': '3'},...

Answer 2

i could not comment to your question due to some privileges, try

for line in filehandle.readlines(): #instead of just filehandle

Answer 3

This oughta do the trick. ast.literal_eval() should be able to parse dict strings like those in the data file:

import ast


with open(os.path.join(path_to_data, 'data.txt')) as filehandle:
    lista = [
        ast.literal_eval(line.strip())
        for line
        in filehandle
    ]

Answer 4

This is how i would approach:

   f = open("data.txt", "r")
   my_list = []

   for x in f: #read the sheet line by line
      dict_x = eval(x) #convert each line to dict
      my_list.append(dict_x)

   f.close()

Alternatively, eval() can also be substituted by dict()