How to convert a string containing a list of dict into Python object?

Question

ugly = "[{'ride': 1, 'pickup_time': datetime(2016, 3, 17, 15, 36, 35, 976202)},
         {'ride': 2, 'pickup_time': datetime.datetime(2016, 3, 17, 15, 41, 35, 976202)}]"
# The actual variable contains a lot more dictionnaries...`

I'd like to convert the ugly variable as a real Python object. I tried json.loads() and ast.literal_eval() but it's for one dict only.

That's why, before that, I tried to split this string into several dictionnaries, but the split() method has only one delimiter, so it seems I could need a REGEX to do this.

What is the simplest way to do it? Thanks.

Answer 1

Well first you should be consistent in you calls to datetime method. In first dict, you use datetime and in second datetime.datetime .

Whatever way you try to have Python eval the string, you cannot have at the same time datetime be a function (first dict) and a module (second). Once you fix that, you will be forced to use the evil eval function, because neither json.loads not ast.litteral_eval will accept a function. They are normally used precisely to avoid the evaluation to call any function...

But if this is what you want to do (and you are sure that ugly contains no harmful code), this works:

>>> ugly = "[{'ride': 1, 'pickup_time': datetime.datetime(2016, 3, 17, 15, 36, 35, 976202)},{'ride': 2, 'pickup_time': datetime.datetime(2016, 3, 17, 15, 41, 35, 976202)}]"
>>> import datetime
>>> dlist = eval(ugly)
>>> dlist
[{'ride': 1, 'pickup_time': datetime.datetime(2016, 3, 17, 15, 36, 35, 976202)}, {'ride': 2, 'pickup_time': datetime.datetime(2016, 3, 17, 15, 41, 35, 976202)}]

Answer 2

You can use eval

a = eval(ugly)

At this point a is a list of dictionaries and I am sure you got it from there.

Answer 3

You can use eval to get the array of dictionaries (after fixing the datetime issue), and then use setattr to turn the dictionaries into real class objects:

from datetime import datetime

ugly = "[{'ride': 1, 'pickup_time': datetime(2016, 3, 17, 15, 36, 35, 976202)}, {'ride': 2, 'pickup_time': datetime(2016, 3, 17, 15, 41, 35, 976202)}]"
array = eval(ugly)

class Ride(object):
    pass

rides = []
for record in array:
    ride = Ride()
    for k, v in record.iteritems():
        setattr(ride, k, v)
    rides.append(ride)

for ride in rides:
    print "ride: {0}: pickup_time: {1}".format(ride.ride, ride.pickup_time)

Answer 4

Depending on how you want to import datetime :

import datetime

ugly = "[{'ride': 1, 'pickup_time': datetime(2016, 3, 17, 15, 36, 35, 976202)},{'ride': 2, 'pickup_time': datetime.datetime(2016, 3, 17, 15, 41, 35, 976202)}]"
ugly = ugly.replace(" datetime(", " datetime.datetime(")
ugly = eval(ugly)

or

from datetime import datetime

ugly = "[{'ride': 1, 'pickup_time': datetime(2016, 3, 17, 15, 36, 35, 976202)},{'ride': 2, 'pickup_time': datetime.datetime(2016, 3, 17, 15, 41, 35, 976202)}]"
ugly = ugly.replace("datetime.", "")
ugly = eval(ugly)

Both work without prior cleaning of ugly

Answer 5

You could do it with a regex by extracting the datetimes and their keys only calling eval on those pairings:

from datetime import datetime
from ast import literal_eval
import re


def parse(ug):
    ug = ug.replace("datetime.", "")
    pairs = ("{{{}}}".format(p) for p in re.findall("('\w+':\s+datetime\(.*\))", ug))
    _parsed = literal_eval(re.sub("datetime\(.*\)","''", ug))
    for d in _parsed:
        d.update(eval(next(pairs)))
    return _parsed

The lists are going to be ordered so the correct pairing will be put back in the correct dicts:

In [4]: parse(ugly)
Out[4]: 
[{'pickup_time': datetime.datetime(2016, 3, 17, 15, 36, 35, 976202),
  'ride': 1},
 {'pickup_time': datetime.datetime(2016, 3, 17, 15, 41, 35, 976202),
  'ride': 2}]