Python String to List of Dicts

Question

I have reviewed a number of similar questions on stackoverflow, but been unable to locate an answer which applies to my data/string.

I have a string which is effectively a list of dictionaries. In the fields, numbers are not surrounded by double quotes. If I try to use ast to evaluate the string, part of the string is cut off and I am unsure why. Could someone help me determine an appropriate way to read in this string and create a list of dicts.

Thanks,

>>> print(ascii_data)
   [{"measurement": "cpu_load_short","tags": {"host": "server999","region": "us-west-1"},"fields": {"value": 0.99}},{"measurement": "cpu_load_short","tags": {"host": "server888","region": "us-east-1"},"fields": {"value": 0.88}}]
>>> x = ast.literal_eval(ascii_data)
>>> print(x)
   [{'fields': {'value': 0.99}, 'tags': {'host': 'server999', 'region': 'us-west-1'}, 'measurement': 'cpu_load_short'}, {'fields': {'value': 0.88}, 'tags': {'host': 'server888', 'region': 'us-east-1'}, 'measurement': 'cpu_load_short'}]

Answer 1

Use json.

In [1]: s = '''[{"measurement": "cpu_load_short","tags": {"host": "server999","region": "us-west-1"},"fields": {"value": 0.99}},{"measuremen
   ...: t": "cpu_load_short","tags": {"host": "server888","region": "us-east-1"},"fields": {"value": 0.88}}]'''

In [2]: import json

In [3]: import pprint

In [4]: pprint.pprint(json.loads(s))
[{'fields': {'value': 0.99},
  'measurement': 'cpu_load_short',
  'tags': {'host': 'server999', 'region': 'us-west-1'}},
 {'fields': {'value': 0.88},
  'measurement': 'cpu_load_short',
  'tags': {'host': 'server888', 'region': 'us-east-1'}}]
In [11]: json.loads(s)[0]['tags']['host']
Out[11]: 'server999'

Answer 2

How about json.loads ?

j = json.loads(ascii_data)

ast.literal_eval may not be the best option. If your data source comes from some API, it would definitely be json format.

And if the order of dict keys matters to you, try specifying the object_pairs_hook argument to JSONDecoder. (ref: Can I get JSON to load into an OrderedDict in Python? )

Answer 3

Given:

>>> s
'[{"measurement": "cpu_load_short","tags": {"host": "server999","region": "us-west-1"},"fields": {"value": 0.99}},{"measurement": "cpu_load_short","tags": {"host": "server888","region": "us-east-1"},"fields": {"value": 0.88}}]'

You can use json

>>> import json
>>> json.loads(s)
[{u'fields': {u'value': 0.99}, u'tags': {u'host': u'server999', u'region': u'us-west-1'}, u'measurement': u'cpu_load_short'}, {u'fields': {u'value': 0.88}, u'tags': {u'host': u'server888', u'region': u'us-east-1'}, u'measurement': u'cpu_load_short'}]

Or ast :

>>> import ast
>>> ast.literal_eval(s)
[{'fields': {'value': 0.99}, 'tags': {'host': 'server999', 'region': 'us-west-1'}, 'measurement': 'cpu_load_short'}, {'fields': {'value': 0.88}, 'tags': {'host': 'server888', 'region': 'us-east-1'}, 'measurement': 'cpu_load_short'}]

And they produce the same Python data structure (at least with ascii input...):

>>> json.loads(s)==ast.literal_eval(s)
True

Since in each case the result is a Python dict know that the order may be different than the string's order. Python dicts are unordered and will usually be different than the creation order (at least prior to Python 3.6).

---

Under Python 3.6, they resulting dict is in the same order:

>>> json.loads(s)
[{'measurement': 'cpu_load_short', 'tags': {'host': 'server999', 'region': 'us-west-1'}, 'fields': {'value': 0.99}}, {'measurement': 'cpu_load_short', 'tags': {'host': 'server888', 'region': 'us-east-1'}, 'fields': {'value': 0.88}}]
>>> ast.literal_eval(s)
[{'measurement': 'cpu_load_short', 'tags': {'host': 'server999', 'region': 'us-west-1'}, 'fields': {'value': 0.99}}, {'measurement': 'cpu_load_short', 'tags': {'host': 'server888', 'region': 'us-east-1'}, 'fields': {'value': 0.88}}]

Python 3.6 is great...