使用ast.literal_eval时格式错误的字符串

Question

I used json dump and then json load on the same data. The data is unicode so I converted it to string. Using the ast.literla_eval I tried to get the type of the string to dict but I am getting error Malformed String.

output of json load is below

('data', u'{\n  "a": "spawning", \n  "addresses": "", \n  "image": "b", \n  "OS-EXT-STS:vm_state": "building", \n  "c:launched_at": null, \n  "d": "e (fgh)", \n  "user_id": "hhh", \n 
    "accessIPv4": "", \n  "accessIPv6": "", \n  "name": "kk", \n  "created": "2017-12-08T07:52:44Z", \n  "z:xyz": []\n}', <type 'unicode'>)

What I tried?

        with open('openstack_list.json', 'w') as e:
            json.dump(check_output(['openstack', 'server', 'show', i, '-f', 'json']), e)
        with open('openstack_list.json', 'r') as a:
            data = json.load(a)
            new_data = data.encode('utf-8')  # output type is unicode
            dict_data = ast.literal_eval(new_data) # output type is string

I want output to be dictionary but I didnt get it. Also, json load gives a unicode data so I believe new_data = data.encode('utf-8') is redundant. But if I use ast.literal_eval without encoding I get Malformed string error. In any case, I am not able to get the data type to be dictionary.

Edits:

Error:

Traceback (most recent call last):
  File "openstack_resource_list.py", line 84, in <module>
    output = get_resources()
  File "openstack_resource_list.py", line 47, in get_resources
    dict_data = ast.literal_eval(new_data)
  File "/usr/lib64/python2.7/ast.py", line 80, in literal_eval
    return _convert(node_or_string)
  File "/usr/lib64/python2.7/ast.py", line 63, in _convert
    in zip(node.keys, node.values))
  File "/usr/lib64/python2.7/ast.py", line 62, in <genexpr>
    return dict((_convert(k), _convert(v)) for k, v
  File "/usr/lib64/python2.7/ast.py", line 79, in _convert
    raise ValueError('malformed string')
ValueError: malformed string

The data before json dump :

"{\n  \"aaa\": null, \n  \"addresses\": \"inner-net=192.168.0.173, x.x.x.x\", \n  \"image\": \"aaa (aaa)\", 
    \n  \"aaa:vm_state\": \"active\", \n  \"aaa:launched_at\": \"2017-12-08T08:21:45.000000\", \n  \"flavor\": \"aaa4 (aaa)\", 
    \n  \"id\": \"aaa\", \n  \"security_groups\": [\n    {\n      \"name\": \"default\"\n    }\n  ], \n  \"user_id\": \"aaa\", 
    \n  \"OS-DCF:diskConfig\": \"MANUAL\", \n  \"accessIPv4\": \"\", \n  \"accessIPv6\": \"\", \n  \"progress\": 0, \n  \"Oaa:power_state\": 1, \n  \"project_id\": \"aaa\", 
    \n  \"config_drive\": \"\", \n  \"status\": \"ACTIVE\", \n  \"updated\": \"2017-12-08T08:21:45Z\", \n  \"hostId\": \"aaa\", \n  \"OS-SRV-USG:terminated_at\": null, 
    \n  \"key_name\": \"pg_ci\", \n  \"properties\": \"\", \n  \"OS-EXT-AZ:availability_zone\": \"nova\", \n  \"name\": \"taaa\", \n  \"created\": \"2017-12-08T08:21:31Z\", \n 
    \"os-extended-volumes:volumes_attached\": [\n    {\n      \"id\": \"aaa\"\n    }\n  ]\n}"

Answer 1

That data returned by check_output is already JSON, so you should not JSON-ify it again with json.dump . You can just write it to the file as is, and the file will be a valid JSON file. And you can load it into a Python object with json.loads :

import json
from pprint import pprint

s = """{\n  \"aaa\": null, \n  \"addresses\": \"inner-net=192.168.0.173, x.x.x.x\", \n  \"image\": \"aaa (aaa)\", 
    \n  \"aaa:vm_state\": \"active\", \n  \"aaa:launched_at\": \"2017-12-08T08:21:45.000000\", \n  \"flavor\": \"aaa4 (aaa)\", 
    \n  \"id\": \"aaa\", \n  \"security_groups\": [\n    {\n      \"name\": \"default\"\n    }\n  ], \n  \"user_id\": \"aaa\", 
    \n  \"OS-DCF:diskConfig\": \"MANUAL\", \n  \"accessIPv4\": \"\", \n  \"accessIPv6\": \"\", \n  \"progress\": 0, \n  \"Oaa:power_state\": 1, \n  \"project_id\": \"aaa\", 
    \n  \"config_drive\": \"\", \n  \"status\": \"ACTIVE\", \n  \"updated\": \"2017-12-08T08:21:45Z\", \n  \"hostId\": \"aaa\", \n  \"OS-SRV-USG:terminated_at\": null, 
    \n  \"key_name\": \"pg_ci\", \n  \"properties\": \"\", \n  \"OS-EXT-AZ:availability_zone\": \"nova\", \n  \"name\": \"taaa\", \n  \"created\": \"2017-12-08T08:21:31Z\", \n 
    \"os-extended-volumes:volumes_attached\": [\n    {\n      \"id\": \"aaa\"\n    }\n  ]\n}"""

d = json.loads(s)
pprint(d)

output

{'OS-DCF:diskConfig': 'MANUAL',
 'OS-EXT-AZ:availability_zone': 'nova',
 'OS-SRV-USG:terminated_at': None,
 'Oaa:power_state': 1,
 'aaa': None,
 'aaa:launched_at': '2017-12-08T08:21:45.000000',
 'aaa:vm_state': 'active',
 'accessIPv4': '',
 'accessIPv6': '',
 'addresses': 'inner-net=192.168.0.173, x.x.x.x',
 'config_drive': '',
 'created': '2017-12-08T08:21:31Z',
 'flavor': 'aaa4 (aaa)',
 'hostId': 'aaa',
 'id': 'aaa',
 'image': 'aaa (aaa)',
 'key_name': 'pg_ci',
 'name': 'taaa',
 'os-extended-volumes:volumes_attached': [{'id': 'aaa'}],
 'progress': 0,
 'project_id': 'aaa',
 'properties': '',
 'security_groups': [{'name': 'default'}],
 'status': 'ACTIVE',
 'updated': '2017-12-08T08:21:45Z',
 'user_id': 'aaa'}

And if you want to make it into clean JSON, pass that Python object to json.dump or json.dumps

print(json.dumps(d, indent=4))

output

{
    "aaa": null,
    "addresses": "inner-net=192.168.0.173, x.x.x.x",
    "image": "aaa (aaa)",
    "aaa:vm_state": "active",
    "aaa:launched_at": "2017-12-08T08:21:45.000000",
    "flavor": "aaa4 (aaa)",
    "id": "aaa",
    "security_groups": [
        {
            "name": "default"
        }
    ],
    "user_id": "aaa",
    "OS-DCF:diskConfig": "MANUAL",
    "accessIPv4": "",
    "accessIPv6": "",
    "progress": 0,
    "Oaa:power_state": 1,
    "project_id": "aaa",
    "config_drive": "",
    "status": "ACTIVE",
    "updated": "2017-12-08T08:21:45Z",
    "hostId": "aaa",
    "OS-SRV-USG:terminated_at": null,
    "key_name": "pg_ci",
    "properties": "",
    "OS-EXT-AZ:availability_zone": "nova",
    "name": "taaa",
    "created": "2017-12-08T08:21:31Z",
    "os-extended-volumes:volumes_attached": [
        {
            "id": "aaa"
        }
    ]
}

In the original JSON the keys are sorted alphabetically. To do that in the cleaned-up JSON, just pass sort_keys=True to json.dumps .