Updating complex JSON object in Python

Question

I am grabbing sort of a complex MongoDB document with Python (v3.5) and I should update some values in it which are scattered all around the object and have no particular pattern in the structure and save it back to a different MongoDB collection. The object looks like this:

# after json.loads(mongo_db_document) my dict looks like this
notification = {
    '_id': '570f934f45213b0d14b1256f',
    'key': 'receipt',
    'label': 'Delivery Receipt',
    'version': '0.0.1',
    'active': True,
    'children': [
        {
            'key': 'started',
            'label': 'Started',
            'children': [
                'date',
                'time',
                'offset'
            ]
        },
        {
            'key': 'stop',
            'label': 'Ended',
            'children': [
                'date',
                'time',
                'offset'
            ]
        },
        {
            'label': '1. Particulars',
            'template': 'formGroup',
            'children': [
                {
                    'children': [
                        {
                            'key': 'name',
                            'label': '2.1 Name',
                            'value': '********** THIS SHOULD BE UPDATED **********',
                            'readonly': 'true'
                        },
                        {
                            'key': 'ims_id',
                            'label': '2.2 IMS Number',
                            'value': '********** THIS SHOULD BE UPDATED **********',
                            'readonly': 'true'
                        }
                    ]
                },
                {
                    'children': [
                        {
                            'key': 'type',
                            'readonly': '********** THIS SHOULD BE UPDATED **********',
                            'label': '2.3 Type',
                            'options': [
                                {
                                    'label': 'Passenger',
                                    'value': 'A37'
                                },
                                {
                                    'label': 'Cargo',
                                    'value': 'A35'
                                },
                                {
                                    'label': 'Other',
                                    'value': '********** THIS SHOULD BE UPDATED **********'
                                }
                            ]
                        }
                    ]
                }
            ]
        },
        {
            'template': 'formGroup',
            'key': 'waste',
            'label': '3. Waste',
            'children': [
                {
                    'label': 'Waste',
                    'children': [
                        {
                            'label': 'Plastics',
                            'key': 'A',
                            'inputType': 'number',
                            'inputAttributes': {
                                'min': 0
                            },
                            'value': '********** THIS SHOULD BE UPDATED **********'
                        },
                        {
                            'label': 'B. Oil',
                            'key': 'B',
                            'inputType': 'number',
                            'inputAttributes': {
                                'min': 0
                            },
                            'value': '********** THIS SHOULD BE UPDATED **********'
                        },
                        {
                            'label': 'C. Operational',
                            'key': 'C',
                            'inputType': 'number',
                            'inputAttributes': {
                                'min': 0
                            },
                            'value': '********** THIS SHOULD BE UPDATED **********'
                        }
                    ]
                }
            ]
        },
        {
            'template': 'formRow',
            'children': [
                'empty',
                'signature'
            ]
        }
    ],
    'filter': {
        'timestamp_of_record': [
            'date',
            'time',
            'offset'
        ]
    }
}

My initial idea was to put placeholders (like $var_name ) in places where I need to update values, and load the string with Python's string.Template , but that approach unfortunately breaks lots of stuff to other users of the same MongoDB document for some reason.

Is there a solution to simply modify this kind of object without "hardcoding" path to find the values I need to update?

Answer 1

Not sure if I understood correctly, but this will dynamically find all keys "value" and "readonly" and print out the paths to address the fields.

def findem(data, trail):
    if isinstance(data, dict):
        for k in data.keys():
            if k in ('value', 'readonly'):
                print("{}['{}']".format(trail, k))
            else:
                findem(data[k], "{}['{}']".format(trail, k))
    elif isinstance(data, list):
        for k in data:
            findem(k, '{}[{}]'.format(trail, data.index(k)))

if __name__ == '__main__':
    findem(notification, 'notification')

notification['children'][2]['children'][0]['children'][0]['readonly']
notification['children'][2]['children'][0]['children'][0]['value']
notification['children'][2]['children'][0]['children'][1]['readonly']
notification['children'][2]['children'][0]['children'][1]['value']
notification['children'][2]['children'][1]['children'][0]['readonly']
notification['children'][2]['children'][1]['children'][0]['options'][0]['value']
notification['children'][2]['children'][1]['children'][0]['options'][1]['value']
notification['children'][2]['children'][1]['children'][0]['options'][2]['value']
notification['children'][3]['children'][0]['children'][0]['value']
notification['children'][3]['children'][0]['children'][1]['value']
notification['children'][3]['children'][0]['children'][2]['value']

Answer 2

There's this small script that I had written a couple years ago - I used it to find entries in some very long and unnerving JSONs. Admittedly it's not beautiful, but it might help in your case, perhaps?
You can find the script on Bitbucket, here (and here is the code). Unfortunately it's not documented; at the time I wasn't really believing other people would use it, I guess.
Anyways, if you'd like to try it, save the script in your working directory and then use something like this:

from RecursiveSearch import Retriever

def alter_data(json_data, key, original, newval):
    '''
    Alter *all* values of said keys
    '''
    retr = Retriever(json_data)
    for item_no, item in enumerate(retr.__track__(key)): # i.e. all 'value'
        # Pick parent objects with a last element False in the __track__() result,
        # indicating that `key` is either a dict key or a set element
        if not item[-1]: 
            parent = retr.get_parent(key, item_no)
            try:
                if parent[key] == original:
                    parent[key] = newval
            except TypeError:
                # It's a set, this is not the key you're looking for
                pass

if __name__ == '__main__':
    alter_data(notification, key='value', 
               original = '********** THIS SHOULD BE UPDATED **********',
               newval = '*UPDATED*')

Unfortunately as I said the script isn't well documented, so if you want to try it and need more info, I'll be glad to provide it.

Answer 3

Add another list to the JSON object. Each item in that list would be a list of keys that lead to the values to be changed. An example for one such list is: ['children', 2, 'children', 'children', 0, 'value'] . Then, to access the value you could use a loop:

def change(json, path, newVal):
    cur = json
    for key in path[:-1]:
        cur = cur[key]
    cur[path[-1]] = newVal

path = notification['paths'][0]
#path, for example, could be ['children', 2, 'children', 'children', 0, 'value']
newVal = 'what ever you want'
change(notification, path, newVal)