How to update DynamoDB nested attribute in python?

Question

I have scenario where i need to update an attribute in a nested dynamodb entry.

Bellow is my sample DynamoDB Entry.

The table has: pipeline_name as Partition Key and pipeline_run_id as Sort Key.

Considering i want to use pipeline_name , pipeline_run_id , name and validation_method_results[ columnCheck ] as my predicates to update the value for any of the key value pairs inside my validation_method_results with another string value.

my python function would receive something like:

update_status(pipeline_name,pipeline_run_id,dependency_name,validation_method,'new_value')

So i would provide it with inputs as such(this is to reflect my example bellow):

update_status('demo','demo-2021-04-29T16:48:46.295','dep-demo','columnCheck','new_value')

I am having dificulties grasping the Condition Expressions constructs in DynamoDb, any help would be appreciated.

{
  "force_run": 0,
  "last_updated_ts": "2021-04-29T16:48:46.295",
  "pipeline_dependencies": [
    {
      "name": "dep-demo", // name 
      "status": "inactive",
      "validation_method_results": [
        {
          "columnCheck": "success" // example key to update with 'new_value' 
        },
        {
          "fileexists": "success" // need to be able to update any of this 
        }
      ]
    },
    {
      "name": "dep-demo-1",
      "status": "active",
      "validation_method_results": [
        {
          "columnCheck": "none" // need to be able to update any of this 
        },
        {
          "fileexists": "none" // need to be able to update any of this 
        },
        {
          "othermethod": "none" // need to be able to update any of this 
        }
      ]
    }
  ],
  "pipeline_name": "demo", // Partition Key 
  "pipeline_run_id": "demo-2021-04-29T16:48:46.295", // Sort Key
  "pipeline_run_status": "pending",
  "pipeline_stage": "dependency_validation"
} 

Answer 1

I found the answer/solution for my use case.

I ll post it here in case somebody might want to use it or improve it.

The issue with very nested attributes in dynamodb seems to be finding the right array index location you want to make an update on.

So i had to do a bit of extra work to find those idexes prior to triggering my update_item statement.

I have created this two simple functions to help me with this proccess.

    def find_dep_index_location(lst, key, key_value):
        for i, dic in enumerate(lst):
            if dic[key] == key_value:
                return i
        return -1
    
    def find_validation_index_location(lst, key_value):
        for i, dic in enumerate(lst):
            if key_value in list(dic):
                return i
# update function
def update_ed_pipeline_log_runs_validation_status(log_table, pipeline_name, pipeline_run_id, dependency_name,validation_method, validation_method_state):
    session = boto3.session.Session()
    dynamodb = session.resource('dynamodb')
    dynamotable = dynamodb.Table(log_table)

    # i first have to go and fetch this record from dynamodb first
    # this will be used by help function to find the location idx
    run_config = get_runConf(pipeline_name, pipeline_run_id)
    # Start Find nested array list locations
    lst = run_config['pipeline_dependencies']
    key = 'name'
    key_value = dependency_name
    dep_index = find_dep_index_location(lst, key, key_value)

    lst = put_item['pipeline_dependencies'][dep_index]['validation_method_results']
    key_value = validation_method
    method_index = find_validation_index_location(lst, key_value)
    # End Find nested array list locations
    new_status = {key_value : validation_method_state}

    try:
        dynamotable.update_item(
            Key={
                'pipeline_name': pipeline_name,
                'pipeline_run_id': pipeline_run_id
            },
            UpdateExpression="SET #dep[" + str(dep_index) + "].#method[" + str(method_index) + "] = :locVal",
            ExpressionAttributeNames={
                '#dep': 'pipeline_dependencies',
                '#method': 'validation_method_results'
            },
            ExpressionAttributeValues={
                ':locVal': new_status,
            },
        )
    except Exception as e:
        logging.error(' -- ERROR -- ' + str(e))
        raise e

But again this is for my use case, none the less you can change the keys to apply to your own use case. This look highly unoptimized but it does the work, if anybody there has any ideas how i can by-pass that first get and do my updates in a single step i would appreciate it.

Thx