Ho to read AWS Athena response effectively in Python Lambda

Question

I get below type response from Athena:

{
    'UpdateCount': 0,
    'ResultSet': {
        'Rows': [{
            'Data': [{
                'VarCharValue': 'cc_code'
            }, {
                'VarCharValue': 'inv_id'
            }, {
                'VarCharValue': 'sno'
            }, {
                'VarCharValue': 'lrt'
            }, {
                'VarCharValue': 'lat'
            }, {
                'VarCharValue': 'long'
            }, {
                'VarCharValue': 'add'
            }]
        }, {
            'Data': [{
                'VarCharValue': 'YYY'
            }, {
                'VarCharValue': '2222'
            }, {
                'VarCharValue': '20:0100'
            }, {
                'VarCharValue': '2019-12-18T02:03:32Z'
            }, {
                'VarCharValue': '144.9194613051093'
            }, {
                'VarCharValue': '-22.879158430049'
            }, {
                'VarCharValue': 'UK'
            }]
        }, {
            'Data': [{
                'VarCharValue': 'ZZZZ'
            }, {
                'VarCharValue': '3333'
            }, {
                'VarCharValue': '10:010030015943960'
            }, {
                'VarCharValue': '2020-02-27T23:34:50Z'
            }, {
                'VarCharValue': '45.5014953316149'
            }, {
                'VarCharValue': '-122.562085095768'
            }, {
                'VarCharValue': 'IND'
            }]
        }],
        'ResultSetMetadata': {
            'ColumnInfo': [{
                'CatalogName': 'hive',
                'SchemaName': '',
                'TableName': '',
                'Name': 'cc_code',
                'Label': 'cc_code',
                'Type': 'varchar',
                'Precision': 2147483647,
                'Scale': 0,
                'Nullable': 'UNKNOWN',
                'CaseSensitive': True
            }, {
                'CatalogName': 'hive',
                'SchemaName': '',
                'TableName': '',
                'Name': 'inv_id',
                'Label': 'inv_id',
                'Type': 'varchar',
                'Precision': 2147483647,
                'Scale': 0,
                'Nullable': 'UNKNOWN',
                'CaseSensitive': True
            }, {
                'CatalogName': 'hive',
                'SchemaName': '',
                'TableName': '',
                'Name': 'sno',
                'Label': 'sno',
                'Type': 'varchar',
                'Precision': 2147483647,
                'Scale': 0,
                'Nullable': 'UNKNOWN',
                'CaseSensitive': True
            }, {
                'CatalogName': 'hive',
                'SchemaName': '',
                'TableName': '',
                'Name': 'lrt',
                'Label': 'lrt',
                'Type': 'varchar',
                'Precision': 2147483647,
                'Scale': 0,
                'Nullable': 'UNKNOWN',
                'CaseSensitive': True
            }, {
                'CatalogName': 'hive',
                'SchemaName': '',
                'TableName': '',
                'Name': 'lat',
                'Label': 'lat',
                'Type': 'varchar',
                'Precision': 2147483647,
                'Scale': 0,
                'Nullable': 'UNKNOWN',
                'CaseSensitive': True
            }, {
                'CatalogName': 'hive',
                'SchemaName': '',
                'TableName': '',
                'Name': 'long',
                'Label': 'long',
                'Type': 'varchar',
                'Precision': 2147483647,
                'Scale': 0,
                'Nullable': 'UNKNOWN',
                'CaseSensitive': True
            }, {
                'CatalogName': 'hive',
                'SchemaName': '',
                'TableName': '',
                'Name': 'add',
                'Label': 'add',
                'Type': 'varchar',
                'Precision': 2147483647,
                'Scale': 0,
                'Nullable': 'UNKNOWN',
                'CaseSensitive': True
            }]
        }
    },
    'ResponseMetadata': {
        'RequestId': '2b9dabae',
        'HTTPStatusCode': 200,
        'HTTPHeaders': {
            'x-amzn-requestid': '2b9dabae-1',
            'content-type': 'application/x-amz-json-1.1',
            'content-length': '7500',
            'date': 'Tue, 03 Mar 2020 08:19:45 GMT'
        },
        'RetryAttempts': 0
    }
}

I want to get some values from this response and do some further processing with those attributes. I tried with below approach but it seems not a clean approach.

query_result['ResultSet']['Rows'][1]['Data'][1]['VarCharValue']

Problem is first "Data" is coming of Column names and actual data comes from Data[1- so on]. I don't want to put 2 for loops to read my attributes and also don't want to get columns by giving hard coded indexes.Is there any better way to to read Athena response in python lambda.

Answer 1

You have to use indexes in loop or out of loop, but to structure your json you can use a helper method like

import json
from types import SimpleNamespace as Namespace
def json2obj(data):
    return json.loads(data, object_hook=lambda d: Namespace(**d))

Call this method and it will parse your json like for example

parsed_result = json2obj(query_result)

Now you can access your values with a dot like

parsed_result.UpdateCount
parsed_result.ResponseMetadata.RequestId