How to Create a Dynamic Path for Grabbing Variables from JSON in Python

Question

I'm calling many web services in Python which return JSON formatted text describing the same data, but each are structured differently. Rather than write a dozen ways to loop through the data, I'm attempting to create a common base path and common field name, but not understanding now to create a dynamic name to grab the desired field.

service1 calls the street address, "addr1" while service2 calls it, "address1".

service1 JSON path to the street address: ["results"][0]["locations"][0]["addr1"] .

service2 JSON path to the street address: ["query"][0]["address1"] .

service1 response:

{
    "results":[
        {
            "providedLocation":{
                "location":"123 Main St. Whoville"
            },
            "locations":[
                {
                "addr1":"123 Main St.",
                "city":"Whoville"
                }
            ]
        }
    ]
}

From my common terms translation.ini file

[service1]
base_path = '["results"][0]["locations"]' 
field     = "addr1"
[service2]
base_path = '["query"]'
field     = 'address1'

I'm looping through the address locations returned like this:

j         = json.loads(response.text)
try:
    for i in range(0,count_the_results()):
        street_number = j[base_path][i][field]  # multiple locations returned

except KeyError:
    print("street_number error again")

I tried using eval(), but it drops the brackets ([0]) which breaks the path

j[eval(base_path)]

I tried breaking the base_path into pieces but eval still gets me

base_path = '["results"][0]'
locations = '["locations"]'
j[eval(str(base_path)+str(locations))]

I could use keys, but still have to build the unique path somehow

j.get("results",{})[0].get("locations") [i][field]

Could someone point out what I'm missing in creating the path, please?

Answer 1

You could do it like that:

def get_item(response, path):
    for item in path:
        response = response[item]
    return response

# Assume we need the string 'qwerty'
spam = {'spam': ['eggs', {'13': 'qwerty'}], 'eggs': 123}
# Then that will be our path:
path = ['spam', 1, '13']

print get_item(spam, path)

It prints 'qwerty' .

---

The shorted and more efficient (but probably less understandable) way of doing this is to use builtin reduce function like that:

def get_item(response, path):
    return reduce(lambda x, y: x[y], path, response)

---

The other problem is how to store the path in the .ini file. My suggestion would be to write it in JSON format, then pass it to json.loads(), which will get you the path like in my example.

base_path = '["results", 0, "locations"]'

You get the string '["results", 0, "locations"]' from your config (it might be a problem, is it?), pass it to json.loads

path = json.loads(path_string)

and you can use it in get_item function.