Grab all nested keys in json structure

Question

Let's say I read in the following json file.

  text = "NASCAR"
  with urllib.request.urlopen(f'https://en.wikipedia.org/w/api.php?action=query&generator=search&gsrsearch=morelike:{text}&format=json') as url:
        more_like_data = json.loads(url.read().decode())

I am trying to extract each of the "titles" contained in query >> pages >> [random page number] and store that in a list. My attempt to do so looked like this

  more_like_titles = list([page_number.get('title') for page_number in more_like_data.get('query').get('pages')])

  print(more_like_titles)

I get the error

"AttributeError: 'str' object has no attribute 'get'"

I'm not sure why it's reading the value in as a string, as in the JSON file that is loaded it clearly appears as a dictionary. See here:

{'batchcomplete': '',
 'continue': {'continue': 'gsroffset||', 'gsroffset': 10},
 'query': {'pages': {'147515': {'index': 6,
                                'ns': 0,
                                'pageid': 147515,
                                'title': 'NASCAR Xfinity Series'},
                     '14855318': {'index': 4,
                                  'ns': 0,
                                  'pageid': 14855318,
                                  'title': 'Criticism of NASCAR'},
                     '17138753': {'index': 9,
                                  'ns': 0,
                                  'pageid': 17138753,
                                  'title': 'List of NASCAR drivers who have '
                                           'won in each of top three series'},
                     '2201365': {'index': 5,
                                 'ns': 0,
                                 'pageid': 2201365,
                                 'title': 'Buschwhacker'},
                     '35514289': {'index': 1,
                                  'ns': 0,
                                  'pageid': 35514289,
                                  'title': 'List of female NASCAR drivers'},
                     '40853273': {'index': 7,
                                  'ns': 0,
                                  'pageid': 40853273,
                                  'title': 'Daniel Hemric'},
                     '43410277': {'index': 10,
                                  'ns': 0,
                                  'pageid': 43410277,
                                  'title': '2015 NASCAR Camping World Truck '
                                           'Series'},
                     '47112554': {'index': 8,
                                  'ns': 0,
                                  'pageid': 47112554,
                                  'title': 'Ryan Preece'},
                     '47828021': {'index': 3,
                                  'ns': 0,
                                  'pageid': 47828021,
                                  'title': '2016 NASCAR Xfinity Series'},
                     '5082163': {'index': 2,
                                 'ns': 0,
                                 'pageid': 5082163,
                                 'title': 'NASCAR Whelen Modified Tour'}}}}

Any thoughts?

Answer 1

When you're having trouble with a list comprehension, breaking it down is probably a good idea. That being said, your issue was that you were trying to iterate over a dictionary directly which can give some unexpected results. I've fixed your list comprehension below using pythons built in .items

more_like_titles = list([vals.get('title') for page_number, vals in more_like_data.get('query').get('pages').items()])