Python nested yield in for loop

Question

I have a program i need to write that, it works like this:

1. send POST request to get token
2. use token to GET all main categories in ecommerce platform
3. then i have to travel down all categories and get all the leaves - that is final categories in tree without any subcategory - that also means i send GET request for each subcategory and then check if it returns "leaf" == True (this is flag in returned JSON)
4. if it is a "leaf", it is a final action and for them I have to GET all required parameters
5. subcategories might be of varying lenghts - eg"Electronics > Smartphone accessories > Earphones > Wireless" = length 4, but "House and garden > garden tools > lawn mowers" = length 3
6. I want to save it all in JSON

This should expand to nice nested JSON that is a collection of all categories and their subcategories. That on the end points - that is final subcategories there shoud be "parameters": Table that stores all the parameters.

So to me (and I have started seriously learning programming with Scheme/Racket) it seems like a recursive problem that should call itself in all subcategories and then end in the "leaf" for which I just make a last call to GET parameters and end recurrsion. I use yield statement instead of return with for loop inside of recurrsive function...... Yes this is what I thought should work.

def get_all_leaves(category_tree):
for category in category_tree["categories"]:
    if category["leaf"] == True:
        request_parameters = requests.get("https://api.allegro.pl.allegrosandbox.pl/sale/categories/" + category["id"] + "/parameters",
                                            headers=headers_main_category_tree)
        dict_parameters = json.loads(request_parameters.text)
        yield {"name":category["name"], "id":category["id"], "parameters":dict_parameters["parameters"]}
    else:
        request_id = category["id"]
        next_category_tree = requests.get("https://api.allegro.pl.allegrosandbox.pl/sale/categories?parent.id=" + request_id, 
                                           headers=headers_main_category_tree)
        dict_next_category_tree = json.loads(next_category_tree.text)

        yield {"name":category["name"], "id":category["id"], "subcategory":get_all_leaves(dict_next_category_tree)}

And this a result:

And this is result of print(list(get_all_leaves(json_main_category_tree)))

[{{'name': 'Dom i Ogród', 'id': '5', 'subcategory': <generator object get_all_leaves at 0x03F7D6F0>},
{'name': 'Dziecko', 'id': '11763', 'subcategory': <generator object get_all_leaves at 0x03F7D878>},
{'name': 'Elektronika', 'id': '42540aec-367a-4e5e-b411-17c09b08e41f', 'subcategory': <generator object get_all_leaves at 0x03F7D808>},
{'name': 'Firma i usługi', 'id': '4bd97d96-f0ff-46cb-a52c-2992bd972bb1', 'subcategory': <generator object get_all_leaves at 0x03F7DA70>},
...
...
...
{'name': 'Wyposażenie', 'id': '123', 'subcategory': <generator object get_all_leaves at 0x03FCF760>}]

That is, it worked in someway, it looks like a recurssion but it put generators inside a generator? They are not expanding themselves.

Answer 1

yield {"name":category["name"], "id":category["id"], "subcategory":get_all_leaves(dict_next_category_tree)}

The recursive call creates a generator object, which is stored in the dict that is yielded.

To get fully expanded results, the simplest thing is to just expand the result from the recursive call here, thus:

yield {"name":category["name"], "id":category["id"], "subcategory":list(get_all_leaves(dict_next_category_tree))}

In general you can pass generators to list to expand them, without needing an explicit loop. It works because the list constructor expects any sort of iterable - which generators are - and does the iteration and builds the list of iterated-over items. (As another example, the readlines() method of file objects is unnecessary; you can just pass the open file object to list in the same way.)