如何在Python的嵌套defaultdict结构中有效地遍历列表？

Question

I am processing a large amount of urls (from a schedule) and I've categorized them in a nested defaultdict structure as follows:

My categories are:

• Options: 3 possibilities
• Quarter: 4 possibilities
• Weeks: 52 possibilities

The value for weeks should be a list.

My code

def setup_urls(option):

    urls = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))

    for quarter in range(1, 5):
        for week in range( 1, 53):
        // logic of computing url goes here
        links[option][quarter][week].append(url)

    return urls

Output

A list with multiple defaultdict in it.

[defaultdict( < function setup_urls. < locals > . < lambda > at 0x7fd30ed2d488 > , {
    'option': defaultdict( < function setup_urls. < locals > . < lambda > . < locals > . < lambda > at 0x7fd3122a2158 > , {
        1: defaultdict( < class 'list' > , {
            45: [ url1, url2, url3, url4, url5 ]
        }),
    })
})]

I didn't want to use standard dictionaries because of efficieny with large datasets. I have to store around 5000-10000 urls. In the future this could be around 100000.

With some research of my own the usage of defaultdict should be good for performance, but the usage of lambda's doesn't seem to be very Pythonic. Not sure if there are better solutions, but it's not my main question.

I currently how this code to access all urls, but it feels to like a lot of dirty code and specially not very Pythonic at all.

    for dict in result:
        for quarter in dict.values():
            for week in quarter.values():
                for url in week.values():
                    print(url)

I'd like to know what the better way is to access these urls in order to make use of the map-function? (And is this the best way to store the urls?)

Answer 1

You can structure your logic for an arbitrary level of nested dictionaries via a recursive function. Below is an example using itertools.chain .

from collections import defaultdict
from itertools import chain

def get_values(d, res=[]):
    for k, v in d.items():
        if isinstance(v, dict):
            get_values(v, res=res)
        else:
            res.append(v)
    return list(chain.from_iterable(res))

d = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))

d[1][2][3].append(343)
d[1][2][3].append(1245)
d[1][2][4].append(563)
d[1][2][4].append(763)

res = list(get_values(d))
# [343, 1245, 563, 763]