setdefault vs defaultdict performance

Question

I am writing code for an application where performance is important. I am wondering why defaultdict seems to be faster then setdefault .

I would like to be able to use setdefault , mostly because i do not like the print output of the nested defaultdict (see implementation below).

In my code, i need to test if element_id is already a key of the dict.

Here are the two functions that i am testing:

def defaultdictfunc(subcases,other_ids,element_ids):
    dict_name= defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
    for subcase in subcases:
        for other_id in other_ids:
            for element_id in element_ids: 
                if element_id in dict_name[subcase][other_id]:
                    # error duplicate element_id
                    pass
                else:
                    dict_name[subcase][other_id][element_id]=0
    return dict_name

def setdefaultfunc(subcases,other_ids,element_ids):
    dict_name={}
    for subcase in subcases:
        for other_id in other_ids:
            for element_id in element_ids: 
                if element_id in dict_name.setdefault(subcase,{}).setdefault(other_id,{}):
                    # error duplicate element_id
                    pass
                else:
                    dict_name[subcase][other_id][element_id]=0

    return dict_name

IPython input and output:

In [1]: from numpy.random import randint

In [2]: subcases,other_ids,element_ids=(randint(0,100,100),randint(0,100,100),randint(0,100,100))

In [5]: from collections import defaultdict

In [6]: defaultdictfunc(subcases,other_ids,element_ids)==setdefaultfunc(subcases,other_ids,element_ids)
Out[6]: True

In [7]: %timeit defaultdictfunc(subcases,other_ids,element_ids)
10 loops, best of 3: 177 ms per loop

In [8]: % timeit setdefaultfunc(subcases,other_ids,element_ids)
1 loops, best of 3: 351 ms per loop

Why is setdefaultfunc slower. I thought the underlying code would be the same. Is there a way to improve its speed?

Thanks

Answer 1

According to user aneroid :

It would make sense that defaultdict is faster that dict.setdefault() since the former sets its default for the entire dict at creation time, whereas setdefault() does it per element when it is read. One reason to use setdefault is when the default you assign is based on the key (or something) rather than a generic default for the entire dict.

Answer 2

The setdefaultfunc is worst because you call the dict constructor several times in the loop (since {} is equivalent to dict() ), while defaultdict avoid this by its own design.

With a small change, you can easily improve the setdefaultfunc :

def setdefaultfunc2(subcases,other_ids,element_ids):
    dict_name={}
    for subcase in subcases:
        subcase_dict = dict_name.setdefault(subcase,{})
        for other_id in other_ids:
            other_id_dict = subcase_dict.setdefault(other_id,{})
            for element_id in element_ids: 
                if element_id in other_id_dict:
                    # error duplicate element_id
                    pass
                else:
                    other_id_dict[element_id]=0
    return dict_name

With this change, the results in my machine were:

In [37]: defaultdictfunc(subcases,other_ids,element_ids)==setdefaultfunc2(subcases,other_ids,element_ids)
Out[37]: True

In [38]: %timeit defaultdictfunc(subcases,other_ids,element_ids)
286 ms ± 8.55 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [39]: %timeit setdefaultfunc(subcases,other_ids,element_ids)
434 ms ± 1.78 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [40]: %timeit setdefaultfunc2(subcases,other_ids,element_ids)
174 ms ± 348 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

IMO, defaultdict does not provide enought performance gain to make worth using it.

Answer 3

val = 20_000_000


def defaultdict():
    """
    defaultdict 1000000: 0.4460279941558838
    defaultdict 10000000: 4.371468782424927
    defaultdict 20000000: 8.807381391525269
    """
    from collections import defaultdict
    import time
    a = defaultdict(list)
    t = time.time()
    for i in range(val):
        key = i % (val / 2)
        a[key].append(i)
    print(f'defaultdict {val}:', time.time() - t)


def setdefault():
    """
    setdefault 1000000: 0.3767530918121338
    setdefault 10000000: 4.230009078979492
    setdefault 20000000: 8.19938588142395
    """
    import time
    a = {}
    t = time.time()
    for i in range(val):
        key = i % (val / 2)
        a.setdefault(key, []).append(i)
    print(f'setdefault {val}:', time.time() - t)

Python 3.10