python - using copy.deepcopy on dotdict

Question

I have used dotdict in various locations around my app to enhance readability of my code. Little did I know that this would cause many problems down the road. One particularly annoying case is the fact that it does not seem to be compatible with the copy library.

This is what I mean by dotdict

class DotDict(dict):
    """dot.notation access to dictionary attributes"""
    __getattr__ = dict.get
    __setattr__ = dict.__setitem__
    __delattr__ = dict.__delitem__

ie a way of accessing dictionary attributes as such: dictionary.attribute

When I try

nested_dico = DotDict({'example':{'nested':'dico'}})
copy.deepcopy(nested_dico)

I get the following error:

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
    167                     reductor = getattr(x, "__reduce_ex__", None)
    168                     if reductor:
--> 169                         rv = reductor(4)
    170                     else:
    171                         reductor = getattr(x, "__reduce__", None)

TypeError: 'NoneType' object is not callable

I assume this is because it does not recognise my class DotDict and thus considers it to be NoneType.

Does anyone know a way around this? Maybe override the copy library's valid types?

Answer 1

Looking into the spot where the error occurs, reductor is not None , but it's a built-in function, meaning the error occurs somewhere in C code where we can't see a traceback. But my guess is that it tries to get a method that it's not sure exists, ready to catch an AttributeError if not. Instead this calls .get which returns None without an error, so it tries to call the method.

This behaves correctly:

class DotDict(dict):
    """dot.notation access to dictionary attributes"""

    def __getattr__(self, item):
        try:
            return self[item]
        except KeyError as e:
            raise AttributeError from e

    __setattr__ = dict.__setitem__
    __delattr__ = dict.__delitem__

a = DotDict(x=1, b=2)

print(deepcopy(a))

I know that's not the same behaviour as your original class, and you won't be able to use . for optional keys, but I think it's necessary to avoid errors like this with external code that expects your class to behave in a predictable way.

EDIT: here is a way to implement deepcopy and preserve the original __getattr__ :

class DotDict(dict):
    """dot.notation access to dictionary attributes"""

    __getattr__ = dict.get
    __setattr__ = dict.__setitem__
    __delattr__ = dict.__delitem__

    def __deepcopy__(self, memo=None):
        return DotDict(deepcopy(dict(self), memo=memo))

Tests:

dd = DotDict(x=1, b=2, nested=DotDict(y=3))

copied = deepcopy(dd)
print(copied)
assert copied == dd
assert copied.nested == dd.nested
assert copied.nested is not dd.nested
assert type(dd) is type(copied) is type(dd.nested) is type(copied.nested) is DotDict

Answer 2

implementing a copy for custom data structure is fairly easy, smth like so

def my_dict_copy(d):
    return {k:v for k,v in d.items()}

d1 = { 'a' :1,'b':2}
d2 = my_dict_copy(d1)
d3 = d1
d1['a'] = 2

print(d1)
print(d2)
print(d3)

output

{'a': 2, 'b': 2}
{'a': 1, 'b': 2}
{'a': 2, 'b': 2}

you didn't provided your implementation of dict, I assume it respond to class methods like items() if not, there must be a way you can iterate all the keys and values

also I assume the key and values are immutable objects and not data structure otherwise you need to make a copy of 'k' and 'v' also (maybe using the origin copy lib)