如何编写__dict__的有效重载函数？

Question

I want to implement a to_dict function that behaves similarly to the built-in __dict__ attribute but allows me to have custom logic. (It is used for construct a pandas DataFrame. See the example below. )

However I find out that my to_dict function is ~25% slower than __dict__ even when they do exactly the same thing. How can I improve my code?

class Foo:
    def __init__(self, a,b,c,d):
        self.a = a
        self.b = b
        self.c = c
        self.d = d

    def to_dict(self):
        return {
            'a':self.a,
            'b':self.b,
            'c':self.c,
            'd':self.d,
        }

list_test = [Foo(i,i,i,i)for i in range(100000)]

%%timeit
pd.DataFrame(t.to_dict() for t in list_test)
# Output: 199 ms ± 4.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
pd.DataFrame(t.__dict__ for t in list_test)
# Output: 156 ms ± 948 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

A digress to this question but related to my final goal: what is the most efficient way to construct a pandas DataFrame from a list of custom objects? My current approach is taken from https://stackoverflow.com/a/54975755/1087924

Answer 1

__dict__ does not “convert” an object to a dict (unlike __int__ , __str__ , etc), it's where the object's (writable) attributes are stored.

I think your implementation is reasonably efficient. Consider this simplified example:

import dis

class Foo:
    def __init__(self, a):
        self.a = a
    def to_dict(self):
        return {'a': self.a}

foo = Foo(1)

dis.dis(foo.to_dict)
dis.dis('foo.__dict__')

We can see that Python looks up the attributes and creates a new dict every time (plus you'd need to make a call to .to_dict , not shown here):

  7           0 LOAD_CONST               1 ('a')
              2 LOAD_FAST                0 (self)
              4 LOAD_ATTR                0 (a)
              6 BUILD_MAP                1
              8 RETURN_VALUE

while accessing an existing attribute is much simpler:

  1           0 LOAD_NAME                0 (foo)
              2 LOAD_ATTR                1 (__dict__)
              4 RETURN_VALUE

You could however store your custom representation on the instance, achieving the same exact bytecode as with __dict__ , but then you need to update it correctly on all changes to Foo (which will cost some speed and memory). If updates are uncommon in your use-case, this could be an acceptable trade-off.

In your example, a simple option is to override __getattribute__ , but I'm guessing Foo has other attributes, so having setters is probably going to be more convenient:

class Foo:
    def __init__(self, a):
        self.dict = {}
        self.a = a

    @property
    def a(self):
        return self._a

    @a.setter
    def a(self, value):
        self._a = value
        self.dict['a'] = value

foo = Foo(1)
print(foo.dict)  # {'a': 1}
foo.a = 10
print(foo.dict)  # {'a': 10}