对象的Python列表占用了太多内存

Question

I have the following code, that creates a million objects of a class foo:

for i in range(1000000):
    bar = foo()
    list_bar.append(bar)

The bar object is only 96 bytes, as determined by getsizeof() . However, the append step takes almost 8GB of ram. Once the code exits the loop, the ram usage drops to expected amounts (size of the list + some overhead ~103MB). Only while the loop is running does the ram usage skyrocket. Why does this happen? Any workarounds? PS: Using a generator is not an option, it has to be a list.

EDIT: xrange doesn't help, using Python 3. The memory usage stays high only during the loop execution, and drops after the loop is through. Could append have some non-obvious overhead?

Answer 1

Most probably this is due to some unintended cyclical references made by the foo() constructor; as normally Python objects will release memory instantly when the reference count drops to zero; now these would be freed later when the garbage collector gets a chance to run.

You can try to force the GC run after say 10000 iterations to see if it keeps the memory usage constant.

import gc
n = 1000000
list_bar = [ None ] * n
for i in range(n):
    list_bar[i] = foo()
    if i % 10000 == 0:
        gc.collect()

If this relieves memory pressure then the memory usage is because of some reference cycles.

---

The resizing of a list has some overhead. If you know how many elements, then you can create the list beforehand, eg:

list_bar = [ foo() for _ in xrange(1000000) ]

should know the size of the array and not need to resize it; or create the list filled with None :

n = 1000000
list_bar = [ None ] * n
for i in range(n):
    list_bar[i] = foo()

---

append should be using realloc to grow the list, but old memory ought to be released as soon as possible; and all in all the overhead of all memory allocated should not sum to 8G for a list that is 100 MB at the end; it can be possible that the operating system is miscalculating the memory used.

Answer 2

How are you measuring the memory usage?

I suspect your usage of a 3rd party module might be the cause. Perhaps the 3rd party module is temporarily using a lot of memory when initialised.

Besides, sys.getsizeof() is not an accurate indication of the memory used by an object.

For example:

from sys import getsizeof

class A(object):
    pass

class B(object):
    def __init__(self):
        self.big = 'a' * 1024*1024*1024    # approx. 1 GiB

>>> getsizeof(A)
976
>>> a = A()
>>> getsizeof(a)
64
>>> 
>>> getsizeof(B)
976
>>> b = B()
>>> getsizeof(b)
64
>>> getsizeof(b.big)
1073741873

After instantiating b = B() , top reports approx 1GiB resident memory usage. Obviously this is not reflected by getsizeof(b) which returns only 64 bytes.