帶有12GB RAM的Numpy陣列內存錯誤

Question

以下是當計數器達到約53009525時向我顯示“內存錯誤”的代碼段。我正在具有12GB內存的Ubuntu虛擬機上運行此代碼。

from collections import defaultdict
from collections import OrderedDict
import math
import numpy as np
import operator
import time
from itertools import product

class State():
   def __init__(self, field1, field2, field3, field4, field5, field6):
       self.lat = field1
       self.lon = field2
       self.alt = field3
       self.temp = field4
       self.ws = field5
       self.wd = field6
...
trans = defaultdict(dict)
freq = {}
...
matrix_col = {}
matrix_col = OrderedDict(sorted(freq.items(), key=lambda t: t[0].lat,    reverse=True))

trans_mat = []
counter = 0 
for u1, u2 in product(matrix_col, matrix_col):
    print counter, time.ctime()
    if (u1 in trans) and (u2 in trans[u1]):
        trans_mat.append([trans[u1][u2]*1.0])
    else:
        trans_mat.append([[0.0001]])
    counter += 1

trans_mat = np.asarray(trans_mat)
trans_mat = np.reshape(trans_mat, (10734, 10734))
print trans_mat

freq和trans都存儲“狀態”類型。 感謝螞蟻的幫助。 這是錯誤：...

53009525 Mon Oct 12 18:11:16 2015
Traceback (most recent call last):
  File "hmm_freq.py", line 295, in <module>
     trans_mat.append([[0.0001]])
MemoryError

Answer 1

似乎在每次迭代中，您都將在兩個嵌套列表內的Python浮點數附加到trans_mat 。 您可以使用sys.getsizeof檢查每個元素的大小：

import sys

# each of the two nested lists is 80 bytes
print(sys.getsizeof([]))
# 80

# a Python float object is 24 bytes
print(sys.getsizeof(0.0001))
# 24

在每次迭代中，您都將2 * 80 + 24 = 184個字節附加到trans_mat 。 在53009525迭代之后，您將追加9753752600字節或9.75GB。

一種使內存效率更高的非常簡單的方法是將結果直接存儲到numpy數組中，而不是存儲在嵌套列表中：

trans_mat = np.empty(10734 * 10734, np.double)
counter = 0
for u1, u2 in product(matrix_col, matrix_col):
    print counter, time.ctime()
    if (u1 in trans) and (u2 in trans[u1]):

        # you may need to check that this line yields a float
        # (it's hard for me to tell exactly what `trans` is from your code snippet)
        trans_mat[counter] = trans[u1][u2]*1.0

    else:
        trans_mat[counter] = 0.0001
    counter += 1

以供參考：

# the size of the Python container is 80 bytes
print(sys.getsizeof(trans_mat))
# 80

# the size of the array buffer is 921750048 bytes or 921 MB
print(trans_mat.nbytes)
# 921750048