NN 数字识别 - 内存错误，有什么想法吗？

Question

我试图为数字制作训练有素的 OCR，但我不断收到内存分配错误并且似乎无法弄清楚出了什么问题，开始时工作得很好，但在运行 2 次后它突然开始吐出这些错误，尝试删除 zips 和重新下载可能损坏的文件，但没有结果甚至无法完成首次运行。

定位错误：

Traceback (most recent call last):
  File "/home/pcname/PycharmProjects/Bot/venv/lib/python3.6/site-packages/theano/compile/function_module.py", line 607, in __call__
    outputs = self.fn()
MemoryError: Unable to allocate array with shape (60000, 800) and data type float64

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pcname/PycharmProjects/Bot/Utility/TextHandler.py", line 90, in <module>
    train_err = train_fn(training_x, training_y)
  File "/home/pcname/PycharmProjects/Bot/venv/lib/python3.6/site-packages/theano/compile/function_module.py", line 618, in __call__
    storage_map=self.fn.storage_map)
  File "/home/pcname/PycharmProjects/Bot/venv/lib/python3.6/site-packages/theano/gof/link.py", line 269, in raise_with_op
    storage_map_list.sort(key=itemgetter(3), reverse=True)
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'

代码：

import os
import urllib.request
import gzip
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

matplotlib.use("TkAgg")
import lasagne
import theano
import theano.tensor as T

def build_nn(input_var=None):
        l_in = lasagne.layers.InputLayer(shape=(None, 1, 28, 28), input_var=input_var)
        l_in_drop = lasagne.layers.DropoutLayer(l_in, p=0.2)

        # Verborgen lagen.
        l_hid1 = lasagne.layers.DenseLayer(l_in_drop, num_units=800,
                                           nonlinearity=lasagne.nonlinearities.rectify,
                                           W=lasagne.init.GlorotUniform())

        l_hid1_drop = lasagne.layers.DropoutLayer(l_hid1, p=0.5)

        l_hid2 = lasagne.layers.DenseLayer(l_hid1_drop, num_units=800,
                                           nonlinearity=lasagne.nonlinearities.rectify,
                                           W=lasagne.init.GlorotUniform())

        l_hid2_drop = lasagne.layers.DropoutLayer(l_hid2, p=0.5)

        l_out = lasagne.layers.DenseLayer(l_hid2_drop, num_units=10,
                                          nonlinearity= lasagne.nonlinearities.softmax)

        return l_out


def load_dataset(file_path="../Lib/Zips/"):
    first_download = True
    def download(filename, source='http://yann.lecun.com/exdb/mnist/'):
        print("Downloading set: %s" % filename)
        urllib.request.urlretrieve(source+filename, file_path+filename)

    def load_imageset(file):
        # Indien het bestand nog niet gevonden is downloaden
        if not os.path.exists(file_path+file):
            download(file)

        # Openen van archief training set afbeeldingen.
        with gzip.open(file_path+file, 'rb') as reader:
            data = np.frombuffer(reader.read(), np.uint8, offset=16)
            data = data.reshape(-1, 1, 28, 28)

            return data/np.float32(256)

    def load_labelset(file):
        if not os.path.exists(file_path+file):
            download(file)

        with gzip.open(file_path+file, "rb") as reader:
            data = np.frombuffer(reader.read(), np.uint8, offset=8)

        return data

    training = ("train-images-idx3-ubyte.gz", "train-labels-idx1-ubyte.gz" )
    test = ("t10k-images-idx3-ubyte.gz", "t10k-labels-idx1-ubyte.gz")
    return load_imageset(training[0]),load_labelset(training[1]), load_imageset(test[0]), load_labelset(test[1])


x_train, y_train, x_test, y_test = load_dataset()


input_var = T.tensor4('inputs')
target_var = T.ivector('targets')

network = build_nn(input_var)

prediction = lasagne.layers.get_output(network)
loss = lasagne.objectives.categorical_crossentropy(prediction, target_var)
loss = loss.mean()

params = lasagne.layers.get_all_params(network, trainable=True)
updates = lasagne.updates.nesterov_momentum(loss, params, learning_rate=0.01, momentum=0.9)

theano.config.optimizer='fast_compile'
theano.config.exception_verbosity='high'
theano.config.compute_test_value = 'warn'

train_fn = theano.function([input_var, target_var], loss, updates=updates)
num_training_steps = 10

for step in range(num_training_steps):
    train_err = train_fn(x_train, y_train)

Answer 1

您的内存错误可能是因为您试图同时通过网络放置所有数据。

你的代码：

for step in range(num_training_steps):
    train_err = train_fn(x_train, y_train)

更好的训练方法是使用小批量。 文档在这里： https ://lasagne.readthedocs.io/en/latest/user/tutorial.html

要快速试验您是否有足够的内存来处理小批量，您可以尝试将上面的代码替换为：

train_err = train_fn(x_train[0:8], y_train[0:8])

其中 8 将模拟 8 的批处理大小。如果代码运行无误，您可以确信一旦您浏览了文档链接，它就会运行良好。

你需要做的是实现一个生成器函数iterate_minibatches

然后像这样训练...

batch_size = 8 #you can edit
num_epochs = 4 #you can edit
for epoch in range(num_epochs):
    for batch in iterate_minibatches(x_train, y_train, batch_size, shuffle=True):
        inputs, targets = batch
        train_fn(inputs, targets)

或者尝试一些有点老套的东西（用这个替换你的代码）：

batch_size = 8
ix = 0
for step in range(num_training_steps):
    train_err = train_fn(x_train[ix*batch_size:ix*(batch_size + 1)], y_train[ix*batch_size:ix*(batch_size + 1)])

注意..未经测试的代码没有任何错误检查或数组边界检查......只是一些让你继续前进的想法。

Answer 2

我同意 guidot，这可能是正常的内存不足错误。

您的输入是一个 384MB 的数组，后面是一些 dropout 和宽度很大的密集层。

也许将前两个 Dense 层从 800 减少到其他值，然后再试一次。 也许将它减少到 600 并查看它是否运行更长时间但最终给出相同的错误将提供更多证据。