Error - Theano+Keras

Question

I am trying to run a Keras script on an AWS instance. While the script runs fine on my own computer (Python 2.7 - no CPU) it causes some error on a GPU enabled AWS instance. I have installed the latest version of Theano - and other scripts (eg the mnist tutoral) do not give errors.

The script that is causing the issue is a standard Keras tutoral script ( https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py ). I have already moved, based on Stack Overflow guidance, the border mode to "valid" - which seems to remove one issue. however - I immediately encounter the following problem (error stack below). i have run the following line at bash "THEANO_FLAGS=optimizer=fast_compile,device=gpu,floatX=float32 cifar10.py" but this didnt give more information. Maybe i should move to nolearn / lasagne package - but please let me know if there are easy ways to troubleshoot this.

Using Theano backend.
Using gpu device 0: GRID K520 (CNMeM is disabled)
X_train shape: (50000, 3, 32, 32)
50000 train samples
10000 test samples
Using real time data augmentation
----------------------------------------
Epoch 0
----------------------------------------
Training...
Testing...
Traceback (most recent call last):
  File "keras_python_4.py", line 152, in <module>
    score = model.test_on_batch(X_batch, Y_batch)
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 445, in test_on_batch
    return self._test(ins)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/theano_backend.py", line 357, in __call__
    return self.function(*inputs)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 871, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 314, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 859, in __call__
    outputs = self.fn()
ValueError: GpuElemwise. Input dimension mis-match. Input 1 (indices start at 0) has shape[0] == 32, but the output's size on that axis is 16.
Apply node that caused the error: GpuElemwise{Composite{(i0 * log(clip((i1 / i2), i3, i4)))}}[(0, 0)](GpuFromHost.0, GpuSoftmaxWithBias.0, GpuDimShuffle{0,x}.0, CudaNdarrayConstant{[[  1.00000001e-07]]}, CudaNdarrayConstant{[[ 0.99999988]]})
Toposort index: 95
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, col), CudaNdarrayType(float32, (True, True)), CudaNdarrayType(float32, (True, True))]
Inputs shapes: [(16, 10), (32, 10), (32, 1), (1, 1), (1, 1)]
Inputs strides: [(10, 1), (10, 1), (1, 0), (0, 0), (0, 0)]
Inputs values: ['not shown', 'not shown', 'not shown', <CudaNdarray object at 0x7f2165bb7730>, <CudaNdarray object at 0x7f2165bb7970>]
Outputs clients: [[GpuCAReduce{add}{0,1}(GpuElemwise{Composite{(i0 * log(clip((i1 / i2), i3, i4)))}}[(0, 0)].0)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Answer 1

When you change your border_mode from "same" to "valid", the dimension of your feature map will gradually shrink. At some point, the dimension of a layer becomes so small (looks like it became 1x1 already in your case) that you can't add extra layers on it anymore.

If you are specifying certain strides of a layer, the size of the filter and the size of the input also need to be compatible with your stride length.