Using LSTM layers in combination with CNN

Question

I am trying to improve the accuracy of an EEG classifier. Currently I was only classifying with conv layers and fully connected ones. In the literature I found people using LSTM layers in their CNN model. I would like to try this out but I get following error.

The data that I'm using is time series EEG data 64 channels x 325 samples (sampled at 500 Hz so 650 ms). X = (1923,63,325,1) and y = (1923,)

model = Sequential()

model.add(TimeDistributed(Conv2D(64, (1, 3), input_shape=X.shape[1:])))
model.add(Activation('relu'))

model.add(TimeDistributed(Conv2D(64, (1, 3))))
model.add(Activation('relu')) 
model.add(MaxPooling2D(pool_size=(1, 2)))

model.add(TimeDistributed(Conv2D(64, (1, 3))))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(1, 2)))

model.add(TimeDistributed(Flatten()))

model.add(LSTM(128,return_sequences=True))

model.add(LSTM(128,return_sequences=True))

model.add(Dense(16))
model.add(Activation('relu'))
model.add(Dropout(0.3)) 

model.add(Dense(2))
model.add(Activation('softmax'))

model.compile(loss='sparse_categorical_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])
return model

The error that I'm getting:

IndexError: list index out of range

I have seen some other questions about this online but most of them do not really apply on my application I think.

Edit: error gets thrown

  File "<ipython-input-8-0b3d7307ea53>", line 1, in <module>
    runfile('D:/ AA TestPrograms/LALALAL/ModelV20.py', wdir='D:/ AA TestPrograms/LALALAL')

  File "d:\anaconda\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 668, in runfile
    execfile(filename, namespace)

  File "d:\anaconda\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 108, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "D:/ AA TestPrograms/LALALAL/ModelV20.py", line 115, in <module>
    history = train_model(model, xtrain, ytrain, xval, yval)

  File "D:/ AA TestPrograms/LALALAL/ModelV20.py", line 94, in train_model
    shuffle=True

  File "d:\anaconda\lib\site-packages\keras\engine\training.py", line 950, in fit
    batch_size=batch_size)

  File "d:\anaconda\lib\site-packages\keras\engine\training.py", line 671, in _standardize_user_data
    self._set_inputs(x)

  File "d:\anaconda\lib\site-packages\keras\engine\training.py", line 577, in _set_inputs
    self.build(input_shape=(None,) + inputs.shape[1:])

  File "d:\anaconda\lib\site-packages\keras\engine\sequential.py", line 225, in build
    x = layer(x)

  File "d:\anaconda\lib\site-packages\keras\engine\base_layer.py", line 457, in __call__
    output = self.call(inputs, **kwargs)

  File "d:\anaconda\lib\site-packages\keras\layers\wrappers.py", line 248, in call
    y = self.layer.call(inputs, **kwargs)

  File "d:\anaconda\lib\site-packages\keras\layers\convolutional.py", line 168, in call
    dilation_rate=self.dilation_rate)

  File "d:\anaconda\lib\site-packages\keras\backend\tensorflow_backend.py", line 3565, in conv2d
    data_format=tf_data_format)

  File "d:\anaconda\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 779, in convolution
    data_format=data_format)

  File "d:\anaconda\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 828, in __init__
    input_channels_dim = input_shape[num_spatial_dims + 1]

  File "d:\anaconda\lib\site-packages\tensorflow\python\framework\tensor_shape.py", line 615, in __getitem__
    return self._dims[key]

IndexError: list index out of range

I am a beginner when it comes to Python and Tensorflow but I think it is the last line where it checks the dimensions? If I look in tensor_shape.py I can't find an IndexError.

Thanks for the help!

Answer 1

I tried to reproduce it and got this error after I replaced model.add(TimeDistributed(Conv2D(64, (1, 3), input_shape=X.shape[1:]))) with this model.add(TimeDistributed(Conv2D(64, (1, 3)), input_shape=X.shape[1:])) with X being a random array of shape (1923,63,325,1). The error is raised because the dimension of the array must equal 4 in order to apply Conv2D sequentially for each sample (or without the dimension of a batch). In other words, X must have shape (batch dimension, temporal dimension, row dimension, col dimension, channel dimension).