combining RNN and CNN with lasagne

Question

I am trying to run a 1D CNN on 10s segments of EEG data and then cover a temporal connection between the segments using an RNN.

The problem is, that the RNN expects input with batch_size x sequence_length x num_inputs while the CNN outputs batch_size x num_filters x sequence_length

This can be solved by a dim-shuffle layer

network = L.InputLayer(shape=(None, data_size[1], data_size[2]), input_var=input_var)
network = L.Conv1DLayer( network, num_filters=32, filter_size = 5) 
network = L.DimshuffleLayer(network, (0, 2, 1))
network = L.LSTMLayer(network, 200)

But to my understanding the RNN will now cover temporal connections only within the sequence_length, but not between the different batches, is that right?

How can I get the temporal connection between segments?

Answer 1

Answering my own question:

The RNN will indeed only learn dependencies within one batch. However, Keras has a mode that allows for states to transition between batches: stateful=True

network = keras.layers.LSTM(network, stateful=True)

Now it is important to feed the batches in the right order: The i-th element of each batch will be learned with the state of the i-th batch at time t-1. That means you need to be very careful when feeding your batches.

Note, this will only transition cell states and not backpropagate between batches. As a side-effect your initial state while predicting will have to be set and bias your results.