How do I properly package multi-channel time series data and build the network for a Tensorflow CNN-LSTM?

Question

I have data from three sensors, and have broken it into 21,190 938-sample sequences. I have stored this in a tensor, xt, with dimensions (21190, 938, 3), and created a tensor of labels, yt, with dimensions (21190,1). With this data, I would like to create a CNN-LSTM binary classifier. I believe I have found the solution to my problem, but I wanted to post it here to ensure that I am correctly understanding the error I was encountering, and have made the proper fix.

My intention is for the convolution layer to apply filters across the time (938-length) dimension, and for the output from each stride of those filters in the convolution layer to be fed into the LSTM layer as a sample in a temporal sequence. I have put together the network based on an example I found online that seemed to be working with a similar dataset (time series from multiple sensors) and objective (using the CNN to identify/extract features, and the LSTM to find patterns in the sequence of features); but based on my reading of the Tensorflow documentation for the TimeDistributed wrapper, and the errors I am getting, I'm not confident that I have my network set up properly.

Here is the code I am using to create and train the CNN-LSTM:

model = models.Sequential()
    model.add(layers.TimeDistributed(layers.Conv1D(filters=32,kernel_size=64),input_shape=(None,938,3)))
    model.add(layers.TimeDistributed(layers.Conv1D(filters=32,kernel_size=64)))
    model.add(layers.TimeDistributed(layers.Dropout(0.5)))
    model.add(layers.TimeDistributed(layers.MaxPooling1D(pool_size=2)))
    model.add(layers.TimeDistributed(layers.Flatten()))
    model.add(layers.LSTM(100))
    model.add(layers.Dense(2))
    model.add(layers.Softmax())
    model.compile(
        loss = losses.SparseCategoricalCrossentropy(from_logits=True),
        optimizer="adam",
        metrics=["accuracy"],
        )
    model.fit(
        xt,
        yt,
        epochs=10,
        )

When I run this, I get the following error:

ValueError: Input 0 of layer sequential is incompatible with the layer: expected ndim=4, found ndim=3. Full shape received: (None, 938, 3)

If I remove the TimeDistributed wrappers and the Flatten layer, the network trains with no errors. Am I correct in understanding that using TimeDistributed causes the network to try to apply 1D convolutions on a single sample at a time from each of the three channels, and that this is why I am getting the error? And am I correct that by removing the wrappers, I will get the functionality I desire, with each stride in the convolution layer providing one time-step of data to the LSTM? Thanks in advance for any input.

Answer 1

Issue is with input data shape.

Working sample code

import tensorflow as tf
inputs = tf.keras.Input(shape=(None, 938, 3))
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Conv1D(filters=32,kernel_size=64),input_shape=(None,938,3)))
model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Conv1D(filters=32,kernel_size=64)))
model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Dropout(0.5)))
model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.MaxPooling1D(pool_size=2)))
model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Flatten()))
model.add(tf.keras.layers.LSTM(100))
model.add(tf.keras.layers.Dense(2))
model.add(tf.keras.layers.Softmax())
output = model(inputs)

Output

output.shape
TensorShape([None, 2])