Keras and input shape to Conv1D issues

Question

First off, I am very new to Neural Nets and Keras.

I am trying to create a simple Neural Network using Keras where the input is a time series and the output is another time series of same length (1 dimensional vectors).

I made dummy code to create random input and output time series using a Conv1D layer. The Conv1D layer then outputs 6 different time series (because I have 6 filters) and the next layer I define to add all 6 of those outputs into one which is the output to the entire network.

import numpy as np
import tensorflow as tf
from tensorflow.python.keras.models import Model
from tensorflow.python.keras.layers import Conv1D, Input, Lambda


def summation(x):
    y = tf.reduce_sum(x, 0)
    return y


time_len = 100  # total length of time series
num_filters = 6 # number of filters/outputs to Conv1D layer
kernel_len = 10 # length of kernel (memory size of convolution)

# create random input and output time series
X = np.random.randn(time_len)
Y = np.random.randn(time_len)

# Create neural network architecture
input_layer = Input(shape = X.shape)
conv_layer = Conv1D(filters = num_filters, kernel_size = kernel_len, padding = 'same')(input_layer)
summation_layer = Lambda(summation)(conv_layer)

model = Model(inputs = input_layer, outputs = summation_layer)

model.compile(loss = 'mse', optimizer = 'adam', metrics = ['mae'])

model.fit(X,Y,epochs = 1, metrics = ['mae'])

The error I get is:

ValueError: Input 0 of layer conv1d_1 is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [None, 100]

Looking at the Keras documentation for Conv1D, the input shape is supposed to be a 3D tensor of shape (batch, steps, channels) which I don't understand if we are working with 1 dimensional data.

Can you explain the meaning of each of the items: batch, steps, and channels? And how should I shape my 1D vectors to allow my network to run?

Answer 1

What is a (training) sample?

The (training) data may consists of tens, hundreds or thousands of samples. For example, each image in an image dataset like Cifar-10 or ImageNet is a sample. As another example, for a timseries dataset which consists of weather statistics recorded during the days over 10 years, each training sample may be a timeseries of each day. If we have recorded 100 measurements during the day and each measurement consists of temperature and humidity (ie we have two features per measurement) then the shape of our dataset is roughly (10x365, 100, 2) .

What is batch size?

The batch size is simply the number of samples that can be processed by the model at a single time. We can set the batch size using the batch_size argument of fit method in Keras. The common values are 16, 32, 64, 128, 256, etc (though you must choose a number such that your machine could have enough RAM to allocate the required resources).

Further, the "steps" (also called "sequence length") and "channels" (also called "feature size") are the number of measurements and the size of each measurement, respectively. For example in our weather example above, we have steps=100 and channels=2.

To resolve the issue with your code you need to define your training data (ie X ) such that it has a shape of (num_samples, steps or time_len, channels or feat_size) :

n_samples = 1000   # we have 1000 samples in our training data
n_channels = 1     # each measurement has one feature
X = np.random.randn(n_samples, time_len, n_channels)

# if you want to predict one value for each measurement
Y = np.random.randn(n_samples, time_len)

# or if you want to predict one value for each sample
Y = np.random.randn(n_samples)

Edit:

One more thing is that you should pass the shape of one sample as the input shape of the model. Therefore, the input shape of Input layer must be passed like shape=X.shape[1:] .