Error while using Convolutional layers on CPU

Question

Can I use Convolutional layers of keras without gpu support? I am getting errors when I use it on Colab with runtime as None. My code looks like this:

model = tf.keras.Sequential()
model.add(layers.Conv1D(1,5, name='conv1', padding="same", activation='relu',data_format="channels_first", input_shape=(1,2048)))
# model.add(layers.LSTM(5, activation='tanh'))
model.add(layers.Flatten())
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(num_classes, activation='softmax'))
#model.summary()
model.compile(loss=tf.keras.losses.categorical_crossentropy,
              optimizer=tf.keras.optimizers.SGD(lr=0.001, momentum=0.9),
              metrics=['accuracy'])

x_train = train_value
y_train = train_label
x_test = test_value
y_test = test_label
print(np.shape(x_train)) #shape of x train is (4459, 1, 2048)
print(np.shape(x_test)) #shape of test is (1340,1,2048)

history = model.fit(x_train, y_train,
          batch_size=100,
          epochs=30, 
          verbose=1,
          validation_data=(x_test, y_test)

          )

It is running fine on GPU but gives following error on CPU:

InvalidArgumentError: Conv2DCustomBackpropFilterOp only supports NHWC. [[{{node training/SGD/gradients/gradients/conv1/conv1d_grad/Conv2DBackpropFilter}}]]

UnimplementedError: The Conv2D op currently only supports the NHWC tensor format on the CPU. The op was given the format: NCHW [[{{node conv1_1/conv1d}}]]

I have figured out that the problem is with the format of Input Data. My input data are vectors of size (1,2048). Can you please guide me on how to convert these vectors to NHWC format? I would really appreciate it, if someone can clear this up for me. Thanks in advance.

Answer 1

Per the Keras documentation

data_format: A string, one of "channels_last" (default) or "channels_first". The ordering of the dimensions in the inputs. "channels_last" corresponds to inputs with shape (batch, steps, channels) (default format for temporal data in Keras) while "channels_first" corresponds to inputs with shape (batch, channels, steps)

Now Keras in TensorFlow appears to implement Conv1D in terms of a Conv2D operator - basically forming an "image" with 1 row, W columns, and then your C "channels". That's why your getting error messages about image shapes when you don't have image data.

In the docs above "channels" are the number of data items per time step (eg perhaps you have 5 sensor readings at each time step so you'd have 5 channels). From your answers above I believe you're passing tensors with shape (n, 1, 2048) where n is your batch size. So, with channels_last TensorFlow thinks that means you have n examples in your batch each with a sequence length of 1 and 2048 data items per time step - that is only a single time step with 2048 data items per observation (eg 2048 sensor readings taken at each time step) in which case the convolution won't be doing a convolution - it'd be equivalent to a single dense layer taking all 2048 numbers as input.

I think in reality you have only a single data item per time step and you have 2048 time steps. That explains why passing channels_first improves your accuracy - now TensorFlow understand that your data represents 1 data item samples 2048 times and it can do a convolution over that data.

To fix you can just tf.reshape(t, (1, 2048, 1)) - and remove the channels_first (that code assumes you're doing batches of size 1 and your tensor is named t ). Now it's in the format (n, s, 1) where n is the batch size ( 1 here), s is the number of time steps (2048), and 1 indicates one data point per time step. You can now run the same model on the GPU or CPU.