Input Layer Incompatible with Tensorflow 2D CNN

Question

I'm trying to train a CNN model for a speech emotion recognition task using spectrograms as input. I've reshaped the spectrograms to have the shape (num_frequency_bins, num_time_frames, 1) which I thought would be sufficient, but upon trying to fit the model to the dataset, which is stored in a Tensorflow dataset, I got the following error:

Input 0 of layer "sequential_12" is incompatible with the layer: expected shape=(None, 257, 1001, 1), found shape=(257, 1001, 1)

I tried reshaping the spectrograms to have the shape (1, num_frequency_bins, num_time_frames, 1) , but that produced an error when creating the Sequential model:

ValueError: Exception encountered when calling layer "resizing_14" (type Resizing).

'images' must have either 3 or 4 dimensions.

Call arguments received:
  • inputs=tf.Tensor(shape=(None, 1, 257, 1001, 1), dtype=float32)

So I passed in the shape as (num_frequency_bins, num_time_frames, 1) when creating the model, and then fitted the model to the training data with the 4-dimensional data, but that raised this error:

InvalidArgumentError: slice index 0 of dimension 0 out of bounds. [Op:StridedSlice] name: strided_slice/

So I'm kind of at a loss now. I genuinely have no idea what to do and how I can go about fixing this. I've read around but haven't come across anything useful. Would really appreciate any help.

Here's some of the code for context.

dataset = [[specgram_files[i], labels[i]] for i in range(len(specgram_files))]
specgram_files_and_labels_dataset = tf.data.Dataset.from_tensor_slices((specgram_files, labels))

def read_npy_file(data):
    # 'data' stores the file name of the numpy binary file storing the features of a particular sound file
    # item() returns numpy array of size 1 as a suitable python scalar.
    # data.item() then returns the bytes string stored in the numpy array.
    # decode() is then called on the bytes string to decode it from a bytes string to a regular string
    # so that it can be passed as a parameter in np.load()
    data = np.load(data.item().decode())
    # Shape of data is now (1, rows, columns)
    # Needs to be reshaped to (rows, columns, 1):
    data = np.reshape(data, (data.shape[0], data.shape[1], 1))
    return data.astype(np.float32)

specgram_dataset = specgram_files_and_labels_dataset.map(
                    lambda file, label: tuple([tf.numpy_function(read_npy_file, [file], [tf.float32]), label]),
                    num_parallel_calls=tf.data.AUTOTUNE)

num_files = len(train_df)
num_train = int(0.8 * num_files)
num_val = int(0.1 * num_files)
num_test = int(0.1 * num_files)

specgram_dataset.shuffle(buffer_size=1000)
specgram_train_ds = specgram_dataset.take(num_train)
specgram_test_ds = specgram_dataset.skip(num_train)
specgram_val_ds = specgram_test_ds.take(num_val)
specgram_test_ds = specgram_test_ds.skip(num_val)

batch_size = 32
specgram_train_ds.batch(batch_size)
specgram_val_ds.batch(batch_size)

specgram_train_ds = specgram_train_ds.cache().prefetch(tf.data.AUTOTUNE)
specgram_val_ds = specgram_val_ds.cache().prefetch(tf.data.AUTOTUNE)

for specgram, label in specgram_train_ds.take(1):
    input_shape = specgram.shape

num_emotions = len(train_df["emotion"].unique())

model = models.Sequential([
    layers.Input(shape=input_shape),
    # downsampling the input. 
    layers.Resizing(32, 128),
    layers.Conv2D(32, 3, activation="relu"),
    layers.MaxPooling2D(),
    layers.Conv2D(64, 3, activation="relu"),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dense(128, activation="softmax"),
    layers.Dense(num_emotions)
])

model.compile(
    optimizer=tf.keras.optimizers.Adam(0.01),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(),
    metrics=["accuracy"]
)

EPOCHS = 10

model.fit(
    specgram_train_ds,
    validation_data=specgram_val_ds,
    epochs=EPOCHS,
    callbacks=tf.keras.callbacks.EarlyStopping(verbose=1, patience=2)
)

Answer 1

Assuming you know your input_shape , I would recommend first hard-coding it into your model:

model = models.Sequential([
    layers.Input(shape=(257, 1001, 1),
    # downsampling the input. 
    layers.Resizing(32, 128),
    layers.Conv2D(32, 3, activation="relu"),
    layers.MaxPooling2D(),
    layers.Conv2D(64, 3, activation="relu"),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dense(128, activation="softmax"),
    layers.Dense(num_emotions)
])

Also, when using tf.data.Dataset.batch , you should assign the Dataset output to a variable:

batch_size = 32
specgram_train_ds = specgram_train_ds.batch(batch_size)
specgram_val_ds = specgram_val_ds.batch(batch_size)

Afterwards, make sure that specgram_train_ds really does have the correct shape:

specgrams, _ = next(iter(specgram_train_ds.take(1)))
assert specgrams.shape == (32, 257, 1001, 1)