TensorFlow2 - Model 子类化 ValueError

Question

I am trying to experiment creating a LeNet-300-100 dense neural network using TensorFlow 2's model sub-classing. The code that I have is as follows:

batch_size = 32
num_epochs = 20


# Load MNIST dataset-
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()

X_train = X_train.astype('float32') / 255.0
X_test = X_test.astype('float32') / 255.0

# Convert class vectors/target to binary class matrices or one-hot encoded values-
y_train = tf.keras.utils.to_categorical(y_train, num_classes)
y_test = tf.keras.utils.to_categorical(y_test, num_classes)

X_train.shape, y_train.shape
# ((60000, 28, 28), (60000, 10))

X_test.shape, y_test.shape
# ((10000, 28, 28), (10000, 10)) 




class LeNet300(Model):
    def __init__(self, **kwargs):
        super(LeNet300, self).__init__(**kwargs)
        
        self.flatten = Flatten()
        self.dense1 = Dense(units = 300, activation = 'relu')
        self.dense2 = Dense(units = 100, activation = 'relu')
        self.op = Dense(units = 10, activation = 'softmax')

    def call(self, inputs):
        x = self.flatten(inputs)
        x = self.dense1(x)
        x = self.dense2(x)
        return self.op(x)




# Instantiate an object using LeNet-300-100 dense model-
model = LeNet300()

# Compile the defined model-
model.compile(
        optimizer=tf.keras.optimizers.Adam(),
        loss=tf.keras.losses.SparseCategoricalCrossentropy(),
        metrics=['accuracy']
        )


# Define early stopping callback-
early_stopping_callback = tf.keras.callbacks.EarlyStopping(
        monitor = 'val_loss', min_delta = 0.001,
        patience = 3)

# Train defined and compiled model-
history = model.fit(
    x = X_train, y = y_train,
    batch_size = batch_size, shuffle = True,
    epochs = num_epochs,
    callbacks = [early_stopping_callback],
    validation_data = (X_test, y_test)
    )

On calling "model.fit()", it gives the following error:

ValueError: Shape mismatch: The shape of labels (received (320,)) should equal the shape of logits except for the last dimension (received (32, 10)).

What's going wrong?

Thanks

Answer 1

The loss SparseCategoricalCrossentropy doesn't take one-hot encoding to calculate loss. In the documentation, they mention that

Use this crossentropy loss function when there are two or more label classes. We expect labels to be provided as integers. If you want to provide labels using one-hot representation, please use CategoricalCrossentropy loss. There should be # classes floating point values per feature for y_pred and a single floating point value per feature for y_true.

As a result of this you are getting the error. If you observe the stacktrace the error arises in the loss function,

    /home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/losses.py:1569 sparse_categorical_crossentropy
        y_true, y_pred, from_logits=from_logits, axis=axis)
    /home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/backend.py:4941 sparse_categorical_crossentropy
        labels=target, logits=output)
    /home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py:4241 sparse_softmax_cross_entropy_with_logits_v2
        labels=labels, logits=logits, name=name)
    /home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py:4156 sparse_softmax_cross_entropy_with_logits
        logits.get_shape()))

    ValueError: Shape mismatch: The shape of labels (received (320,)) should equal the shape of logits except for the last dimension (received (32, 10)).

I would suggest using CategoricalCrossentropy .

Answer 2

This is because input to first Dense layer should be flattened. MNIST data has 28x28 grid/image for every digit. This 28x28 data should be flattened to 784 input numbers.

So just before first Dense(...) layer insert Flatten() keras layer ie do Flatten()(inputs) .

See this doc of Flatten layer for reference.