Tensorflow - I don't get the right shapes - `ValueError: Shapes (9, 1) and (8, 9) are incompatible`

Question

I want to train a Sequential Neural Net (NN) with Tensorflow.

import tensorflow as tf


bidding_nn = tf.keras.Sequential([
    tf.keras.layers.Dense(
        units=128, activation='elu', kernel_initializer='he_uniform'),
    tf.keras.layers.Dense(
        units=128, activation='elu', kernel_initializer='he_uniform'),
    tf.keras.layers.Dense(
        units=9, activation='softmax', kernel_initializer='he_uniform'),
    ])

opt = tf.keras.optimizers.Adam(lr=0.02, decay=0.01)
bidding_nn.compile(optimizer=opt,
                   loss=tf.keras.losses.CategoricalCrossentropy(),
                   metrics=["accuracy"])
bidding_nn.fit(train_dataset, epochs=10)

It should tell me probabilities for the 9 categories I have.

So as input for the NN, I have 8 numpy arrays of lengths 32 (one-hot encoded) and as output 1 numpy array of lengths 9 (one-hot encoded).

(Pdb) train_dataset
<TensorSliceDataset shapes: ((8, 32), (9,)), types: (tf.float64, tf.float64)>

However, at bidding_nn.fit(train_dataset, epochs=10) I get the error message

ValueError: Shapes (9, 1) and (8, 9) are incompatible

Answer 1

output should be (Batch_size,num_output)

change your dataset generator to expand the dimension then append each label together

labels = [] 
for i in one_hot:
   lbs.append(np.expand_dims(i,0))
labels = np.array(lbs)
print(labels.shape) # (8,9)

Answer 2

Your first problem is that the dataset you have created is only spitting out a single training instance. try:

print(train_dataset)
# <TensorSliceDataset shapes: ((8, 32), (9,)), types: (tf.float32, tf.float32)>

Your network is expecting an input of ((batch_size, 8, 32), (batch_size, 9)) . To fix this add .batch() to the end of your dataset.

train_dataset = train_dataset.batch(32)
print(train_dataset)
# <BatchDataset shapes: ((None, 8, 32), (None, 9)),
# types: (tf.float32, tf.float32)>

This should fix your original problem. Your next problem will be that you are putting 3D input into a network but your need 2D output. You need a Flatten() layer (or some other dimensional reduction layer) before you create your logits.

import tensorflow as tf


# fake X
X = tf.one_hot(tf.random.uniform(
    [100, 8], minval=0, maxval=33, dtype=tf.int64), 32)

print(X.shape)
# (100, 8, 32)

# fake y
y = tf.one_hot(tf.random.uniform(
    [100, ], minval=0, maxval=10, dtype=tf.int64), 9)

print(y.shape)
# (100, 9)

# network layers
bidding_nn = tf.keras.Sequential([
    tf.keras.layers.Dense(
        units=128, activation='elu', kernel_initializer='he_uniform'),
    tf.keras.layers.Dense(
        units=128, activation='elu', kernel_initializer='he_uniform'),
    tf.keras.layers.Flatten(),  # <= add this
    tf.keras.layers.Dense(
        units=9, activation='softmax', kernel_initializer='he_uniform'),
])

# compile model
bidding_nn.compile(optimizer=tf.keras.optimizers.Adam(lr=0.02, decay=0.01),
                   loss=tf.keras.losses.CategoricalCrossentropy(),
                   metrics=["accuracy"])

# create dataset
train_dataset = tf.data.Dataset.from_tensor_slices((X, y)).batch(32)

# train
bidding_nn.fit(train_dataset, epochs=10)

# Epoch 1/10
# 4/4 [========================] - 1s 3ms/step - loss: 2.2432 - accuracy: 0.0738
# Epoch 2/10
# 4/4 [========================] - 0s 2ms/step - loss: 0.1858 - accuracy: 0.8672
# ...