Tensorflow/Keras：训练数据和预测数据中的 Output 形状不匹配

Question

I have an LSTM model defined in tensorflow/keras as below. I am including only the relevant details pertaining to the question.

t_steps = 60
n_features = 3

def LSTMModel():
  input = Input(shape=(t_steps, n_features))
  l1 = BatchNormalization()(input)
  l2 = LSTM(160,return_sequences=True)(l1)
  l3 = LSTM(80,return_sequences=True)(l2)
  l4 = LSTM(10,return_sequences=True)(l3)
  l5 = Dense(1, activation = 'relu')(l4)
  model = Model(inputs = input, outputs = l5)
  model.compile(loss="mean_squared_error", optimizer='adam')
  return model

# shape of X_train_seq is (275268, 60, 3)
# shape of Y_train_seq is (275268,)

model = LSTMModel()
model.fit(X_train_seq, Y_train_seq, epochs=n_epochs,
  batch_size=batch_size,verbose=1,initial_epoch=init_epoch)

Then when I predict using this model on a X_test_seq of shape (30355, 60, 3), I get Y_test_seq_pred of shape (30355, 60, 1) while I expect the prediction of shape (30355,). This happened because my l4 line in the above code should have been

l4 = LSTM(10,return_sequences=False)(l3)

My question is, with the original code, why didnt Tensorflow/Keras give an error or produce any sort of warning during training as the shape of Y_train_seq that is passed to the fit() method is (275268,) and during training, it must have been internally predicting a Y of shape (batch_size, 60, 1) for every batch during optimization and comparing it with a slice of shape (batch_size,) from Y_train_seq. How come this dimension mismatch in Y still let the code continue only for me to find out about it at the end of the training? I am sure there must be some reason behind it and I want to know what is going on. Thank you!!

Answer 1

That's occurs due to what's called on Numpy broadcasting, which describes how Numpy treats arrays with different shapes.

Let first define two tensors A and B tensors with the following shapes,

dim(A) = (32, 60, 1)
dim(B) = (32,)

For example, if you try to do some operations such subtraction , addition or any element-wise operation between A and B , in such case the way how Tensorflow represent B dimensions, something like replacing dimensions with don't care.

dim(B) = (do_not_care, do_not_care, 32)

So at iteration, scalar value x subtracted from an array of 32 elements (values), Therefor the output is a column matrix contains 32 different values.

dim(output) = (32, 60, 32) , which is valid, but unexpected behavior.

Such operation could be implemented recursively , we are iterating through the tensor with a higher dimension, starting from the 1st dimension ( axis ) to the known dimension of the second tensor, then we can apply the required operation to the resultant chunked tensor, if their shapes match, in other words the broadcasting succeeds .

---

The the implementation of a broadcasting for subtraction could be done as follows.

def supstract(a, b): # a and b are numpy.array()

  # check if shape valid
  if a.squeeze().ndim == b.squeeze().ndim:
    if b.squeeze().shape[-1] == b.squeeze().shape[-1]:
      print(a - b, '\n')
      return
    else:
      raise ValueError(f'could not Broadcasting, {a.shape} != {b.shape}')

  elif a.ndim > b.ndim:
    for sub_a in a:
      supstract(sub_a, b) # recursive call
  else:
    for sub_b in b:
      supstract(a, sub_b) # recursive call

I have used the following example to check the implementation, so I may have missed some conditions (different cases handling).

a = np.random.randint(1, 4, (2, 2, 6, 1))
b = np.random.randint(1, 4, (4, )) 

---

In your case Tensorflow need Y to compute the loss function, you are using MSE which just a scalar value, computed using Keras backend as follows.

from tensorflow.keras import backend as K

def mean_squared_error(y_true, y_pred):
  return K.mean((y_pred - y_true) ** 2) # broadcasting (y_pred - y_true)

---

Update

|1 2 3| + |7 8 9|
|4 5 6|

|1 2 3| + 7 = |8  9  10|
|4 5 6|       |11 12 13|

Based on the Tensorflow documentation for describing the broadcasting . Numpy (broadcasting) handle any of the above operations, but mathematically you can't apply any of them. If you want to have warning for any broadcasting operation, you might expect to get tons of warnings.