如何将掩蔽损失与 tensorflow2 TimeSeriesGenerator 结合起来

Question

我们正在尝试使用卷积 LSTM 来预测给定过去 7 个时间步长的图像的值。 我们使用了 tensorflow2 TimeSeriesGenerator 方法来创建我们的时间序列数据：

    train_gen = TimeseriesGenerator(
        data,
        data,
        length=7,
        batch_size=32,
        shuffle=False
    )

每个图像（时间步长）都具有形状 (55, 50, 1)，因此生成器生成了具有形状 (32, 7, 55, 50, 1) 及其目标 (32, 55, 50, 1) 的数据。 然而，有一个转折点，我们只想计算图像屏蔽区域的预测损失。 这个掩码是常数，我们通过以下方式将它存储在张量常数中：

mask = tf.keras.backend.constant(mask)

然后我们的想法是将此常量作为我们 model 的第二个输入，并使用它来计算使用自定义损失 function 的屏蔽损失：

def masked_MSE_loss(y_true, y_pred, mask):
    y_pred_masked = tf.math.multiply(y_pred, mask)
    mse = tf.keras.losses.mean_squared_error(y_true = y_true, y_pred = y_pred_masked)
    return mse

我们的 model 看起来像下面这样：

# Define the input tensors
inputs = Input(shape=(lookback, 55, 50, 1))
input_mask = Input(tensor=mask)

# First stack of convlstm layers
convlstm1 = layers.ConvLSTM2D(filters=128, kernel_size=(3, 3), padding='same', activation='tanh', return_sequences=True)(inputs)
bathnorm1 = layers.BatchNormalization()(convlstm1)
convlstm2 = layers.ConvLSTM2D(filters=128, kernel_size=(3, 3), padding='same', activation='tanh', return_sequences=False)(bathnorm1)

# Second stack of convlstm layers
convlstm3 = layers.ConvLSTM2D(filters=128, kernel_size=(3, 3), padding='same', activation='tanh', return_sequences=True)(inputs)
batchnorm2 = layers.BatchNormalization()(convlstm3)
convlstm4 = layers.ConvLSTM2D(filters=128, kernel_size=(3, 3), padding='same', activation='tanh', return_sequences=False)(batchnorm2)

# Concatenate outputs of two stacks
concatenation = layers.concatenate([convlstm2, convlstm4])
outputs = layers.Conv2D(filters=1, kernel_size=1, padding="same", activation='tanh')(concatenation)

# Create the model
model = Model(inputs=[inputs, input_mask], outputs=outputs)
model.add_loss(masked_MSE_loss(inputs, outputs, input_mask))

# Compile the model
model.compile(optimizer='adam', loss=None, metrics=['mae'])

最后，我们尝试以一种相当独特的方式拟合 model，以尝试将我们的 TimeSeriesGenerator 与我们的常量输入合并：

for batch in train_gen:
    batch_input, batch_target = batch
    model.fit(x=[batch_input, np.repeat(mask[np.newaxis, :, :, :], len(batch), axis=0)], y=batch_target, epochs=1)

我们遍历每个批次并重复恒定的len(batch)次，然后将其提供给 our.network 并对其进行 1 个时期（每个批次）的训练。 这给了我们以下错误：

ValueError: Input 1 of layer "model" is incompatible with the layer: expected shape=(None, 50, 1), found shape=(32, 55, 50, 1)

这让我们认为我们只需要输入一次掩码常量：

for batch in train_gen:
    batch_input, batch_target = batch
    model.fit(x=[batch_input, mask], y=batch_target, epochs=100)

但这给了我们错误：

ValueError: Data cardinality is ambiguous:
  x sizes: 32, 55
  y sizes: 32
Make sure all arrays contain the same number of samples

显然， model 期望第一个参数是批量大小，但感觉这个错误几乎与之前尝试的解决方案相矛盾，我们根据len(batch)重复常量。

所以我们的问题是：我们如何将我们的 model 与我们的 TimeSeriesGenerator 数据相结合作为我们的第二个输入（计算我们预测的掩蔽损失）的常数张量？

Answer 1

也许尝试这样的事情：

import tensorflow as tf
from tensorflow.keras import Input, layers, Model
from tensorflow.keras.preprocessing.sequence import TimeseriesGenerator
import numpy as np

################################ Creating Mask #################################
mask = np.ones((55,50))
mask = tf.keras.backend.constant(mask)
mask = tf.expand_dims(mask, -1)
################################################################################

lookback = 7
batch_size = 16

# Create data with shape (3653, 55, 50, 1) with 3653 timesteps
data = np.random.random((3653, 55, 50, 1))

train_gen = TimeseriesGenerator(
data,
data,
length=lookback,
batch_size=batch_size,
shuffle=False
)

def masked_MSE_loss(y_true, y_pred):
  y_pred_masked = tf.math.multiply(y_pred, mask)
  y_true = tf.math.multiply(y_true, mask)
  mse = tf.keras.losses.mean_squared_error(y_true = y_true, y_pred = y_pred_masked)
  return mse

# Define the input tensors
inputs = Input(shape=(lookback, 55, 50, 1))

# First stack of convlstm layers
convlstm1 = layers.ConvLSTM2D(filters=128, kernel_size=(3, 3), padding='same', activation='tanh', return_sequences=True)(inputs)
bathnorm1 = layers.BatchNormalization()(convlstm1)
convlstm2 = layers.ConvLSTM2D(filters=128, kernel_size=(3, 3), padding='same', activation='tanh', return_sequences=False)(bathnorm1)

# Second stack of convlstm layers
convlstm3 = layers.ConvLSTM2D(filters=128, kernel_size=(3, 3), padding='same', activation='tanh', return_sequences=True)(inputs)
batchnorm2 = layers.BatchNormalization()(convlstm3)
convlstm4 = layers.ConvLSTM2D(filters=128, kernel_size=(3, 3), padding='same', activation='tanh', return_sequences=False)(batchnorm2)

# Concatenate outputs of two stacks
concatenation = layers.concatenate([convlstm2, convlstm4])
outputs = layers.Conv2D(filters=1, kernel_size=1, padding="same", activation='tanh')(concatenation)

model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam', loss=masked_MSE_loss, metrics=['mae'])
model.fit(train_gen, epochs=100)

请注意，您的面具不再是您的 model 的一部分，因为它是不变的。