如何批量训练具有多个输入的 model？

Question

I have a model like

inputA = tf.keras.layers.Input(shape=(40,40))
inputB = tf.keras.layers.Input(shape=6)

a = tf.keras.layers.Conv2D(...)(inputA)
...
b = tf.keras.layers.Dense(...)(inputB)
combined = tf.keras.layers.concatenate([a, b])
....

It has 2 inputs, a preprocessed image, and some extra info.

My problem is to train more than one instance at a time. I have this train function:

def train_step(observations, actions, rewards):
    with tf.GradientTape() as tape:
        logits = my_model(observations)
        loss = self.compute_loss(logits, actions, rewards)
    ....

In the line logits = my_model(observations) if I give just 1 sample it works, but if I send a list it gives the error " Layer model expects 2 input(s), but it received 34 input tensors ".

I suspect I'm not passing the arguments correctly (it used to work when I had 1 input model).

How should I pass a batch of observations to this model?

To clarify:

• observations is a list of numpy arrays.
• my_model(observations) throws an error " Layer model expects... "
• my_model([observations[0][0], observations[0][1]]) works
• my_model([[observations[0][0], observations[0][1]],[observations[1][0], observations[1][1]]]) throws an error Layer model expects 2 input(s), but it received 4 input tensors .

I can just train using a call to my_model([observations[i][0], observations[i][1]]) , in a loop (where i is my n-th observation in this batch. But I suppose that there will be performance issues if I do this compared to just send all the training batch in a call.

EDIT

To be more clear. I can:

input_a = np.random.randint(0, 255, size=(1, 40, 40, 3)).astype("int8")
input_b = np.random.randint(0, 1, size=(1, 6)).astype("int8")
logits = my_model([input_a, input_b])

This works perfectly.

But if I have more than one observation in a batch, I don't know how to pass the parameters:

input_a0 = np.random.randint(0, 255, size=(1, 40, 40, 3)).astype("int8")
input_b0 = np.random.randint(0, 1, size=(1, 6)).astype("int8")
input_a1 = np.random.randint(0, 255, size=(1, 40, 40, 3)).astype("int8")
input_b1 = np.random.randint(0, 1, size=(1, 6)).astype("int8")
logits = my_model([[input_a0, input_a0], [input_a1, input_a1]])

Throws:

ValueError: Layer model expects 2 input(s), but it received 4 input tensors. Inputs received: ...

Answer 1

If I understand the main problem that you've faced is to pass the input data correctly for the multiple inputs model. As you mentioned, observations is a list of numpy arrays, and you passed the data to a multi-input model as follows my_model(observations) - which seems problematic. What you probably need to do is to unpack the observations and pass the data as a list , eg my_model([input1, input2]) . Here is a possible solution.

CustomFit

from tensorflow import keras
import tensorflow as tf 

class CustomModel(keras.Model):
    def train_step(self, data):
        # Unpack the data. Its structure depends on your model and
        # on what you pass to `fit()`.
        (x, z), y = data

        with tf.GradientTape() as tape:
            y_pred = self([x, z], training=True)  # Forward pass
            # Compute the loss value
            # (the loss function is configured in `compile()`)
            loss = self.compiled_loss(y, y_pred, regularization_losses=self.losses)

        # Compute gradients
        trainable_vars = self.trainable_variables
        gradients = tape.gradient(loss, trainable_vars)
        # Update weights
        self.optimizer.apply_gradients(zip(gradients, trainable_vars))
        # Update metrics (includes the metric that tracks the loss)
        self.compiled_metrics.update_state(y, y_pred)
        # Return a dict mapping metric names to current value
        return {m.name: m.result() for m in self.metrics}

Base Model

import numpy as np

# Construct and compile an instance of CustomModel
inputA = keras.Input(shape=(28, 28))
inputB = keras.Input(shape=(32))

a = tf.keras.layers.Flatten()(inputA)
b = tf.keras.layers.Dense(16)(inputB)
combined  = tf.keras.layers.concatenate([a, b])
outputs = keras.layers.Dense(1)(combined)

# without custom fit 
# model = tf.keras.Model([inputA, inputB], outputs)

model = CustomModel([inputA, inputB], outputs)
model.compile(optimizer="adam", loss="mse", metrics=["mae"])

Train

x = np.random.randint(0, 256, size=(50, 28, 28)).astype("float32")
z = np.random.random((50, 32))
y = np.random.random((50, 1))
print(x.shape, z.shape, y.shape)

model.fit([x, z], y, epochs=3, verbose=2)

(50, 28, 28) (50, 32) (50, 1)
Epoch 1/3
7ms/step - loss: 19227.8789 - mae: 112.9769
Epoch 2/3
8ms/step - loss: 16268.3952 - mae: 98.0642
Epoch 3/3
10ms/step - loss: 14247.0742 - mae: 95.9225