What should the generator return if it is used in a multi-input/output Keras model built with functional API?

Question

Following this article , I'm trying to implement a generative RNN. In the mentioned article, the training and validation data are passed as fully loaded np.array s. But I'm trying to use the model.fit_generator method and provide a generator instead.

I know that if it was a straightforward model, the generator should return:

def generator():
    ...
    yield (samples, targets)

But this is a generative model which means there are two models involved:

encoder_inputs = Input(shape=(None,))
x = Embedding(num_encoder_tokens, embedding_dim)(encoder_inputs)
x.set_weights([embedding_matrix])
x.trainable = False
x, state_h, state_c = LSTM(embedding_dim, return_state=True)(x)
encoder_states = [state_h, state_c]

decoder_inputs = Input(shape=(None,))
x = Embedding(num_decoder_tokens, embedding_dim)(decoder_inputs)
x.set_weights([embedding_matrix])
x.trainable = False
x = LSTM(embedding_dim, return_sequences=True)(x, initial_state=encoder_states)
decoder_outputs = Dense(num_decoder_tokens, activation='softmax')(x)

model = Model([encoder_inputs, decoder_inputs], decoder_outputs)

model.fit([encoder_input_data, decoder_input_data], decoder_target_data,
          batch_size=batch_size,
          epochs=epochs,
          validation_split=0.2)

As mentioned before, I'm trying to use a generator:

model.fit_generator(generator(),
                   steps_per_epoch=500,
                   epochs=20,
                   validation_data=generator(),
                   validation_steps=val_steps)

But what should the generator() return? I'm a little confused since there are two input collections and one target.

Answer 1

Since your model has two inputs and one output, the generator should return a tuple with two elements where the first element is a list containing two arrays, which corresponds to two input layers, and the second element is an array corresponding to output layer:

def generator():
    ...
    yield [input_samples1, input_samples2], targets

Generally, in a model with M inputs and N outputs, the generator should return a tuple of two lists where the first one has M arrays and the second one has N arrays:

def generator():
        ...
        yield [in1, in2, ..., inM], [out1, out2, ..., outN]

If instead of a Python generator, an instance of Sequence class is used to generate data, the format is the same with the only difference that you would use return instead of yield in the __getitem__ method.