How to evaluate a multi input/output model in keras?

Question

I've followed the description on this guide by keras to build the following model with multi-input and multi-output.

## define the model
EMBEDDING_SIZE = 128
HIDDEN_LAYER_SIZE = 64
BATCH_SIZE = 32
NUM_EPOCHS = 10

# first input model
main_input = Input(shape=(50,), dtype='int32', name='main_input')
embedding = Embedding(MAX_NB_WORDS, EMBEDDING_SIZE,
                    input_length=MAX_SEQUENCE_LENGTH)(main_input)
lstm_out = LSTM(HIDDEN_LAYER_SIZE)(embedding)
auxiliary_output = Dense(4, activation='sigmoid', name='aux_output')(lstm_out)
# second input model
auxiliary_input = Input(shape=(5,), name='aux_input')
x = concatenate([lstm_out, auxiliary_input])

x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)

main_output = Dense(4, activation='sigmoid', name='main_output')(x)

model = Model(inputs=[main_input, auxiliary_input], outputs=[main_output, auxiliary_output])

model.compile(optimizer='rmsprop',
              loss={'main_output': 'categorical_crossentropy', 'aux_output': 'categorical_crossentropy'},
              loss_weights={'main_output': 1., 'aux_output': 0.2})

model.fit([x1train, x2train], [ytrain, ytrain],
                    epochs=NUM_EPOCHS, batch_size=BATCH_SIZE,
                    validation_data=([x1test, x2test], [ytest, ytest]))

In the next step, I want to evaluate my model as well. I suggested running this code for it:

score, acc = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                            y={'main_output': ytest, 'aux_output': ytest},
                            batch_size=BATCH_SIZE)

But with that code I am getting the error "ValueError: too many values to unpack (expected 2)"

So I thought I maybe get a score and accuracy for both outputs and tried this code, too:

score1, score2, acc1, acc2 = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                            y={'main_output': ytest, 'aux_output': ytest},
                            batch_size=BATCH_SIZE)

but now I am getting the error "ValueError: not enough values to unpack (expected 4, got 3)"

So what am I doing wrong? I am just interested in the accuracy of my main_output to be honest.

Answer 1

From keras documentation for evaluate the can be found here

Returns

Scalar test loss (if the model has a single output and no metrics) or list of scalars (if the model has multiple outputs and/or metrics). The attribute model.metrics_names will give you the display labels for the scalar outputs.

According to your model, if you do print(model.metrics_names) you will get ['loss', 'main_output_loss', 'aux_output_loss'] .

The model.evaluate produces a scalar of this format, which indicates what each of those numbers you see in the output of evaluate method corresponds to.

Hence your code,

score1, score2, acc1, acc2 = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                            y={'main_output': ytest, 'aux_output': ytest},
                            batch_size=BATCH_SIZE)

will result in an error because there are only 3 indices in the scalar and the code expects to find 4 .

Also,

score, acc = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                            y={'main_output': ytest, 'aux_output': ytest},
                            batch_size=BATCH_SIZE)

will result in an error due to the fact there are more values returned by the evaluate .

You can do something like this if you want to unpack the result of evaluate in your model directly.

loss, main_loss, aux_loss = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                            y={'main_output': ytest, 'aux_output': ytest},
                            batch_size=BATCH_SIZE)

Also in your code, I can see acc1 and acc2 which makes me assume that you are expecting to evaluate the model accuracy.

Having said that, I can see that you have not used any metrics for compilation of the model. If you want to evaluate acc of the model then compile your model like this.

model.compile(optimizer='rmsprop',
              loss={'main_output': 'categorical_crossentropy', 'aux_output': 'categorical_crossentropy'},
              loss_weights={'main_output': 1., 'aux_output': 0.2}, metrics=['acc'])

Then on model.evaluate() you will get a scalar corresponding to

['loss',
 'main_output_loss',
 'aux_output_loss',
 'main_output_acc',
 'aux_output_acc']

Hence, you can unpack it like this,

loss, main_loss, aux_loss, main_acc, aux_acc = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                                y={'main_output': ytest, 'aux_output': ytest},
                                batch_size=BATCH_SIZE)

Answer 2

You haven't defined in the model compile that you want accuracy, so when you use evaluate the function will return the loss. You have 3 losses here which evaluate returns:

• (1) Weighted averaged loss of (2) and (3), in your case: 1. * (2) + 0.2 * (3)
• (2) Loss of main_output - categorical_crossentropy
• (3) Loss of aux_output - categorical_crossentropy