How to give labels of multiple inputs in Keras to model.fit() function?

Question

I am using functional API of Keras. I want to create a model that will work on multiple inputs where inputs have different labels. Below is the model that I want to create:

Here is what I have tried:

import numpy as np
from keras import backend as K
from keras import losses
from keras.models import Model
from keras.utils import to_categorical, plot_model
from keras.layers import Input, Dense, Conv1D, Flatten
from keras.layers.merge import concatenate


# Create data
NUM_TRAINING_SAMPLES_INPUT_1 = 100
NUM_TRAINING_SAMPLES_INPUT_2 = 100
NUM_FEATURES_INPUT_1 = 144
NUM_FEATURES_INPUT_2 = 145
NUM_CLASSES = 15

# Create train data
train_X_1 = np.random.randint(low=0, high=100, size=(NUM_TRAINING_SAMPLES_INPUT_1, NUM_FEATURES_INPUT_1, 1), dtype='int64')
train_X_2 = np.random.randint(low=0, high=100, size=(NUM_TRAINING_SAMPLES_INPUT_2, NUM_FEATURES_INPUT_2, 1), dtype='int64')

# Create labels
train_Y_1 = np.random.randint(low=0, high=NUM_CLASSES, size=(NUM_TRAINING_SAMPLES_INPUT_1, 1), dtype='int64')
train_Y_2 = np.random.randint(low=0, high=NUM_CLASSES, size=(1, 5), dtype='int64')

# Convert labels to categorical
train_Y_1 = to_categorical(train_Y_1, num_classes=NUM_CLASSES)
train_Y_2 = to_categorical(train_Y_2, num_classes=NUM_CLASSES)

# Create model architecture
input_1 = Input(shape=(train_X_1.shape[1], train_X_1.shape[2]))
convl1_input_1 = Conv1D(filters=96, kernel_size=12, strides=2, padding='valid', activation='relu')(input_1)
flat1 = Flatten()(convl1_input_1)

input_2 = Input(shape=(train_X_2.shape[1], train_X_2.shape[2]))
convl1_input_2 = Conv1D(filters=96, kernel_size=13, strides=2, padding='valid', activation='relu')(input_2)
flat2 = Flatten()(convl1_input_2)

# Merge inputs
merge = concatenate([flat1, flat2])

# interpretation model
hidden = Dense(10, activation='relu')(merge)
output = Dense(NUM_CLASSES, activation='sigmoid')(hidden)
model = Model(inputs=[input_1, input_2], outputs=output)

# Plot model
plot_model(model, to_file='multiple_inputs_dummy.png')

# Compile model
model.compile(loss=losses.categorical_crossentropy, optimizer='adam', metrics=["accuracy"])

# Fit model 
BATCH_SIZE = 32
EPOCHS = 3
SHUFFLE = False
VERBOSE = 1
model.fit([train_X_1, train_X_2], [train_Y_1, train_Y_2], epochs=EPOCHS, batch_size=BATCH_SIZE, verbose=VERBOSE, shuffle=SHUFFLE)

This gives me the below error,

ValueError: Error when checking model target: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 arrays but instead got the following list of 2 arrays: [array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       ..., 
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0...

What is the proper way of giving labels in model.fit() function?

Answer 1

Your model only has one output. That means that only one "label" is produced for each pair of inputs. Using standard loss functions, this label can only be compared to a single ground truth label, so each pair of inputs can only have one label.

To fix this, you have several options, but none are a quick and easy fix.

1. Figure out how to combine each pair of ground truth labels into a single label, and then pass those combined labels into fit() .
2. Modify your model to have two outputs.
3. I haven't tried this, but you might be able to make a custom loss function that compares each output label to a pair of ground truth labels.

Which one of these works best depends on what your data is, and why you want only a single output despite having two labels.