如何使用预训练的 model 进行双输入迁移学习

Question

I am going to use a pretrained model (which was previously saved using save_best_only argument of ModelCheckpoint ) for dual input transfer learning. I have the following:

pretrained_model = load_model('best_weight.h5')

def combined_net(): 
    
    u_model = pretrained_model
    u_output = u_model.layers[-1].output
    
    v_model = pretrained_model
    v_output = v_model.layers[-1].output


    concat = concatenate([u_output, v_output])
    #hidden1 = Dense(64, activation=activation)(concat) #was 128
    main_output = Dense(1, activation='sigmoid', name='main_output')(concat) # pretrained_model.get_layer("input_1").input

    model = Model(inputs=[u_model.input, v_model.input], outputs=main_output)
    opt = SGD(lr=0.001, nesterov=True)
    model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
    return model

And when I try to fit using:

best_weights_file="weights_best_of_pretrained_dual.hdf5"
checkpoint = ModelCheckpoint(best_weights_file, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks = [checkpoint]

base_model = combined_net()
print(base_model.summary)

history = base_model.fit([x_train_u, x_train_v], y_train,
                         batch_size=batch_size,
                         epochs=epochs,
                         callbacks=callbacks, 
                         verbose=1,
                         validation_data=([x_test_u, x_test_v], y_test), 
                         shuffle=True)

I have the the following error:

ValueError: The list of inputs passed to the model is redundant. All inputs should only appear once. Found: [<tf.Tensor 'input_1_5:0' shape=(None, None, None, 3) dtype=float32>, <tf.Tensor 'input_1_5:0' shape=(None, None, None, 3) dtype=float32>]

Apparently, model = Model(inputs=[u_model.input, v_model.input], outputs=main_output) line seems to cause an error.

All I want to do is to use a pretrained model ("best_weight.h5") for dual input to single output model. Both inputs are the same as previously initialized and the concatenate layer should concatenate the layers before the last layer of each model constructed by loaded model.

I have tried several ways I found online but was not able to properly set the model.

I hope one can help me

EDIT:

The pretrained model is shown below:

def vgg_16():
    b_model = VGG16(weights='imagenet', include_top=False)
    x = b_model.output
    x = GlobalAveragePooling2D()(x)
    x = Dense(256, activation=activation)(x)
    predictions = Dense(1, activation='sigmoid')(x)
    model = Model(inputs=b_model.input, outputs=predictions)
    for layer in model.layers[:15]:  #
        layer.trainable = False
    opt = SGD(lr=init_lr, nesterov=True)
    model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
    return model

main_model = vgg_16()
history = main_model.fit(X_train, y_train, batch_size=batch_size, 
          epochs=EPOCHS, validation_data=(X_test, y_test), verbose=1, 
          callbacks=[es, mc, l_r])

Answer 1

here the correct way to do it. when I define the combined_net I define 2 new inputs which are used to feed the pre_trained model in the same way

def vgg_16():
    
    b_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False)
    x = b_model.output
    x = GlobalAveragePooling2D()(x)
    x = Dense(256, activation='relu')(x)
    predictions = Dense(1, activation='sigmoid')(x)
    model = Model(inputs=b_model.input, outputs=predictions)
    
    for layer in model.layers[:15]:
        layer.trainable = False
        
    opt = SGD(lr=0.003, nesterov=True)
    model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
    
    return model

main_model = vgg_16()
# main_model.fit(...)

pretrained_model = Model(main_model.input, main_model.layers[-2].output)

def combined_net(): 
    
    inp_u = Input((224,224,3)) # the same input dim of pretrained_model
    inp_v = Input((224,224,3)) # the same input dim of pretrained_model
    
    u_output = pretrained_model(inp_u)
    v_output = pretrained_model(inp_v)


    concat = concatenate([u_output, v_output])
    main_output = Dense(1, activation='sigmoid', name='main_output')(concat)

    model = Model(inputs=[inp_u, inp_v], outputs=main_output)
    opt = SGD(lr=0.001, nesterov=True)
    model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
    return model

base_model = combined_net()
base_model.summary()