具有两个预训练的ResNet 50的暹罗神经网络-测试模型时的奇怪行为

Question

I built siamese neural network, using Keras lib for it. My model has two inputs with shape (64,64,3), two pre-trained ResNet-50. Loss function is binary cross entropy.

The model is based on this paper a link

During train I have very good trait/val accuracy, about 0.99/0.98, and low loss 0.01/0.05.

But when I test my saved model, I get bad results. The model can't recognize even two the same pictures.

Also I noticed strange behavior: the greater the number of epochs the result is worse. For example, comparing two identical images, trained model with 10 epoch gives prediction: "8.jpg": 0.5180479884147644 but the same model trained with 100 epoch gives "8.jpg": 5.579867080537926E-13 However for 100 epoch I have better train results.

I've tried different model for CNN: ResNet18, different input shapes, like (224,224,3) or (128,128,3) .

Also I've triad use not pre-train model, only ResNet50/ResNet18 without pre-trained weights. But I have the same bad results while testing real model.

My code is

def create_base_model(image_shape, dropout_rate, suffix=''):
    I1 = Input(shape=image_shape)
    model = ResNet50(include_top=False, weights='imagenet', input_tensor=I1, pooling=None)
    model.layers.pop()
    model.outputs = [model.layers[-1].output]
    model.layers[-1].outbound_nodes = []

    for layer in model.layers:
        layer.name = layer.name + str(suffix)
        layer.trainable = False

    flatten_name = 'flatten' + str(suffix)

    x = model.output
    x = Flatten(name=flatten_name)(x)
    x = Dense(1024, activation='relu')(x)
    x = Dropout(dropout_rate)(x)
    x = Dense(512, activation='relu')(x)
    x = Dropout(dropout_rate)(x)

    return x, model.input


def create_siamese_model(image_shape, dropout_rate):

    output_left, input_left = create_base_model(image_shape, dropout_rate)
    output_right, input_right = create_base_model(image_shape, dropout_rate, suffix="_2")

    L1_layer = Lambda(lambda tensors: tf.abs(tensors[0] - tensors[1]))
    L1_distance = L1_layer([output_left, output_right])
    L1_prediction = Dense(1, use_bias=True,
                          activation='sigmoid',
                          kernel_initializer=RandomNormal(mean=0.0, stddev=0.001),
                          name='weighted-average')(L1_distance)

    prediction = Dropout(0.2)(L1_prediction)

    siamese_model = Model(inputs=[input_left, input_right], outputs=prediction)

    return siamese_model

siamese_model = create_siamese_model(image_shape=(64, 64, 3),
                                         dropout_rate=0.2)

siamese_model.compile(loss='binary_crossentropy',
                      optimizer=Adam(lr=0.0001),
                      metrics=['binary_crossentropy', 'acc'])
siamese_model.fit_generator(train_gen,
                            steps_per_epoch=1000,
                            epochs=10,
                            verbose=1,
                            callbacks=[checkpoint, tensor_board_callback, lr_reducer, early_stopper, csv_logger],
                            validation_data=validation_data,
                            max_q_size=3)

siamese_model.save('siamese_model.h5')



# and the my prediction
siamese_net = load_model('siamese_model.h5', custom_objects={"tf": tf})

X_1 = [image, ] * len(markers)
batch = [markers, X_1]
result = siamese_net.predict_on_batch(batch)

# I've tried also to check identical images 
markers = [image]
X_1 = [image, ] * len(markers)
batch = [markers, X_1]
result = siamese_net.predict_on_batch(batch)

I have some doubts about my prediction method. Could someone please help me to find what is wrong with predictions?

Answer 1

What you are getting is expected. I'm not sure what you mean by

Also I noticed strange behavior: the greater the number of epochs the result is worse.

But the results you shown are valid and expected. Let's start with what the model is outputting. Your model output is (normalized)distance between the first and second inputs. If the inputs are similar, then the distance should be close to zero. As number of training step increases the model learns to identify the inputs, ie if the inputs are similar the model learns to output values close to zero, and if the inputs are different the model learns to output values close to one. So,

... trained model with 10 epoch gives prediction: "8.jpg": 0.5180479884147644 but the same model trained with 100 epoch gives "8.jpg": 5.579867080537926E-13 However for 100 epoch I have better train results.

, confirms that the model has learned that the two inputs are similar and outputs 5.579867080537926E-13 ~ 0 (approximately close to 0).

Although the model is performing well, there is one issue I've observed in the model definition:- The output layer is dropout layer. Dropout is not valid output layer. What you are doing by this setting is, randomly with probability 0.2 you are setting the output of the model to be zero.

Let's assume the target variable has 1(the two inputs are different), and model has learnt to identify the images correctly and outputs value close to 1 before the dropout layer. Let's further assume that the dropout layer has decided to set the output to be zero. So the model output will be zero. Even though the layers before dropout layer have performed well, because of the dropout layer, they will be penalized. If this is not what you are looking then remove the last dropout layer.

L1_prediction = Dense(1, use_bias=True,
                    activation='sigmoid',
                    kernel_initializer=RandomNormal(mean=0.0, stddev=0.001),
                    name='weighted-average')(L1_distance)


siamese_model = Model(inputs=[input_left, input_right], outputs=L1_prediction)

However, sometimes this behavior is needed if one want to add noise to the model. This has the same effect with randomly altering the target variable when the value is 1.