如何在训练连体网络后生成测试三元组数据集的预测

Question

I have a dataset of images and two txt files in which each line contains the id of three pictures, the first one is for training and tells me that the first picture is most similar to the second one than to the third one. The second one is for testing: I have to predict wether the first image is most similar to the first or the second one for each line. To do this I have trained a siamese network utilising triplet loss using as guideline this article: https://keras.io/examples/vision/siamese_network/

After training the network I do not know how to proceed to evaluate my testing dataset, to prepare the data I have done:

with open('test_triplets.txt') as f:
    lines2 = f.readlines()
lines2 = [line.split('\n', 1)[0] for line in lines2]
anchor2 = [line.split()[0] for line in lines2]
pic1 = [line.split()[1] for line in lines2]
pic2  = [line.split()[2] for line in lines2]

anchor2 = ['food/' + item + '.jpg' for item in anchor2]
pic1 = ['food/' + item + '.jpg' for item in pic1]
pic2 = ['food/' + item + '.jpg' for item in pic2]

anchor2_dataset = tf.data.Dataset.from_tensor_slices(anchor2)
pic1_dataset = tf.data.Dataset.from_tensor_slices(pic1)
pic2_dataset = tf.data.Dataset.from_tensor_slices(pic2)

test_dataset = tf.data.Dataset.zip((anchor2_dataset, pic1_dataset, pic2_dataset))
test_dataset = test_dataset.map(preprocess_triplets)
test_dataset = test_dataset.batch(32, drop_remainder=False)
test_dataset = test_dataset.prefetch(8)

I have then tried to utilise a for loop as follows, but the running time is too high since I have around 50000 lines in the txt file.

n_images = len(anchor2)
results  = np.zeros((n_images,2))
for i in range(n_images):
    sample = next(iter(test_dataset))
    anchor, positive, negative = sample
    anchor_embedding, positive_embedding, negative_embedding = (
        embedding(resnet.preprocess_input(anchor)),
        embedding(resnet.preprocess_input(positive)),
        embedding(resnet.preprocess_input(negative)),
    )
    cosine_similarity = metrics.CosineSimilarity()

    positive_similarity = cosine_similarity(anchor_embedding, positive_embedding)
    results[i,0] = positive_similarity.numpy()

    negative_similarity = cosine_similarity(anchor_embedding, negative_embedding)
    results[i,1] = negative_similarity.numpy()

How can I do to be able to generate predictions on my testing triplets ? My objective would be to have a vector [n_testing_triplets x 1] where each line is 1 if the first pic is most similar to the anchor or 0 otherwise.

Answer 1

You can stack your images first , then calculate all embedings in parallel like this :

import numpy as np
stack = np.stack([anchor0, positive0, negative0, ..., anchor999, positive999, negative999])

# then you calculate all embeding at the same time like this 
embeddings = list(embedding(resnet.preprocess_input(stack)).numpy())

Then you compare the embeding as you want, in a loop :

cosine_similarity = metrics.CosineSimilarity()

positive_similarity = cosine_similarity(embeddings [0] , embeddings [1])
whatever_storage = positive_similarity.numpy()

negative_similarity = cosine_similarity(embeddings [0] , embeddings [2])
whatever_storage  = negative_similarity.numpy()