特征相似性的连体网络

Question

I have around 20k images of different domains with the features already extracted using GLCM and HOG. The dimensions of features are around 2000 for each image. I want to find similarity between features using Siamese network.I stored all in a dataframe. I'm not sure how we can give input features to neural net. There is only one possibilty of using 1DCNN / Dense layers.

encoder = models.Sequential(name='encoder')
encoder.add(layer=layers.Dense(units=1024, activation=activations.relu, input_shape=[n_features]))
encoder.add(layers.Dropout(0.1))
encoder.add(layer=layers.Dense(units=512, activation=activations.relu))
encoder.add(layers.Dropout(0.1))
encoder.add(layer=layers.Dense(units=256, activation=activations.relu))
encoder.add(layers.Dropout(0.1))

In this above code we only we give number of features as input to encoder, But number of features for my both images are same. Should I train two encoders separately and join them at the end to form a embedding layer? But how should I test?

Answer 1

For a siamese network you would want to have one network, and train it on different sets of data.

So say you have two sets of data X0 and X1 that have the same shape, you would do

from tensorflow.keras import models
from tensorflow.keras import layers
from tensorflow.python.keras.utils import losses_utils


# number of features
n_features = 2000

# fake data w/batch size 4
X0 = tf.random.normal([4, n_features])
X1 = tf.random.normal([4, n_features])

# siamese encoder model
encoder = models.Sequential(name='encoder')
encoder.add(layer=layers.Dense(
    units=1024, activation="relu", input_shape=[n_features]))
encoder.add(layers.Dropout(0.1))
encoder.add(layer=layers.Dense(units=512, activation="relu"))
encoder.add(layers.Dropout(0.1))
encoder.add(layer=layers.Dense(units=256, activation="relu"))
encoder.add(layers.Dropout(0.1))

# send both sets of data through same model
enc0 = encoder(X0)
enc1 = encoder(X1)

# compare the two outputs
compared = tf.keras.losses.CosineSimilarity(
    reduction=losses_utils.ReductionV2.NONE)(enc0, enc1)
print(f"cosine similarity of output: {compared.numpy()}")
# cosine similarity of output: [-0.5785658, -0.6405066, -0.57274437, -0.6017716]

# now do optimization ...

There are numerous way to compare the output, cosine similarity being one of them, but I just included it for illustration and you may require some other metric.

Answer 2

There is only one network which is just duplicated. All weights are shared. so you are training one network, just run it twice at each step of learning. you should pick two sample from your dataset and label it to 1 if came from same class and 0 otherwise.

from tensorflow.keras import models
from tensorflow.keras import layers
import tensorflow.keras.backend as K

n_features = 2000

def cos_similarity(x):
    x1,x2 = x
    return K.sum(x1*x2)/(K.sqrt(K.sum(x1*x1))*K.sqrt(K.sum(x2*x2)))

inp1 = layers.Input(shape=(n_features))
inp2 = layers.Input(shape=(n_features))

encoder = models.Sequential(name='encoder')
encoder.add(layer=layers.Dense(
    units=1024, activation="relu", input_shape=[n_features]))
encoder.add(layers.Dropout(0.1))
encoder.add(layer=layers.Dense(units=512, activation="relu"))
encoder.add(layers.Dropout(0.1))
encoder.add(layer=layers.Dense(units=256, activation="relu"))
encoder.add(layers.Dropout(0.1))

out1 = encoder(inp1)
out2 = encoder(inp2)

similarity = layers.Lambda(cos_similarity)([out1,out2])

model = models.Model(inputs=[inp1,inp2],outputs=[similarity])

model.compile(optimizer='adam',loss='mse')

For testing, first of all you should compute HOG features which you said it has 2000 features. Then run

model.predict(hog_feature)

and you have output feature. By the way I recommend to do not use hog feature and siamese network. Extract image feature just using this network. change input shape and train with images.