How to get a binary output from a Siamese Neural Network

Question

I'm trying to train a Siamese network to check if two images are similar. My implementation is based on this. I find the Euclidian distance of the feature vectors(the final flattened layer of my CNN) of my two images and train the model using the contrastive loss function.

My question is, how do I get a binary output from the Siamese network for testing (1 if it two images are similar, 0 otherwise). Is it just by thresholding the Euclidian distance to check how similar the images are? If so, how do I go about selecting the threshold? If I wanted to measure the training and validation accuracies, the threshold would have to be increased as the network learns better. Is there a way to learn this threshold for a given dataset?

I would appreciate any leads, thank you.

Answer 1

I am working on something rather similar in the sense that I get a continuous value (between 0 and 1) as output that I want to be binary. I think you should be using the ROC curve (Receiver-Operating-Characteristic) instead of the accuracy. If you want a single value take the area-under-the-curve. The ROC curve is basically true-prositive-rate vs false-positive-rate as you test for different values of threshold. You could also (or instead) use precision vs recall.

Also, the optimal threshold does not depend as much on the dataset as it depends on your neural network. As you said this value will change as your network learns. Simply use the outputted value and test different thresholds.

If do want to keep the accuracy, simply pick the threshold for which is it best (at the given iteration). Keep in mind that you should not define the threshold from the validation set as you would be cheating.

ps: from rereading it seems like you were implying to learn the threshold from inside the network. I believe it should just be a post-process

Answer 2

Don't use the Euclidian distance as the similar/dissimilar factor, you'll get better results if you put a couple Dense layers at the top of your Siamese network. You don't mention how large your feature vectors are, but if they are 128D face encodings, this is what I used:

# dist_euclid_layer is a layers.Lambda that performs the euclid distance on the input
first_dense = layers.Dense(512, activation='relu')(dist_euclid_layer)
drop_one = layers.Dropout(0.5)(first_dense)
output_dense = layers.Dense(2, activation='sigmoid')(drop_one)
model = models.Model(inputs=[input_layer], outputs=output_dense)
model.compile(loss=losses.binary_crossentropy, optimizer=opt, metrics=['accuracy'])

In training, use [0,1] or [1,0] for Y to indicate different / similar.

Then in production, use the np.argmax to find the positive match:

predictions_raw = model.predict(vstack_batch_input, batch_size=len(batch_input))
predictions = np.argmax(predictions_raw,axis=1)
tot_matches = np.sum(predictions)
if tot_matches == 1:
    # ... good times, whatever face[N] has prediction[N] == 1 is the matching face