Can CNNs detect image similarity?

Question

I have been running some experiments to see whether a CNN can detect whether two images are the same. However, I can't seem to make it work. I am wondering whether CNNs are not able to do what I am trying, or whether I am making mistakes.

This is the setup: I take two 128x128 images and append them to be 128x256. These two images are either exactly the same: or different:

I have also experimented with a black bar in the middle. The entire appended 128x256 image(thus containing two images) is fed into a CNN with the following setup:

Depending on the dropout usage, the model is able to learn training data between 80-100% accuracy. But the validation accuracy never gets above around 65%.

I do not understand why the model is not able to get to 100% very easily, as the input images are either EXACTLY the same, or very different. Seems like a trivial task to me. I would assume that the convolutional kernels would create exactly the same feature representations of the images and then feed that into the dense layers.

ps. I know there are better ways to do this but I wanted to experiment with this.

Answer 1

An interesting issue: the model is seemingly under-fitting to the train data (since the accuracy isn't that high), but it is also underfitting to the training data. :)

When you flatten the data and use dense layers, you lose all of the spatial information. Before you do that, you might want to subtract the left side of the output (I calculated it having a size of 15x30x128, so a half would be 15x15x128?) from the right side. This is a bit tricky with the Sequential model, but it can be done with a Lambda layer. Note that the tensor's first dimension is None, which is actually the batch size during training.

After doing this you may keep the current flatten + dense structure and see how it goes. Although I suppose it can be much simpler at this point, Conv2d is doing the heavy lifting.

You might want to add dropouts to the earlier part of the model as well, or use enough data augmentation to avoid overfitting. The black bar in the middle is obstructing the image, making them very non-identical.

Alternatively you could make a single feature extraction model and feed it these two inputs separately. This architecture is called a Siamese Network, and requires a bit more code to implement (example in the article, however it uses PyTorch which seems quite verbose when compared to Keras).

Actually the first suggestion is a Siamese Network as well, just in a bit odd structure having just one input. I'd be interested to hear if this works better for you.

Edit: I decided to try out the problem myself. Using the idea described above, I got 99% validation accuracy on images which aren't even identical, using a model with 194 parameters. Half of the examples have a different image, and the other half is a modified copy of the reference (random +/- 10 degrees of rotation, +/- 10% of zoom with TF's ImageDataGenerator).

I won't share the code yet in case you want to figure it out yourself, but here is the summary which shows the dataflow (uses the Sequential API):

Model: "sequential_173"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_132 (Conv2D)          (None, 126, 254, 3)       84        
_________________________________________________________________
max_pooling2d_88 (MaxPooling (None, 63, 127, 3)        0         
_________________________________________________________________
conv2d_133 (Conv2D)          (None, 61, 125, 3)        84        
_________________________________________________________________
max_pooling2d_89 (MaxPooling (None, 30, 62, 3)         0         
_________________________________________________________________
lambda_60 (Lambda)           (None, 30, 30, 3)         0         
_________________________________________________________________
average_pooling2d_47 (Averag (None, 5, 5, 3)           0         
_________________________________________________________________
lambda_61 (Lambda)           (None, 5, 5)              0         
_________________________________________________________________
flatten_52 (Flatten)         (None, 25)                0         
_________________________________________________________________
dense_435 (Dense)            (None, 1)                 26        
=================================================================
Total params: 194
Trainable params: 194
Non-trainable params: 0
_________________________________________________________________

It takes a "twin" RGB image in, and applies two Conv2D layers with just 3 kernels on each of them. This leads to 30 x 31 x 3 shaped image features (the width of max_pooling2d_89's output is actually 62 since there are two images side-by-side).

The two sides are subtracted from each other (cropping out the middle column) and the result is downsampled and aggregated, resulting in a 5 x 5 output. The final Dense layers is just a sigmoid to predict whether the images are nearly identical or not.

Example result is shown below. It has six columns of images: left image, right image, left's features (conv2d_133's output), right's features, feature difference and weighted feature difference. The last column shows that the model has learned to focus more on the center of the image rather than corners, because the used image augmentation distorts the corners the most.

If the image is downsampled to just 1 x 1 before the Flatten layer then the model has only 170 parameters, but the validation accuracy drops a bit to 97%. (edit 2: using elu rather than tanh activation brings this model's accuracy to 99.3%, I originally used tanh since it is easy to visualize as an image due to its limited range)

The used dataset is Indoor Scene Recognition, of which I used 2000 random images.