An interesting issue: the model is seemingly under-fitting to the train data (since the accuracy isn't that high), but it is also underfitting to the training data. :)
When you flatten the data and use dense layers, you lose all of the spatial information. Before you do that, you might want to subtract the left side of the output (I calculated it having a size of 15x30x128, so a half would be 15x15x128?) from the right side. This is a bit tricky with the
Sequential model, but it can be done with a
Lambda layer. Note that the tensor's first dimension is
None, which is actually the batch size during training.
After doing this you may keep the current flatten + dense structure and see how it goes. Although I suppose it can be much simpler at this point, Conv2d is doing the heavy lifting.
You might want to add dropouts to the earlier part of the model as well, or use enough data augmentation to avoid overfitting. The black bar in the middle is obstructing the image, making them very non-identical.
Alternatively you could make a single feature extraction model and feed it these two inputs separately. This architecture is called a Siamese Network, and requires a bit more code to implement (example in the article, however it uses PyTorch which seems quite verbose when compared to Keras).
Actually the first suggestion is a Siamese Network as well, just in a bit odd structure having just one input. I'd be interested to hear if this works better for you.
Edit: I decided to try out the problem myself. Using the idea described above, I got 99% validation accuracy on images which aren't even identical, using a model with 194 parameters. Half of the examples have a different image, and the other half is a modified copy of the reference (random +/- 10 degrees of rotation, +/- 10% of zoom with TF's
I won't share the code yet in case you want to figure it out yourself, but here is the summary which shows the dataflow (uses the
Layer (type) Output Shape Param #
conv2d_132 (Conv2D) (None, 126, 254, 3) 84
max_pooling2d_88 (MaxPooling (None, 63, 127, 3) 0
conv2d_133 (Conv2D) (None, 61, 125, 3) 84
max_pooling2d_89 (MaxPooling (None, 30, 62, 3) 0
lambda_60 (Lambda) (None, 30, 30, 3) 0
average_pooling2d_47 (Averag (None, 5, 5, 3) 0
lambda_61 (Lambda) (None, 5, 5) 0
flatten_52 (Flatten) (None, 25) 0
dense_435 (Dense) (None, 1) 26
Total params: 194
Trainable params: 194
Non-trainable params: 0
It takes a "twin" RGB image in, and applies two
Conv2D layers with just 3 kernels on each of them. This leads to
30 x 31 x 3 shaped image features (the width of
max_pooling2d_89's output is actually
62 since there are two images side-by-side).
The two sides are subtracted from each other (cropping out the middle column) and the result is downsampled and aggregated, resulting in a
5 x 5 output. The final
Dense layers is just a
sigmoid to predict whether the images are nearly identical or not.
Example result is shown below. It has six columns of images: left image, right image, left's features (
conv2d_133's output), right's features, feature difference and weighted feature difference. The last column shows that the model has learned to focus more on the center of the image rather than corners, because the used image augmentation distorts the corners the most.
If the image is downsampled to just
1 x 1 before the
Flatten layer then the model has only 170 parameters, but the validation accuracy drops a bit to 97%. (edit 2: using
elu rather than
tanh activation brings this model's accuracy to 99.3%, I originally used
tanh since it is easy to visualize as an image due to its limited range)
The used dataset is Indoor Scene Recognition, of which I used 2000 random images.