I am trying to build a neural network suitable to measure similarity between pairs of images. In particular I am interested in shoes. I have a query image (e.g. a shoe that I just took a picture of) and I want to find similar shoes in a database (several thousands of images).
I tried using MAC feature (e.g. max pool over the entire spacial dimension on last (or some other) convolution layer of say VGG16) (here is a link to a paper https://arxiv.org/pdf/1511.05879.pdf). The two MAC vectors are compared using cosine similarity. That works, but among the top matches there are always a few very strange shoes (e.g when I submit a query image with a boot I find sandals among other boots with extremely high similarity score).
What would be a better way of doing that? Something more robust to finding shoes similar in shape to the query image. Thanks!