# Given a query image Q and two other images X and Y, how to determine which one is most similar to Q?

Given a query image Q and two other images X and Y (you can assume they have more or less the same resolutions if that simplifies the problem), which algorithm would perform extremely well at determining which image between X and Y is most similar to Q, even when the differences are rather subtle? For example, a trivial case would be:

• Q = image of mountains, X = image of mountains, Y = image of dogs, therefore it is clear that sim(Q,X) > sim(Q,Y).

However, examples of trickier cases would be:

• Q = image of a yellow car, X = image of a red car, Y = image of a yellow car, therefore sim(Q,Y) > sim(Q,X) (assuming the car shapes are more or less the same).
• Q = image of a man standing up in the middle with a black background, X = image of another man standing up in the middle with a black background, Y = image of a woman standing up in the middle with a black background, therefore sim(Q,X) > sim(Q,Y).

Which algorithm (or combination of algorithms) would be robust enough to handle even the tricky cases with very high accuracy?

• Are you commenting in the wrong question by any chance? I don't understand why you are talking about time, simultaneity, past, present and future. May 20, 2018 at 19:48
• @Bobs, I really think you are just trolling. Your comments have nothing to do with my original question. Please stay on-topic. May 20, 2018 at 20:49
• @Bobs you have gone very deep with the thought process...I appreciate it, you opened up a new viewpoint for me....In short you are trying to say that the OP should come up with a function/measure to define similarity or dissimilarity of images..and then train that function...correct?
– user9947
May 21, 2018 at 10:27
• A simpler way of putting this: The OP has to define "similarity" by some kind of measure. There is no single correct, objective measure. The section labelled "trickier cases" gets closer to it. Consider the two men and woman against plain background. Which is more important: 1) pose of person (is one man standing with back facing camera, woman and man both facing camera) 2) colour or nature of background 3) colour of hair/skin/eye, clothing or some cultural similarity between the people . . . etc, etc. There is no "true" answer, the goal can be highly subjective, depending on the data set May 21, 2018 at 15:07

From your examples I assume you presuppose image recognition in the sense that you don't compare the actual images, but the descriptions of what the images contain.

For comparing images there are various algorithms working on the visual similarity. This can sometimes lead to interesting results, as you probably have seen images on the internet like "dog or muffin". A purely visual approach would find this hard to do.

However, if you do have the description of the image (as there are ways of getting captions from images), then it would just be a text comparison between three sentences: the one describing your query image, and those describing your images X and Y. There are ways of getting at the semantic similarity of sentences. The simplest way (from your examples) would be to look at the overlap in words: [yellow, car]/Q, [red, car]/X, and [yellow, car]/Y obviously has the largest overlap with Q and Y. This is rather simplistic, but it is what your examples suggest you are dealing with.

But, getting a proper description from an image is a hard task, of the calibre that Google and Instagram are still working on; they have large amounts of training data and huge resources they can throw at the problem. So, unless that is your starting point, it will not be easy to achieve.

Overall I do not think it is currently possible to solve this problem with high accuracy.

• I would go further and say that it is not even possible to define the problem with high accuracy, although I like the approach of comparing open-ended text annotations, because in theory these will emulate the subjectivity of comparing photos done by people (at least in a given context, such as how they would be labelled when uploaded to the internet). May 21, 2018 at 15:11
• Thanks for the idea. In fact I do have access to some textual metadata, but unfortunately that would only help me solve the easy cases, e.g. "a car is probably more similar to another car than to a dog", however for the trickier cases textual metadata falls short, which makes me believe I need to resort to some kind of fine-grained visual features to break ties. In fact my goal is to learn an embedding with a Neural Network using a Triplet Loss, but the crucial problem lies in generating accurate triplet samples, including both easy and tricky cases. Hence my question. May 21, 2018 at 21:57
• One idea I have in mind is to extract a ton of features using different techniques (pre-trained CNNs, handcrafted features, etc.) and somehow figure out a hybrid score that combines the "opinions" of all the individual features. A sort of "mixture of experts". This hybrid score would let me judge which image is the most similar to the query image Q. But I have no idea of how to define such a hybrid score. May 21, 2018 at 22:21