Given a query image Q and two other images X and Y (you can assume they have more or less the same resolutions if that simplifies the problem), which algorithm would perform extremely well at determining which image between X and Y is most similar to Q, even when the differences are rather subtle? For example, a trivial case would be:

  • Q = image of mountains, X = image of mountains, Y = image of dogs, therefore it is clear that sim(Q,X) > sim(Q,Y).

However, examples of trickier cases would be:

  • Q = image of a yellow car, X = image of a red car, Y = image of a yellow car, therefore sim(Q,Y) > sim(Q,X) (assuming the car shapes are more or less the same).
  • Q = image of a man standing up in the middle with a black background, X = image of another man standing up in the middle with a black background, Y = image of a woman standing up in the middle with a black background, therefore sim(Q,X) > sim(Q,Y).

Which algorithm (or combination of algorithms) would be robust enough to handle even the tricky cases with very high accuracy?

  • $\begingroup$ Cant you decide on the simultanetity of each image upon a quantifiable account upon a measure of difference in the pixel return in each image, pattern recognition would emerge in the differenciation of each image and be revealed unless each image is uncorrelated despite a pretence of simultainitey that would question the truth of the pattern and a nature in indifferentiation, Knowledge cannot determine simultainity only from a reference to contain knowledge and its comparison to the present, the present is not a function of the past or future but is transient in communicable information. $\endgroup$ – Bobs May 20 '18 at 19:37
  • $\begingroup$ Are you commenting in the wrong question by any chance? I don't understand why you are talking about time, simultaneity, past, present and future. $\endgroup$ – Pablo Messina May 20 '18 at 19:48
  • $\begingroup$ Perception is a product of the present and comparison is a product of a division in a sumultaneos observation of a causal sample of the division in a continuous world. Everything is a product of phillosophical analysis although not objective and absolute in quantification but its subjective nature adds a quality to its reasoning. Lol your right similarity may not be a property of phillosophy but a nature of its determination from classification and objection should be a product of speculation and subject to argument my good fellow lol $\endgroup$ – Bobs May 20 '18 at 20:44
  • $\begingroup$ @Bobs, I really think you are just trolling. Your comments have nothing to do with my original question. Please stay on-topic. $\endgroup$ – Pablo Messina May 20 '18 at 20:49
  • 1
    $\begingroup$ A simpler way of putting this: The OP has to define "similarity" by some kind of measure. There is no single correct, objective measure. The section labelled "trickier cases" gets closer to it. Consider the two men and woman against plain background. Which is more important: 1) pose of person (is one man standing with back facing camera, woman and man both facing camera) 2) colour or nature of background 3) colour of hair/skin/eye, clothing or some cultural similarity between the people . . . etc, etc. There is no "true" answer, the goal can be highly subjective, depending on the data set $\endgroup$ – Neil Slater May 21 '18 at 15:07

From your examples I assume you presuppose image recognition in the sense that you don't compare the actual images, but the descriptions of what the images contain.

For comparing images there are various algorithms working on the visual similarity. This can sometimes lead to interesting results, as you probably have seen images on the internet like "dog or muffin". A purely visual approach would find this hard to do.

However, if you do have the description of the image (as there are ways of getting captions from images), then it would just be a text comparison between three sentences: the one describing your query image, and those describing your images X and Y. There are ways of getting at the semantic similarity of sentences. The simplest way (from your examples) would be to look at the overlap in words: [yellow, car]/Q, [red, car]/X, and [yellow, car]/Y obviously has the largest overlap with Q and Y. This is rather simplistic, but it is what your examples suggest you are dealing with.

But, getting a proper description from an image is a hard task, of the calibre that Google and Instagram are still working on; they have large amounts of training data and huge resources they can throw at the problem. So, unless that is your starting point, it will not be easy to achieve.

Overall I do not think it is currently possible to solve this problem with high accuracy.

  • 2
    $\begingroup$ I would go further and say that it is not even possible to define the problem with high accuracy, although I like the approach of comparing open-ended text annotations, because in theory these will emulate the subjectivity of comparing photos done by people (at least in a given context, such as how they would be labelled when uploaded to the internet). $\endgroup$ – Neil Slater May 21 '18 at 15:11
  • $\begingroup$ Thanks for the idea. In fact I do have access to some textual metadata, but unfortunately that would only help me solve the easy cases, e.g. "a car is probably more similar to another car than to a dog", however for the trickier cases textual metadata falls short, which makes me believe I need to resort to some kind of fine-grained visual features to break ties. In fact my goal is to learn an embedding with a Neural Network using a Triplet Loss, but the crucial problem lies in generating accurate triplet samples, including both easy and tricky cases. Hence my question. $\endgroup$ – Pablo Messina May 21 '18 at 21:57
  • $\begingroup$ One idea I have in mind is to extract a ton of features using different techniques (pre-trained CNNs, handcrafted features, etc.) and somehow figure out a hybrid score that combines the "opinions" of all the individual features. A sort of "mixture of experts". This hybrid score would let me judge which image is the most similar to the query image Q. But I have no idea of how to define such a hybrid score. $\endgroup$ – Pablo Messina May 21 '18 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.